04 thematic

Northcove AI Engineering Training — What to Focus On

Synthesis of interview notes with three engineers: Nina Park (full-stack, ~1 year, self-described "basic" AI user, building an LLM-powered feature), Ben Walsh (senior, Business group, self-described power user), and Raj (org-wide view; read as engineering leadership — see assumptions).

The short version

Northcove is not a "convince them AI is useful" company. Everyone is on Cursor, tab-complete is universal, engineers already describe themselves as "editors rather than authors." The gap is between that baseline and trusting AI on the work that matters — production features in a coupled monolith, anything touching regulated data, non-trivial changes. Three things block that, and all three engineers named them independently:

1. The codebase's context isn't in a form AI can use. Docs are stale and scattered; there's no structured way to feed standards to Cursor; AI "isn't good at Northcove-specific implementations."
2. There's no verification story. AI output "looks plausible" but may hide subtle issues; E2E/integration coverage is thin; CI takes 20–30 min and deploys 40 min–2 h, so mistakes are expensive to catch late.
3. Most of the org is "AI-second." One power user (Ben) is well ahead; the median engineer uses chat for scaffolding and tests; agents are barely adopted.

Recommendation: build the training around a single arc — make the codebase legible to AI → make AI output verifiable → then delegate bigger work — with a separate, smaller track on evals for the people building LLM features. Those four modules, in that order, cover ~80% of what was raised. Below is the evidence and the detail.

---

What we heard — themes by interviewee

Theme	Nina	Ben	Raj
Context / company-specific knowledge is the bottleneck	No structured way to feed docs to Cursor; basic system prompt only; AI cherry-picks wrong legacy patterns	"Context is king"; can't supply all the context for production code; needs org buy-in on rules/commands/reusable prompts	AI "not good at Northcove-specific implementations"; needs "very prescriptive custom rules"; knowledge management is the major gap
Trust / verification	AI code plausible but subtly wrong; weak E2E/integration tests; risk of accepting "good enough"	Doesn't use AI for production features — coupling, hidden side effects, "turn one knob, something falls down elsewhere"	Senior devs most suspicious; skeptical of non-trivial features, migrations, regulated data; slow CI/deploy
Workflow maturity	Considers herself basic; wants background agents and better incremental-change prompting	Most of org "AI-second"; agents just starting (test coverage, flag removal, debugging)	Tab + chat + code search popular; agents lightly adopted by early adopters
Building LLM features	Manually labeled 100–250 docs; iterative prompt tweaking; non-determinism from small changes; moving to proxy metrics	Aware of DSPy / automated prompt optimization, no time to implement	—
Compliance / data	Anthropic with org data-isolation contracts	—	Prior third-party data-sharing incident; litigation concern; regulated data always a factor
Prototyping for alignment	PM uses v0 mockups in user research	Proposes AI prototypes at quarterly onsite; "directionally correct is the name of the game"	PRDs thin; engineers do much of the scoping; 1-week bets
Tooling friction	—	MCP for Confluence/Jira/GitHub had perf issues; prefers CLI tools	—
Meetings / knowledge load	—	Wants AI-first for questions, docs, Jira, calendar	Meetings eat senior IC time; knowledge goes stale

Everything in the first three rows was raised by all three people. That's the signal.

---

Recommended focus areas, ranked

1. Context engineering for the Northcove codebase (highest leverage — do this first)

Why: It's the one thing every interview named, and it's upstream of everything else. Ben won't use AI on production code because he can't load the context; Nina gets over-engineered or wrong-pattern output because the AI has no guidance; Raj says the org needs prescriptive rules. A 100-engineer org that doubled in a year and has a "strong autonomy culture" has also almost certainly drifted in conventions, which makes this worse.

What the module should be: A working session, not a lecture, whose output is committed artifacts:

• Repo-level rules files (Cursor rules / AGENTS.md-style) for the monolith: stack conventions, the Python/TS standards Nina is trying to cram into a system prompt, and the known antipatterns and coupling hotspots Ben is carrying in his head. "Here's what will fall down if you touch X" is the most valuable context in the company and it currently lives in senior engineers.
• Reusable commands/prompts for the recurring tasks people already do with AI (scaffold a component, add tests to an existing suite, remove a feature flag, reproduce a bug).
• A pattern for pointing AI at examples from the codebase rather than general prompts — Nina found this is what actually works; make it the taught default.
• Using AI to generate the missing docs from the code itself (module overviews, "how X works" pages) so the stale-docs problem shrinks rather than waiting on a documentation initiative that won't happen.

Organizational note: Ben explicitly said this needs org buy-in. Treat the rules files as owned, reviewed artifacts with a maintainer per silo, otherwise they go stale like everything else in Confluence.

2. Verification: make AI output trustworthy enough for production

Why: This is the blocker for the skeptical senior half of the org and for Ben's production work. It's also the right answer to Nina's "accepting good-enough output" worry. Slow CI and deploy mean the feedback loop has to move left — into tests and review — for AI-assisted production work to be safe.

What the module should be:

• The plan → implement → verify loop as the standard workflow, including writing the verification before the implementation (AI-generated characterization tests around the code you're about to change, then the change).
• Using agents to raise E2E/integration coverage on the correctness-critical paths (Ben's multi-party calculations, invoicing, Stripe) — this is the task Ben already uses agents for, it's low-risk, and it's what makes later AI use on those paths possible. It's a flywheel: coverage enables delegation enables more coverage.
• AI-assisted code review of AI-generated code: a second pass prompted specifically to hunt for the failure modes people named (over-engineering, copied legacy patterns, hidden side effects).
• An explicit risk ladder for what gets AI-first treatment vs. human-first: throwaway scripts and internal tools → tests and refactors → feature code with coverage → migrations and regulated-data paths (human-led, AI-assisted only). Raj and Ben both already operate on something like this implicitly; making it explicit gives the cautious engineers permission to use AI more within clear bounds and the enthusiastic ones guardrails.

3. From AI-second to AI-first: planning and delegation

Why: The median engineer is at "chat for scaffolding and tests." Nina wants background agents and better incremental-change prompting but doesn't know how; Ben has a workflow worth spreading. The goal is to move the middle of the distribution, not to push the frontier.

What the module should be:

• Planning-first with AI (Cursor's own recommended plan-then-build flow, which Raj says the team already uses for some things): decompose, tag relevant files up front ("don't just say do the thing"), then execute.
• Delegating to background agents with a concrete starter list of tasks that are known to work at Northcove: test coverage, feature-flag removal, bug reproduction, one-off scripts, internal tools. These are the things Ben has validated; name them.
• Prompting for incremental change in an existing codebase vs. greenfield (Nina's specific ask — and the over-engineering failure she sees is a symptom of greenfield-style prompts on brownfield code).
• Ben's broader "AI-first for everything" stance: questions, reproduction, documentation, Jira. Worth 20 minutes, not a module.

Pair this module with internal champions. Ben-style power users plus the existing Slack channels (cursor-nerds, background-agents) are the distribution mechanism; training should hand them material to keep going after the session.

4. Evals for LLM-powered features (separate track, smaller audience)

Why: This is the only area that's "AI engineering" in the strict sense, and Nina's experience shows the team is doing it by hand: manual labeling, eyeballing prompt tweaks, getting burned by non-determinism, then reaching for proxy metrics. Ben knows DSPy exists and hasn't had time. As more teams ship LLM features this gets repeated per team.

What the module should be:

• Building an eval harness: ground-truth datasets (Nina's 100–250 labeled docs are a great seed), scoring functions, running it on every prompt change so non-determinism shows up as a distribution rather than a surprise.
• Prompt versioning and structured outputs so document-interpretation-style features are testable.
• Proxy metrics for scale and when they're trustworthy.
• Automated prompt optimization (DSPy or similar) as an advanced session, once a harness exists to optimize against.
• Data handling for LLM features given the isolation contracts and compliance posture — what can go to the model, how to test without regulated data.

Audience: engineers actually building LLM features (Nina's team and similar). Don't put all 100 engineers through this.

---

Cross-cutting: compliance and data handling

Not a module, but a thread through every module. Raj's notes mention a prior third-party data-sharing incident, litigation sensitivity, and regulated data as a constant factor; Nina mentions data-isolation contracts. The more experienced engineers are the most suspicious, and some of that suspicion is legitimately about data exposure, not model quality. Every session should be explicit about: what's covered under the enterprise license, what can and can't go into a prompt or an agent's context, and how the risk ladder in Module 2 maps to data classes. Getting this wrong once would set adoption back more than any training gains.

Secondary: AI prototyping for scoping and alignment

Ben's quarterly-onsite proposal and the PM's v0 usage point at a real need: PRDs are thin, engineers do much of the scoping, bets go PRD-to-deploy in a week, and "people are visual." A short session on throwaway prototypes for alignment ("directionally correct, doesn't need to compile") fits well, ideally cross-functional with PMs. It's a good second-phase addition; it doesn't compete with the four modules above for the core curriculum.

What to deprioritize

• MCP setup. Ben tried it for Confluence/Jira/GitHub and hit performance problems; he prefers CLI tools that models already know. Teach the CLI pattern (gh, Jira CLI, etc.) and skip MCP beyond the database connection that already works.
• AI for user research — Raj is right that n=3 interviews isn't a volume problem AI solves.
• AI-generated PRDs — blocked by product-org workflow, not by engineering skill.
• Non-engineering adoption (PMs using Cursor Slack integration, BI hookup) — real and popular, but a different audience and program. Note it as demand; don't fold it into engineering training.
• Tool evangelism. They're on Cursor with some Claude Code experimentation. Training should be tool-agnostic on workflow and meet them where they are.

---

Suggested shape

• Everyone (100 engineers): Modules 1–3, ideally as hands-on sessions on the real monolith, so the output is committed rules files, commands, and a test-coverage PR — not slides. Sequence them 1 → 2 → 3; each enables the next.
• LLM feature builders (small cohort): Module 4.
• Champions (Ben and the early adopters in the Slack channels): a short enablement session beforehand so they can co-teach and carry it forward; they are the reason the median moves.
• Seniors specifically: Module 2's risk ladder and the compliance thread are what earn their participation. Don't lead with agents for this group.

Success looks like: rules files exist and are maintained per silo; Ben-tier engineers start using AI on production code because they can load context and verify; Nina-tier engineers are running background agents on the validated task list; LLM features ship with an eval harness.

---

Assumptions and gaps

• Raj's role isn't stated. The notes read as an org-level view (headcount, silos, pipeline, compliance), so I've treated him as engineering leadership. If he's an IC, the org-level claims carry less weight but the themes still triangulate with Nina and Ben.
• Three interviews, one stack. All three are on the Python/TS monolith; Ben is in Business, Nina's silo isn't stated, and Platform and Consumer aren't clearly represented. Before finalizing, a short survey or two more interviews across those silos would confirm whether the context/verification problems look the same there (likely, given the shared monolith, but unconfirmed).
• Experience skew. Raj says seniors are more AI-suspicious; the interviews include one power user and one self-described basic user, but no explicitly skeptical senior. Module 2's framing is built from Raj's and Ben's second-hand account of that group and would benefit from one direct conversation.
• Training is assumed to be for engineers using AI in their work, with one track on building AI products; the interviews support both but weight heavily toward the former.