04 thematic

AI Engineering Training for Northcove: What to Focus On

Synthesized from meeting notes with Nina Park (full stack, Consumer-ish product work), Ben Walsh (Senior SWE, Business group, power user), and Raj (engineering org overview, ~100 engineers).

Assumptions: the training is for Northcove's engineering org (Python backend, TypeScript frontend, monolith, Cursor enterprise license, some Claude Code experimentation). Audience is roughly 100 engineers with 4-10 years experience, split across Consumer, Business, and Platform. Training time is limited, so this prioritizes rather than lists everything.

The one-paragraph answer

Northcove is already past "should we use AI." Everyone is on Cursor, tab-complete and chat are standard, and engineers describe themselves as editors rather than authors. The gap is not adoption, it is trust and leverage: engineers do not trust AI output on real production code because the model lacks Northcove-specific context and because verification is weak, so AI stays confined to scaffolding, tests, scripts, and throwaway work. The training should therefore center on three things: (1) context engineering so the model knows how Northcove code works, (2) verification discipline so AI-written code can be trusted in a correctness-critical, regulated codebase, and (3) delegation patterns (plan-then-build, background agents) so engineers move from back-and-forth chat to handing off larger units of work. A fourth, smaller track on building and evaluating LLM features should be offered to the subset of engineers shipping AI-powered product features.

Recommended focus areas, ranked

1. Context engineering for the Northcove codebase (highest priority)

All three engineers independently named this as the core blocker.

• Nina: AI "cherry-picks wrong context from legacy code patterns," works far better with concrete examples than general prompts, and there is "no structured way to feed documentation as context." She struggles to write a comprehensive system prompt for coding standards.
• Ben: "Context is king." He avoids AI for production features because he "can't supply all the context" about antipatterns, coupling, and hidden side effects. He explicitly wants organizational buy-in for rules, commands, and reusable prompts.
• Raj: AI is "not good at Northcove-specific implementations" and the team needs "very prescriptive custom rules."

What to teach:

• How to write and maintain repo-level rules files (Cursor rules / CLAUDE.md-style) that encode Northcove conventions, known antipatterns, and "if you touch X, also check Y" coupling notes.
• Tagging files and providing exemplars up front instead of describing the task in prose. Turn Nina's observation ("better results with similar examples") into a taught habit.
• Building a shared prompt and command library, owned like code, reviewed in PRs. This addresses Ben's buy-in ask and Raj's prescriptive-rules ask in one artifact.
• Using AI itself to bootstrap context: generate module summaries, document hidden side effects, and convert stale Confluence/Google Docs content into living in-repo docs.

Make this the capstone deliverable: each team leaves the training with a real, merged rules file and a starter prompt library for their area of the monolith. That turns training output into infrastructure the whole org inherits.

2. Verification and guardrails for AI-generated code

This is the trust problem, and it is sharpest in Northcove's correctness-critical and regulated domains.

• Nina: AI code "looks plausible but may introduce subtle issues," end-to-end and integration testing are weak, and there is a "risk of accepting good enough AI output without proper verification."
• Ben: works on "what users owe across complex multi-party calculations," where turning one knob breaks something elsewhere. He keeps AI away from production logic for exactly this reason.
• Raj: codegen "needs human review, especially for database/regulated data." The org is skeptical of AI for non-trivial features and database migrations. More experienced engineers are more suspicious.

What to teach:

• Spec-first and test-first workflows: have the model write the failing test and the acceptance criteria before the implementation, then review the test rather than the diff.
• Characterization tests for legacy code before AI-assisted refactors, so coupling and hidden side effects become visible to both the human and the model.
• Small, reviewable increments. Nina wants "better prompt engineering for incremental changes"; teach diff-sized tasks rather than feature-sized prompts, which also fixes her over-engineering complaint.
• A concrete review checklist for AI-written code: data access, migrations, regulated fields, side effects on shared modules. Tie it to the existing CI (20-30 minutes) and deploy (40 minutes to 2 hours) times: local verification matters because the feedback loop is slow.
• Using AI as a reviewer and reproducer, not just an author. Ben already does this for debugging; make it standard.

Framing matters for the skeptical senior engineers Raj mentioned. Position this track as "how to make AI output safe enough for your code," not "how to use AI more." Skeptics are the right people to help write the checklist.

3. Delegation: plan-then-build and background agents

This is where the leverage jump is, and it is the most requested "next step."

• Nina: workflow is "very back-and-forth, time consuming." Wants to "delegate larger features to independent agents" and explore background agents.
• Ben: "starting to use agents for background work: test coverage, feature flag removal, debugging." Uses subagents but they are "not core to workflow yet." Most of the org is "AI-second" rather than AI-first.
• Raj: agents are "less adopted, some early adopters experimenting." The Cursor team already recommended a chat-for-planning, then build workflow.

What to teach:

• The plan-then-execute loop as the default: have the model produce a plan, edit the plan, then execute. This is cheap to teach and Raj notes it is already the recommended pattern.
• Which tasks are safe to hand to a background agent today: test coverage backfill, feature flag removal, dependency bumps, lint migrations, one-off scripts. Ben's list is a ready-made curriculum.
• Task decomposition: how to split a feature into agent-sized units with clear acceptance criteria, and how to write the handoff prompt.
• AI-first for non-code work: reproductions, documentation, Jira grooming, answering "how does X work" questions. Ben's principle, and it lowers the risk profile for skeptics.
• Prefer CLI tools over MCP servers for now. Ben tried MCP for Confluence/Jira/GitHub and hit performance issues. Do not spend training time on MCP configuration beyond database connections.

4. Building and evaluating LLM-powered features (targeted track)

Only a subset of engineers need this, but for them it is a real, current pain.

• Nina's document interpretation feature: hand-labeled 100-250 documents, iterative prompt tweaking, non-deterministic results from small prompt changes, a recent shift to proxy metrics at scale. She notes it is "more labor intensive than traditional ML training."
• Ben: aware of DSPy-style prompt optimization, "many projects would benefit," but no time to implement.

What to teach:

• Eval-driven development: golden datasets, regression evals in CI, and how to size a labeled set. Nina's dataset is a great worked example.
• Structured outputs and schema enforcement to reduce variance from prompt drift.
• Prompt versioning and A/B evaluation, plus a lightweight introduction to automated prompt optimization (DSPy or equivalent) with a Northcove use case.
• Production monitoring for LLM features: Nina asked for "better monitoring and validation systems."

5. Data handling and compliance (short, mandatory module for everyone)

Not a large topic, but it must be explicit and early.

• Raj: regulated-data exposure is a standing concern, a prior third-party data-sharing incident forced rework, and litigation risk makes the team cautious. Legal and Compliance "always factor into decisions."
• Nina: the Anthropic integration relies on organizational data isolation contracts.

Cover: which tools are covered by enterprise agreements, what data may go into prompts, how regulated fields are handled in prompts and in evals, and how to keep test datasets compliant. Skipping this leaves the skeptics with a valid objection and leaves the org exposed.

6. AI-assisted planning and prototyping (light touch)

Worth a session, not a track. The org has an autonomy culture where engineers do a lot of scoping (Raj), tech specs are "too much" (Ben), and PMs already prototype in v0 (Nina).

• Teach throwaway AI prototypes for alignment ("directionally correct, doesn't need to compile").
• Teach AI-assisted spec drafting that separates technical alignment from resourcing and sequencing, which is Ben's diagnosis of why specs are hard to write and review.

What to de-emphasize

• Autocomplete and basic chat. Already adopted org-wide. Do not spend time here.
• MCP server setup. Tried, performance disappointed, CLI preferred. Revisit later.
• AI for user research. Raj: only three customer conversations, not enough volume.
• AI for PRD generation. Blocked by product org workflow, not by skill.
• Fine-tuning or custom models. Nothing in the notes suggests it is needed; evals and context are the higher-leverage moves.

Suggested shape

• Segment by adoption level. The notes describe three clear personas: power users (Ben), basic users (Nina, who still gets real value), and skeptical seniors (Raj's observation). One-size training will bore the first group and lose the third. Run a shared core (context, verification, compliance) and then split into "delegation and agents" for power users and "verification and incremental workflows" for the cautious.
• Hands-on in the Northcove monolith, not toy repos. Every complaint in the notes is about Northcove-specific context. Generic exercises will not transfer.
• Produce artifacts, not just skills. Rules files, prompt library, review checklist, one eval harness. These outlive the training and give the org the "foundation" Ben says everyone, including non-engineers, needs.
• Use existing channels to sustain it. Ben mentions cursor-nerds, background-agents, and general eng Slack channels. Seed them with the training artifacts and nominate power users as owners.
• Measure something. The Business group already thinks in metrics (loss per transaction, processing accuracy). Pick two or three leading indicators for the training: share of PRs with AI-authored tests, number of teams with a merged rules file, background-agent tasks completed per week.

Open questions to resolve before finalizing

• Is Claude Code in scope alongside Cursor, or is Cursor the single supported tool? Raj notes both.
• How many engineers currently ship LLM-powered features? This sizes track 4.
• Is there a documentation cleanup effort the training can piggyback on? All three flagged stale, scattered knowledge as a root cause, and training alone will not fix it.
• Who from Legal and Compliance can co-own the data handling module?