04 thematic

Recommended focus for Northcove’s AI engineering training

Executive recommendation

Northcove does not need an introductory “how to generate code with AI” course. Cursor is already broadly adopted, autocomplete and chat are common, and some engineers rarely write more than 20 lines manually. The training should instead teach engineers how to make AI effective and safe inside Northcove’s real constraints: a coupled Python/TypeScript monolith, scattered and stale knowledge, correctness-critical business logic, regulated data, and uneven adoption of agents.

The central workflow should be:

Frame the task → assemble the right context → set boundaries and acceptance criteria → delegate in small units → verify with tests and domain invariants → capture reusable context.

This moves the organization from “AI-assisted typing” to reliable AI-assisted engineering without implying that every task should be AI-led.

What the three interviews collectively say

Signal	Evidence from the interviews	Training implication
Adoption is broad but depth is uneven	Nina calls herself a basic user; Ben is a power user; Raj sees heavy autocomplete/chat use but limited agent adoption	Use a shared core workflow, then differentiated labs rather than basic tool orientation
Company context is the main constraint	Nina sees legacy examples being copied incorrectly; Ben cannot load coupling and hidden side effects; Raj says models struggle with Northcove-specific implementation	Make context engineering the largest module
Plausible output is not trustworthy output	Nina worries about subtle defects and weak integration/E2E coverage; Ben works in money calculations; Raj is skeptical of migrations and regulated-data work	Teach verification, invariants, and risk-based review before autonomous agents
Agents are promising but not yet routine	Nina wants larger independent tasks; Ben uses agents for tests, cleanup, and debugging; Raj reports adoption mainly among early adopters	Teach bounded delegation on well-specified, low-to-medium-risk work
Knowledge is fragmented and stale	All three mention difficulty finding or supplying organizational context across code, Confluence, Slack, and other docs	Teach lightweight reusable context artifacts; treat broader knowledge infrastructure as an organizational follow-up, not a prompt trick
Northcove also builds AI product features	Nina owns document interpretation and manual evals; Ben is interested in prompt optimization	Include a separate production-LLM evaluation track

Curriculum priorities

1. Context engineering for the Northcove codebase — highest priority

Teach engineers to give the model a deliberately curated working set rather than asking it to infer the whole monolith.

Key practices:

• Start with repository search and trace the relevant call path, tests, data model, and downstream side effects.
• Provide one known-good, current example and explicitly identify legacy patterns the model must not imitate.
• Ask for a plan and assumptions before code on non-trivial work; make the model surface missing context and likely blast radius.
• Add context progressively: product intent and constraints first, relevant files second, then feedback from tests and runtime behavior.
• Turn recurring guidance into small, owned repository-level rules, commands, checklists, and examples. Avoid one enormous system prompt that will become stale.
• Prefer tools already reliable in Northcove’s environment—repository search and approved CLI workflows—over making MCP configuration a prerequisite.

Hands-on lab: Take a real Python/TypeScript change in the monolith. Compare a vague request with a context packet containing the goal, non-goals, relevant files, a current exemplar, invariants, and acceptance tests. Review not only whether the second answer compiles, but whether it avoids legacy anti-patterns and catches downstream effects.

2. Verification, testing, and safe use by task risk — equally important

Northcove’s largest danger is not obviously broken code; it is plausible code that is locally correct but violates a financial, compliance, or system invariant. Training should make verification part of the prompt and workflow, not a final human glance.

Teach a provisional three-tier model:

• Lower risk: prototypes, throwaway analysis, internal scripts, documentation, test scaffolding, feature-flag cleanup, and bug reproduction. AI may do most of the first pass with normal review.
• Moderate risk: ordinary product code. Require an explicit plan, focused diff, automated tests, review of adjacent behavior, and human ownership.
• High risk: money calculations, Stripe/transaction lifecycle behavior, database migrations, permissions, regulated data, and legal/compliance-sensitive flows. Humans retain design and approval responsibility; require domain invariants, integration/E2E coverage, data-handling checks, staged rollout, and monitoring. AI output is a draft, not evidence of correctness.

The course should cover:

• Writing acceptance criteria and domain invariants before generation.
• Asking AI to identify failure modes and propose tests, then independently checking both code and tests.
• Reviewing diffs for unnecessary abstraction, scope expansion, copied legacy patterns, migration/data risk, and missing observability.
• Using existing test suites, targeted integration tests, and production guardrails rather than relying on model confidence.
• Approved-model and data-boundary rules: what code, logs, documents, customer data, and regulated data may be supplied to each tool.

Hands-on lab: Give participants a plausible but subtly wrong change to transaction or document-processing logic. Have them use AI to map invariants, generate adversarial cases, run the relevant tests, and explain why “the tests pass” may still be insufficient.

3. Bounded agent delegation and task decomposition

Agents should be taught after context and verification. The goal is not “delegate an entire feature,” but select work whose boundaries and completion criteria can be observed.

Teach engineers to:

• Choose agent-friendly work: test expansion, mechanical cleanup, feature-flag removal, bug reproduction, isolated internal tools, documentation updates, and well-bounded implementation slices.
• Write a delegation brief with objective, scope, non-goals, allowed files, relevant context, commands to run, acceptance criteria, and conditions that require stopping for a human.
• Split larger features into inspectable stages: investigate, propose, implement a narrow slice, test, and summarize the diff.
• Set checkpoints for long-running work and require a final evidence report containing changed files, tests run, unresolved assumptions, and risks.
• Keep high-coupling or high-risk decisions human-led even if an agent performs research, test creation, or a bounded implementation step.

Hands-on lab: Run a background task from a real maintenance backlog item, then score it with a delegation rubric: stayed in scope, used the right context, produced a reviewable diff, ran meaningful checks, and reported uncertainty.

4. AI-assisted planning and disposable prototypes

Northcove engineers do substantial scoping because PRDs are thin, while current technical specs try to serve too many audiences. Training should show how AI can reduce alignment cost without turning generated plans into false certainty.

Focus on:

• Turning an ambiguous PRD into questions, assumptions, user flows, system impacts, risks, and a thin technical decision record.
• Separating product/strategic alignment from implementation decisions instead of forcing both into one oversized spec.
• Generating directionally correct, disposable prototypes to make tradeoffs concrete—especially for quarterly alignment and user-facing flows.
• Using prototypes as conversation artifacts, not production foundations; label assumptions and expected throwaway code.
• Asking AI to summarize decisions and open questions after meetings, with a named owner and freshness date for durable artifacts.

Hands-on lab: Convert a thin PRD into (1) an assumptions/questions brief, (2) a disposable prototype or interaction sketch, and (3) a short implementation decision record. Participants identify which outputs are evidence and which are hypotheses.

5. Production LLM feature engineering — a separate advanced track

This material is essential for engineers building features such as document interpretation, but it need not occupy the general course for all 100 engineers.

Cover:

• Versioned golden datasets with representative slices by document source, format, terminology, and known failure mode.
• Clear task-level metrics and thresholds; do not rely only on one aggregate score or weak proxy metrics.
• Repeated runs or uncertainty analysis where nondeterminism matters.
• Prompt/model/configuration versioning and regression evaluation before rollout.
• Error taxonomy and slice-based analysis to determine whether to change the prompt, context, parser, model, or product fallback.
• Human review/fallback paths, monitoring, data isolation, retention, and incident response.
• Automated prompt optimization (including DSPy-style approaches) only after the evaluation set and objective are trustworthy; otherwise it optimizes the wrong target.

Hands-on lab: Use a sanitized subset of the document corpus to define slices and an error taxonomy, run a prompt change against a fixed baseline, and write a ship/no-ship recommendation that includes regressions—not just the average gain.

Suggested delivery design

Use Northcove code and sanitized cases throughout; generic toy examples will not address the stated problem.

1. Pre-work (30–45 minutes): Each participant brings one recently successful AI interaction and one failure, with sensitive data removed. Collect baseline data on task type, time, rework, and escaped defects.
2. Core workshop (one day): Context engineering; risk classification; verification; a paired monolith exercise.
3. Applied workshop (half to one day): Agent delegation plus planning/prototyping. Participants work on real, bounded backlog items.
4. Advanced clinic: Production LLM evaluation for the smaller group building AI features.
5. 30-day reinforcement: Office hours, a shared prompt/rules cookbook, and review of a small set of real AI-assisted changes.

Pair less-experienced users with power users, but standardize the review rubric so success does not depend on personal prompting folklore.

Concrete artifacts the training should leave behind

• A Northcove AI task-risk matrix ratified by Engineering, Security, Legal, and Compliance.
• A reusable context-packet template.
• An agent delegation brief and completion-report template.
• A PR review checklist for AI-assisted changes.
• A small, curated set of current Python/TypeScript implementation exemplars and explicit “do not copy” legacy patterns, each with an owner and review date.
• An approved-data/tool policy in plain language.
• For LLM features: an evaluation harness template, prompt/model version log, and ship/no-ship scorecard.

These artifacts matter more than a long list of clever prompts: they convert individual expertise into organizational capability.

What not to emphasize

• Basic autocomplete or generic prompt-writing tips; adoption has moved beyond that.
• “Build a whole feature from one prompt” demonstrations; they ignore the coupling and risk the engineers described.
• MCP setup as the centerpiece; current integrations are unreliable and CLI/repository workflows appear more practical.
• A giant universal Northcove system prompt; it will become another stale document.
• Raw output volume, lines of AI-written code, or prompt count as success metrics.
• Autonomous database migrations, regulated-data work, or financial logic as beginner agent exercises.

How to measure whether the training worked

Evaluate outcomes by task and risk tier rather than seeking one organization-wide “AI productivity” number.

Track for 30–60 days:

• Cycle time and human hands-on time for selected repeatable tasks.
• Review rounds, rework rate, escaped defects, and rollback rate for AI-assisted versus comparable work.
• Percentage of agent tasks that remain in scope and meet their stated acceptance criteria.
• Adoption and reuse of approved rules, context packets, delegation briefs, and exemplars.
• Test quality and invariant coverage, not merely test count or coverage percentage.
• For production LLM features: performance by critical data slice, regression count, human-review/fallback rate, and production incident rate.

A good pilot would choose two or three repeatable task types—such as test expansion, feature-flag cleanup, and bug reproduction—plus one document-interpretation evaluation. Establish a pre-training baseline, run the new workflow for a month, and expand only where speed improves without worse quality or risk.

Assumptions and boundaries

• This synthesis assumes the three interviews are directional rather than a statistically representative survey of all 100 engineers. Validate the curriculum with a short pre-training survey and task sampling, but do not delay the pilot.
• The risk tiers above are a proposed training heuristic, not Northcove policy. Security, Legal, and Compliance should ratify the actual tool and data boundaries.
• “Northcove” is used as the company name because it appears consistently in the supplied notes.
• The recommendation assumes the goal is engineering enablement, not selecting or purchasing a new AI platform.