04 thematic

AI Engineering Training Focus for Northcove — Synthesis from 3 Engineer Interviews

Assumption: Audience is Northcove's 100-engineer org (Python backend / TypeScript frontend, monolithic app, 4-10 YOE, doubled YoY) context.md:132-143. Training goal is applied AI engineering productivity with safety, not generic LLM theory. Noted where evidence is thin.

Executive Summary

Focus training on 5 convergent needs, in priority order:

1. Context engineering for Northcove-specific code — unanimous #1 blocker
2. Evals, testing & verification for AI output — both LLM features and AI-generated code
3. Compliance-safe AI use (regulated data) — prerequisite for trust, especially with seniors
4. AI-first + agentic workflows — close the basic → power-user gap
5. AI prototyping for planning/alignment — cheap, already working, reduces meeting load

De-emphasize: deep MCP setup, DSPy optimization, automated DB migrations, AI user-research, PRD generation.

1. What all three agree on

A. "Context is king" but Northcove context is hard to supply

• Ben explicit principle: tag relevant files upfront, don't just say "do the thing" context.md:110. Needs org-standard rules/commands/reusable prompts context.md:113-114.
• Nina: AI over-engineers complex prompts, cherry-picks wrong legacy patterns; works better with similar concrete examples vs. general prompts context.md:31-34. Only basic system prompt today, can't encode detailed standards context.md:40-42.
• Raj: AI "not good at Northcove-specific implementations," needs "very prescriptive custom rules" context.md:171-174. Knowledge siloed across Confluence/Slack/docs, goes stale context.md:167,181.
• Ben adds: production code has antipatterns/coupling/hidden side effects from startup speed; "if you turn one knob, something else falls down" context.md:83-85.

Training implication: This is module #1. Teach: file-tagging, minimal high-signal context, example-driven prompting, writing/maintaining Cursor rules + commands for Python/TS monolith, how to exclude legacy antipatterns. Build a shared Northcove prompt/rule library during training for buy-in.

B. Verification gap: plausible-looking output + weak guardrails

• Nina: AI code looks plausible but subtle issues; lacks robust e2e/integration testing and monitoring; risk of accepting "good enough" without verification context.md:57-61.
• Nina's production LLM feature (document interpretation, Anthropic with isolation contracts) context.md:9-14: manual 100-250 labeled docs, iterative prompt tweaking "more labor intensive than traditional ML" context.md:16-18, non-determinism on small changes context.md:20, now shifting to proxy metrics at scale context.md:19.
• Ben works in correctness-critical space: multi-party calculations, estimates/invoicing/Stripe context.md:71-73. He avoids AI for production features for context reasons, but uses it heavily for throwaways/scripts context.md:83-90.
• Raj confidence scale: codegen OK "needs human review, especially for database/regulated data" context.md:165; "skeptical" on non-trivial features, migrations, regulated-data work context.md:168.

Training implication: Module #2. Two tracks: (a) evals for LLM features — ground-truth sets, prompt versioning, non-determinism handling, proxy metrics; (b) verification for AI-generated code — targeted tests, review checklists for DB/regulated paths, when to reject. Use Nina's doc-interpretation case as live example. Introduce DSPy/prompt-optimization only as awareness (Ben aware but no time context.md:107), not hands-on.

C. Compliance caution is load-bearing

• Raj: regulated-data exposure concern, prior third-party sharing incident required rework, litigation caution context.md:175-178. Legal/compliance always factors in context.md:153.
• Nina already uses isolation contracts context.md:14; Raj notes review needed for DB/regulated data context.md:165.

Training implication: Module #3 (short, mandatory, upfront). What data can go to which model/tool, Cursor enterprise boundaries, Stripe/invoicing/regulated-data red lines, incident lesson. Required to win over suspicious experienced devs context.md:135.

D. Org is "AI-second," split between basic and power users

• Ben: never does rough drafts by hand, most org still "AI-second" not AI-first context.md:77-78. Everyone on Cursor, value in autocomplete context.md:79. Agents for background work (coverage, flag removal, debugging) just starting context.md:80.
• Nina self-describes basic vs. Ben power user context.md:26; uses chat for scaffolding/baselines/tests/debugging context.md:22-30. Wants background agents, better incremental-change prompting, delegating larger features context.md:52-56,63-66.
• Raj: tab-complete popular, chat for plan-then-build, code search; "editors rather than authors — rarely >20 lines manually" context.md:158-162; agents less adopted context.md:162.

Training implication: Module #4. Teach AI-first loop explicitly (Ben's list: questions, reproduction, docs, calendar/Jira context.md:111), plan-then-build chat workflow context.md:160, background agents/subagents hands-on with narrow tasks (coverage, flag cleanup, debug). Address senior skepticism with correctness-critical demos, not hype.

E. Prototyping works and solves a planning pain

• Ben: current tech specs "too much" — strategy+resourcing+tech in one doc context.md:94; proposes 3-4 day onsite + AI prototypes that are "directionally correct, don't need to compile" because "people are visual" context.md:97-101. "Directionally correct is the name of the game" for internal scripts context.md:112.
• Nina: PM-built Vercel v0 mockups for user testing, lets non-technical iterate without Figma/eng context.md:44-47.
• Raj context: thin PRDs, eng does scoping context.md:136-138; 1-week PRD-to-deploy for bets context.md:146; too many meetings for senior ICs context.md:180-182.

Training implication: Module #5. AI throwaway prototypes + one-off scripts/internal tools as sanctioned high-ROI use (Ben context.md:86-90). Tie to planning reform: slim tech specs, visual alignment. Include non-eng adjacent value (Cursor Slack integration popular context.md:117, PM copy/changes context.md:118, BI hookup needed context.md:121) so engineers know what to support vs. own.

2. Divergences to handle in training design

• Production code: Ben avoids AI for it; Nina uses for scaffolding/baselines; Raj says OK with review. Don't teach "AI writes your feature E2E." Teach scoped use + human ownership.
• Tooling depth: Ben tried MCP for Confluence/Jira/GitHub, hit perf issues, prefers CLI tools context.md:103-105; subagents early context.md:106. Don't center training on MCP. Teach CLI + DB connection (what works) and note MCP limits honestly.
• Infra: Raj flags DB migrations pain + slow pipeline (CI 20-30min, deploy 40m-2h context.md:150-152). Don't promise AI fixes migrations; teach skepticism + guardrails there.

3. Recommended curriculum (prioritized)

Must-have (Day 1):

1. Context Engineering for monolith (90 min, hands-on in own repo): tagging, example-first prompts, Cursor rules/commands, anti-pattern exclusion. Output: contributed team rule.
2. Verification & Evals (90 min): review checklist for AI diffs; eval harness demo using doc-interpretation pattern (100-250 labels → proxy metrics); e2e/monitoring minimums.
3. Safe & Compliant Use (30 min): data boundaries, isolation contracts, regulated-data decision tree + incident recap.

High-value (Day 2 / follow-up):
4. Agentic Workflows (60 min): background agents for coverage/flag-removal/debug; prompting for incremental changes; when to delegate vs. pair.
5. Prototype-to-Align (60 min): directionally-correct sketches, v0 for PMs, slim specs; internal scripts clinic.
6. Knowledge workaround (30 min): pragmatic retrieval from stale docs/Slack without heavy MCP; team norm for keeping rules/docs fresh + Slack channels (cursor-nerds, background-agents context.md:81).

Defer / light touch only: MCP server building, DSPy full implementation, AI PRD generation (blocked by product workflows context.md:166), AI user research (n=3 volume too low context.md:169), DB-migration automation.

4. Format notes

• Hands-on in Northcove monolith, single-engineer scope (matches "most projects are single engineer" context.md:148); avoid toy repos.
• Level-split exercises: Nina-basic track (scaffold/test/debug) + Ben-power track (agents/subagents, prototype seeding).
• Leave with artifacts: shared rules/commands library, eval template, compliance one-pager — addresses "need foundation so everyone can benefit" context.md:120.
• Measure: % PRs with AI-assist + human review note, eval coverage on LLM features, reuse of shared prompts, reduction in spec churn/meeting load — aligns to Ben's platformy metric focus (loss/accuracy/recoup timeliness context.md:74) and Raj's OKR structure context.md:139.

5. Assumptions noted

• No direct access to codebase, Cursor config, or compliance policy; assumes Cursor enterprise default + Anthropic isolation as stated.
• Assumes training time is limited (1-2 days); if only half-day available, do modules 1-3 only.
• Assumes Platform/Business/Consumer silos context.md:128-131 share stack enough for common rules; may need per-silo customization.