Thematic
meta-muse-spark-1.3run 8 Sep 202615,780 tokens
Instructions
Instructions — the case's current instructions
help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers
help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers
AI Engineering Training Focus for Northcove — Synthesis from 3 Engineer Interviews
Assumption: Audience is Northcove's 100-engineer org (Python backend / TypeScript frontend, monolithic app, 4-10 YOE, doubled YoY) context.md:132-143. Training goal is applied AI engineering productivity with safety, not generic LLM theory. Noted where evidence is thin.
Executive Summary
Focus training on 5 convergent needs, in priority order:
- Context engineering for Northcove-specific code — unanimous #1 blocker
- Evals, testing & verification for AI output — both LLM features and AI-generated code
- Compliance-safe AI use (regulated data) — prerequisite for trust, especially with seniors
- AI-first + agentic workflows — close the basic → power-user gap
- AI prototyping for planning/alignment — cheap, already working, reduces meeting load
De-emphasize: deep MCP setup, DSPy optimization, automated DB migrations, AI user-research, PRD generation.
1. What all three agree on
A. "Context is king" but Northcove context is hard to supply
- Ben explicit principle: tag relevant files upfront, don't just say "do the thing"
context.md:110. Needs org-standard rules/commands/reusable promptscontext.md:113-114. - Nina: AI over-engineers complex prompts, cherry-picks wrong legacy patterns; works better with similar concrete examples vs. general prompts
context.md:31-34. Only basic system prompt today, can't encode detailed standardscontext.md:40-42. - Raj: AI "not good at Northcove-specific implementations," needs "very prescriptive custom rules"
context.md:171-174. Knowledge siloed across Confluence/Slack/docs, goes stalecontext.md:167,181. - Ben adds: production code has antipatterns/coupling/hidden side effects from startup speed; "if you turn one knob, something else falls down"
context.md:83-85.
Training implication: This is module #1. Teach: file-tagging, minimal high-signal context, example-driven prompting, writing/maintaining Cursor rules + commands for Python/TS monolith, how to exclude legacy antipatterns. Build a shared Northcove prompt/rule library during training for buy-in.
B. Verification gap: plausible-looking output + weak guardrails
- Nina: AI code looks plausible but subtle issues; lacks robust e2e/integration testing and monitoring; risk of accepting "good enough" without verification
context.md:57-61. - Nina's production LLM feature (document interpretation, Anthropic with isolation contracts)
context.md:9-14: manual 100-250 labeled docs, iterative prompt tweaking "more labor intensive than traditional ML"context.md:16-18, non-determinism on small changescontext.md:20, now shifting to proxy metrics at scalecontext.md:19. - Ben works in correctness-critical space: multi-party calculations, estimates/invoicing/Stripe
context.md:71-73. He avoids AI for production features for context reasons, but uses it heavily for throwaways/scriptscontext.md:83-90. - Raj confidence scale: codegen OK "needs human review, especially for database/regulated data"
context.md:165; "skeptical" on non-trivial features, migrations, regulated-data workcontext.md:168.
Training implication: Module #2. Two tracks: (a) evals for LLM features — ground-truth sets, prompt versioning, non-determinism handling, proxy metrics; (b) verification for AI-generated code — targeted tests, review checklists for DB/regulated paths, when to reject. Use Nina's doc-interpretation case as live example. Introduce DSPy/prompt-optimization only as awareness (Ben aware but no time context.md:107), not hands-on.
C. Compliance caution is load-bearing
- Raj: regulated-data exposure concern, prior third-party sharing incident required rework, litigation caution
context.md:175-178. Legal/compliance always factors incontext.md:153. - Nina already uses isolation contracts
context.md:14; Raj notes review needed for DB/regulated datacontext.md:165.
Training implication: Module #3 (short, mandatory, upfront). What data can go to which model/tool, Cursor enterprise boundaries, Stripe/invoicing/regulated-data red lines, incident lesson. Required to win over suspicious experienced devs context.md:135.
D. Org is "AI-second," split between basic and power users
- Ben: never does rough drafts by hand, most org still "AI-second" not AI-first
context.md:77-78. Everyone on Cursor, value in autocompletecontext.md:79. Agents for background work (coverage, flag removal, debugging) just startingcontext.md:80. - Nina self-describes basic vs. Ben power user
context.md:26; uses chat for scaffolding/baselines/tests/debuggingcontext.md:22-30. Wants background agents, better incremental-change prompting, delegating larger featurescontext.md:52-56,63-66. - Raj: tab-complete popular, chat for plan-then-build, code search; "editors rather than authors — rarely >20 lines manually"
context.md:158-162; agents less adoptedcontext.md:162.
Training implication: Module #4. Teach AI-first loop explicitly (Ben's list: questions, reproduction, docs, calendar/Jira context.md:111), plan-then-build chat workflow context.md:160, background agents/subagents hands-on with narrow tasks (coverage, flag cleanup, debug). Address senior skepticism with correctness-critical demos, not hype.
E. Prototyping works and solves a planning pain
- Ben: current tech specs "too much" — strategy+resourcing+tech in one doc
context.md:94; proposes 3-4 day onsite + AI prototypes that are "directionally correct, don't need to compile" because "people are visual"context.md:97-101. "Directionally correct is the name of the game" for internal scriptscontext.md:112. - Nina: PM-built Vercel v0 mockups for user testing, lets non-technical iterate without Figma/eng
context.md:44-47. - Raj context: thin PRDs, eng does scoping
context.md:136-138; 1-week PRD-to-deploy for betscontext.md:146; too many meetings for senior ICscontext.md:180-182.
Training implication: Module #5. AI throwaway prototypes + one-off scripts/internal tools as sanctioned high-ROI use (Ben context.md:86-90). Tie to planning reform: slim tech specs, visual alignment. Include non-eng adjacent value (Cursor Slack integration popular context.md:117, PM copy/changes context.md:118, BI hookup needed context.md:121) so engineers know what to support vs. own.
2. Divergences to handle in training design
- Production code: Ben avoids AI for it; Nina uses for scaffolding/baselines; Raj says OK with review. Don't teach "AI writes your feature E2E." Teach scoped use + human ownership.
- Tooling depth: Ben tried MCP for Confluence/Jira/GitHub, hit perf issues, prefers CLI tools
context.md:103-105; subagents earlycontext.md:106. Don't center training on MCP. Teach CLI + DB connection (what works) and note MCP limits honestly. - Infra: Raj flags DB migrations pain + slow pipeline (CI 20-30min, deploy 40m-2h
context.md:150-152). Don't promise AI fixes migrations; teach skepticism + guardrails there.
3. Recommended curriculum (prioritized)
Must-have (Day 1):
- Context Engineering for monolith (90 min, hands-on in own repo): tagging, example-first prompts, Cursor rules/commands, anti-pattern exclusion. Output: contributed team rule.
- Verification & Evals (90 min): review checklist for AI diffs; eval harness demo using doc-interpretation pattern (100-250 labels → proxy metrics); e2e/monitoring minimums.
- Safe & Compliant Use (30 min): data boundaries, isolation contracts, regulated-data decision tree + incident recap.
High-value (Day 2 / follow-up):
4. Agentic Workflows (60 min): background agents for coverage/flag-removal/debug; prompting for incremental changes; when to delegate vs. pair.
5. Prototype-to-Align (60 min): directionally-correct sketches, v0 for PMs, slim specs; internal scripts clinic.
6. Knowledge workaround (30 min): pragmatic retrieval from stale docs/Slack without heavy MCP; team norm for keeping rules/docs fresh + Slack channels (cursor-nerds, background-agents context.md:81).
Defer / light touch only: MCP server building, DSPy full implementation, AI PRD generation (blocked by product workflows context.md:166), AI user research (n=3 volume too low context.md:169), DB-migration automation.
4. Format notes
- Hands-on in Northcove monolith, single-engineer scope (matches "most projects are single engineer"
context.md:148); avoid toy repos. - Level-split exercises: Nina-basic track (scaffold/test/debug) + Ben-power track (agents/subagents, prototype seeding).
- Leave with artifacts: shared rules/commands library, eval template, compliance one-pager — addresses "need foundation so everyone can benefit"
context.md:120. - Measure: % PRs with AI-assist + human review note, eval coverage on LLM features, reuse of shared prompts, reduction in spec churn/meeting load — aligns to Ben's platformy metric focus (loss/accuracy/recoup timeliness
context.md:74) and Raj's OKR structurecontext.md:139.
5. Assumptions noted
- No direct access to codebase, Cursor config, or compliance policy; assumes Cursor enterprise default + Anthropic isolation as stated.
- Assumes training time is limited (1-2 days); if only half-day available, do modules 1-3 only.
- Assumes Platform/Business/Consumer silos
context.md:128-131share stack enough for common rules; may need per-silo customization.