Thematic

meta-muse-spark-1.3run 8 Sep 202615,780 tokens

Instructions

Instructions — the case's current instructions

help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers

help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers

7 of 7 checks passed

ANSWER.mdthe model's output · 9.25 KBdownload

AI Engineering Training Focus for Northcove — Synthesis from 3 Engineer Interviews

Assumption: Audience is Northcove's 100-engineer org (Python backend / TypeScript frontend, monolithic app, 4-10 YOE, doubled YoY) context.md:132-143. Training goal is applied AI engineering productivity with safety, not generic LLM theory. Noted where evidence is thin.

Executive Summary

Focus training on 5 convergent needs, in priority order:

  1. Context engineering for Northcove-specific code — unanimous #1 blocker
  2. Evals, testing & verification for AI output — both LLM features and AI-generated code
  3. Compliance-safe AI use (regulated data) — prerequisite for trust, especially with seniors
  4. AI-first + agentic workflows — close the basic → power-user gap
  5. AI prototyping for planning/alignment — cheap, already working, reduces meeting load

De-emphasize: deep MCP setup, DSPy optimization, automated DB migrations, AI user-research, PRD generation.

1. What all three agree on

A. "Context is king" but Northcove context is hard to supply

  • Ben explicit principle: tag relevant files upfront, don't just say "do the thing" context.md:110. Needs org-standard rules/commands/reusable prompts context.md:113-114.
  • Nina: AI over-engineers complex prompts, cherry-picks wrong legacy patterns; works better with similar concrete examples vs. general prompts context.md:31-34. Only basic system prompt today, can't encode detailed standards context.md:40-42.
  • Raj: AI "not good at Northcove-specific implementations," needs "very prescriptive custom rules" context.md:171-174. Knowledge siloed across Confluence/Slack/docs, goes stale context.md:167,181.
  • Ben adds: production code has antipatterns/coupling/hidden side effects from startup speed; "if you turn one knob, something else falls down" context.md:83-85.

Training implication: This is module #1. Teach: file-tagging, minimal high-signal context, example-driven prompting, writing/maintaining Cursor rules + commands for Python/TS monolith, how to exclude legacy antipatterns. Build a shared Northcove prompt/rule library during training for buy-in.

B. Verification gap: plausible-looking output + weak guardrails

  • Nina: AI code looks plausible but subtle issues; lacks robust e2e/integration testing and monitoring; risk of accepting "good enough" without verification context.md:57-61.
  • Nina's production LLM feature (document interpretation, Anthropic with isolation contracts) context.md:9-14: manual 100-250 labeled docs, iterative prompt tweaking "more labor intensive than traditional ML" context.md:16-18, non-determinism on small changes context.md:20, now shifting to proxy metrics at scale context.md:19.
  • Ben works in correctness-critical space: multi-party calculations, estimates/invoicing/Stripe context.md:71-73. He avoids AI for production features for context reasons, but uses it heavily for throwaways/scripts context.md:83-90.
  • Raj confidence scale: codegen OK "needs human review, especially for database/regulated data" context.md:165; "skeptical" on non-trivial features, migrations, regulated-data work context.md:168.

Training implication: Module #2. Two tracks: (a) evals for LLM features — ground-truth sets, prompt versioning, non-determinism handling, proxy metrics; (b) verification for AI-generated code — targeted tests, review checklists for DB/regulated paths, when to reject. Use Nina's doc-interpretation case as live example. Introduce DSPy/prompt-optimization only as awareness (Ben aware but no time context.md:107), not hands-on.

C. Compliance caution is load-bearing

  • Raj: regulated-data exposure concern, prior third-party sharing incident required rework, litigation caution context.md:175-178. Legal/compliance always factors in context.md:153.
  • Nina already uses isolation contracts context.md:14; Raj notes review needed for DB/regulated data context.md:165.

Training implication: Module #3 (short, mandatory, upfront). What data can go to which model/tool, Cursor enterprise boundaries, Stripe/invoicing/regulated-data red lines, incident lesson. Required to win over suspicious experienced devs context.md:135.

D. Org is "AI-second," split between basic and power users

  • Ben: never does rough drafts by hand, most org still "AI-second" not AI-first context.md:77-78. Everyone on Cursor, value in autocomplete context.md:79. Agents for background work (coverage, flag removal, debugging) just starting context.md:80.
  • Nina self-describes basic vs. Ben power user context.md:26; uses chat for scaffolding/baselines/tests/debugging context.md:22-30. Wants background agents, better incremental-change prompting, delegating larger features context.md:52-56,63-66.
  • Raj: tab-complete popular, chat for plan-then-build, code search; "editors rather than authors — rarely >20 lines manually" context.md:158-162; agents less adopted context.md:162.

Training implication: Module #4. Teach AI-first loop explicitly (Ben's list: questions, reproduction, docs, calendar/Jira context.md:111), plan-then-build chat workflow context.md:160, background agents/subagents hands-on with narrow tasks (coverage, flag cleanup, debug). Address senior skepticism with correctness-critical demos, not hype.

E. Prototyping works and solves a planning pain

  • Ben: current tech specs "too much" — strategy+resourcing+tech in one doc context.md:94; proposes 3-4 day onsite + AI prototypes that are "directionally correct, don't need to compile" because "people are visual" context.md:97-101. "Directionally correct is the name of the game" for internal scripts context.md:112.
  • Nina: PM-built Vercel v0 mockups for user testing, lets non-technical iterate without Figma/eng context.md:44-47.
  • Raj context: thin PRDs, eng does scoping context.md:136-138; 1-week PRD-to-deploy for bets context.md:146; too many meetings for senior ICs context.md:180-182.

Training implication: Module #5. AI throwaway prototypes + one-off scripts/internal tools as sanctioned high-ROI use (Ben context.md:86-90). Tie to planning reform: slim tech specs, visual alignment. Include non-eng adjacent value (Cursor Slack integration popular context.md:117, PM copy/changes context.md:118, BI hookup needed context.md:121) so engineers know what to support vs. own.

2. Divergences to handle in training design

  • Production code: Ben avoids AI for it; Nina uses for scaffolding/baselines; Raj says OK with review. Don't teach "AI writes your feature E2E." Teach scoped use + human ownership.
  • Tooling depth: Ben tried MCP for Confluence/Jira/GitHub, hit perf issues, prefers CLI tools context.md:103-105; subagents early context.md:106. Don't center training on MCP. Teach CLI + DB connection (what works) and note MCP limits honestly.
  • Infra: Raj flags DB migrations pain + slow pipeline (CI 20-30min, deploy 40m-2h context.md:150-152). Don't promise AI fixes migrations; teach skepticism + guardrails there.

3. Recommended curriculum (prioritized)

Must-have (Day 1):

  1. Context Engineering for monolith (90 min, hands-on in own repo): tagging, example-first prompts, Cursor rules/commands, anti-pattern exclusion. Output: contributed team rule.
  2. Verification & Evals (90 min): review checklist for AI diffs; eval harness demo using doc-interpretation pattern (100-250 labels → proxy metrics); e2e/monitoring minimums.
  3. Safe & Compliant Use (30 min): data boundaries, isolation contracts, regulated-data decision tree + incident recap.

High-value (Day 2 / follow-up):
4. Agentic Workflows (60 min): background agents for coverage/flag-removal/debug; prompting for incremental changes; when to delegate vs. pair.
5. Prototype-to-Align (60 min): directionally-correct sketches, v0 for PMs, slim specs; internal scripts clinic.
6. Knowledge workaround (30 min): pragmatic retrieval from stale docs/Slack without heavy MCP; team norm for keeping rules/docs fresh + Slack channels (cursor-nerds, background-agents context.md:81).

Defer / light touch only: MCP server building, DSPy full implementation, AI PRD generation (blocked by product workflows context.md:166), AI user research (n=3 volume too low context.md:169), DB-migration automation.

4. Format notes

  • Hands-on in Northcove monolith, single-engineer scope (matches "most projects are single engineer" context.md:148); avoid toy repos.
  • Level-split exercises: Nina-basic track (scaffold/test/debug) + Ben-power track (agents/subagents, prototype seeding).
  • Leave with artifacts: shared rules/commands library, eval template, compliance one-pager — addresses "need foundation so everyone can benefit" context.md:120.
  • Measure: % PRs with AI-assist + human review note, eval coverage on LLM features, reuse of shared prompts, reduction in spec churn/meeting load — aligns to Ben's platformy metric focus (loss/accuracy/recoup timeliness context.md:74) and Raj's OKR structure context.md:139.

5. Assumptions noted

  • No direct access to codebase, Cursor config, or compliance policy; assumes Cursor enterprise default + Anthropic isolation as stated.
  • Assumes training time is limited (1-2 days); if only half-day available, do modules 1-3 only.
  • Assumes Platform/Business/Consumer silos context.md:128-131 share stack enough for common rules; may need per-silo customization.