Thematic

meta-muse-wonderrun 13 Oct 202645,774 tokens

Instructions

Instructions — what the model was asked

help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers

help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers

8 of 8 checks passed

ANSWER.mdthe model's output · 18.5 KBdownload

AI Engineering Training for Northcove — Synthesis & Recommendations

Based on meeting notes with Nina Park (Full-Stack Engineer), Ben Walsh (Senior Engineer, Business), and Raj (Engineering Org overview). Assumption: The audience is Northcove's ~100 engineers. Duration/format not specified, so this recommends a core curriculum + advanced track that can be delivered as a half-day intensive or a 3-4 session series.


1. Executive Summary: What to Focus On

Northcove is not starting from zero — and the training shouldn't act like it is.

Everyone is on Cursor Enterprise. Tab-complete is ubiquitous. The team is, in Raj's words, now "editors rather than authors — rarely write more than 20 lines by hand." But adoption is uneven and shallow:

  • Ben is a power user (never hand-writes a rough draft, experimenting with background agents/subagents, wants DSPy). Nina self-describes as a "basic user" (Cursor chat for scaffolding, tests, debugging).
  • Most of the org is still "AI-second, not AI-first" (Ben).
  • Agents are being experimented with, but are not yet core workflow for anyone.
  • Production-feature coding is the least trusted use case, due to Northcove-specific coupling, legacy antipatterns, and regulated data.

The training gap is not "how to prompt ChatGPT." It's Northcove-specific context, verification, and evals.

Prioritized focus for the training, in order:

# Focus Area Why It's #1 Driver
1 Context Engineering All 3 cite this unsolved; it's Ben's "Context is king" principle and Nina/Raj's #1 failure mode
2 Verification, Testing & Guardrails Correctness-critical business + plausible-but-wrong AI code + weak E2E coverage = biggest risk
3 Evals for LLM Features Only live LLM product (doc interpretation) has manual, labor-intensive evals; this is Northcove's differentiation skill
4 Background Agents & Incremental Delegation Highest curiosity (Nina + Ben), lowest current competence, high payoff if done safely
5 AI-First Planning & Prototyping Already emerging organically (v0 mockups, Ben's prototypes) but not standardized; fixes bloated tech specs
6 Northcove Rules, Prompts & Tooling (MCP vs CLI) No shared rules/commands library today — everyone is re-inventing and hitting legacy patterns
7 Compliance & Data Boundaries Cross-cutting constraint. Regulated data, prior 3rd-party incident, litigation caution — must be taught inside every module, not as an afterthought

Do NOT center the training on: generic prompt-engineering tricks, writing code from scratch with AI, or autonomous building of non-trivial features / DB migrations / user research. All three engineers are skeptical or say the tools aren't ready there, and centering on it will lose the experienced (and already AI-skeptical) senior cohort.


2. Company Context the Training Must Be Built Around

These facts from the notes constrain what will work:

  • Scale & culture: 100 engineers (Consumer / Business / Platform silos), doubled YoY, typically 4–10 yrs experience. Strong autonomy culture; PRDs are thin, engineers do a lot of scoping. More experienced engineers are more suspicious of AI.
  • Stack: Python backend, TypeScript frontend, monolithic app, standard libraries only. CI 20–30 min, deploy 40 min–2 hrs, continuous deployment. Projects are mostly single-engineer, 1 week (bets) to 5–6 weeks (infra).
  • The one production LLM feature: Document interpretation (Nina, v2) — LLM scans uploaded documents with varying layouts/terminology, using an Anthropic model under organizational data-isolation contracts. This is the best internal case study you have. Use it.
  • Business risk profile: Business group does correctness-critical, multi-party money calculations (estimates, invoicing, Stripe, loss per transaction, recoups/refunds). "If you turn one knob, something else falls down elsewhere." This is not a "move fast and let AI ship it" org.
  • Knowledge problem: Docs are stale and scattered across Google Docs, Confluence, and Slack. There is no structured way to feed docs to Cursor. This is an engineering-systems problem, not just a training problem.
  • Compliance overhang: Prior third-party data-sharing incident requiring rework, plus litigation concerns. Legal/compliance factors into every decision. Any training that doesn't give engineers a crisp "what can / cannot go into a model" rule will either be ignored or will create fear.

3. Thematic Synthesis — What the Three Engineers Converge On

Theme 1: Context is the bottleneck, not prompting cleverness

  • Ben: "Context is king — don't just say 'do the thing,' tag relevant files upfront." Quality output requires standardized Northcove-specific prompts/guidelines. Org buy-in needed for rules/commands/reusable prompts.
  • Nina: AI over-engineers on complex prompts and "cherry-picks wrong context from legacy code patterns." Works far better with concrete examples from the codebase than general prompts. Has only a basic system prompt (Python/framework specs); can't yet encode detailed standards.
  • Raj: AI is "not good at Northcove-specific implementations." Needs "very prescriptive custom rules." Knowledge in Confluence/Slack/docs goes stale and is hard to find.

Implication: The single highest-leverage module is teaching engineers to curate context: which files to tag, how to write Cursor Rules / CLAUDE.md-style project instructions for Python/TS conventions, how to give one golden example instead of a paragraph of instructions, and how to recognize when the model has latched onto a legacy antipattern and intervene.

Theme 2: AI output looks right and is subtly wrong — verification is underbuilt

  • Nina: "AI code looks plausible but may introduce subtle issues." Lack of robust E2E/integration testing. Risk of accepting "good enough" output. Need better monitoring/validation.
  • Ben: Deliberately does not use AI much for production features — can't supply all context, hidden side effects and coupling from fast startup development. Uses it heavily for throwaway sketches, planning, debugging, one-off scripts instead.
  • Raj: Confidence is highest for dev-assistance codegen with human review, especially for database/regulated data. Skeptical for significant non-trivial features and DB migrations.

Implication: Training must teach a verification discipline, not just generation. Diff-review patterns, writing the test first and having AI pass it, using AI to write tests for existing suites (already Nina's workflow — formalize it), and a clear tiering: throwaway/internal (directionally correct is fine, per Ben) vs. production correctness-critical (adversarial review mandatory).

Theme 3: Evals are manual, painful, and where the real LLM-product skill lives

Only Nina is building an LLM feature, but her pain is the clearest signal of a capability Northcove lacks org-wide:

  • Manually labeled 100–250 documents as ground truth. Iterative prompt tweaking is more labor-intensive than traditional ML training. Small prompt changes cause non-deterministic swings. Only recently shifting to larger-scale proxy metrics.

Implication: If Northcove ships more LLM features (and document intake sits in Ben's group too), "prompt and pray" will not scale. Engineers need eval literacy: building golden datasets, choosing proxy vs. human labels, regression-testing prompts, and (advanced) automated prompt optimization. Ben already knows about DSPy but has had no time to implement it — this is the advanced-track hook.

Theme 4: Everyone wants background agents; nobody has a safe pattern yet

  • Nina: Current workflow is "very back-and-forth, time-consuming." Wants long-running background processes, better prompting for incremental changes, delegating larger features to independent agents.
  • Ben: Starting to use agents for test coverage, feature-flag removal, debugging. Using subagents, but "not core to workflow yet." Active Slack channels (cursor-nerds, background-agents) show grassroots demand.
  • Raj: "Agents less adopted, some early adopters experimenting."

Implication: Teach agents as scoped, reviewable delegation — small briefs, checkpoints, incremental diffs — not "give it a feature and walk away." Anchor on Ben's proven use cases (test coverage, flag removal, debugging) and Nina's pain (incremental changes), and explicitly pair this module with Theme 2 guardrails.

Theme 5: AI is already reshaping planning — formalize it

  • Ben: Quarterly tech specs are "too much" — one doc trying to do strategy, deadlines, resourcing, sequencing, and technical alignment. Hard to write, hard to review. He uses AI-generated prototypes for alignment: "People are visual… prototypes just need to be directionally correct, don't need to compile. Cheap to throw away now."
  • Nina: PM uses Vercel v0 mockups for interactive user testing, enabling non-technical iteration without Figma or engineer-built prototypes.
  • Raj: PRD development "could work" with AI but isn't used due to product-org workflows; most projects start from thin PRDs anyway.

Implication: A short, practical module on AI-first planning will land well because it solves a felt process pain (bloated specs, slow alignment), not just a coding task. Teach: prototype-to-align, disposable sketches before specs, splitting the mega-spec into smaller AI-drafted artifacts, and Ben's "directionally correct is the name of the game" for internal tools.

Theme 6: Tooling beyond autocomplete is fragmented (MCP, rules, sharing)

  • Ben: Not effectively using MCP beyond DB connections; tried Confluence/Jira/GitHub MCP but hit performance issues; prefers CLI tools (in the training set) over configuring MCP servers. Aware of DSPy, no time to implement.
  • Raj: Early Cursor adopters, experimented with Claude Code. Chat-for-planning-then-building (Cursor-team recommended) and code search are established.
  • All: Sharing happens ad-hoc in Slack (cursor-nerds, background-agents, eng channels). Raj flags siloed knowledge management as a "major gap."

Implication: Don't teach MCP configuration as a centerpiece — Ben's experience suggests CLI-first will get more traction. Instead, teach when to reach for MCP vs CLI vs native search, and spend the time on building a shared, version-controlled library of Northcove Rules, commands, and reusable prompts that the org actually maintains.

Theme 7: Compliance is the constraint that can veto everything else

  • Raj: Concern about regulated-data exposure; prior incident; litigation caution shapes decisions.
  • Nina: Document feature runs on Anthropic with organizational data isolation contracts — engineers need to understand what that does and doesn't cover.
  • Ben: DB/regulated work is where human review is non-negotiable.

Implication: Embed a 15-minute "data boundaries" segment up front (what data classes exist at Northcove, what's approved for Cursor/Anthropic under current contracts, what never gets pasted) and repeat the relevant rule inside each hands-on lab. A standalone legal lecture at the end will be tuned out.


4. Recommended Training Design

Audience segmentation

Segment Who What they need
Core (all engineers) ~100 engineers, mixed AI confidence Modules 1–3, 5, 7 below
Power-user / champion track Ben and peers, Slack-channel regulars, early agent experimenters Modules 4, 6 + DSPy/eval-optimization deep dive
Optional extension PMs, BI, non-eng Cursor users (already asking via Slack integration) Lightweight version of Module 5 (prototyping) only — don't dilute the engineering core for this

Explicitly plan for the skeptical senior cohort Raj flagged: lead with limitations and review discipline, use their correctness-critical examples, and recruit Ben-type power users as co-instructors/demo-ers rather than external evangelists.

Core curriculum (priority order)

Module 1 — Context Engineering at Northcove (hands-on, Python/TS monolith)
Tagging relevant files upfront; writing and maintaining Cursor Rules for Northcove Python/TS conventions; example-driven prompting (one golden file > long instructions); spotting legacy-antipattern contamination and correcting it; building a team context pack where docs are stale. Lab: take a real Business-group-style coupled change, do it once with a vague prompt (watch it over-engineer), once with curated context + example.

Module 2 — Verification & Guardrails for AI-Generated Code
Review-tier framework (throwaway / internal tool / production / correctness-critical & regulated). Test-first AI workflows; using AI to raise test coverage safely; diff-review checklists for subtle issues (side effects, coupling, DB access); when not to use AI (non-trivial features, migrations, regulated data — per Raj's confidence ladder). Lab: seed a plausible-but-buggy AI diff in a money-calculation path; teams find the subtle bug and write the guardrail test.

Module 3 — Evals for LLM Features (using document interpretation as the case study)
Golden datasets (how Nina's 100–250 doc set was built, and its limits); human labels vs proxy metrics and when to switch; prompt regression testing under non-determinism; error analysis on varying layouts/terminology. Lab: given a small labeled doc set, tweak a prompt, measure against the set, observe the non-deterministic swing, add a regression check. Advanced follow-on: DSPy-style automated prompt optimization (Ben's stated interest).

Module 4 — Background Agents & Incremental Delegation [Champion track, then roll out]
Scoping an agent brief; incremental prompting for small diffs vs big-bang feature requests; checkpoint/review cadence; proven starter tasks (test coverage, feature-flag removal, debugging, one-off scripts). Failure modes and abort criteria. Lab: delegate a flag-removal or test-coverage task to a background agent with a checkpoint plan.

Module 5 — AI-First Planning & Prototyping
Disposable prototypes for alignment (v0, Cursor sketches); replacing the mega tech-spec with smaller artifacts; Ben's prototype-before-spec workflow and Raj's thin-PRD reality; "directionally correct" standard for internal tools. Exercise: turn a bloated spec excerpt into a throwaway prototype + 1-page alignment doc in 30 minutes.

Module 6 — Shared Northcove Toolkit [Champion track + working session, not a lecture]
Building the org's version-controlled Rules/commands/prompt library; MCP vs CLI decision guide (CLI-first per Ben's experience, MCP where it earns its keep, e.g. DB); how to contribute patterns back via existing Slack channels into the library. Output of the session is the actual v1 library — training that ships an asset.

Module 7 — Data Boundaries & Compliance (15 min up front + reinforced in every lab)
Data classification at Northcove; what the Anthropic isolation contract covers; regulated-data handling and human-review requirements; lessons from the prior third-party incident (at whatever level of detail Legal approves). Deliverable: a one-page "safe / review / never" card engineers keep open while coding.

What to cut or defer

  • Generic "10 prompt hacks" content — this audience is past it and seniors will disengage.
  • Autonomous end-to-end feature building and DB-migration automation — explicitly marked "not ready" by Raj; teaching it now undermines credibility.
  • Deep MCP server configuration — low traction, performance complaints; cover as a decision guide only.
  • AI for user research — Raj notes only ~3 research contacts; insufficient volume.

5. Sequencing & Format Recommendations

  1. Start with champions. Run Module 4/6 with Ben and the background-agents/cursor-nerds regulars first. They become demo-ers and library maintainers for the core rollout, and their real Northcove examples replace generic ones.
  2. Core delivery: 3 sessions × ~90 min over 2–3 weeks (Modules 1+7, 2, 3+5 interleaved), each with a lab on the real Python/TS monolith — not toy examples. Single-engineer project norms mean labs should be solo/pair-sized and finishable.
  3. Ship assets, not just knowledge: Cursor Rules v1, prompt/command library, eval template from the doc-interpretation case, and the compliance one-pager. The stale-docs problem means unmaintained training material will rot — assign owners (champions) and a review cadence.
  4. Measure it: Baseline and follow up on (a) % of PRs using shared Rules, (b) test coverage on AI-touched code, (c) eval coverage on LLM features (golden set size, regression runs per prompt change), (d) background-agent tasks completed with review pass. Avoid vanity metrics like "AI lines generated" — Raj's "editors not authors" shift makes line counts meaningless.

6. Risks If You Train the Wrong Things

  • Training only generation, not verification, in a correctness-critical money system with weak E2E coverage, increases the "plausible but subtly wrong" incident rate Nina flagged.
  • Ignoring the context/docs problem means every technique taught degrades against stale Confluence/Slack knowledge. Pair the training with even a minimal effort to curate a Cursor-feedable context source for the highest-traffic domains.
  • One-size-fits-all delivery will bore power users (Ben) and overwhelm basic users (Nina) simultaneously. The champion/core split above is the fix.
  • Compliance as an afterthought risks either a regulated-data incident or blanket fear that kills adoption. Give engineers crisp rules early.

7. One-Paragraph Brief (for stakeholders)

Northcove's engineers already generate most of their code with AI but lack shared, Northcove-specific practices for the things that actually determine quality and risk: curating context in a coupled Python/TypeScript monolith with stale docs, verifying plausible-but-subtly-wrong output in correctness-critical money flows, and evaluating their one production LLM feature (document interpretation) beyond manual prompt-tweaking. The training should therefore focus, in order, on context engineering, verification/guardrails, and LLM evals, with background-agent delegation and AI-first prototyping as high-interest follow-ons, a shared Rules/prompt library built during the training, and data-boundary rules embedded throughout given regulated-data and litigation sensitivity. Generic prompt training, autonomous feature building, and DB-migration automation should be explicitly deferred — the org's own engineers rate them not-ready, and leading with them would lose the experienced cohort whose buy-in the rollout depends on.