Mike's Checks/deepseek-v4-flash/04 thematic
04 thematic
deepseek-v4-flashCodex CLIhigh effortrun 4 Sep 202647,491 tokens
▸Instructions — the case's current instructions; none were saved with this result
help me synthesize what to focus on in our AI engineering training for this company based on meeting notes with three engineers
Answer
AI Engineering Training Recommendations for Northcove
Based on interviews with Nina (Full Stack), Ben (Senior Software Engineer, Business group), and Raj (Engineering leadership context), the following cross-cutting themes emerged. Below is a prioritized recommendation for training focus areas.
1. Context Engineering & Knowledge Management
Why it matters: All three engineers independently identified context as the #1 bottleneck. Poor documentation, scattered knowledge (Confluence, Slack, Google Docs, stale repos), and inability to feed company-specific context to AI tools consistently degrades output quality.
What to train:
- Techniques for tagging relevant files upfront when prompting (Ben: "don't just say 'do the thing'")
- Building and maintaining structured system prompts with coding standards, framework versions, and architectural conventions
- Using project-level rules files (e.g.,
AGENTS.md,.cursorrules) to encode Northcove-specific patterns - Strategies for keeping context artifacts fresh alongside code
2. From AI-Second to AI-First Workflow Integration
Why it matters: Ben describes most of the org as still in "AI-second" mode. Raj notes the team has transitioned to "editors rather than authors" but agents are still underutilized. There's an appetite to go further.
What to train:
- Practical AI-first patterns: using AI for planning, debugging, reproduction, documentation, and one-off scripts — not just codegen
- Embedding AI into daily rituals: standups, ticket creation, code review prep
- Knowing when to use AI vs. when to write manually (correctness-critical, regulated data, database migrations)
- Organizational playbooks for reusable prompts and commands (Ben's recommendation)
3. Prompt Engineering Beyond Chat
Why it matters: Nina identifies herself as a "basic user" and wants better prompt engineering for incremental changes. Ben is experimenting with DSPy and subagents but hasn't operationalized them. There's a spectrum of skill that training can level up.
What to train:
- Structuring prompts for incremental, surgical edits vs. big batch generation (Nina's pain point)
- Providing concrete examples from the codebase rather than abstract descriptions (common failure pattern)
- Techniques for prompt iteration and evaluation (especially for LLM-powered features like Nina's document interpretation system)
- Introduction to automated prompt optimization (DSPy, prompt management tools)
4. Testing, Validation & Guardrails for AI-Generated Code
Why it matters: Nina worries about "plausible but subtly wrong" code and a lack of robust end-to-end testing. Raj flags compliance risks with regulated data exposure. Ben notes he can't use AI for production features because of hidden side effects and coupling.
What to train:
- Building verification workflows: AI generates code → human-approved test plans → automated validation
- Techniques for generating test suites alongside code changes
- Identifying high-risk areas where AI output needs extra scrutiny (database migrations, financial calculations, regulated data)
- Monitoring and evaluation systems for AI-generated code in production
- Data exposure awareness when using third-party AI models
5. Agent Workflows & Background Automation
Why it matters: All three show interest in moving beyond chat and tab-complete toward autonomous agents. Nina wants long-running background processes for feature development. Ben uses agents for test coverage and feature flag removal. Raj notes agents are less adopted but early adopters are experimenting.
What to train:
- When to delegate to an agent vs. work interactively
- Structuring tasks for autonomous execution: clear scope, success criteria, boundaries
- Patterns for background agents: test writing, refactoring, code cleanup, documentation sync
- Safety mechanisms: sandboxed execution, review gates, rollback plans
6. Code Review Mindset for AI Outputs
Why it matters: The team is "editors rather than authors" (Raj). This shifts the skill required from writing code to evaluating code — a different muscle.
What to train:
- Reviewing AI-generated code with appropriate scrutiny levels based on risk
- Recognizing common AI failure patterns: over-engineering, wrong context cherry-picking, plausible but incorrect logic
- Balancing speed (AI generates faster) with quality (human review must adapt)
- Building checklists for AI code review tailored to Northcove's stack and compliance requirements
Suggested Training Format
| Module | Audience | Format | Estimated Time |
|---|---|---|---|
| Context Engineering | All engineers | Workshop + hands-on rules file creation | 2 hours |
| AI-First Workflows | All engineers | Lunch & learn + team playbooks | 1 hour |
| Prompt Engineering | Intermediate users (Nina's cohort) | Hands-on lab with real Northcove code | 2-3 hours |
| Testing & Guardrails | All engineers + QA | Workshop + checklist creation | 2 hours |
| Agent Workflows | Power users (Ben's cohort) | Demo + guided sandbox experimentation | 2 hours |
| Code Review for AI | All engineers | Pair-review sessions + guidelines doc | 1.5 hours |
Quick Wins (Do First)
- Create a Northcove
.cursorrules/AGENTS.mdtemplate with stack versions, architectural patterns, and compliance notes — every engineer can adopt immediately. - Run a "context sprint" where each team documents its top 5-10 conventions in a machine-readable format.
- Establish an internal Slack channel (Ben's
cursor-nerdsmodel) for sharing effective prompts and agent recipes. - Pick 2-3 agent use cases (test generation, feature flag removal, docs sync) and run a pilot with Ben's cohort, then socialize results.
Assumptions: Northcove uses Python/TypeScript on a monolithic architecture, has ~100 engineers, and operates with strong engineering autonomy and continuous deployment. Training should accommodate 4-10 years experience range and varying levels of AI skepticism.
Checks
6/7Analysis existsexists
a synthesis document of >= 100 words was produced
Theme counttheme-count
it groups its findings into at least 3 themes. FAIL when no theme structure can be parsed out of it at all -- see the ladder in pick_themes for what counts as one.
Q1Goes beyond repetition
Judge's reasoning
Theme 2 lifts Ben's exact phrasing ('AI-second mode', 'AI-first workflow integration' appears verbatim under 'Key AI Principles Ben Would Teach') as a theme title, and what sits behind it is a motivational grab-bag — 'use AI for planning, debugging, documentation', 'embed AI into daily rituals', 'organizational playbooks for reusable prompts' — that mostly duplicates themes 1, 3 and 5 rather than establishing independent substance. This is the canonical restated-phrase failure the rubric names by example.
▸Rubric
No shallow echo. FAIL if any theme "just repeats a theme it heard in the document without thinking about whether there's enough material there for a training." Mike's canonical fail is a theme like "how to build your company context" — "Okay, but that's like... What would you actually teach there, right? There's not that much to teach that is universal." Restating a phrase from the notes ("context is king", "documentation is stale", "AI-first workflow") as a theme title, without the output having done the work of asking whether there is enough substance behind it for a training, is a FAIL. Note that the underlying subject can be legitimate — "context engineering" is a good topic — so judge the treatment, not just the words.
Q2Enough depth to teach
Judge's reasoning
Each of the six themes carries concrete teachable content and a stated exercise/format — rules-file creation for context engineering, hands-on prompt iteration lab tied to Nina's document-interpretation evals, sandboxed agent delegation, AI-code-review checklists — so none survives only as a heading, though 'AI-First Workflow Integration' is the thinnest and is itself scoped to a 1-hour lunch-and-learn.
▸Rubric
Every theme is deep enough to teach. Mike's bar: "these are actually topics that you could do a whole training on." FAIL if any single theme is a pleasant-sounding course name that is not really a real topic — a label you could not build a session's worth of exercises, demos and decisions around. Ask of each theme literally: what would you actually teach in that session, and for how long? A theme that survives only as a heading fails.
Q3Supported by the interviews
Judge's reasoning
Every theme traces to specific interview material: context engineering to Nina's stale-docs/no-structured-context problem and Ben's 'context is king'; prompt engineering to Nina's incremental-change wish and Ben's DSPy interest; guardrails to Nina's 'plausible but subtly wrong' worry and Raj's regulated-data and migration skepticism; code-review to Raj's 'editors rather than authors'. No unsupported claims about Northcove are introduced.
▸Rubric
Every theme is grounded in what the three engineers said. FAIL if any theme cannot be traced to specific material in `context.md` — a generic AI-training theme bolted on that the interviews do not support, or a claim about Northcove that no engineer made. Being a good idea in general is not grounding.
Q4Uses all three interviews
Judge's reasoning
All three altitudes visibly shape the set — Nina supplies the basic-user prompt gap, over-engineering/cherry-picking failure modes and testing anxiety; Ben supplies AI-first workflow, MCP/subagent/DSPy gaps and reusable-prompt buy-in; Raj supplies the 100-engineer org, editors-not-authors shift, compliance/regulated-data risk and the prior data-sharing incident. The rationale under each theme cites different engineers rather than one interview's summary.
▸Rubric
The synthesis uses all three interviews. FAIL if the themes are effectively a summary of one engineer's interview, with the other two contributing nothing distinct. Nina, Ben and Raj sit at different altitudes (hands-on feature work, power-user workflows, org-wide constraints) and the synthesis should show it. This does not require every theme to cite all three — it requires that all three engineers visibly shaped the set.
Q5Clear priorities
Judge's reasoning
The output is framed as prioritized, orders context engineering first as the bottleneck all three named, differentiates audience and time per module (1–3 hours, all-engineers vs power-user cohorts), and closes with an explicit 'Quick Wins (Do First)' sequence naming the rules-file template and a piloted agent use case. That is sequencing and emphasis, not a flat catalogue.
▸Rubric
It is a set of priorities, not a catalogue. FAIL if the output lists themes without any signal about what matters most for this company — no ordering, no emphasis, no reasoning about which gaps are urgent. A flat, undifferentiated list of equally-weighted topics fails; the ask was "what to focus on."