Mike's Checks
Checks

Mike's Checks/claude-opus-5.5/06 execercise

06 execercise

claude-opus-5.5Claude Codehigh effortrun 20 Sep 2026629,821 tokens

Compare models
6/11
checks passed
55%
▸Instructions — what the model was asked

I have added a bunch of context on this exec offsite session in the context file and I want you to create a folder for Adrian Vale (SVP Strategic Initiatives, initials AV) with 2-3 ideas he could build, with any synthetic data he needs to build it, in a way that matches his role, with a well written prompt for each idea. Put everything in a folder with his initials. Give realistic file formats for the synthetic data.

Answer

The model left no answer file or screenshots.

Checks

6/11
Script checks 2/4answered by a program
fail

Idea folders existfolderthis task

an AV / adrian-vale folder exists with 2-3 idea subfolders

fail

Prompt for each ideaprompt-per-ideathis task

every idea has a non-trivial prompt file

pass

No AI-tell wordingai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

inspected context.md, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/channels.json, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/users.json, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/si-integration-pmo/2026-03-06.json, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/si-integration-pmo/2026-03-03.json, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/si-integration-pmo/2026-03-02.json, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/si-integration-pmo/2026-03-05.json, AV/01_Integration_and_Market_Activation_Dashboard/data/slack_export_aurex_2026-03-02_to_2026-03-06/si-integration-pmo/2026-03-04.json

pass

Usable data filesreal-datathis task

every idea has >=1 data file in a realistic format that parses

Judge checks 4/7judged by Claude
fail

Q1Different ideasthis task

Judge's reasoning

Only one idea folder exists (`AV/01_Integration_and_Market_Activation_Dashboard`); the requested 2-3 distinct ideas were never delivered, so there is no distinct second or third problem addressed.

▸Rubric

Ideas are distinct. Do the 2-3 ideas solve different problems for him? FAIL if two of them are the same artifact with different input data (e.g. two dashboards that differ only in subject).

fail

Q2Self-contained promptsthis task

Judge's reasoning

There is not a single prompt file anywhere under `AV/` — the only .md files in the workspace are the task's own `prompt.md` and `context.md` at the root, so Vale has nothing to paste.

▸Rubric

Prompt is self-contained. Could Vale paste each prompt into a fresh Claude session, with only the files in that idea's folder, and get the thing built without adding anything? FAIL if any prompt has unfilled placeholders (`[INSERT ...]`, `<your company>`), leans on conversation context ("as we discussed", "the ideas above"), or never states what should be produced.

fail

Q3Prompts match the datathis task

Judge's reasoning

No prompt exists to name any file, yet the folder ships 22 data files (xlsx, .ics, 9 .eml, Slack export JSONs) that no prompt mentions.

▸Rubric

Prompt and data match. Does each prompt name the data files that are actually present in that idea's folder, and does each supplied data file get used by the prompt? FAIL if a prompt references a file that does not exist, or the folder ships data the prompt never mentions.

pass

Q4Fits the person's rolethis task

Judge's reasoning

The single delivered idea, 'Integration and Market Activation Dashboard' (SI Program Portfolio Tracker with exec owner 'Adrian Vale, SVP Strategic Initiatives'), is squarely M&A-integration/cross-company-program work he owns.

▸Rubric

Role fit. Is every idea something this specific exec — SVP Strategic Initiatives — would own, rather than another function's job (a marketing campaign calendar, an HR onboarding tracker, an engineering ticket triager) or generic executive-productivity filler (inbox summarizer, meeting-notes cleaner) that any exec at any company could have been handed? Mike: "what I'm looking for here is like, does it come up with interesting ideas that are relevant." FAIL if any one idea sits outside the strategy / corp-dev / cross-company-programs remit.

pass

Q5Specific to the companythis task

Judge's reasoning

Data is saturated with Aurex specifics — 'AUSD cross-margining GA moved to Apr 30', 'Aurex Prime (Northwind) Integration', 'Treasura Integration', 'Settlement migration to AUSD rails 45% complete' — and would not survive a company-name swap.

▸Rubric

Company-specific. Are the ideas built on Aurex's actual situation — its products, the AUSD token, the acquisition-integration program, the payments and digital-asset competitive set? FAIL if the deliverable would read identically with the company name swapped for any other mid-size B2B company.

pass

Q6Realistic practice datathis task

Judge's reasoning

Content is genuine, varied and detailed — e.g. 'MRC session on Mar 3 did not approve... AUSD depeg to 0.97', 'KYC migration: 194 of 312 clients done (62%)', six xlsx sheets (Programs/Milestones/RAID/Synergies/Market KPIs) with named leads and budgets.

▸Rubric

The data is real data, not props. Mike: "the thing I look for here is, um, does it, does it create real data?" Format is already checked by script — judge the *content*. FAIL if values are obvious filler (`Company A`, `Competitor 1`, lorem text, the same row repeated, all-identical dates/amounts) or if there is too little of it to build the thing the prompt asks for (e.g. a trend dashboard shipped with four rows).

pass

Q7Buildable during the sessionthis task

Judge's reasoning

The dashboard's inputs are all included as local files (SI_Program_Portfolio_Tracker.xlsx, gmail_label_SI-Weekly-Status/*.eml, slack_export_aurex_*/, .ics calendar) with no live system or paid feed required.

▸Rubric

Buildable in the session. Could this be built during a workshop session against the supplied synthetic data? FAIL if any idea's core function requires live credentials or system access the exec won't have in the room (a real NetSuite/Snowflake connection, his production inbox, a paid market-data feed) instead of working off the included files.

Notes

1

Also didn't get the final thing here unfortunately. It did produce a lot of good training data, but my diagnosis is that it let realism and file generation work consume the task, and that left no time to finish the final thing. So in its defence, it wasn't we didn't tell it how much of a time budget it had, but compared to other models, we had never had this issue before. This seems to massively overthink.

21 Sep 2026