Dan’s Editorial Checks
Checks

Dan’s Editorial Checks/Sonnet 5.5/01 article-opening-feedback

01 article-opening-feedback

Sonnet 5.5Claude Codehigh effortrun 23 Sep 202652,348 tokens

Compare models
8/12
checks passed
67%
▸Instructions — what the model was asked

Article opening feedback: Token Math

Review the headline, deck, and opening paragraph for readers of Every, an AI-focused publication. Identify the strongest idea, assess the framing and wording, and give concise editorial feedback with suggested replacement copy where useful. Use the rest of the draft as context. Preserve the factual qualifications.
Use Every’s house style: headlines should express the most interesting or surprising claim in concise, punchy language that mostly fits an email subject line. Let the dek develop the headline with a clear implication or detail. Headlines can be bold, with qualifications developed in the body. The first sentence should be strong and catchy, and continue naturally from the hed/dek. Any problems you find you should explain and offer alternatives.
The draft is frozen below. This case is for editorial feedback; it does not ask for a whole-article rewrite.

Headline selection captured in the source

HED: Token Math
Alt Hed: How Many Tokens are Too Many?

DEK: What’s the ROI on experimentation?
Alt DEK: When expensive experimentation is worth it and when to rein your tokenmaxxing in.

Opening paragraph

Often if you are the highest spender at your company you might think about reining yourself in. But when it comes to tokens, Every CEO Dan Shipper thinks everyone else should consider ramping up. He consumes more than three times as many tokens as the next-highest user at the company. The size of the gap has him wondering whether the rest of us are being ambitious enough with AI.

Full draft without comments

HED: Token Math
DEK: What’s the ROI on experimentation?

Often if you are the highest spender at your company you might think about reining yourself in. But when it comes to tokens, Every CEO Dan Shipper thinks everyone else should consider ramping up. He consumes more than three times as many tokens as the next-highest user at the company. The size of the gap has him wondering whether the rest of us are being ambitious enough with AI.

Every’s OpenAI token leaderboard (Screenshot courtesy of Laura Entis.)

The goal isn't tokenmaxxing, or the practice of torching tokens for sport. Instead, he wants to give people room to experiment with radical new strategies and ways of working that, should they pan out, could vastly improve their productivity and more than justify the token costs.

In a conversation with head of platform Willie Williams, the two settled on a tangible starting point: use AI to attack the engineering team’s bottlenecks, see how far they get, and only then do the token math.

Inside Every

Token budgets at the frontier

Dan’s encouragement to experiment comes with an expectation: When people spend company money, they should assess what that spending produced and whether they could be more efficient next time.

Turning that philosophy into spending decisions falls to head of operations Arielle Shipper. For now, she evaluates expensive runs case by case, using the results to decide when to keep funding the work and when to change course.

Her early-warning system is the company card. She keeps the ChatGPT automatic credit refill relatively low, so an unusually rapid succession of charges tells her that someone is building or testing something big.

When she’s alerted to changes over the set amount Arielle typically asks the team on Slack if anyone is in the middle of a particularly big run or check the usage leaderboard and message the people at the top. “Then Dan always responds, ‘It’s me running Ultra.’” she says, which explains why he’s always on the top of the leaderboard.

A common occurrence. (Screenshot courtesy of Laura Entis.)

She wants answers to three questions: What did the run cost? What did it buy us? And what did we learn?

The team has started offering these retrospectives unprompted. After a string of Astra experiments racked up billions of tokens, head of video Randy Counsman messaged her detailing what had worked, what hadn’t, and what he’d learned—which, it turns out, was a lot.

More on that below.

Data point

4.5 billion

That’s how many OpenAI tokens Randy burned through in an attempt to make a 3D model of his face with Astra.

The project started innocently enough: Inspired by social media posts of impressive-looking AI-generated demos made with Blender, a free 3D tool, he wanted to try it himself. Experimenting with AI, after all, is an important component of his job.

Using strategies shared on X, he set up an orchestrator agent to maintain the plan, an implementer to assign tasks, and subagents to execute them, and instructed Codex to keep looping through improvements until the model was “implemented very well,” Randy says. In retrospect, “It was an ambiguous goal.”

Then he let the model run. And run. And run.

By the time he pressed pause on the project, the token spend was in the billions and the result was, as he puts it, “very janky.”

The output, rendered in four different shaders. (Image courtesy of Randy Counsman.)

An AI audit of his Codex sessions revealed some glaring inefficiencies: Layers of agents passed growing amounts of context back and forth, including messages checking whether a subagent had completed a task. He’d unwittingly created a compute-hungry “unruly swarm” of agents.

Randy has since rebuilt his project setup. He dropped the implementer, caped the orchestrator at five Sol subagents to control costs, and built in explicit feedback checkpoints. For a recent project, he also generated an image of the design he wanted, so a judge agent could check the model's work against a concrete target instead of an open-ended instruction to keep improving.

A failed experiment is still a good investment if it shows where an AI system falls short. Randy now has a benchmark to run new models against—how well they turn an organic 2D image into an organic 3D model—and a leaner setup to get a better result with fewer tokens.

Unwittingly large token spends via experimentation is an Every rite of passage. (Screenshot courtesy of Laura Entis.)
Try it yourself: Pick an AI task that burned through more tokens than expected and paste the following into your agent of choice:
Review the available session history and token usage records for [task/project] during [date range], including any subagent sessions.
I was trying to [goal]. The result was [what happened]. Help me understand where the tokens went and what I could do differently next time.

  • Break down usage by activity and model where the records allow.
  • Look for repeated work, unnecessary status checks, and large amounts of context passed between agents. Show specific examples.
  • Distinguish necessary work from likely waste. Don’t assume high usage or cached input was wasteful.
  • Recommend the three most useful changes to my instructions, model choices, or agent setup.
    If you can’t access the records you need, tell me what to provide.

Steal this workflow

Don’t send Fable to do a Sonnet’s job

Spiral general manager Marcus Moretti’s token spend strategy is straightforward: Stay within the weekly usage limits for his Claude Code Max plan.

It’s a simple goal that requires active management; Marcus is selective about which assignments go to Fable 5.1 and which go to a cheaper model.

Here’s what that looks like in practice:

Step 1. Learn what each model is good at. Marcus has used Anthropic's models long enough to know on sight whether a job needs Sonnet, Opus, or Fable-level intelligence.

Step 2. Match the model to the assignment. Basic tasks—analytics checks, for example—go to Sonnet. Work with a clear objective, such as a tightly scoped product change, goes to Opus. For a larger feature—like the new billing implementation he’s working on—Marcus uses Fable 5.1 to develop the specification: what needs to be built and how it should work.

Step 3. For complex work, let the model delegate. Once the plan is ready, Marcus tells Fable, “Kick this off, and for all coding tasks, use your judgment about delegating to a lower model.” Fable assigns pieces to Opus and Sonnet, then checks their work. This lets the strongest model manage the project without executing every task.

Try it this week: Choose a recurring task and run it with a cheaper model than you usually use. Check whether the result meets your needs before making it your default for that task.

The daily driver
The models the team is using this week:

  • Douglas Brundage, head of marketing: Using Fable 5.1 more for visual work, plus Grok Bot to monitor social media and build a tool for visuals.
  • Randy Counsman, head of video: Astra (high) for complex work and medium for basic tasks; Sol (extra high) for voice mode orchestration, where Astra feels too slow and heavy-handed. Fable for code-related planning and Opus when he runs out of Astra tokens.
  • Kieran Klaassen, Cora general manager: Fable 5.1 (extra high, 1M context) for coding and knowledge work, switching to Astra for some research and writing.
  • Tyler Nishida, design engineer: Fable 5.1 (high) as the coordinator/orchestrator for projects in Cursor, with Grok 4.6 (extra high) subagents. Astra (ultra) for personal projects. “It’s not a model, but Cursor projects are amazing as a daily driver for your preferred models. My sidebar is so clean, and it's so much easier to manage work.”

Tyler’s Cursor setup. (Image courtesy of Tyler Nishida.)

  • Katie Parrott, staff writer: “The Astra fever has broken for me,” she says. These days, she’s back to Sol (high) for task management and has switched to Claude models for writing because she finds Sol and Astra “too flat—they just report the facts instead of shaping a narrative that brings the reader along.”
  • Natalia Quintero, head of consulting: Astra (medium).
  • Yash Poojary, growth engineer: Astra (high) and Fable 5.1.
  • Arielle Shipper, head of operations: Sol (medium). “Sol forever. SOLoyalist,” she says.
  • Loren Stewart, engineer: Downgraded from Astra to Sol (medium) for coding to reduce costs and finds the switch is working well. Fable 5.1 “remains the champion thought partner.”

Answer

ANSWER.md the model's output · 6.56 KBdownload

Token Math: opening feedback

Bottom line

The piece has a strong idea, but the hed and dek don't use it. The strongest idea is an inversion: the person with the biggest token footprint at the company (the CEO, at more than 3x the next user) thinks everyone else should use more. The body then adds the counterweight that makes this more than a boast: Arielle's three questions (what did it cost, what did it buy, what did we learn) and Randy's 4.5-billion-token "very janky" face.

The current package leaves that out:

  • Hed: "Token Math" names a topic, not a claim.
  • Dek: "What's the ROI on experimentation?" is an abstract question, and the piece never answers it in those terms.
  • Lede: it opens with a hedged, generic sentence and gets to the surprising part in sentence three.

Headline

"Token Math" (current): It's a good phrase, and it pays off in the body ("only then do the token math"). It works better as a section label or kicker than as the hed. As a hed it gives a reader in an inbox no reason to open, and no claim to be surprised by.

"How Many Tokens Are Too Many?" (alt): This is better, since it has a tension, but it points the wrong way. It frames the piece as a limit or a warning. Dan's point is closer to the opposite: he wonders whether people aren't ambitious enough. It also promises a number the piece doesn't give. (Small fix: "Are" is a verb, so it takes a capital in title case.)

Recommended hed: Every's CEO Wants You to Burn More Tokens

  • It has the inversion, it's under 50 characters, and it works as a subject line.
  • It's a little bolder than the body's "should consider ramping up." Per house style, that qualification lives in the lede. The dek signals it's not blanket advice.

Other options:

  • Are You Burning Enough Tokens? This is more reader-facing and drops the Dan hook.
  • 4.5 Billion Tokens for One Janky 3D Face. This is the most surprising detail in the piece, and it would make a great hed. But it makes Randy's story the piece's center, so the lede would have to change too. I'd hold it for a pull quote, social copy, or the "Data point" callout.

Dek

"What's the ROI on experimentation?" (current): This is a question, it's jargon-y ("ROI on experimentation"), and it adds nothing to the hed. A dek should develop the hed with an implication or detail.

Alt dek ("When expensive experimentation is worth it and when to rein your tokenmaxxing in"): The content is right, since it carries the "worth it or not" stakes. Two problems:

  • "Tokenmaxxing" is undefined jargon at this point in the piece. The body defines it in paragraph two as what Dan is not advocating, so using it in the dek muddies the message.
  • "Rein in" repeats the lede's "reining yourself in."

Recommended dek: Big token bills can be smart bets or expensive mistakes. Here's how Every tells the difference.

If you'd rather keep your alt's structure: When expensive experimentation is worth it, and when it's time to rein it in.

Opening paragraph

What works:

  • The setup-and-flip structure (top spender → reins in, but Dan says spend more) is the right instinct.
  • "The size of the gap has him wondering whether the rest of us are being ambitious enough with AI" is the best sentence in the paragraph. It's the thesis in Dan's own uncertainty, and it makes the "3x" fact mean something.

What doesn't:

  1. "Often if you are the highest spender at your company you might think about reining yourself in." This is stacked with hedges ("Often," "if," "might"), reads awkwardly (no comma, slow start), and is generic. It's the least catchy sentence and it's first.
  2. "But when it comes to tokens" is filler, and the "but" leans on a sentence that wasn't strong enough to lean on.
  3. "Spender" vs. "tokens." The opening says highest spender, then measures in tokens consumed. The two aren't the same, because token counts aren't dollars and models are priced differently. This matters because the piece is about cost vs. return. Use "token leaderboard" or "top token user" to keep it accurate.
  4. The 3x fact comes third. The surprising number should sit next to the claim.
  5. "OpenAI" is missing. The caption calls this "Every's OpenAI token leaderboard," but the lede just says "tokens." I've added "OpenAI" to the suggested copy because that's the qualification in the draft. Please confirm the 3x holds on that leaderboard specifically and not across all providers.
  6. Small: "thinks... should consider" is a double hedge, but it's the sourced qualification, so I kept it.

Suggested replacement (keeps the qualifications: "more than three times," "next-highest user," "should consider," "wondering"):

If you topped your company's token leaderboard, you might think about reining yourself in. Every CEO Dan Shipper has the opposite reaction. He consumes more than three times as many OpenAI tokens as the next-highest user at the company, and he thinks everyone else should consider ramping up. The size of the gap has him wondering whether the rest of us are being ambitious enough with AI.

Why it flows from the package: The hed makes the claim, and the dek says a big bill isn't automatically good or bad. The lede then answers "who says, and why should I believe them?" with the inversion and the number. The paragraph still ends on the "ambitious enough" question, which is where the next paragraph (the "not tokenmaxxing" clarification) picks up.

Tighter alternative if you want a shorter, punchier version:

Every CEO Dan Shipper uses more than three times as many OpenAI tokens as the next-highest user at the company. Most people in that spot would think about reining themselves in. He thinks everyone else should consider ramping up.

Assumptions and things to confirm

  • I assumed the "3x" figure refers to the OpenAI leaderboard shown in the first screenshot (per the caption). If it covers all providers, drop "OpenAI" from the lede.
  • I kept "burn" in the hed because the body itself uses "burned through." If you'd rather not echo "torching tokens for sport," swap in "Spend": Every's CEO Wants You to Spend More on Tokens. It's less punchy, but it is closer to the dollars-vs-tokens point.

Noticed in passing (outside the scope of this review)

  • Paragraph starting "When she's alerted…": the sentence structure breaks ("asks... or check... and message"), and there's a stray period inside the quote.
  • "caped the orchestrator" should be "capped."
  • "Unwittingly large token spends… is an Every rite of passage": subject-verb agreement.
  • Paragraph two's "he" follows a sentence about tokenmaxxing, so make sure it clearly refers to Dan.

Checks

8/12
Script checks 0/1answered by a program
fail

ai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

shared AI-tell phrases: when it comes to

Judge checks 8/11judged by Claude
fail

Q1S05. Weasel wordsthis case

Judge's reasoning

It correctly flags the hedges ("Often," "if," "might"), but its primary suggested replacement reproduces the empty lead-in verbatim: "If you topped your company's token leaderboard, you might think about reining yourself in."

▸Rubric

S05. Does the AI’s feedback call out weasel words in the headline, dek, or first line, like “often” and “you might think about”, that make the point tentative? PASS if: The feedback identifies unnecessary hedging, tentativeness or needless indirection in the original lead-in, tied to an actual phrase such as often or you might think about. Equivalent wording counts; it need not say weasel words or name both examples. Where it offers replacement copy, that copy removes the empty lead-in. Preserve uncertainty that carries real meaning: possible, probably, might or similar words can be appropriate for a genuine inference. FAIL if: The feedback ignores the unnecessarily tentative lead-in, discusses only a different issue, or repeats equivalent empty hedging in its replacement. Do not apply a blanket ban on uncertainty words. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

fail

Q2S07. Interesting setup immediatelythis case

Judge's reasoning

It praises the "ambitious enough" line but leaves it last in both rewrites—one leads with the familiar rein-yourself-in warning and reaches the gap in sentence three, the other states the stat then interposes "Most people in that spot would think about reining themselves in" before the implication.

▸Rubric

S07. Does the AI’s feedback identify the interesting gap and what it implies, and recommend setting that up really quickly—in the first sentence, or at least most of it in the first sentence? PASS if: The feedback identifies the surprising possibility that the rest of the team is not being ambitious enough and calls for presenting it immediately. Any replacement opening it offers delivers that direction: it can combine the implication with the gap or lead with a provocation whose supporting gap follows promptly. In a gap-led version, connect the ambition implication in the first sentence or the immediately following sentence. Dan’s If your team is not wasting tokens, there’s probably something wrong is a passing provocation-first direction; numerical evidence can follow. Preserve the source’s more-than-three-times comparison when stating the number and the uncertainty of the inference. FAIL if: The feedback misses the gap’s ambition implication, leaves it buried, or recommends an opening that leads with a familiar spending warning or states the statistic and delays its interesting implication behind another familiar reaction. Do not require both number and implication in one literal sentence or exact wording from an example. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

fail

Q3S08. CEO-specific cutback insightthis case

Judge's reasoning

Both offered openings hook on the unsupported cutback default ("you might think about reining yourself in"; "Most people in that spot would think about reining themselves in"), and the feedback endorses that setup-and-flip as "the right instinct."

▸Rubric

S08. Does the AI’s feedback recognize and resolve the CEO-specific flaw in a default cutback setup? PASS if: The feedback recognizes that Dan is the CEO, so a generic claim that the highest spender would normally be asked or expected to cut back does not fit this case, and every usable offered replacement opening deals with that insight. It may drop the cutback contrast and lead with the more-than-three-times gap and what it makes Dan wonder about the team’s ambition. It may instead use a source-supported account of Dan himself weighing whether to spend less, or Arielle’s actual card and leaderboard process. The feedback need not use Dan’s exact words, but a generic rewrite that merely omits the cutback phrase without showing the CEO-aware insight does not meet this high bar. This check tests contextual understanding and repair; S07 separately tests how early the ambition point lands. FAIL if: The feedback misses the CEO-specific mismatch, or any usable offered replacement opening still uses an imagined order or default expectation to rein in the biggest spender as the hook for Dan’s case. Equivalent unsupported defaults such as “the obvious move is to rein it in” or “cue to cut back” also fail. A good alternative does not rescue another usable opening that retains the false setup. A quotation used only to criticize the original wording, or a clearly rejected example, is not an offered opening. If no replacement opening is offered, diagnosis alone does not demonstrate the repair this check asks for. Return PASS or FAIL and one sentence citing the CEO-specific diagnosis and offered opening, or the missing insight or flawed setup.

pass

Q4S02c. Headline makes sense on its ownthis task

Judge's reasoning

Each usable hed states a clear subject and action on its own — "Every's CEO Wants You to Burn More Tokens," "Are You Burning Enough Tokens?", the "Spend More on Tokens" variant, and "4.5 Billion Tokens for One Janky 3D Face" — with no opaque slogan or missing referent for an AI-literate Every reader.

▸Rubric

S02c. Does each headline offered as a usable replacement make sense on its own to the intended reader? PASS if: Each proposed headline communicates an intelligible claim, question, or subject without requiring the dek, article, or accompanying explanation to supply missing meaning. The subject, action, referents, and any comparison or scope essential to understanding are clear enough that the reader can restate the intended meaning without guessing or filling in an unstated premise. Recognizing a topic or familiar phrase is not enough when the central action or relationship remains unspecified. The reader must understand what is being acted on and what any linked evaluation, comparison, or consequence refers to; a plausible interpretation supplied by the reviewer is insufficient. Normal audience knowledge and familiar vocabulary are allowed. FAIL if: Any headline offered as usable depends on an opaque slogan, an unspecified action or referent, or omitted context essential to understanding. A stronger alternative or explanatory dek does not rescue that headline. Clearly rejected examples are excluded. A headline may withhold an answer to create curiosity, but its premise must be understandable. Essential context is different from hedging or detailed qualifications; do not require a complete summary or every factual qualification. Assess clarity here, separately from interest and factual support. If no replacement headline is offered, do not fail this check for the omission; S02b assesses that. Return PASS or FAIL and one sentence of evidence from the proposed headline copy.

pass

Q5S01. On topicthis case

Judge's reasoning

Treats "Token Math" as an on-topic phrase that "pays off in the body" but works better as a kicker, and all proposed heds stay on token spending/ambitious experimentation.

▸Rubric

S01. Does the AI’s feedback assess whether the headline talks about the main thing in the article? PASS if: The feedback correctly treats Token Math as on topic and relates the framing to the article’s main subject or idea: token spending, ambitious AI experimentation and what experimentation buys or teaches the team. Any replacement headline stays on that main thing. Interest is assessed separately in S02a and S02b. FAIL if: The feedback wrongly calls the original headline off topic, misses the article’s main subject or proposes a replacement headline about a tangent. Do not fail merely because the original headline is on topic but uninteresting. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

pass

Q6S02a. Interesting headline: diagnosisthis case

Judge's reasoning

Says "'Token Math' names a topic, not a claim" and "gives a reader in an inbox no reason to open, and no claim to be surprised by."

▸Rubric

S02a. Does the AI’s feedback identify why the headline isn’t interesting enough—why it fails to capture the most interesting, counterintuitive or surprising part of the article that makes me feel, “ooh, I need to read that”? PASS if: The feedback recognizes that Token Math merely names the topic and does not convey the article’s interesting provocation, tension or payoff. Equivalent descriptions count; it need not use Dan’s exact words. This check assesses recognition, so no replacement is required to pass it. FAIL if: The feedback misses the interest problem, treats being on topic as sufficient, or only offers vague advice without identifying what is missing. Do not fail this diagnosis check because a replacement headline is weak; assess that separately in S02b. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

pass

Q7S02b. Interesting headline: replacementthis case

Judge's reasoning

Proposes "Every's CEO Wants You to Burn More Tokens" and "Are You Burning Enough Tokens?"—both concise, subject-line length, and reader-facing provocations; the Randy 4.5B line is explicitly held back for a pull quote/callout rather than offered as a hed.

▸Rubric

S02b. Does the AI’s feedback propose better headline options that all capture the interesting provocation in a concise, punchy way that would fit in an email subject line (at least mostly)? PASS if: The feedback actually proposes at least one replacement headline rather than merely recommending improvement, and every headline it proposes as an option meets the bar. This applies to recommended choices, ranked options, alternates and unranked lists. Options count even if they are not labeled usable; calling an alternate weaker or less punchy does not exempt it. A stronger option cannot rescue a weak proposed option. Clearly rejected examples are excluded. Each proposed option gives the intended Every reader a short, compelling, draft-supported angle, expressed concisely and punchily and mostly fitting in an email subject line. It may make the article’s argument about ambitious AI experimentation, foreground a specific experiment and what it taught us, or take another supported angle; no particular thesis wording, person or number is required. Each headline must give a reader outside Every a reason to care through a useful implication for their AI work, a compelling argument they can apply, or a vivid example with a transferable stake. Second person is not required, but it does help. The reader stake must be apparent from the headline itself; a dek cannot supply a missing reader stake. “The Case for a Bigger Token Bill” and “Why You Should Use More Tokens” are nonexhaustive passing directions; similar supported formulations pass without exact matching. Internal examples are allowed when they signal the larger argument; this is not a blanket ban on CEO or company anecdotes. If a proposed headline shifts the center away from the current Dan-led opening, the feedback must explain the restructuring needed, such as bringing Randy’s experiment into the introduction. A clearly conditional idea described as incompatible with the current opening is exploratory: assess whether its headline itself is concise, compelling, supported and has a reader stake, but do not require a paired dek. If the feedback actually recommends using that shifted angle as a publication-ready package, require a dek explicitly paired with that headline; it must stay on the focal story or question and add a specific supported answer, consequence or stake. For a Randy 4.5-billion-token package, the dek should say what the run bought or taught, such as a benchmark and a leaner agent setup; a generic spending-process dek does not continue that promise. In a two-part headline, the second part must tell the reader something specific about this article’s angle, question or payoff. “Spend the Tokens, Then Do the Math” fails the interest bar: the idiom is comprehensible, but only says evaluation will happen afterward and adds no information about what this article asks or reveals. Do not fail an idiom merely because it is an idiom; assess its information in this context, separately from S02c comprehension. Keep hedging and detailed qualifications out of the headline. Essential identifying context needed to understand the subject, action, or comparison is allowed and is not a qualification; boldness does not require ambiguity. FAIL if: The feedback proposes no replacement, or any headline it offers as an option fails the bar: proposed options remain topic labels, generic spending/value claims or questions, isolated facts, unsupported broad claims, or weak side stories without a clear restructuring caveat or plan, or off-center headlines recommended for use without an explicitly paired dek that continues their particular promise; only report an internal directive without an apparent stake for an outside reader; add an empty second clause that tells the reader nothing specific about this article; bury the claim in qualification; or sprawl beyond mostly fitting in an email subject line. “Our CEO Wants Us to Use More Tokens” fails this reader-interest bar because it only reports an internal directive; basic clarity and topicality are insufficient. Do not impose an invented exact character limit. Recognizing the original problem alone does not pass this replacement check. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

pass

Q8S03. Promise cashed outthis case

Judge's reasoning

Explicitly checks payoff ("the piece never answers it in those terms," "promises a number the piece doesn't give") and its recommended pair's promise—how Every tells smart bets from mistakes—is delivered by Arielle's three questions and Randy's retrospective.

▸Rubric

S03. Does the AI’s feedback assess whether the headline/subhead combination makes a promise that the article cashes out, using a loose standard? PASS if: The feedback considers whether the pair’s interest, answer or payoff is broadly delivered by the article, and any replacement pair it offers promises a broadly supported payoff. Equivalent discussion of framing accuracy, payoff or reader expectations counts. The body can supply nuance and qualifications; the headline need not mirror it literally. Qualitative returns such as learning, reusable benchmarks and better workflows can cash out an experimentation/ROI promise. A numerical ROI calculation is not required. FAIL if: The feedback ignores the promise/payoff question, recommends a promise the article does not come close to delivering, or insists on headline qualifications for literal faithfulness. Do not fail merely because the original article offers qualitative rather than numerical returns. The token maxing backlash is wrong remains borderline rather than an unconditional pass or fail. Also FAIL if an offered headline makes a supporting detail look like the article’s central story and leads the reader to expect a materially different kind of article, even when the literal claim appears in the body. “Our CEO Wants Us to Use More Tokens” frames this draft as a workplace story about how employees respond to a CEO directive; the draft uses his usage gap to make a broader case for ambitious AI experiments and examining what each experiment produced. A dek cannot fully rescue a headline whose primary frame points at the wrong story. Do not fail every headline that mentions Dan or Every: a headline can and should use internal examples when they accurately signal the larger argument. This does not require a full article summary or detailed headline qualifications. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

pass

Q9S04a. Headline and dek connect: diagnosisthis case

Judge's reasoning

Says the dek "adds nothing to the hed. A dek should develop the hed with an implication or detail," naming the missing connection.

▸Rubric

S04a. Does the AI’s feedback identify whether the headline and subhead form a connected whole, where the headline raises a question or topic of interest that the dek deepens and gives more details on? PASS if: The feedback recognizes that Token Math and What’s the ROI on experimentation? relate but their connection is not obvious, or gives equivalent criticism that the dek does not clearly develop the headline. This check assesses recognition, so no replacement pair is required to pass it. FAIL if: The feedback reviews the two in isolation without addressing their relationship, treats the original pairing as fine, or misses the difference between being loosely related and forming a connected whole. A good replacement pair alone does not establish that the feedback identified this problem; assess its copy separately in S04b. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

pass

Q10S04b. Headline and dek connect: replacementthis case

Judge's reasoning

Pairs "Every's CEO Wants You to Burn More Tokens" with "Big token bills can be smart bets or expensive mistakes. Here's how Every tells the difference," where the dek adds the stake and mechanism behind the hed's claim.

▸Rubric

S04b. Does the AI’s feedback propose a headline/subhead pair that actually forms a connected whole, with the dek deepening the headline and giving more details on it? PASS if: The feedback actually proposes at least one usable headline/subhead pair. The connection is clear and the dek develops the headline’s question or claim with an implication, mechanism, stake or detail in concise, compelling language. A headline and dek recommended together can count without a prescribed presentation format. For Why you should be spending more on tokens, Maximizing your company’s AI use requires risky experiments illustrates the relationship. Exact wording or a risk theme is not required. Judge the relationship here; headline interest is assessed separately in S02b. FAIL if: The feedback proposes no usable pair, or none of its proposed pairs clearly develops the headline: the dek is unrelated, merely restates it, leaves its connection unclear, or introduces a different promise. Merely identifying the original problem does not pass this replacement check. The spoken four-X-engineers example illustrates structure; it is not an established source fact or grading target. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

pass

Q11S06. Diagnoses delayed hookthis case

Judge's reasoning

Human adjudication under revised S06: the feedback says the lede reaches its surprising part only in sentence three, identifies the three-times gap and ambition question as the payoff, and recommends moving the fact forward. S07 separately grades the imperfect rewrite.

▸Rubric

S06. Does the AI’s feedback diagnose that the opening delays its real point or hook? PASS if: The feedback identifies what the reader is made to wait for and why the lead-in is slow. It may notice that the first sentence’s condition-heavy setup postpones “reining yourself in” until the end, that a generic opening delays the more-than-three-times gap, or that the gap’s interesting ambition implication is buried behind familiar setup. Equivalent descriptions count; it need not name the first sentence’s exact last phrase. The feedback should connect the delay to the actual payoff in this draft, not merely call the prose weak. This check scores diagnosis only; S07 separately scores whether an offered replacement delivers the gap and ambition implication quickly enough. FAIL if: The feedback only calls the opening weak, wordy or hedged without identifying the delayed point or hook, or comments on the statistic’s position without showing why it matters to the article’s provocation. Do not fail this diagnosis check because a proposed replacement still delays the point; score that under S07. Return PASS or FAIL and one sentence of evidence from the AI’s feedback.

Notes

1

Adjudication update: The original 7/12 score has been corrected to 8/12 under revised S06. The feedback correctly diagnoses that the lede reaches its surprising point late; S07 still fails because the offered rewrites delay the ambition implication, and S08 still fails because they use a generic cutback expectation that does not fit a CEO. The Beignet answer and trace are unchanged.

24 Sep 2026