13 ai-village

No AI-tell wordingai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

shared AI-tell phrases: furthermore; moreover

Substantial reportreportthis task

a report of at least 300 words exists in the workspace: REPORT.md, else any report-named document, else ANSWER.md.

no REPORT.md, report-named document or ANSWER.md

Three measured variationsresults-datathis task

results.json parses and lists at least three variations, each carrying the ad text and an end-of-run awareness figure: a share in [0, 1] or a count no larger than the town.

no results.json in the workspace

Distinct ad variationsvariants-differthis task

no two variations are the same: the ad text, once case, punctuation and spacing are normalised, together with whatever the entry names as the lever -- seed persona, messenger, channel, incentive, framing. The same flyer handed to a different persona is a variation (the prompt says so); three labels on one setup are not. Whether a variation tests anything is the judge's call.

fewer than 3 ad texts to compare

Report numbers matchnumbers-tie-outthis task

every variation's final awareness in results.json is in the report, as a percentage within one point or as "k of n" / "k/n" / "k out of n".

needs a report and 3 scored variations

Interactive diffusion mapinteractive-mapthis task

a self-contained map.html is linked from the report, includes every variation, draws a map with SVG or canvas, has controls for switching and inspection, and can replay the diffusion in chronological order.

no map.html in the workspace

Enough persona subagentssubagent-personasthis task

the run's own session log shows at least 8 subagent spawns -- Agent/Task tool calls in a Claude transcript, spawn_agent in a Codex rollout, spawn_subagent in a Grok session -- or at least 8 subagent transcripts were captured beside it. The prompt makes subagents the only model available for the personas, so a run that never spawned one did not simulate anything. FAIL when no trace was captured.

148 subagent spawns in the trace, 0 subagent transcripts

Memory, reflection, and plansmemory-streamsthis task

files whose path names a memory stream (or JSON files keeping one under a memory-named key) exist and, read together, hold all three record kinds the paper defines: observation, reflection and plan.

22 memory files with observations, reflections and plans

Q1Controlled comparisonthis task

Judge's reasoning

Only variation_1 was actually run to completion (variation_2 halted mid-block, variation_3 never started) and no report exists to state that the town, entry point and length were held constant, so no controlled comparison was delivered or documented.

▸Rubric

A controlled comparison. Read the report and the run configuration. FAIL if the variations differ in anything besides the advertisement — a different seeding persona or location, a different town, a different number of ticks — and the report does not acknowledge it as a confound. PASS if the same town, the same entry point and the same simulated length were used for every variation and the report says so.

Q2Actionable marketing findingsthis task

Judge's reasoning

No REPORT.md or any written conclusions exist, so there is no statement of which variation spread furthest, no mechanism drawn from the transcripts, and no next steps.

▸Rubric

Findings a marketer could act on. Read the report's conclusions. FAIL if it only restates the numbers, or explains the difference between variations with generalities that are not grounded in what happened in the runs. PASS if it says which variation spread furthest, points to a mechanism visible in the transcripts — who relayed it, in what setting, what about the ad made it worth repeating — and names what to try next.

Q3Honest about scale and noisethis task

Judge's reasoning

No report or limitations section was produced, so the scale (8 personas, 14 encounters, one run), the simplifications from the paper, and the noise caveat are nowhere stated.

▸Rubric

Honest about scale and noise. Read the report's limitations. FAIL if it presents the result as if it were the paper's twenty-five agents over two days, or omits that each variation was run once with a handful of personas so the differences may be noise. PASS if it states the scale — personas, ticks, roughly how many subagent calls — what was simplified from the paper, and what that does to the confidence a reader should have.

Q4Uses the paper's architecturethis task

Judge's reasoning

memory.py/retrieval.py/engine.py implement timestamped observation/reflection/plan records, a recency+importance+relevance retrieval score, importance-threshold reflections with pointers to source records, and a per-persona day plan broken into timed actions — all visible in maya_lin.txt.

▸Rubric

The paper's architecture, not a chat loop. Read the simulation code and one persona's memory stream. FAIL if a persona is just a system prompt plus the transcript so far — no memory stream that accumulates records, no retrieval step that selects which memories a persona sees, no reflection, no plan. PASS if all four are present in some form: a per-persona stream of timestamped records, retrieval that scores or filters memories on recency, importance and relevance (an approximation of all three is fine if it is named as one), reflections synthesised from earlier records, and a day plan that is broken into actions.

Q5Subagents drive behaviorthis task

Judge's reasoning

The orchestrator emits per-persona prompts containing that persona's retrieved memories (pending_tasks.json) and ingests free-text subagent replies (task_responses.json), and the dialogue in conversations.json is varied free text, not templated.

▸Rubric

Subagents produced the personas' behaviour. Read the code and a few conversations. FAIL if what the personas say or decide comes from templates, a rule table, random draws, or the orchestrator writing the lines itself in bulk for everyone. PASS if each persona's dialogue and choices are free text that a subagent produced with that persona's memory in front of it, and the code or run log shows the loop that hands memory out and takes answers back.

Q6Believable connected communitythis task

Judge's reasoning

All eight Oakwood personas have a paragraph of identity/occupation/traits plus 6-8 seeded initial memories naming specific mutual relationships (Maya-Arthur, Sam-Nora, Marcus-Lucas, Chloe-Elena) that link them into one connected town.

▸Rubric

Seeded like Smallville. Read the persona definitions. FAIL if the personas are names with a job title, or strangers with no ties to each other. PASS if each has a paragraph of identity — occupation, traits, who they live with, who they know — entered as initial memories, and the relationships link them into one community.

Q7Awareness verified from memorythis task

Judge's reasoning

For the run that completed, all eight personas were interviewed at 19:30 and each affirmative answer was checked against that persona's own memory stream with a supporting_memory_id and a VERIFIED_YES/HALLUCINATED_YES status, not read off the orchestrator's awareness set.

▸Rubric

Diffusion measured the way section 7.1.1 measures it. Read the interview material. FAIL if awareness was read off the orchestrator's own record of who was told, or if the personas were not each asked at the end whether they know about the advertisement, or if a "yes" was counted without checking that persona's memory stream for where it came from. PASS if every persona was interviewed at the end of every variation and each yes was verified against memory, with any hallucinated yes excluded and said so.

Q8Advertisement diffuses naturallythis task

Judge's reasoning

seed_advertisement() writes the flyer into Maya Lin's memory alone at 08:00 at Oakwood Cafe, and every subsequent diffusion event in diffusion_path.json is attributed to a persona-to-persona utterance.

▸Rubric

The advertisement entered the world, it was not broadcast. Trace how each variation was introduced. FAIL if the ad was written into every persona's memory by the orchestrator, or told to most of the town at tick zero — that is not diffusion. PASS if it reached one or two personas or one location and everything beyond that came from personas talking to each other.

Q9Meaningful tested variationsthis task

Judge's reasoning

The three ad texts each move one nameable lever (neutral information, bilateral $50 referral incentive, scarcity/civic-urgency framing) and each carries an explicit pre-registered hypothesis with a predicted reach in VARIATION_SPECS written before the run.

▸Rubric

Variations that test something. Read the ad texts and the report's setup. FAIL if the variations are cosmetic rewordings with no hypothesis, or the report states no expectation before the results. PASS if each variation changes one nameable lever — framing, messenger, channel, incentive, social proof — and the report says what was expected of it before the numbers.

Q10Diffusion paths shownthis task

Judge's reasoning

diffusion_path.json records the full ordered who-told-whom chain (flyer→Maya→Arthur/Marcus/Chloe/Nora→Sam/Lucas/Elena) with times, locations and quoted utterances rather than a bare count.

▸Rubric

Diffusion paths shown. Look for a who-told-whom path per variation, the way Figure 9 draws Isabella's party. FAIL if the report gives only a count or percentage per variation. PASS if, for each variation, the report or its attachments show the chain of personas the ad passed through, as a graph, a list or a table.

Q11Believable grounded dialoguethis task

Judge's reasoning

Dialogue is anchored in seeded facts and prior turns — Arthur's dark roast and ecology framing, Lucas's oak book stands and workshop bills, Lucas correctly citing Marcus as his earlier source — with no claims outside each persona's memory.

▸Rubric

Believable, non-hallucinated dialogue. Read three conversations from different variations. FAIL if the dialogue is generic filler that any two people could say, or a persona asserts something that is in neither its seed nor its memory stream — an event that never happened, a relationship it does not have. PASS if the lines draw on the personas' seeded facts and earlier interactions and stay inside what each persona could know.