13 ai-village

No AI-tell wordingai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

shared AI-tell phrases: furthermore; moreover

Substantial reportreportthis task

a report of at least 300 words exists in the workspace: REPORT.md, else any report-named document, else ANSWER.md.

REPORT.md, 1664 words

Enough persona subagentssubagent-personasthis task

the run's own session log shows at least 8 subagent spawns -- Agent/Task tool calls in a Claude transcript, spawn_agent in a Codex rollout, spawn_subagent in a Grok session -- or at least 8 subagent transcripts were captured beside it. The prompt makes subagents the only model available for the personas, so a run that never spawned one did not simulate anything. FAIL when no trace was captured.

48 subagent spawns in the trace, 48 subagent transcripts

Three measured variationsresults-datathis task

results.json parses and lists at least three variations, each carrying the ad text and an end-of-run awareness figure: a share in [0, 1] or a count no larger than the town.

3 variations with ad text and awareness in results.json

Distinct ad variationsvariants-differthis task

no two variations are the same: the ad text, once case, punctuation and spacing are normalised, together with whatever the entry names as the lever -- seed persona, messenger, channel, incentive, framing. The same flyer handed to a different persona is a variation (the prompt says so); three labels on one setup are not. Whether a variation tests anything is the judge's call.

3 variations, each a different ad or lever

Memory, reflection, and plansmemory-streamsthis task

files whose path names a memory stream (or JSON files keeping one under a memory-named key) exist and, read together, hold all three record kinds the paper defines: observation, reflection and plan.

27 memory files with observations, reflections and plans

Report numbers matchnumbers-tie-outthis task

every variation's final awareness in results.json is in the report, as a percentage within one point or as "k of n" / "k/n" / "k out of n".

in the report: practical=75%, savings=38%, social=38%

Interactive diffusion mapinteractive-mapthis task

a self-contained map.html is linked from the report, includes every variation, draws a map with SVG or canvas, has controls for switching and inspection, and can replay the diffusion in chronological order.

map.html is self-contained, replayable, linked, and covers 3 variations

Q1Uses the paper's architecturethis task

Judge's reasoning

run.py maintains per-persona timestamped memory streams with observations/plans/reflections, a retrieve() scoring normalized recency (0.995^hours), persona-rated importance and lexical cosine relevance, evidence-cited reflections, and a day plan decomposed into timed actions (see faye.json faye-005..011, faye-014).

▸Rubric

The paper's architecture, not a chat loop. Read the simulation code and one persona's memory stream. FAIL if a persona is just a system prompt plus the transcript so far — no memory stream that accumulates records, no retrieval step that selects which memories a persona sees, no reflection, no plan. PASS if all four are present in some form: a per-persona stream of timestamped records, retrieval that scores or filters memories on recency, importance and relevance (an approximation of all three is fine if it is named as one), reflections synthesised from earlier records, and a day plan that is broken into actions.

Q2Subagents drive behaviorthis task

Judge's reasoning

Python makes no model calls; prepare() writes a per-persona prompt with that persona's retrieved memories, a subagent authors the free-text openings/replies/reflections into responses/*.json, and ingest() feeds them back into the world state.

▸Rubric

Subagents produced the personas' behaviour. Read the code and a few conversations. FAIL if what the personas say or decide comes from templates, a rule table, random draws, or the orchestrator writing the lines itself in bulk for everyone. PASS if each persona's dialogue and choices are free text that a subagent produced with that persona's memory in front of it, and the code or run log shows the loop that hands memory out and takes answers back.

Q3Believable connected communitythis task

Judge's reasoning

Each of the eight personas has a seed paragraph covering age, occupation, traits, named friends/relatives and personal concerns, split into four initial memories, and the relationship ties (Ada–Ben–Cora–Dev–Eli–Faye–Gus–Hana) form one connected community.

▸Rubric

Seeded like Smallville. Read the persona definitions. FAIL if the personas are names with a job title, or strangers with no ties to each other. PASS if each has a paragraph of identity — occupation, traits, who they live with, who they know — entered as initial memories, and the relationships link them into one community.

Q4Controlled comparisonthis task

Judge's reasoning

All three variations share the same eight personas, the same sole seed (Ada's private flyer at 09:00), the same ten contact opportunities and the same 09:00–14:00 window, and the report and PRE_REGISTRATION.md state this explicitly.

▸Rubric

A controlled comparison. Read the report and the run configuration. FAIL if the variations differ in anything besides the advertisement — a different seeding persona or location, a different town, a different number of ticks — and the report does not acknowledge it as a confound. PASS if the same town, the same entry point and the same simulated length were used for every variation and the report says so.

Q5Awareness verified from memorythis task

Judge's reasoning

All eight personas were interviewed at 14:00 in every variation, and a 'yes' is only counted when a cited memory ID has heard_from plus an exact stimulus identifying the workshop (own speech/plans/reflections rejected); the report explains the rule and reports zero rejections.

▸Rubric

Diffusion measured the way section 7.1.1 measures it. Read the interview material. FAIL if awareness was read off the orchestrator's own record of who was told, or if the personas were not each asked at the end whether they know about the advertisement, or if a "yes" was counted without checking that persona's memory stream for where it came from. PASS if every persona was interviewed at the end of every variation and each yes was verified against memory, with any hallucinated yes excluded and said so.

Q6Advertisement diffuses naturallythis task

Judge's reasoning

The ad enters only as Ada's private flyer (private_stimulus is None for everyone else) and every later acquisition is traced to a persona-authored line in a recorded conversation.

▸Rubric

The advertisement entered the world, it was not broadcast. Trace how each variation was introduced. FAIL if the ad was written into every persona's memory by the orchestrator, or told to most of the town at tick zero — that is not diffusion. PASS if it reached one or two personas or one location and everything beyond that came from personas talking to each other.

Q7Meaningful tested variationsthis task

Judge's reasoning

The offer is held constant while the closing appeal varies along a named lever (practical skill / savings / social), and PRE_REGISTRATION.md records per-framing expectations about who would carry each before the runs.

▸Rubric

Variations that test something. Read the ad texts and the report's setup. FAIL if the variations are cosmetic rewordings with no hypothesis, or the report states no expectation before the results. PASS if each variation changes one nameable lever — framing, messenger, channel, incentive, social proof — and the report says what was expected of it before the numbers.

Q8Diffusion paths shownthis task

Judge's reasoning

The report lists a timestamped who-told-whom chain with location and conversation ID per variation (e.g. Ada→Cora→Faye→Hana→Gus), mirrored by diffusion_path in results.json.

▸Rubric

Diffusion paths shown. Look for a who-told-whom path per variation, the way Figure 9 draws Isabella's party. FAIL if the report gives only a count or percentage per variation. PASS if, for each variation, the report or its attachments show the chain of personas the ad passed through, as a graph, a list or a table.

Q9Actionable marketing findingsthis task

Judge's reasoning

The conclusions point to a mechanism visible in the transcripts — Ada's choice of confidant differing by framing, and Faye converting the practical ad into a personal invitation to Hana that Hana then relayed to Gus — and name a concrete next test (vary the invitation independently of the headline).

▸Rubric

Findings a marketer could act on. Read the report's conclusions. FAIL if it only restates the numbers, or explains the difference between variations with generalities that are not grounded in what happened in the runs. PASS if it says which variation spread furthest, points to a mechanism visible in the transcripts — who relayed it, in what setting, what about the ad made it worth repeating — and names what to try next.

Q10Honest about scale and noisethis task

Judge's reasoning

The report states 8 personas, 09:00–14:00, 48 subagent calls, lists the simplifications from the paper (batched decisions, lexical retrieval, fixed contact graph, two-turn conversations), and says one run per wording means the 6-vs-3 gap is descriptive and may be noise.

▸Rubric

Honest about scale and noise. Read the report's limitations. FAIL if it presents the result as if it were the paper's twenty-five agents over two days, or omits that each variation was run once with a handful of personas so the differences may be noise. PASS if it states the scale — personas, ticks, roughly how many subagent calls — what was simplified from the paper, and what that does to the confidence a reader should have.

Q11Believable grounded dialoguethis task

Judge's reasoning

Dialogue in all three variations draws on seeded facts (Ada's torn apron, Ben's split glove seam, Dev's course savings, Hana's ripped denim bag) and on earlier recorded exchanges, with no claims outside each persona's memory.

▸Rubric

Believable, non-hallucinated dialogue. Read three conversations from different variations. FAIL if the dialogue is generic filler that any two people could say, or a persona asserts something that is in neither its seed nor its memory stream — an event that never happened, a relationship it does not have. PASS if the lines draw on the personas' seeded facts and earlier interactions and stay inside what each persona could know.