13 ai-village

No AI-tell wordingai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

shared AI-tell phrases: furthermore; moreover

Substantial reportreportthis task

a report of at least 300 words exists in the workspace: REPORT.md, else any report-named document, else ANSWER.md.

REPORT.md, 2421 words

Enough persona subagentssubagent-personasthis task

the run's own session log shows at least 8 subagent spawns -- Agent/Task tool calls in a Claude transcript, spawn_agent in a Codex rollout, spawn_subagent in a Grok session -- or at least 8 subagent transcripts were captured beside it. The prompt makes subagents the only model available for the personas, so a run that never spawned one did not simulate anything. FAIL when no trace was captured.

404 subagent spawns in the trace, 402 subagent transcripts

Three measured variationsresults-datathis task

results.json parses and lists at least three variations, each carrying the ad text and an end-of-run awareness figure: a share in [0, 1] or a count no larger than the town.

3 variations with ad text and awareness in results.json

Distinct ad variationsvariants-differthis task

no two variations are the same: the ad text, once case, punctuation and spacing are normalised, together with whatever the entry names as the lever -- seed persona, messenger, channel, incentive, framing. The same flyer handed to a different persona is a variation (the prompt says so); three labels on one setup are not. Whether a variation tests anything is the judge's call.

3 variations, each a different ad or lever

Memory, reflection, and plansmemory-streamsthis task

files whose path names a memory stream (or JSON files keeping one under a memory-named key) exist and, read together, hold all three record kinds the paper defines: observation, reflection and plan.

31 memory files with observations, reflections and plans

Report numbers matchnumbers-tie-outthis task

every variation's final awareness in results.json is in the report, as a percentage within one point or as "k of n" / "k/n" / "k out of n".

in the report: A: plain ad, peripheral seed (Klaus)=1 of 8, B: plain ad, hub seed (Isabel)=5 of 8, C: referral incentive, peripheral seed (Klaus)=1 of 8

Interactive diffusion mapinteractive-mapthis task

a self-contained map.html is linked from the report, includes every variation, draws a map with SVG or canvas, has controls for switching and inspection, and can replay the diffusion in chronological order.

map.html is self-contained, replayable, linked, and covers 3 variations

Q1Uses the paper's architecturethis task

Judge's reasoning

sim.py keeps a per-persona timestamped memory stream (seed/plan/observation/conversation/reflection), retrieval scoring normalized recency (0.8 decay from last_access) + persona-rated importance + keyword-cosine relevance, threshold-triggered reflection with cited evidence ids, and a day plan decomposed into per-slot actions revisable mid-day.

▸Rubric

The paper's architecture, not a chat loop. Read the simulation code and one persona's memory stream. FAIL if a persona is just a system prompt plus the transcript so far — no memory stream that accumulates records, no retrieval step that selects which memories a persona sees, no reflection, no plan. PASS if all four are present in some form: a per-persona stream of timestamped records, retrieval that scores or filters memories on recency, importance and relevance (an approximation of all three is fine if it is named as one), reflections synthesised from earlier records, and a day plan that is broken into actions.

Q2Subagents drive behaviorthis task

Judge's reasoning

Every persona line/decision comes from a packet file containing that persona's identity plus retrieved memories and a matching JSON response written by a subagent (runs/*/packets + responses, 135–146 per run), with the ingest/advance loop in sim.py handing memory out and taking answers back; no templates or rule tables generate dialogue.

▸Rubric

Subagents produced the personas' behaviour. Read the code and a few conversations. FAIL if what the personas say or decide comes from templates, a rule table, random draws, or the orchestrator writing the lines itself in bulk for everyone. PASS if each persona's dialogue and choices are free text that a subagent produced with that persona's memory in front of it, and the code or run log shows the loop that hands memory out and takes answers back.

Q3Believable connected communitythis task

Judge's reasoning

Each of the 8 personas has a Smallville-style identity paragraph (occupation, hours, traits, housemates, friends, employer/employee, marriages) split by sentence into initial memories, and the ties interlock into one community.

▸Rubric

Seeded like Smallville. Read the persona definitions. FAIL if the personas are names with a job title, or strangers with no ties to each other. PASS if each has a paragraph of identity — occupation, traits, who they live with, who they know — entered as initial memories, and the relationships link them into one community.

Q4Controlled comparisonthis task

Judge's reasoning

Same town, same 8 ticks, same pre-ad day plans, same mailbox channel and same 20:00 interview across all three runs; the one difference — B's hub seed (Isabel) vs A/C's peripheral seed (Klaus) — is the deliberate 'messenger' lever, stated in the headline table's Seed column and discussed throughout, with A vs C isolating the incentive.

▸Rubric

A controlled comparison. Read the report and the run configuration. FAIL if the variations differ in anything besides the advertisement — a different seeding persona or location, a different town, a different number of ticks — and the report does not acknowledge it as a confound. PASS if the same town, the same entry point and the same simulated length were used for every variation and the report says so.

Q5Awareness verified from memorythis task

Judge's reasoning

All 8 personas were interviewed at end of each run with retrieved memories and answered in first person; analyze() only counts a yes when the persona's own stream contains ad evidence ids, tallies n_hallucinated_yes (0) and the exposed-but-said-no case (Mei in B), and the report states this.

▸Rubric

Diffusion measured the way section 7.1.1 measures it. Read the interview material. FAIL if awareness was read off the orchestrator's own record of who was told, or if the personas were not each asked at the end whether they know about the advertisement, or if a "yes" was counted without checking that persona's memory stream for where it came from. PASS if every persona was interviewed at the end of every variation and each yes was verified against memory, with any hallucinated yes excluded and said so.

Q6Advertisement diffuses naturallythis task

Judge's reasoning

begin_step() writes the leaflet memory only to cfg['seed_persona'] at step 0; every subsequent knower in results.json is credited to a specific dialogue line containing the ad keywords.

▸Rubric

The advertisement entered the world, it was not broadcast. Trace how each variation was introduced. FAIL if the ad was written into every persona's memory by the orchestrator, or told to most of the town at tick zero — that is not diffusion. PASS if it reached one or two personas or one location and everything beyond that came from personas talking to each other.

Q7Meaningful tested variationsthis task

Judge's reasoning

A = baseline, B changes the messenger (hub vs peripheral seed), C adds a refer-a-friend incentive, and each variation carries a pre-registered numeric expectation in town.json that is reproduced in the report's prediction column before the results.

▸Rubric

Variations that test something. Read the ad texts and the report's setup. FAIL if the variations are cosmetic rewordings with no hypothesis, or the report states no expectation before the results. PASS if each variation changes one nameable lever — framing, messenger, channel, incentive, social proof — and the report says what was expected of it before the numbers.

Q8Diffusion paths shownthis task

Judge's reasoning

The report gives an ordered who-told-whom chain for B (Isabel→Carla/Tomás/Sam/Mei, Sam→Priya with times and locations) and explains the empty path for A and C; results.json carries diffusion_path edges and map.html animates them.

▸Rubric

Diffusion paths shown. Look for a who-told-whom path per variation, the way Figure 9 draws Isabella's party. FAIL if the report gives only a count or percentage per variation. PASS if, for each variation, the report or its attachments show the chain of personas the ad passed through, as a graph, a list or a table.

Q9Actionable marketing findingsthis task

Judge's reasoning

It names B as the widest spread and attributes it to relevance-driven importance ratings (7/10 vs 3/10) visible in the responses, cites Tomás reframing the ad as a Main Street threat and Sam's dinner relay, notes the missed Sam→Klaus grower question, and recommends concrete next moves (seed the person with the problem, arm the seed against the incumbent's objection, budget for one hop).

▸Rubric

Findings a marketer could act on. Read the report's conclusions. FAIL if it only restates the numbers, or explains the difference between variations with generalities that are not grounded in what happened in the runs. PASS if it says which variation spread furthest, points to a mechanism visible in the transcripts — who relayed it, in what setting, what about the ad made it worth repeating — and names what to try next.

Q10Honest about scale and noisethis task

Judge's reasoning

Limitations state 8 personas, one 12-hour day, 8 steps, ~420 subagent calls, one run per variation with explicit 'may be noise' framing, plus the keyword-vs-embedding retrieval simplification and shared day plans, contrasted against the paper's 25 agents over two days.

▸Rubric

Honest about scale and noise. Read the report's limitations. FAIL if it presents the result as if it were the paper's twenty-five agents over two days, or omits that each variation was run once with a handful of personas so the differences may be noise. PASS if it states the scale — personas, ticks, roughly how many subagent calls — what was simplified from the paper, and what that does to the confidence a reader should have.

Q11Believable grounded dialoguethis task

Judge's reasoning

Sampled conversations from all three runs stay inside seeded and remembered facts — Isabel's supplier hunt, Tomás's highway-supermarket grudge and egg/milk deliveries, Carla's book recommendations to budget-conscious Klaus, Joan's library shift feeding her yoga chat — and spot-checked interview claims trace to real memory ids.

▸Rubric

Believable, non-hallucinated dialogue. Read three conversations from different variations. FAIL if the dialogue is generic filler that any two people could say, or a persona asserts something that is in neither its seed nor its memory stream — an event that never happened, a relationship it does not have. PASS if the lines draw on the personas' seeded facts and earlier interactions and stay inside what each persona could know.