15 phylomemetics

Consistent result tablesresult-tablesthis task

the named family, evidence, and edge tables have consistent schemas and keys.

Timeline and lineage figuresfiguresthis task

valid timeline and lineage/network/tree figures contain graphical content.

Message-level evidence trailevidence-trailthis task

at least six report IDs or resolved evidence links have exact message IDs.

Full-corpus accountingcorpus-scalethis task

the report and corpus audit consistently account for at least 100,000 rows.

No AI-tell wordingai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

inspected REPORT.md, README.md, ANSWER.md, results/summary.json, figures/lineage_F4.svg, figures/lineage_F5.svg, figures/org_spread.svg, figures/null_models.svg

Substantial research reportreportthis task

REPORT.md exists and contains at least 600 words.

Executable analysis pipelineanalysis-codethis task

an executable, documented analysis pipeline covers the required stages.

Q1Adapts comparative phylogeneticsthis task

Judge's reasoning

§2 gives an explicit mapping table (tale type→idea family, variant→authored instance, societies→people/units, traits→regex motifs, language tree→time-ordered communication graph, vertical→contact-backed descent edges, horizontal→broadcast/similarity-only, D-statistic→contact permutation, ancestral-state→Fitch root states) and the code actually implements it (lineage.py fitch_root_states, permutation_test_contact); breaks in the analogy (no org chart, censored 260 mailboxes, reticulate graph built from the same data, 2-year depth) are stated and revisited in Limitations.

▸Rubric

Is this a real adaptation of the paper's comparative logic rather than a phylogeny metaphor? PASS only if the work explicitly maps the paper's analytical objects to this setting (at minimum units/taxa, traits or variants, descent/transmission, horizontal diffusion, and root/ancestral-state inference), uses that mapping in the actual analysis, and explains important breaks in the analogy. FAIL if it merely draws a tree, copies biological tests onto emails without justification, or claims an org/language tree that the provided data do not contain.

Q2Defensible idea familiesthis task

Judge's reasoning

Families come from an unsupervised propagating-passage layer (394k near-dup clusters → 21,121 multi-author → PMI-expanded concept ranking for reading), then explicit anchor/exclusion/require/motif definitions in family_defs.py; each family is classified on an articulated spectrum (advocacy F5, misleading framing F2, execution-without-shown-intent F3, anti-concealment F4, external accusation F6) with a neutral United Way control, and the report repeatedly states knowing deception is not establishable rather than accusing individuals.

▸Rubric

Are the candidate "fraudulent ideas" operationalized defensibly? PASS only if families are coherent claims or narratives supported by semantic or textual evidence; the analysis distinguishes ordinary discussion, aggressive advocacy, potentially misleading claims, concealment, and knowing deception; and labels/conclusions are tied to explicit criteria. FAIL for a fraud-keyword search, topic-model labels presented as proof, circular seed selection, or accusations based only on who used a word.

Q3Sound corpus treatmentthis task

Judge's reasoning

parse_corpus.py streams all 517,401 rows from archive.zip (verified: cache has 517,401 copies → 254,222 unique messages), collapses duplicate mailbox copies on a sender/date/subject/body hash, splits quoted/forwarded segments with Lotus/Outlook markers and parses embedded headers, uses own_text only for membership, counts forwards as carriers/exposure rather than new variants, and logs malformed/out-of-range dates (605) and 9,274 recipient-less broadcasts.

▸Rubric

Is the corpus treatment adequate for transmission inference? PASS only if the pipeline parses the full archive (while disclosing any later filtering or sampling), normalizes dates and participants, handles malformed records, and addresses duplicate mailbox copies plus quoted/forwarded text so they do not become false independent transmissions. FAIL if the work silently analyzes a convenience sample, treats every CSV row as independent, or ignores temporal parsing and quote duplication.

Q4Auditable uncertain originsthis task

Judge's reasoning

All six families plus control carry earliest_supported uid + Message-ID + corpus file + date + sender + excerpt with competing roots, confidence and jackknife root stability; I verified uids 23399/2646/218844/80598/113804/130345 against archive.zip (campbell-l/inbox/916., stclair-c/all_documents/40., hain-m/_sent_mail/315., etc.) and report/evidence table/idea_families.csv agree; the report explicitly says the corpus dates first written transmission, not conception (notably F3, whose designers are not mailbox owners).

▸Rubric

Are origin claims auditable and appropriately uncertain? PASS only if at least three families have an earliest supported appearance or root, with exact message IDs and/or corpus file paths, dates, participants, short excerpts, and confidence or competing-root discussion. Spot checks across the report and evidence table must be internally consistent. FAIL if roots are named without primary-message evidence, if "earliest in this corpus" becomes "invented by this person," or if ambiguous roots are forced into certainty.

Q5Evidence-led spread pathsthis task

Judge's reasoning

Edges are graded quoted > received > broadcast > thread > similar using quoted-cluster matching, a recipient+carrier exposure index, thread/subject evidence and time order plus combined shingle-containment/TF-IDF similarity; transmission_edges.csv gives parent/child uids, senders, units, similarity and lag (e.g. Fergus→Belden 22 Dec 2000, sim 0.923, 2.8 h), lineage SVGs show interpretable person paths, and the report separates chain descent (F3/F4), broadcast-then-independent-restatement (F1/F2) and shared-source star diffusion (F6, 0/1120 contact-backed).

▸Rubric

Does the work reconstruct spread rather than just count mentions? PASS only if the lineage/transmission edges use temporal order and textual mutation/similarity together with sender-recipient, thread, or forward evidence; the output shows interpretable paths through people or groups; and it distinguishes direct transmission from independent convergence or shared-source exposure. FAIL for a co-occurrence network, sender leaderboard, or timeline with no evidence-led parent/child logic.

Q6Chance and sensitivity teststhis task

Judge's reasoning

Contact-label permutation (F3 68 obs vs null 20.3, p=0.002; F4 8 vs 1.8), time permutation, random same-window similarity baseline, motif Fitch-parsimony permutation, 20%-deletion root stability, a neutral X1 control and an external F6 control, plus threshold sweeps in sensitivity.csv (θ=0.3/0.4/0.5; edge sim 0.4/0.5/0.6 changing F3 87→79 with roots unchanged); only blemish is that §3.4's recounted θ=0.30 additions and p99.9 range are stale versus sensitivity.csv (F2 17 not 0, F5 50 not 115, range 0.20–0.40 not 0.32–0.44), which does not affect the reported conclusion that θ=0.40 adds zero members.

▸Rubric

Does it test whether the inferred structure is stronger than chance and probe sensitivity? PASS only if the work uses at least one meaningful null model, permutation, negative control, or comparable baseline and reports what changed under plausible clustering/edge thresholds or candidate definitions. The control must bear on an actual inference, not appear as generic methodology prose. FAIL if every observed cluster or edge is assumed meaningful or if robustness is asserted without a reported test.

Q7Answers origin and spreadthis task

Judge's reasoning

§0 and §6 answer the question directly per family — dated internal roots (F1 14 Aug 2001, F2 16 Oct 2001, F4 14 Dec 2000, F5 14 Aug 2000), an idea older than its corpus record (F3), an external root (F6) — with routes, claim mutation (one-time charges → related party → restatement; hedge → credit deficiency → unwind), reach figures, and consistent artifacts; language stays at 'misleading/aggressive advocacy/not establishable', with no guilt asserted as proven.

▸Rubric

Do the results directly answer where the ideas originated and how they spread, without outrunning the evidence? PASS only if the report synthesizes family-specific origins, routes, mutation of claims, reach, and uncertainty into clear conclusions; figures and tables support those conclusions; and the prose avoids unsupported legal or personal claims. FAIL if the answer stays at methods, gives generic Enron history, buries the research question in artifact inventory, or presents guilt/conspiracy as proven when the corpus supports only a candidate interpretation.

Q8Reproducible research bundlethis task

Judge's reasoning

`python3 analysis.py` runs five cached stages from archive.zip with stdlib only, relative paths (os.chdir(HERE)), SEED=20260922 and seeded permutations; thresholds THETA/THETAS/SIM_SIMILAR are module constants; cache/full_run.log shows a real 225 s end-to-end run whose per-family numbers match REPORT.md and the CSVs exactly, and every result row retains uid/Message-ID/file links back to the corpus.

▸Rubric

Is the research bundle reproducible and inspectable? PASS only if a documented offline command can start from `archive.zip` and regenerate the principal tables and figures; paths are relative; randomness is seeded or absent; key thresholds are visible; and outputs retain message-level audit links. FAIL if the code is pseudocode, depends on hidden/manual steps, uses unavailable private data, hard-codes reported results, or cannot connect the report's claims back to generated artifacts. ## Output format Return one line per question: `Q<n>: PASS|FAIL — <specific evidence>` Then: `TOTAL: <passed>/8`.