Consistent result tablesresult-tablesthis task

the named family, evidence, and edge tables have consistent schemas and keys.

Timeline and lineage figuresfiguresthis task

valid timeline and lineage/network/tree figures contain graphical content.

Message-level evidence trailevidence-trailthis task

at least six report IDs or resolved evidence links have exact message IDs.

Full-corpus accountingcorpus-scalethis task

the report and corpus audit consistently account for at least 100,000 rows.

No AI-tell wordingai-wordingevery task

deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Task checks may add narrower contractual bans, but must not repeat this shared floor.

inspected REPORT.md, README.md, ANSWER.md, figures/lineage_document_destruction.svg, figures/lineage_motif_accounting_scandals.svg, figures/lineage_prepay_loans.svg, figures/null_tests.svg, figures/custodian_tree_traits.svg

Substantial research reportreportthis task

REPORT.md exists and contains at least 600 words.

Executable analysis pipelineanalysis-codethis task

an executable, documented analysis pipeline covers the required stages.

Q1Adapts comparative phylogeneticsthis task

Judge's reasoning

REPORT §3 gives an explicit paper→corpus mapping table (taxa=149 custodian mailboxes, traits=family presence/absence, tree=UPGMA of baseline correspondent profiles, spatial=Notes office code, D/autologistic/Mk1 ancestral states) and actually runs those tests in phylomemetics/comparative.py+phylo.py, while stating the breaks (communication tree ≠ genealogy, P(root present)<3e-5 is uninformative so roots are done on the time axis, office/clade confounding).

▸Rubric

Is this a real adaptation of the paper's comparative logic rather than a phylogeny metaphor? PASS only if the work explicitly maps the paper's analytical objects to this setting (at minimum units/taxa, traits or variants, descent/transmission, horizontal diffusion, and root/ancestral-state inference), uses that mapping in the actual analysis, and explains important breaks in the analogy. FAIL if it merely draws a tree, copies biological tests onto emails without justification, or claims an org/language tree that the provided data do not contain.

Q2Defensible idea familiesthis task

Judge's reasoning

Families are defined by operational definitions + TF-IDF centroids with anchors and background subtraction (not raw keyword counts), τ earned per family from 371 hand labels with precision floors, LOSO keyword-dependence tests honestly reporting low recall for MTM/retention, benign controls (United Way, fantasy football, holiday party), plus an explicit evidence ladder (ordinary business → advocacy → misleading claim → concealment cue → concern cue → knowing deception) with 'knowing deception: not found' and no accusations.

▸Rubric

Are the candidate "fraudulent ideas" operationalized defensibly? PASS only if families are coherent claims or narratives supported by semantic or textual evidence; the analysis distinguishes ordinary discussion, aggressive advocacy, potentially misleading claims, concealment, and knowing deception; and labels/conclusions are tied to explicit criteria. FAIL for a fraud-keyword search, topic-model labels presented as proof, circular seed selection, or accusations based only on who used a word.

Q3Sound corpus treatmentthis task

Judge's reasoning

Full 517,401 rows parsed from the zip (corpus_profile.csv: 0 malformed, 1 unparseable date, 605 out-of-range excluded), 263,698 duplicate mailbox copies collapsed via sha1(sender|minute|subject|body) content key to 253,703 canonical, and own-text vs quoted-text are scored separately so quoted matches become 'relays', never origins or independent transmissions.

▸Rubric

Is the corpus treatment adequate for transmission inference? PASS only if the pipeline parses the full archive (while disclosing any later filtering or sampling), normalizes dates and participants, handles malformed records, and addresses duplicate mailbox copies plus quoted/forwarded text so they do not become false independent transmissions. FAIL if the work silently analyzes a convenience sample, treats every CSV row as independent, or ignores temporal parsing and quote duplication.

Q4Auditable uncertain originsthis task

Judge's reasoning

All in-depth families carry root file paths, Message-IDs, dates, senders, excerpts and 0–5 confidence components in idea_families.csv/root_sensitivity.csv/evidence_messages.csv; I verified taylor-m/all_documents/96., farmer-d/2364., jones-t/4399., fossum-d/_sent_mail/174., campbell-l/inbox/916. and buy-r/inbox/1076. against archive.zip — dates, senders and excerpts match exactly; competing/jack-knife roots and censoring (SPE root called 'low 2/5, precursor vocabulary'; Watkins letter 'censored root') are stated rather than forced.

▸Rubric

Are origin claims auditable and appropriately uncertain? PASS only if at least three families have an earliest supported appearance or root, with exact message IDs and/or corpus file paths, dates, participants, short excerpts, and confidence or competing-root discussion. Spot checks across the report and evidence table must be internally consistent. FAIL if roots are named without primary-message evidence, if "earliest in this corpus" becomes "invented by this person," or if ambiguous roots are forced into certainty.

Q5Evidence-led spread pathsthis task

Judge's reasoning

transmission_edges.csv (2,015 edges) assigns each expression a best earlier parent via a typed ladder combining time, shingle re-use, observed receipt, prior baseline tie and forward/quote evidence (FWD/RCV_COPY/TIE_COPY vs RCV exposure-only, SHARED shared-source, COPY, ORIGIN_OR_UNOBSERVED), with per-family lineage SVGs laid out by person/office/date, and the retirement family is explicitly reclassified as 550 SHARED shared-source convergence rather than descent.

▸Rubric

Does the work reconstruct spread rather than just count mentions? PASS only if the lineage/transmission edges use temporal order and textual mutation/similarity together with sender-recipient, thread, or forward evidence; the output shows interpretable paths through people or groups; and it distinguishes direct transmission from independent convergence or shared-source exposure. FAIL for a co-occurrence network, sender leaderboard, or timeline with no evidence-led parent/child logic.

Q6Chance and sensitivity teststhis task

Judge's reasoning

null_model_tests.csv reports a timestamp-permutation null (500, seed 11), month-matched pseudo-family null (200, seed 12), wording-inheritance null (1,000, seed 13), 999-rep D randomizations, 199-perm autologistic and 999-perm Mantel, plus three benign controls; the controls change conclusions (United Way passes S1 at p=0.048, so exposure-driven spread is declared non-diagnostic), and sensitivity is reported at τ±0.05, via mailbox jack-knife roots and LOSO recall.

▸Rubric

Does it test whether the inferred structure is stronger than chance and probe sensitivity? PASS only if the work uses at least one meaningful null model, permutation, negative control, or comparable baseline and reports what changed under plausible clustering/edge thresholds or candidate definitions. The control must bear on an actual inference, not appear as generic methodology prose. FAIL if every observed cluster or edge is assumed meaningful or if robustness is asserted without a reported test.

Q7Answers origin and spreadthis task

Judge's reasoning

§0 and ANSWER.md answer the question directly per family — SPE/MTM/prepay/retention originate as internal work vocabulary in small professional clusters, the famous crisis phrases arrive from press or have censored roots, retirement is one external petition copied 1,113 times — with reach, drift terms, statistical support and limits; prose stays at 'candidate' level, names no one as fraudulent, and flags that no knowing-deception message exists.

▸Rubric

Do the results directly answer where the ideas originated and how they spread, without outrunning the evidence? PASS only if the report synthesizes family-specific origins, routes, mutation of claims, reach, and uncertainty into clear conclusions; figures and tables support those conclusions; and the prose avoids unsupported legal or personal claims. FAIL if the answer stays at methods, gives generic Enron history, buries the research question in artifact inventory, or presents guilt/conspiracy as proven when the corpus supports only a candidate interpretation.

Q8Reproducible research bundlethis task

Judge's reasoning

`python3 analysis.py` starts from archive.zip (streamed, never unpacked), stdlib-only, os.chdir to its own folder with relative paths, every seed tabulated in README, τ grid/precision floors/thresholds visible as module constants, hand labels shipped in annotations/ and kept out of the code; outputs carry file paths and Message-IDs linking every report claim back to inspectable rows.

▸Rubric

Is the research bundle reproducible and inspectable? PASS only if a documented offline command can start from `archive.zip` and regenerate the principal tables and figures; paths are relative; randomness is seeded or absent; key thresholds are visible; and outputs retain message-level audit links. FAIL if the code is pseudocode, depends on hidden/manual steps, uses unavailable private data, hard-codes reported results, or cannot connect the report's claims back to generated artifacts. ## Output format Return one line per question: `Q<n>: PASS|FAIL — <specific evidence>` Then: `TOTAL: <passed>/8`.