# ANSWER: clean-room rewrite of live document collaboration (issue #557 family)

Choice: **greenfield**. The legacy collab engine (live Y.Doc registry, projection/repair workers, leases, quarantine, collab verification
and fallback, dual-write mutation tables) is deleted, not wrapped. `npm run build` and `npm run serve` use the new core through the
existing editor and API URLs. There is no fallback to the legacy implementation: the legacy modules no longer exist in the tree.

## 1. Scope delivered, and where things live

| What | Where |
|---|---|
| The plan (written first): owner's question, diagnosis, invariants I1-I13, architecture, incident mapping, phases, test and retirement rule, as-built notes | `docs/plans/2026-03-18-clean-room-document-core-plan.md` |
| Contract inventories of the old system that the rewrite had to keep (document API, agent API, persistence and surfaces) | `docs/plans/appendix-a-document-api-contract.md`, `appendix-b-agent-api-contract.md`, `appendix-c-persistence-and-surfaces.md` |
| Retirement ledger: every removed/adapted/untouched legacy test, with its replacement | `docs/plans/test-retirement-ledger.md` (scenario-level detail in `docs/plans/ledger-fragments/`) |
| Findings found while porting tests, with repros and status | `docs/plans/ledger-fragments/g1-findings.md`, `g2-findings.md`, `g3-findings.md` |
| New core (store, model, tokens, access, sync, rooms, ops, mark events, presence, service, transport, viewers, side effects, errors, limits) | `server/core/` (about 3.7k lines), `server/health.ts` |
| New API layer on the core (documents, agent, commands, bug-report routes, principal, direct-share, http errors) | `server/api/` (about 2.4k lines) |
| Re-pointed edge modules | `server/index.ts`, `routes.ts`, `bridge.ts`, `snapshot.ts`, `library.ts`, `dashboard.ts`, `share-web-routes.ts`, `bug-reporting.ts`, `db.ts` (slimmed) |
| Client edits (minimal: stop publishing projections / seeding on the collab path) | `src/editor/share-collab-bind-reset.ts`, `src/editor/share-refresh-persist.ts` |
| New tests | `src/tests/core-*.test.ts` (19 files, helpers `core-test-harness.ts`, `core-scenario-helpers.ts`, `core-known-finding.ts`) |
| Docs updated to the new model | `AGENTS.md` (Collab Guardrails), `CLAUDE.md` (notice), `README.md`, `docs/agent-docs.md`, `docs/testing/proof-collab-test-strategy.md`, `.agents/skills/proof-collab-test-author/SKILL.md` |
| `preview.json` | `{"install":"npm ci && npm run build","start":"npm run serve","port":4000}` (the port is `$PORT` when set) |

### The design in five lines
1. One authority per document: a Yjs document whose only durable form is an **append-only update log plus a compacted snapshot** in SQLite
   (`core_updates`, `core_snapshots`, `core_docs`, `core_events`, `core_idempotency`). Markdown, text, block refs, `mt1:` base tokens are projections computed on read; nothing derived is stored or repaired.
2. One write path: browser edits arrive over Hocuspocus, REST/agent/library edits are a single Yjs transaction applied to the same live room under a per-slug FIFO lock with a bounded queue. Every committed change is durable before it is acknowledged (a synchronous `doc.on('update')` append); a failed append faults that one room and nothing else.
3. Failure is scoped per slug (503/413/429 for that document), never a global breaker. `/health` and public routes never wait on a room lock.
4. Authorization is one pure function (`admitSession`/`capabilitiesFor`), applied identically on HTTP and at socket admission; epoch bumps close only that slug's sockets.
5. Graceful shutdown drains rooms, reconciles and compacts; legacy rows are adopted exactly once from their old markdown/marks columns.

## 2. Commands run and results

All run from the repo root. In this sandbox `APPSIGNAL_LOG_PATH=$TMPDIR` is needed (AppSignal cannot log to `/tmp`; it also made `test:pr:collab` fail at the starting commit). Network is blocked, so PostHog/AppSignal calls log `ENOTFOUND` noise; none affect results.

| Command | Result |
|---|---|
| `npm run build` | **pass** (exit 0) |
| `npm run test:ci` | **pass** (exit 0) |
| `npm run test:pr:collab` (`test:collab:contracts` + `test:core`) | **pass** (exit 0): yjs simulation 3, hocuspocus lifecycle 5, yjs history/anchor 4, plus all 19 core files (the per-file lines are in section 4) |
| `npm test` | **exit 1**, stopping at `live-viewer-passive-reconnect-wiring.test.ts` (inside `test:server-routes-share`; Playwright is not installed and the test fails identically at the starting commit). The 76 other commands of the expanded `test` chain were then run by a generated script that only skips the 12 baseline-failing files named below: **0 failures** |
| Every `src/tests/*.test.ts` run one by one (`tsx`, 5-minute alarm), before the last three retirements | 131 pass, 15 fail. After retiring `document-blocks-rebuild`, `mutation-backfill-and-tombstones`, `mutation-dual-write` (they exercised deleted tables) there are 143 files: **131 pass, 12 fail**, and all 12 are files this work did not change |
| `tsc -p tsconfig.json --noEmit` | exit 2, **596** errors (starting commit: 1127). Mostly `TS6059` rootDir config errors and Milkdown module-typing errors that exist at the starting commit; none are logic errors in `server/core` or `server/api` |
| `tsc -p tsconfig.server.json --noEmit` | exit 2, **82** errors (starting commit: 256), same families (47 in `src/editor/schema/proof-marks.ts`, 10 in `milkdown-headless.ts`, a few `string \| string[]` header typings) |
| `tsc -p tsconfig.mcp.json --noEmit` | pass, 0 errors |
| Live run of `npm run serve` plus a REST/WebSocket exerciser, and a SIGKILL-restart exerciser | see section 4 |

The 12 baseline-failing files (not touched by this work; the ledger marks them UNTOUCHED). Nine were run against a clean checkout of the starting commit and fail there too:
`collab-focus-hard-reconnect-runtime-regression`, `collab-hydration-mismatch-regression` (string-matches the unmodified `src/editor/index.ts`), `docker-build-info-wiring` (Dockerfile start command), `editor-activity` and `library-agent-invite-url` (DOM shim lacks `addEventListener`), `library-account-menu-new-doc` (label check), `live-viewer-passive-reconnect-wiring`, `rewrite-mode`, `track-changes`.
The other three need Playwright Chromium, which is not installed here (`collab-focus-refresh-live-regression`, `collab-remote-cursor-runtime-regression`, `share-new-doc-bootstrap-editability-regression`); I did not run those against the starting commit.

### Incident-driven bug fixes made after the first green run
A porting pass over the legacy tests found real defects in the new core; these were fixed and the guarding assertions are now plain assertions (no `knownFinding` wrapper):
parser state poisoning (one unparseable document made the next 1-2 unrelated requests 400), reference-style links/images rejected, untitled images silently dropped, v1 edit and `/rewrite {changes}` deleting explicit blank paragraphs, anchor diagnostics missing from `details`, denied ops no longer traced, paused-library override missing from the ops routes and socket admission, plus the g1 set (accepted suggestions losing edge spaces, formatting lost on replace, markdown-syntax anchors, reads served after a failed commit, `includeTextPreview`). Unrelated harness flake fixed: server start now retries on `EADDRINUSE`.

## 3. Capability inventory

Regression coverage names are test files under `src/tests/`; scenario-level mapping is in the ledger.

### Kept (same URLs and behaviour; covered by new or ported tests)
| Capability | Coverage |
|---|---|
| Create a document from markdown (`POST /share/markdown`, `/documents`), tokens and roles (viewer/commenter/editor/owner), share states (active/paused/revoked/deleted), access links | `core-share-lifecycle`, `core-share-surfaces`, `core-http-ws-integration` |
| Share pages: HTML for browsers (SPA shell) and agents (server-rendered page), JSON, static snapshot fallback, OG pages | `core-share-surfaces`, `core-app-smoke`, `snapshot-html-canonical-fallback` (adapted) |
| Live collaboration over `/ws` (Hocuspocus), awareness, viewer channel; reconnect, offline typing, restart | `core-http-ws-integration`, `core-client-runtime` (real `collab-client`/`share-client` modules), `core-soak`, `core-duplication-regressions` |
| Agent state, snapshot, edit v1 (`/edit`), edit v2 (block refs), rewrite (`content` / `changes`), `PUT` markdown and marks | `core-agent-edit-scenarios`, `core-markdown-fidelity`, `server-routes-and-share` (adapted) |
| Comments, replies, suggestions (add/accept/reject/resolve), marks events, authored provenance in the fragment | `core-marks-scenarios`, `core-mark-write-failure`, `editor-suggestion-api-regression`, `comment-add-edit-sequence-regression`, `reject-suggestion-share-persistence-regression`, `rest-put-comment-events` (all adapted) |
| Idempotency keys, base-token preconditions (`STALE_BASE`), agent presence and activity, events polling | `core-http-ws-integration`, `core-agent-edit-scenarios`, `core-bridge-routes` |
| Bridge routes and auth policy, library documents and sync, Every OAuth flow, dashboard, bug reporting, health | `core-bridge-routes`, `core-library-sync`, `core-library-oauth`, `dashboard-feature-regression`, `agent-bug-report-api`, `health-build-sha-precedence`, `server-incident-tracing` (all adapted) |
| Markdown fidelity: tables, task lists, fenced code with blank lines, frontmatter, HTML spans, footnotes, authored spans, explicit blank paragraphs, images, reference links | `core-markdown-fidelity` (21-document matrix incl. the 25 KB style guide and a newsroom article) |
| Incident #557 families (fault isolation, starvation, oversize, passive viewers, stale epochs, side-effect-free recovery reads) | `core-incident-557` scenarios A-F, `core-rooms`, `core-shutdown`, `core-soak` |
| Graceful shutdown and drain, fail-closed behaviour, DB environment write fence | `core-shutdown`, `db-env-write-fence` (adapted) |

### Changed (deliberate contract changes, asserted by the new tests)
- No collab-verification phase: edits return `200` only after they are durable. `202 pending`, `canonicalStatus`, `collabApplied`-style flags and fallback repair are gone.
- `LIVE_CLIENTS_PRESENT` / `LEGACY_EDIT_UNSAFE` blocks are gone: edits are safe with peers connected because they are base-token-checked Yjs transactions.
- The bridge executes server-side; `NO_VIEWERS` / relay-to-browser behaviour is gone.
- A blind `PUT` with no base is last-writer-wins (applied as a minimal diff under the slug lock). Marks-only `PUT` merges instead of 503-and-rollback.
- `documents.markdown`/`marks` columns are an identity/legacy-adoption record only; they are never read as content after adoption. The `document_blocks`, lease, repair-queue and quarantine tables and their endpoints are removed.
- `help.reportBug` is no longer attached to mutation errors; `document.updated` on the viewer channel carries no markdown; `mt1:` tokens are content hashes without an access epoch.
- A paused ordinary share page with no published snapshot answers 404 with the generic unavailable card (legacy answered 200 from a synchronous snapshot).
- Every removed route is listed as `LEGACY-RECOVERY` in the appendices.

### Missing / not preserved
- Authored (`system:library`) provenance marks for library-managed sections (G3-F5).
- A guard against a buggy editor client wiping a room, and any revision-history API (G2-N1): the log is internal.
- Cross-instance live rooms (single process, single SQLite file, as before).
- Migration of production data and of legacy suggestion wrappers that were truncated (G1-F5): documents are adopted lazily.
- `POST /marks/unresolve` on the bridge is a dead mapping (404 `UNKNOWN_ROUTE`), which was already true at the starting commit.

## 4. Observed directly, versus not tested

### Observed in the running app
`npm run serve` (entry `tsx server/bootstrap.ts`) was started on port 4377 against a fresh SQLite file after `npm run build`. Observed:
- `GET /health` → `{"status":"ok","collab":{"enabled":true,"state":"ready","rooms":0,"faultedRooms":0,...}}`; `GET /` → 200 HTML; the built editor bundle `/assets/editor-*.js` → 200, 3.6 MB.
- `POST /share/markdown` created a document and returned slug, owner secret, token, share URL.
- Two real `HocuspocusProvider` editors joined through `open-context` sessions. Editor A typed a paragraph; B received it. An agent `POST /api/agent/:slug/edit/v2` (`insert_after b2`) reached both editors exactly once. A second edit/v2 with the old base token returned `409 STALE_BASE`. A `comment.add` through `/api/documents/:slug/ops` reached both editors' `marks` map. A viewer token write returned `403`.
- `GET /d/:slug` returned the agent-readable server-rendered page to a UA-less client and the SPA shell (script `/assets/editor-*.js`) for a browser UA; the JSON form returned `success:true`; `events/pending` listed `document.created, agent.edit.v2, comment.added`.
- **Kill and restart** (separate run through the same entry `server/bootstrap.ts`, spawned by the test harness so it could be signalled): two editors and an agent edited; the server was `SIGKILL`ed; one editor typed while it was down; the server restarted on the same port and database; both editors reconnected on their own; one more edit synced to the peer. Final document: every one of the five typed or agent lines appeared **exactly once**, and both editors' fragments equalled the server's. The same scenario runs in `core-soak` and `core-http-ws-integration`.

### Covered by automated tests (all against real SQLite; integration over real HTTP+WS in a child process)
`npm run test:core` / `test:pr:collab`: store, model/ops, rooms, incident #557, http+ws integration, shutdown, client runtime, app smoke, library, soak, agent-edit, marks, fidelity and duplication suites. Per-file summary lines from the last run: agent-edit 18 scenarios, bridge 9, duplication 13, http+ws 14, incident 6, library sync 3, mark write failure 4, markdown fidelity 10 (21 matrix documents), marks 16, model+ops 9, rooms 8, share lifecycle 12, share surfaces 12, shutdown 4, store 7. `core-app-smoke` ("integrated app smoke passed"), `core-client-runtime` ("real client runtime against the core passed"), `core-library-oauth` ("Every sign-in + Library against the core passed") and `core-soak` ("3122 lines typed across 4 docs x 3 peers survived a SIGKILL restart exactly once") also pass.

### Not tested, and fixture/runtime limits
- **No browser.** None is installed (Playwright Chromium is absent), so the editor UI (Milkdown rendering, popovers, mobile UX) was never driven. The real `collab-client`/`share-client` modules run in Node with a window shim against the real server; the client edits are small, but their behaviour in a real browser is unobserved.
- Nothing ran against Railway, production data, the real Every OAuth service (a local mock is used), the macOS app, the MCP server or the Swift file bridge. Staging/prod `test:live-viewer:*` scripts were not run (network blocked); `test:live-viewer:fragment:*` now just run `core-agent-edit-scenarios` because a workflow calls them.
- Performance was checked only qualitatively (event-loop lag and `/health` latency under fault and starvation in `core-incident-557`); there is no load test at production scale.
- The live run used one document and two editors; multi-document load comes only from the test suites.
- A server I started on port 4377 for the live run **is still running**: the sandbox refuses to signal it (`kill` returns "operation not permitted", the same reason the old server on `$PORT` 56220 could not be stopped). It holds only a temp database under `$TMPDIR`.

## 5. Test retirement (summary; the full table is `docs/plans/test-retirement-ledger.md`)

189 legacy test files were removed with `git rm`, each named in the ledger: 71 **PORT** (behaviour re-asserted in a named new core test) and 118 **RETIRE** (the mechanism no longer exists; the ledger names the invariant or test that makes the failure mode impossible or covers its observable remainder). 15 files were **ADAPTed** in place and still run (`comment-add-edit-sequence-regression`, `editor-suggestion-api-regression`, `reject-suggestion-share-persistence-regression`, `rest-put-comment-events`, `track-changes-race`, `dashboard-feature-regression`, `health-build-sha-precedence`, `server-incident-tracing`, `server-routes-and-share`, `share-server-startup`, `snapshot-html-canonical-fallback`, `agent-bug-report-api`, `db-env-write-fence`, `share-collab-bind-reset`, `share-refresh-persist`). 12 files are **UNTOUCHED** and fail for the baseline reasons in section 2. Every other test (about 97 files: homepage, CI config, parsers, client wiring, observability, skills, Hocuspocus/Yjs contract suites, ...) was neither changed nor removed and passes.

Examples: `collab-security` (about 60 source-string assertions over deleted modules) → WS close-code, token and epoch tests in `core-http-ws-integration`/`core-share-lifecycle`; `agent-edit-single-writer-live-clients-skip-stability` (202/pending under churn) → `core-agent-edit-scenarios` "agent edits racing continuous typing lose nothing and duplicate nothing"; `collab-active-connection-registry` (lease table) → the live peer count is Hocuspocus' own connection set (`core-rooms`); `mutation-dual-write` / `mutation-backfill-and-tombstones` → `core-store` "events and idempotency records are durable and scoped" plus the HTTP replay test; `document-blocks-rebuild` → block refs are derived on read (`core-model-ops`).

Scripts changed because they ran retired tests: `test`, `test:server-routes-share`, `test:web-bridge` (retired entries pruned), `test:pr:collab` (now `test:collab:contracts` + `test:core`, replacing the two retired shutdown tests with `core-shutdown`), new `test:core`, and `test:live-viewer:fragment:{local,staging}` / `fragment-base:local` (kept because `.github/workflows` references one of them; they now run `core-agent-edit-scenarios`). Removed scripts: `test:collab:stability-followups`, `test:collab:same-paragraph-race`, `test:collab:nightly:matrix`, `test:migration-verify`, `repair:proof-marks`, `repair:yjs:audit` (their targets were deleted). `.github/workflows` was not touched.

## 6. Remaining limitations and gaps

Open findings (each has a repro in the findings files and, where noted, a `knownFinding` wrapper in a test that prints `! KNOWN FINDING` and passes; `PROOF_STRICT_FINDINGS=1` turns them into failures):
- **G2-F5** (medium, needs a product decision): `PUT markdown` / rewrite `content` strips authorship marks from every paragraph, including unchanged ones. Block edits (v2) and live typing keep authorship.
- **G2-F6** (low): a run of blank lines no longer becomes explicit blank paragraphs (only a literal `<br />` line does).
- **G2-F7** (low): a block-level HTML element (`<div>`, a comment) makes the parser strip inline HTML tags in the whole document.
- **G3-F5**: library-managed sections carry no `system:library` provenance marks, so the provenance gutter will not show them as system-written.
- **G1-F3/F4/F5** (low): emptied inline HTML elements remain; `<p>foo</p><p>bar</p>` collapses through the headless HTML fallback; legacy truncated suggestion wrappers cannot be accepted after adoption.
- **G2-N1** (design note): nothing stops an editor client from wiping a room, and there is no history API to recover it. Worth a product decision if client bugs of that class recur.

Other gaps and assumptions:
- Single-process live rooms. Running two server instances against one database is safe for the log but live broadcast is not shared (stated in the plan, unchanged from before).
- `tsc` is not clean (596 / 82 errors, down from 1127 / 256); the remaining errors are pre-existing configuration and typing issues, not addressed here.
- Twelve baseline-failing test files remain failing for environmental reasons (section 2); I did not repair them because they test pieces I did not change.
- The `npm test` chain stops at the first of them; running it end to end needs Chromium and those fixes.
- Assumptions made without asking: greenfield over incremental; breaking the `202/pending` and viewer-relay contract on purpose (listed above); keeping `documents` rows as identity records so existing slugs, tokens and owner secrets keep working.
- Nothing was committed or pushed; no network, deploy or Railway action was taken.