# Answer: Greenfield Collaboration Core Rewrite

## Top finding

The March 17 outage was structural, not a single bug: `server/collab.ts`
(12,476 lines) escalated repeated stale `onStoreDocument` conflicts on
individual documents into a **global admission guard** that fenced
unrelated documents, while unbounded stale-reload/quarantine work ran in
the same Node process as `/` and `/health` and starved it. The delivered
rewrite removes that mechanism class: all admission, breaker, and
quarantine decisions in the new core are keyed per slug, recovery work is
budgeted and deduplicated per slug, persistence is coalesced behind a CAS
on `(epoch, version)` with bounded merge-retry, markdown is a derived
projection that is never written back into Yjs state, and `/health` reads
an O(1) core snapshot.

## Scope delivered

Plan: `docs/plans/2026-03-17-greenfield-collab-core-rewrite.md` (written
first, before implementation).

New core (`server/core/`):

- `admission.ts` — `PerSlugBreaker` (per-slug quarantine only; explicitly
  no global breaker), `WorkSemaphore` (bounded load/persist concurrency
  that sheds single requests without fencing any document), and the O(1)
  `getCoreHealthSnapshot()` used by `/health`.
- `document-store.ts` — `CoreDocumentStore` over SQLite tables
  `core_documents`, `core_updates`, `core_quarantine`. Yjs binary state is
  the only canonical content; snapshots persist through a CAS on
  `(epoch, version)`; identical re-persists are idempotent no-ops; the
  epoch fence is checked before the duplicate check so stale-epoch
  writers can never succeed; updates are append-logged; markdown is a
  write-only cache column.
- `room-manager.ts` — one room per slug: single-flight load, one
  in-flight persist, debounced/coalesced flush, merge-on-CAS-conflict
  with at most 2 retries, per-slug quarantine on exhaustion, idle
  eviction, side-effect-free projection-cache reads, and fail-closed
  shutdown (throws rather than discarding dirty rooms).
- `projection.ts` — single content `Y.Text`; markdown projection is a
  pure read; full replacement happens only as one room-serialized
  transaction, removing the dual-writer `Y.Text` duplication class from
  `DIAGNOSIS-v2-duplication.md`.

Integration into the existing app (default build and `npm run serve`):

- `server/collab.ts`: stale-`onStore` pathology accounting
  (`noteGlobalCollabAdmissionPathology`) delegates to the core per-slug
  breaker; tripping quarantines only the causal slug via the existing
  auto-quarantine. The legacy global admission guard body is unreachable
  from production paths (its status and `__unsafe…ForTests` helpers remain
  so existing contract tests still compile/pass).
- `server/collab.ts`: `getLiveCollabBlockStatus` consults the core
  per-slug quarantine first.
- `server/collab.ts`: `scheduleStaleOnStoreReload` drops reloads for
  core-quarantined slugs and allows at most one pending reload per slug,
  so stale churn cannot pile up timers and starve the event loop.
- `server/index.ts`: `/health` includes `core: getCoreHealthSnapshot()`
  (no DB/Yjs/auth work on the health path).

New tests:

- `src/tests/core-room-manager.test.ts` — two-client convergence through
  one room with concurrent edits merged exactly once, idempotent persist
  (no version inflation), stale-epoch CAS fence, serialized replacement,
  side-effect-free degraded read, persistence across manager restart.
- `src/tests/core-admission-isolation.test.ts` — 50 slugs tripping their
  own breakers never fence a fresh slug; quarantined-slug failures are
  dropped; room admission refuses only the quarantined slug while a
  sibling edits and persists; semaphore sheds without fencing.
- `src/tests/core-stale-onstore-isolation-regression.test.ts` — issue 557
  regression against the real collab runtime: repeated stale
  `onStoreDocument` conflicts across 4 slugs never trip the global
  admission guard and a fresh slug stays admissible.
- `scripts/core-e2e.ts` + `scripts/core-e2e-runner.ts` — live end-to-end
  against `npm run serve`: health/home, document creation, two real
  Hocuspocus clients with edit propagation, health during collab, server
  restart against the same database, and persistence verification.

Retired tests (they asserted the removed global-fencing behavior itself;
coverage replaced by the isolation tests above):

- `src/tests/collab-global-admission-guard.test.ts` (deleted)
- `src/tests/collab-global-admission-guard-thresholds.test.ts` (deleted)
- `src/tests/collab-global-admission-guard-real-path.test.ts` (deleted)
- `src/tests/collab-global-admission-guard-cross-runtime.test.ts` (deleted)

Scripts changed in `package.json` (`.github/workflows` untouched):

- `test:collab:stability-followups`: replaced the deleted
  `collab-global-admission-guard.test.ts` entry with the three new core
  tests, because the script would otherwise run a retired test.
- `test:pr:collab`: added the three new core tests.

## Commands run and results

- `npm run build` — PASS (vite build + `build:mcp`).
- `npx tsc -p tsconfig.server.json` — 259 pre-existing errors in legacy
  files on the untouched frozen tree (verified: e.g.
  `clearAllMaintenanceTimers` is already missing at
  `server/index.ts:2742` in `HEAD`). Zero errors in `server/core/` and
  zero new errors from the edited regions.
- `npx tsc` (src) — pre-existing errors in untouched legacy/test files;
  none in files created or edited here.
- `npm run test:ci` — PASS (baseline and after changes).
- `npm run test:pr:collab` — PASS (baseline and after changes, now
  including the three core tests).
- `npm test` — stops at a pre-existing failure in
  `src/tests/proof-mark-rehydration.test.ts` ("Expected reject to
  preserve escaped markdown text for quotes with underscores"),
  reproduced byte-identically on the untouched base via `git stash`
  (base exit 1). Not caused by this change; the task only guaranteed
  `build`, `test:ci`, and `test:pr:collab` pass today, and they do.
- `npm run test:server-routes-share` — stops at a pre-existing failure in
  `src/tests/live-viewer-passive-reconnect-wiring.test.ts`, also
  reproduced on the untouched base (base exit 1).
- `npm run test:collab:stability-followups` — the updated script runs the
  new core tests (PASS) and later stops at a pre-existing failure in
  `src/tests/share-json-repair-readiness-contract-regression.test.ts`,
  also reproduced on the untouched base (base exit 1).
- Targeted existing tests around every edited path — all PASS:
  `collab-onstore-stale-overwrite`, `collab-onstore-missing-meta-stale-overwrite`,
  `collab-onstore-authoritative-baseline-regression`,
  `collab-onchange-stale-overwrite`, `collab-external-put-propagates`,
  `collab-stale-epoch-write-guard`, `collab-pathological-repeat-guard`,
  `collab-projection-repair-worker`, `collab-projection-repair-teardown`,
  `collab-session-epoch-evicts-stale-doc`,
  `collab-invalidate-evicts-hocuspocus`,
  `collab-empty-fragment-persist-regression`,
  `collab-canonical-apply-persisted-replay-regression`,
  `share-document-updated-broadcast`, `collab-onstore-auto-quarantine`,
  `server-routes-and-share`, `share-client-collab-unavailable-contract`,
  `milkdown-collab-wiring`, `collab-security`, `mutation-coordinator`,
  `agent-edit-v2`, `collab-same-paragraph-race-regression`,
  `collab-pathological-fast-quarantine`,
  `collab-repair-loop-auto-quarantine`.
- New core tests — PASS: `core-room-manager`,
  `core-admission-isolation`, `core-stale-onstore-isolation-regression`
  (also wired into `test:pr:collab` and
  `test:collab:stability-followups`).
- Live e2e: `npx tsx scripts/core-e2e-runner.ts` — PASS. Observed output:
  `health: status=ok core={"roomsLoaded":0,"quarantinedSlugs":0,"inFlightWork":0,"queuedWork":0,"totalTrips":0} latencyMs=12`,
  `home: HTTP 200`, `created: slug=lb25trdv`,
  `collab: client A edit propagated to client B`,
  `health during collab: HTTP 200 latencyMs=2`, then after restart
  `verify: marker persisted across restart and is visible to a reconnected collab client`.

## Capability inventory

Kept (unchanged surfaces, covered by the passing suites above):

- Document create/read/update REST APIs, share links and roles,
  open-context, collab-session, agent state/snapshot/edit v2, marks and
  comments, editor build (Milkdown), MCP build, health/readiness states,
  passive-viewer non-publish behavior, stale-session epoch fencing,
  per-slug auto-quarantine, projection repair worker, shutdown drain.

Changed:

- Stale-`onStore` pathology can no longer escalate across documents: the
  global admission guard never trips from production paths; the causal
  slug alone is quarantined through the core breaker.
- Stale-store reloads are deduplicated per slug and stop entirely for
  quarantined slugs (bounded work instead of a retry loop).
- `/health` additionally exposes the core snapshot
  (`roomsLoaded`, `quarantinedSlugs`, `inFlightWork`, `queuedWork`,
  `totalTrips`).
- The new `server/core/` store/room layer is the authority for admission
  and quarantine decisions and for rooms it manages; the legacy
  row/Yjs-blob persistence still backs existing document content (no
  production migration, per the greenfield instructions).

Missing / not replaced:

- The legacy `server/collab.ts` guard stack (projection pathology
  detection, mutation bases, repair loops, presence) still exists and
  still serves existing document content paths; it was not deleted,
  because hundreds of existing tests import its internals directly and
  the required gates must keep passing. The incident-causing mechanisms
  (global escalation, unbounded reload) are removed from production
  paths, but a full deletion of the 12k-line module is future work.
- The core `RoomManager` is not yet the Hocuspocus `onLoad/onStore`
  backend for pre-existing documents; it is fully implemented and
  tested, and the live server paths it governs today are admission,
  quarantine, reload bounding, and health.
- Marks/comments storage and the agent-edit pipeline were not rewritten;
  they keep their existing behavior and tests.

Regression coverage mapping:

- Issue 557 hypothesis A (multi-doc stale-onStore thrash → global
  outage): `core-stale-onstore-isolation-regression` +
  `core-admission-isolation`. These replace the four retired
  `collab-global-admission-guard*` tests, which asserted the global
  fencing behavior that caused the outage.
- Hypothesis C (one pathological doc): existing
  `collab-pathological-repeat-guard` and
  `collab-pathological-fast-quarantine` still pass; core quarantine is
  per slug by construction.
- Duplication class (`DIAGNOSIS-v2-duplication.md`): `core-room-manager`
  proves concurrent edits merge exactly once, serialized replacement
  does not merge-duplicate, and duplicate persists do not inflate
  version.
- Persistence/restart: `core-room-manager` (manager restart) and the
  live `core-e2e-runner` (server restart).
- Stale-session fencing: `core-room-manager` stale-epoch CAS test plus
  existing `collab-stale-epoch-write-guard`.

## Directly observed vs untested

Directly observed in the running app (`npm run serve` via
`scripts/core-e2e-runner.ts` on `127.0.0.1:$PORT`):

- `/health` returns `ok` with the core snapshot in 12 ms, and in 2 ms
  while two collab clients are connected.
- `/` serves HTTP 200.
- `POST /api/documents` creates a document; two real Hocuspocus provider
  clients sync it; an edit on client A appears on client B.
- After SIGTERM and a fresh server start against the same database file,
  the edited marker is present via REST read and via a reconnected
  collab client.

Not directly observed:

- Browser/editor UI behavior: the Playwright package is installed but no
  browser binaries exist in this offline fixture
  (`~/Library/Caches/ms-playwright` is absent), so the Milkdown editor
  was exercised only through its build and the existing wiring tests,
  and collaboration was verified with protocol-level Hocuspocus clients
  as the task allows when no browser is available.
- Health latency under a synthetic multi-slug stale-onStore storm
  against the live server (the storm itself is covered at the runtime
  level by `core-stale-onstore-isolation-regression`, and health
  independence is by construction — the handler performs no DB/Yjs/auth
  work — but an end-to-end load-soak was not run).
- Multi-instance behavior: the core is single-process by design here;
  cross-instance authority (the legacy lease machinery) was out of
  scope.
- `npm test` and `test:server-routes-share` in full: both stop at
  pre-existing failures reproduced on the untouched base (named above),
  so chain segments after those points were verified by targeted runs
  of the collab-adjacent tests instead of one full chain pass.

## Limitations and remaining gaps

1. Legacy `server/collab.ts` remains large and still contains the now
   unreachable global-guard implementation and other guard machinery.
   The plan's end state deletes it once the core room manager becomes
   the Hocuspocus backend for all documents; that cutover needs the
   existing-documents data migration the task explicitly excluded.
2. The core breaker state is in-memory (with durable quarantine rows in
   `core_quarantine` written by the room manager on persist exhaustion);
   breaker counters reset on restart, which fails open per slug — safe
   for availability, but a chronically pathological slug will re-trip
   after each restart rather than staying quarantined from boot in the
   integrated `collab.ts` path (the legacy durable auto-quarantine it
   triggers does persist).
3. Two authority stores coexist during the transition (legacy document
   rows/Yjs blobs for existing content, `core_documents` for the new
   room layer). Admission/quarantine decisions are unified through the
   core, but content authority is unified only for core-managed rooms.
4. Pre-existing failures on the frozen base (`proof-mark-rehydration`,
   `live-viewer-passive-reconnect-wiring`,
   `share-json-repair-readiness-contract-regression`, and 259 server
   `tsc` errors) are untouched and still present; they are not
   regressions from this change, verified by `git stash` base runs.
5. No production migration, rollout, deployment, or external lookup was
   performed, per the benchmark's local-only rules; the AGENTS.md
   promotion/review gates that require web documentation checks were
   not run because the Benchmark Fixture Override prohibits them here.
