3. Documents — daemon-first, then echoed to the cache

# 3. Documents — daemon-first, then echoed to the cache

Documents (domains, constellations, extractions, …) are the **only** shape written **daemon-first**: the
record is updated and versioned *before* the cache, so canon is never behind the projection.

**Downstream (edit → record → cache):** An explicit Save/upload/generation boundary uses the compact
Tier-1 action path described in `record-and-write-path.md`: the renderer applies the same pinned reducer
action optimistically, the daemon commits it, and a minimal acknowledgement lets the control finish without
returning the full document. The record-stream echo then projects the committed document. The older
projection-writer path below remains for coalesced/high-frequency renderer edits that have not yet moved to
an explicit transaction boundary:

1. A local edit lands in the reducer; `state.activeDomain` changes.
2. `usePersistence` (`src/lib/client/store/hooks/use-persistence.ts`, keyed on `activeDomain`) fires. It
   first checks `isRecordSynced("domains", id, updatedAt)` — if this version was *just adopted* from the
   record (see below), it skips the daemon and writes IndexedDB only. Otherwise it calls
   `persistDocument()` (`persistence.ts`).
3. `persistDocument()` is **daemon-first**: `await putDocumentToDaemon(cls, id, doc)`
   (`PUT /api/local/doc/:cls/:id`). The daemon runs `writeDocument()` — version bump + atomic write +
   forward-time guard (a stale write gets a **409**; the record already holds newer, and the echo below
   converges the cache).
4. Only after the daemon acks does it write the projection: `idbPut(store, id, doc, { mirror: false })`.
   `mirror: false` is the crucial half — the daemon canon is already written, so this IndexedDB write
   must **not** bounce back to the record (§4 explains the flag).

**Upstream (record → cache, the echo):**

1. A committed mutation calls `publishRecordEvent()` (`record-events.ts`, a `globalThis`-scoped in-process
   bus so every Next route bundle shares the listener set).
2. `GET /api/local/stream` (SSE) relays the frame `{ type:"record", cls, id, version, doc }` to the browser.
3. `useRecordStream` (`src/lib/client/store/hooks/use-record-stream.ts`) turns it into a real action
   (`recordEventToAction` → `SYNC_ADOPT_DOMAIN`), dispatches it, then:
   - `markRecordSynced("domains", id, doc.updatedAt)` — stamps the version so the persist effect in step
     2 above recognises it as *already canon* and doesn't re-write the daemon (**echo suppression**);
   - `saveDomainProjection(doc)` — an IndexedDB-only write (`mirror: false`, safe precisely because the
     doc was just `markRecordSynced`, so the daemon holds it and can't echo it into a loop).

This closes the loop with no ping-pong: an edit writes canon then projection; the echo re-projects
without re-writing canon.

# 4. The `mirror` flag — one switch, two directions

`idbPut` / `idbDelete` (`idb.ts`) take a `mirror` option that decides whether an IndexedDB write also
shadow-writes the record (`record-mirror.ts`). It encodes *which store led*:

| Situation | `mirror` | Meaning |
|---|---|---|
| Browser edit to an **aux** store (members, activity, meta, catalog, …) | `true` (default) | Cache led → propagate to the record. Fire-and-forget, debounced 500ms/key, flushed on page-hide (`keepalive`), retried on 5xx. |
| Browser writes/creates a reference-bearing **asset** | explicit awaited asset write | `assetManager` writes the local projection, then waits for canonical bytes (or client→master acceptance) before returning the ref. |
| **Document** write, after `putDocumentToDaemon` succeeded | `false` | Record already led (daemon-first) — don't bounce back. |
| **Boot hydrate** — cache filled *from* the record | `false` | Record led — the cache is catching up, don't re-push. |
| **SSE echo / catch-up** adopt | `false` | Record led — projection-only write. |

So documents are **daemon-first** (record leads, `mirror:false`); aux stores are **write-through**
(cache leads, `mirror:true`), while asset creation has its own awaited bytes-before-reference path because
its id may immediately enter a document. `setMirrorSuppressed(true)`
turns the mirror off entirely during boot hydrate so freshly-seeded cache data isn't shipped straight
back to the record it came from.

# 5. Catch-up & boot reconciliation (event- and version-driven)

The live SSE stream only carries events received **in real time**. The gaps are filled by one
version-proof discipline, without a time window:

- **Content drift on the open domain** while the tab was briefly disconnected → `catchUpActiveDomain()`
  (run on every SSE `onopen`) re-pulls the active record and re-projects it (`markRecordSynced` +
  `saveDomainProjection`) — the fast targeted path for the doc the user is looking at.
- **Everything else** → `catchUpEntries()` (also on `onopen`) sweeps the daemon's doc **lists** for all
  four classes (domains · constellations · onboardings · extractions) through the pure planner
  `planStaleSweep` (`catch-up.ts`): pull an id we don't hold at all (a headless mint, an import), AND
  pull a held id whose record `updatedAt` has provably moved past the newest version we hold — the
  **record-synced version ledger** (`recordSyncedVersion`) unioned with the in-memory copy's clock, so
  a doc the operator just edited is never re-pulled down over the edit. A held doc whose version is
  *unprovable* (encrypted cache, hydration still decrypting) is **deliberately skipped** — the storm
  guard — and the sweep **re-runs once the ledger warms** (`hydration-signal.ts`,
  `markHydrationStampsWarm` fired by `use-hydration` after the doc stamps land), so the skip is a
  deferral, never a lost catch-up. This is what closes the *closed-window drift* gap: a scheduler run
  or MCP commit to a background world made while no window was open is caught on the next boot's sweep,
  not on the next manual reload.

The sweep is built to be **fast, failure-isolated, and resumable**:

- **Bounded-parallel** (`parallelBatch`, `RECORD_HYDRATE_CONCURRENCY`): a fresh instance's N-doc sweep
  is ~N/slots round-trips, not N serial ones.
- **Per-doc isolated**: each ref is an independent pull → mark → dispatch → mirror; one doc's failure
  (`fetchDocBody` never throws — null on miss) skips only that doc.
- **Resumable by construction**: an interrupted sweep (tab close, dropped socket) keeps every doc it
  completed — each is already record-synced + in IndexedDB, so the next reconnect's presence-planned
  sweep re-pulls only what is still missing. There is no cursor to persist and therefore none to lose.

Its UI is deliberately non-blocking: a sweep that lasts long enough to notice appears in the shared
floating dock, a no-op stays silent, and a material completion holds briefly with deduplicated update /
removal counts and the document payload bytes moved. The active-domain safety pull and the collection
sweep may observe the same document, so the status merges them by class/id rather than double-counting.

**Boot coalescers** — three same-shaped fan-in dedups keep the unsequenced boot paths (reconcile,
hydration, SSE sweep) from doing each other's work twice. Each is an in-flight promise + short TTL,
never a cache; failures are never pinned:

| Coalescer | Shares | Consumers |
|---|---|---|
| `fetchSyncState` (`doc-list.ts`) | the ONE `GET /api/local/sync-state` boot snapshot (lists, aux keys, asset ids, tombstones) | reconcile planning + `catchUpEntries` |
| `fetchDocBody` (`doc-list.ts`) | each `GET /api/local/doc/:cls/:id` body | reconcile hydrate (`httpTransport.get`) + `catchUpActiveDomain` + `catchUpEntries` |
| `sharedLoad` (`load-coalescer.ts`) | the full-cache IndexedDB decrypt (~16MB on a large instance) | reconcile planning + store hydration |

`sharedLoad`'s coherence rule is load-bearing: **every doc-store write invalidates the snapshot at the
`idb.ts` choke point** (put/putMany/delete on domains/constellations/onboardings), and a load already
in flight when a write lands is *not* cached on resolve — so no reader inside the TTL can ever see a
pre-write copy that hides a just-hydrated doc.

> **The store-miss gap (closed).** Reconcile and store hydration run **unsequenced** at boot. When
> hydration's IndexedDB read lands *before* a reconcile-hydrate write, the doc reaches the cache but
> not the in-memory store — and the SSE sweep then skips it (its cache key reads as "already have"),
> leaving it invisible until a reload. `reconcileRecord()` therefore returns the hydrated **document
> objects** (`ReconcileResult.hydratedDocs`), and `RecordCatchupGate` adopts into the store any of them
> the store still lacks (`markRecordSynced` + `SYNC_ADOPT_*`, presence-checked so steady-state boots
> adopt nothing, auth-gated like hydration itself). Whichever order the race resolves, the store ends
> complete.

> **Why both exist — the `activeDomain`-only blind spot.** `usePersistence` and the ordinary publish
> effects are keyed on `state.activeDomain`. A headless commit to a *background* world adopts into the
> in-memory dashboard entry but, without these two, never lands in IndexedDB — a reload would then
> rehydrate from a cache that never got it and the change would "disappear" until that domain is next
> opened. `saveDomainProjection` for **every** adopted domain (not just the active one) closes it.

**Boot reconciliation** (`reconcile.ts`, `reconcileRecord()`) runs once on load, after IndexedDB opens
and the data key loads. Its direction logic is pure and tested (`catch-up.ts`, `planCatchup`/`isAheadOf`,
by `updatedAt` not version):

- **Documents:** a cache document may be **adopted** (cache→record via `putDocumentToDaemon`) only when
  the record lacks that id; `baseHash:null` means create-if-absent and the tombstone guard decides a
  genuine re-create. When both sides exist, **hydrate** (record→cache via `saveDomainProjection`) only
  if the record is provably newer. A cache-ahead body is retained and reported as recovery data, not
  written over canon: a newer timestamp does not prove it descended from the current record. Equal
  clocks with different content remain deliberately untouched for the same reason.
- **Aux + asset stores:** symmetric, keys-only probe first — if the record is empty and the cache has
  it, push; if the cache is empty and the record has it, pull (`idbPutMany(…,{mirror:false})`); if both
  have it, leave it (write-through keeps it current). Never a bidirectional field merge — one direction
  per store.

When boot reconciliation moves document, store, or file payload, the return-to-app gate reports the
landed category breakdown, payload bytes saved to / restored from the record, and elapsed time. Payload
bytes exclude HTTP framing and failed attempts: they describe data successfully moved, not an estimate
of metered network traffic. A no-op boot and a successful stale-key / tombstone cleanup stay silent;
cleanup still converges in the background, while any degraded pass remains visible.

The member registry has a stricter pre-auth read because stale identity is not an acceptable steady
state. Before the access gate decides who may sign in, it calls `hydrateAuxFromRecord("instanceMembers")`
even when the browser projection is non-empty. That targeted read replaces local values from canon and
removes locally cached members canon no longer contains. This closes the multi-browser boot case where
another browser changed People after this browser last projected the registry; an empty record still
falls back to the local first-run projection so it can seed canon normally.

After a clean reconciliation the cache is fully in agreement with the record; steady-state writes
(§3–§4) keep it there. A degraded pass reports the failed direction and remains resumable. Existing-id
cache-ahead divergence intentionally stays degraded until an explicit, provenance-aware recovery choice;
automatic boot work never fabricates a base and overwrites canon.

Reconcile keeps failed outcomes split by direction. A failed **hydrate** leaves this browser behind
canon, so the record-wins Force sync is a safe repair. A failed **adopt** leaves this browser ahead;
Force sync would overwrite the unlanded local value, so the completion surface instead asks the user
to keep the window open and save again. Mixed or otherwise unclassified failures fail safe as
browser-ahead and never recommend a destructive pull.

Asset seed fallback also retains the record route's structured content rejection. A declared MIME that
disagrees with recognisable bytes is corrected to the sniffed safe type and retried. Unsupported media
is removed from the browser projection only when the complete readable ref-bearing cache set (domains,
Constellations, onboarding documents, and the People registry) proves it is unreferenced; a referenced
asset, or any unreadable encrypted record in that set, is retained and reported as an adopt failure.
The removal is projection-only and never emits a canonical delete.

---

# Part B — replication (master ↔ client over a tunnel)

Raw Markdown source