3. Documents — daemon-first, then echoed to the cache
From the Meridians Wiki · Public · Maintained · joint
Documents (domains, constellations, extractions, …) are the only shape written daemon-first: the record is updated and versioned before the cache, so canon is never behind the projection.
Downstream (edit → record → cache): An explicit Save/upload/generation boundary uses the compact
Tier-1 action path described in record-and-write-path.md: the renderer applies the same pinned reducer
action optimistically, the daemon commits it, and a minimal acknowledgement lets the control finish without
returning the full document. The record-stream echo then projects the committed document. The older
projection-writer path below remains for coalesced/high-frequency renderer edits that have not yet moved to
an explicit transaction boundary:
- A local edit lands in the reducer;
state.activeDomainchanges. usePersistence(src/lib/client/store/hooks/use-persistence.ts, keyed onactiveDomain) fires. It first checksisRecordSynced("domains", id, updatedAt)— if this version was just adopted from the record (see below), it skips the daemon and writes IndexedDB only. Otherwise it callspersistDocument()(persistence.ts).persistDocument()is daemon-first:await putDocumentToDaemon(cls, id, doc)(PUT /api/local/doc/:cls/:id). The daemon runswriteDocument()— version bump + atomic write + forward-time guard (a stale write gets a 409; the record already holds newer, and the echo below converges the cache).- Only after the daemon acks does it write the projection:
idbPut(store, id, doc, { mirror: false }).mirror: falseis the crucial half — the daemon canon is already written, so this IndexedDB write must not bounce back to the record (§4 explains the flag).
Upstream (record → cache, the echo):
- A committed mutation calls
publishRecordEvent()(record-events.ts, aglobalThis-scoped in-process bus so every Next route bundle shares the listener set). GET /api/local/stream(SSE) relays the frame{ type:"record", cls, id, version, doc }to the browser.useRecordStream(src/lib/client/store/hooks/use-record-stream.ts) turns it into a real action (recordEventToAction→SYNC_ADOPT_DOMAIN), dispatches it, then:markRecordSynced("domains", id, doc.updatedAt)— stamps the version so the persist effect in step 2 above recognises it as already canon and doesn't re-write the daemon (echo suppression);saveDomainProjection(doc)— an IndexedDB-only write (mirror: false, safe precisely because the doc was justmarkRecordSynced, so the daemon holds it and can't echo it into a loop).
This closes the loop with no ping-pong: an edit writes canon then projection; the echo re-projects without re-writing canon.
4. The mirror flag — one switch, two directions
idbPut / idbDelete (idb.ts) take a mirror option that decides whether an IndexedDB write also
shadow-writes the record (record-mirror.ts). It encodes which store led:
| Situation | mirror | Meaning |
|---|---|---|
| Browser edit to an aux store (members, activity, meta, catalog, …) | true (default) | Cache led → propagate to the record. Fire-and-forget, debounced 500ms/key, flushed on page-hide (keepalive), retried on 5xx. |
| Browser writes/creates a reference-bearing asset | explicit awaited asset write | assetManager writes the local projection, then waits for canonical bytes (or client→master acceptance) before returning the ref. |
Document write, after putDocumentToDaemon succeeded | false | Record already led (daemon-first) — don't bounce back. |
| Boot hydrate — cache filled from the record | false | Record led — the cache is catching up, don't re-push. |
| SSE echo / catch-up adopt | false | Record led — projection-only write. |
So documents are daemon-first (record leads, mirror:false); aux stores are write-through
(cache leads, mirror:true), while asset creation has its own awaited bytes-before-reference path because
its id may immediately enter a document. setMirrorSuppressed(true)
turns the mirror off entirely during boot hydrate so freshly-seeded cache data isn't shipped straight
back to the record it came from.
5. Catch-up & boot reconciliation (event- and version-driven)
The live SSE stream only carries events received in real time. The gaps are filled by one version-proof discipline, without a time window:
- Content drift on the open domain while the tab was briefly disconnected →
catchUpActiveDomain()(run on every SSEonopen) re-pulls the active record and re-projects it (markRecordSynced+saveDomainProjection) — the fast targeted path for the doc the user is looking at. - Everything else →
catchUpEntries()(also ononopen) sweeps the daemon's doc lists for all four classes (domains · constellations · onboardings · extractions) through the pure plannerplanStaleSweep(catch-up.ts): pull an id we don't hold at all (a headless mint, an import), AND pull a held id whose recordupdatedAthas provably moved past the newest version we hold — the record-synced version ledger (recordSyncedVersion) unioned with the in-memory copy's clock, so a doc the operator just edited is never re-pulled down over the edit. A held doc whose version is unprovable (encrypted cache, hydration still decrypting) is deliberately skipped — the storm guard — and the sweep re-runs once the ledger warms (hydration-signal.ts,markHydrationStampsWarmfired byuse-hydrationafter the doc stamps land), so the skip is a deferral, never a lost catch-up. This is what closes the closed-window drift gap: a scheduler run or MCP commit to a background world made while no window was open is caught on the next boot's sweep, not on the next manual reload.
The sweep is built to be fast, failure-isolated, and resumable:
- Bounded-parallel (
parallelBatch,RECORD_HYDRATE_CONCURRENCY): a fresh instance's N-doc sweep is ~N/slots round-trips, not N serial ones. - Per-doc isolated: each ref is an independent pull → mark → dispatch → mirror; one doc's failure
(
fetchDocBodynever throws — null on miss) skips only that doc. - Resumable by construction: an interrupted sweep (tab close, dropped socket) keeps every doc it completed — each is already record-synced + in IndexedDB, so the next reconnect's presence-planned sweep re-pulls only what is still missing. There is no cursor to persist and therefore none to lose.
Its UI is deliberately non-blocking: a sweep that lasts long enough to notice appears in the shared floating dock, a no-op stays silent, and a material completion holds briefly with deduplicated update / removal counts and the document payload bytes moved. The active-domain safety pull and the collection sweep may observe the same document, so the status merges them by class/id rather than double-counting.
Boot coalescers — three same-shaped fan-in dedups keep the unsequenced boot paths (reconcile, hydration, SSE sweep) from doing each other's work twice. Each is an in-flight promise + short TTL, never a cache; failures are never pinned:
| Coalescer | Shares | Consumers |
|---|---|---|
fetchSyncState (doc-list.ts) | the ONE GET /api/local/sync-state boot snapshot (lists, aux keys, asset ids, tombstones) | reconcile planning + catchUpEntries |
fetchDocBody (doc-list.ts) | each GET /api/local/doc/:cls/:id body | reconcile hydrate (httpTransport.get) + catchUpActiveDomain + catchUpEntries |
sharedLoad (load-coalescer.ts) | the full-cache IndexedDB decrypt (~16MB on a large instance) | reconcile planning + store hydration |
sharedLoad's coherence rule is load-bearing: every doc-store write invalidates the snapshot at the
idb.ts choke point (put/putMany/delete on domains/constellations/onboardings), and a load already
in flight when a write lands is not cached on resolve — so no reader inside the TTL can ever see a
pre-write copy that hides a just-hydrated doc.
The store-miss gap (closed). Reconcile and store hydration run unsequenced at boot. When hydration's IndexedDB read lands before a reconcile-hydrate write, the doc reaches the cache but not the in-memory store — and the SSE sweep then skips it (its cache key reads as "already have"), leaving it invisible until a reload.
reconcileRecord()therefore returns the hydrated document objects (ReconcileResult.hydratedDocs), andRecordCatchupGateadopts into the store any of them the store still lacks (markRecordSynced+SYNC_ADOPT_*, presence-checked so steady-state boots adopt nothing, auth-gated like hydration itself). Whichever order the race resolves, the store ends complete.
Why both exist — the
activeDomain-only blind spot.usePersistenceand the ordinary publish effects are keyed onstate.activeDomain. A headless commit to a background world adopts into the in-memory dashboard entry but, without these two, never lands in IndexedDB — a reload would then rehydrate from a cache that never got it and the change would "disappear" until that domain is next opened.saveDomainProjectionfor every adopted domain (not just the active one) closes it.
Boot reconciliation (reconcile.ts, reconcileRecord()) runs once on load, after IndexedDB opens
and the data key loads. Its direction logic is pure and tested (catch-up.ts, planCatchup/isAheadOf,
by updatedAt not version):
- Documents: a cache document may be adopted (cache→record via
putDocumentToDaemon) only when the record lacks that id;baseHash:nullmeans create-if-absent and the tombstone guard decides a genuine re-create. When both sides exist, hydrate (record→cache viasaveDomainProjection) only if the record is provably newer. A cache-ahead body is retained and reported as recovery data, not written over canon: a newer timestamp does not prove it descended from the current record. Equal clocks with different content remain deliberately untouched for the same reason. - Aux + asset stores: symmetric, keys-only probe first — if the record is empty and the cache has
it, push; if the cache is empty and the record has it, pull (
idbPutMany(…,{mirror:false})); if both have it, leave it (write-through keeps it current). Never a bidirectional field merge — one direction per store.
When boot reconciliation moves document, store, or file payload, the return-to-app gate reports the landed category breakdown, payload bytes saved to / restored from the record, and elapsed time. Payload bytes exclude HTTP framing and failed attempts: they describe data successfully moved, not an estimate of metered network traffic. A no-op boot and a successful stale-key / tombstone cleanup stay silent; cleanup still converges in the background, while any degraded pass remains visible.
The member registry has a stricter pre-auth read because stale identity is not an acceptable steady
state. Before the access gate decides who may sign in, it calls hydrateAuxFromRecord("instanceMembers")
even when the browser projection is non-empty. That targeted read replaces local values from canon and
removes locally cached members canon no longer contains. This closes the multi-browser boot case where
another browser changed People after this browser last projected the registry; an empty record still
falls back to the local first-run projection so it can seed canon normally.
After a clean reconciliation the cache is fully in agreement with the record; steady-state writes (§3–§4) keep it there. A degraded pass reports the failed direction and remains resumable. Existing-id cache-ahead divergence intentionally stays degraded until an explicit, provenance-aware recovery choice; automatic boot work never fabricates a base and overwrites canon.
Reconcile keeps failed outcomes split by direction. A failed hydrate leaves this browser behind canon, so the record-wins Force sync is a safe repair. A failed adopt leaves this browser ahead; Force sync would overwrite the unlanded local value, so the completion surface instead asks the user to keep the window open and save again. Mixed or otherwise unclassified failures fail safe as browser-ahead and never recommend a destructive pull.
Asset seed fallback also retains the record route's structured content rejection. A declared MIME that disagrees with recognisable bytes is corrected to the sniffed safe type and retried. Unsupported media is removed from the browser projection only when the complete readable ref-bearing cache set (domains, Constellations, onboarding documents, and the People registry) proves it is unreferenced; a referenced asset, or any unreadable encrypted record in that set, is retained and reported as an adopt failure. The removal is projection-only and never emits a canonical delete.