MeridiansMeridians

3. Documents — daemon-first, then echoed to the cache

From the Meridians Wiki · Public · Maintained · joint

Documents (domains, constellations, extractions, …) are the only shape written daemon-first: the record is updated and versioned before the cache, so canon is never behind the projection.

Downstream (edit → record → cache): An explicit Save/upload/generation boundary uses the compact Tier-1 action path described in record-and-write-path.md: the renderer applies the same pinned reducer action optimistically, the daemon commits it, and a minimal acknowledgement lets the control finish without returning the full document. The record-stream echo then projects the committed document. The older projection-writer path below remains for coalesced/high-frequency renderer edits that have not yet moved to an explicit transaction boundary:

  1. A local edit lands in the reducer; state.activeDomain changes.
  2. usePersistence (src/lib/client/store/hooks/use-persistence.ts, keyed on activeDomain) fires. It first checks isRecordSynced("domains", id, updatedAt) — if this version was just adopted from the record (see below), it skips the daemon and writes IndexedDB only. Otherwise it calls persistDocument() (persistence.ts).
  3. persistDocument() is daemon-first: await putDocumentToDaemon(cls, id, doc) (PUT /api/local/doc/:cls/:id). The daemon runs writeDocument() — version bump + atomic write + forward-time guard (a stale write gets a 409; the record already holds newer, and the echo below converges the cache).
  4. Only after the daemon acks does it write the projection: idbPut(store, id, doc, { mirror: false }). mirror: false is the crucial half — the daemon canon is already written, so this IndexedDB write must not bounce back to the record (§4 explains the flag).

Upstream (record → cache, the echo):

  1. A committed mutation calls publishRecordEvent() (record-events.ts, a globalThis-scoped in-process bus so every Next route bundle shares the listener set).
  2. GET /api/local/stream (SSE) relays the frame { type:"record", cls, id, version, doc } to the browser.
  3. useRecordStream (src/lib/client/store/hooks/use-record-stream.ts) turns it into a real action (recordEventToActionSYNC_ADOPT_DOMAIN), dispatches it, then:
    • markRecordSynced("domains", id, doc.updatedAt) — stamps the version so the persist effect in step 2 above recognises it as already canon and doesn't re-write the daemon (echo suppression);
    • saveDomainProjection(doc) — an IndexedDB-only write (mirror: false, safe precisely because the doc was just markRecordSynced, so the daemon holds it and can't echo it into a loop).

This closes the loop with no ping-pong: an edit writes canon then projection; the echo re-projects without re-writing canon.

4. The mirror flag — one switch, two directions

idbPut / idbDelete (idb.ts) take a mirror option that decides whether an IndexedDB write also shadow-writes the record (record-mirror.ts). It encodes which store led:

SituationmirrorMeaning
Browser edit to an aux store (members, activity, meta, catalog, …)true (default)Cache led → propagate to the record. Fire-and-forget, debounced 500ms/key, flushed on page-hide (keepalive), retried on 5xx.
Browser writes/creates a reference-bearing assetexplicit awaited asset writeassetManager writes the local projection, then waits for canonical bytes (or client→master acceptance) before returning the ref.
Document write, after putDocumentToDaemon succeededfalseRecord already led (daemon-first) — don't bounce back.
Boot hydrate — cache filled from the recordfalseRecord led — the cache is catching up, don't re-push.
SSE echo / catch-up adoptfalseRecord led — projection-only write.

So documents are daemon-first (record leads, mirror:false); aux stores are write-through (cache leads, mirror:true), while asset creation has its own awaited bytes-before-reference path because its id may immediately enter a document. setMirrorSuppressed(true) turns the mirror off entirely during boot hydrate so freshly-seeded cache data isn't shipped straight back to the record it came from.

5. Catch-up & boot reconciliation (event- and version-driven)

The live SSE stream only carries events received in real time. The gaps are filled by one version-proof discipline, without a time window:

  • Content drift on the open domain while the tab was briefly disconnected → catchUpActiveDomain() (run on every SSE onopen) re-pulls the active record and re-projects it (markRecordSynced + saveDomainProjection) — the fast targeted path for the doc the user is looking at.
  • Everything elsecatchUpEntries() (also on onopen) sweeps the daemon's doc lists for all four classes (domains · constellations · onboardings · extractions) through the pure planner planStaleSweep (catch-up.ts): pull an id we don't hold at all (a headless mint, an import), AND pull a held id whose record updatedAt has provably moved past the newest version we hold — the record-synced version ledger (recordSyncedVersion) unioned with the in-memory copy's clock, so a doc the operator just edited is never re-pulled down over the edit. A held doc whose version is unprovable (encrypted cache, hydration still decrypting) is deliberately skipped — the storm guard — and the sweep re-runs once the ledger warms (hydration-signal.ts, markHydrationStampsWarm fired by use-hydration after the doc stamps land), so the skip is a deferral, never a lost catch-up. This is what closes the closed-window drift gap: a scheduler run or MCP commit to a background world made while no window was open is caught on the next boot's sweep, not on the next manual reload.

The sweep is built to be fast, failure-isolated, and resumable:

  • Bounded-parallel (parallelBatch, RECORD_HYDRATE_CONCURRENCY): a fresh instance's N-doc sweep is ~N/slots round-trips, not N serial ones.
  • Per-doc isolated: each ref is an independent pull → mark → dispatch → mirror; one doc's failure (fetchDocBody never throws — null on miss) skips only that doc.
  • Resumable by construction: an interrupted sweep (tab close, dropped socket) keeps every doc it completed — each is already record-synced + in IndexedDB, so the next reconnect's presence-planned sweep re-pulls only what is still missing. There is no cursor to persist and therefore none to lose.

Its UI is deliberately non-blocking: a sweep that lasts long enough to notice appears in the shared floating dock, a no-op stays silent, and a material completion holds briefly with deduplicated update / removal counts and the document payload bytes moved. The active-domain safety pull and the collection sweep may observe the same document, so the status merges them by class/id rather than double-counting.

Boot coalescers — three same-shaped fan-in dedups keep the unsequenced boot paths (reconcile, hydration, SSE sweep) from doing each other's work twice. Each is an in-flight promise + short TTL, never a cache; failures are never pinned:

CoalescerSharesConsumers
fetchSyncState (doc-list.ts)the ONE GET /api/local/sync-state boot snapshot (lists, aux keys, asset ids, tombstones)reconcile planning + catchUpEntries
fetchDocBody (doc-list.ts)each GET /api/local/doc/:cls/:id bodyreconcile hydrate (httpTransport.get) + catchUpActiveDomain + catchUpEntries
sharedLoad (load-coalescer.ts)the full-cache IndexedDB decrypt (~16MB on a large instance)reconcile planning + store hydration

sharedLoad's coherence rule is load-bearing: every doc-store write invalidates the snapshot at the idb.ts choke point (put/putMany/delete on domains/constellations/onboardings), and a load already in flight when a write lands is not cached on resolve — so no reader inside the TTL can ever see a pre-write copy that hides a just-hydrated doc.

The store-miss gap (closed). Reconcile and store hydration run unsequenced at boot. When hydration's IndexedDB read lands before a reconcile-hydrate write, the doc reaches the cache but not the in-memory store — and the SSE sweep then skips it (its cache key reads as "already have"), leaving it invisible until a reload. reconcileRecord() therefore returns the hydrated document objects (ReconcileResult.hydratedDocs), and RecordCatchupGate adopts into the store any of them the store still lacks (markRecordSynced + SYNC_ADOPT_*, presence-checked so steady-state boots adopt nothing, auth-gated like hydration itself). Whichever order the race resolves, the store ends complete.

Why both exist — the activeDomain-only blind spot. usePersistence and the ordinary publish effects are keyed on state.activeDomain. A headless commit to a background world adopts into the in-memory dashboard entry but, without these two, never lands in IndexedDB — a reload would then rehydrate from a cache that never got it and the change would "disappear" until that domain is next opened. saveDomainProjection for every adopted domain (not just the active one) closes it.

Boot reconciliation (reconcile.ts, reconcileRecord()) runs once on load, after IndexedDB opens and the data key loads. Its direction logic is pure and tested (catch-up.ts, planCatchup/isAheadOf, by updatedAt not version):

  • Documents: a cache document may be adopted (cache→record via putDocumentToDaemon) only when the record lacks that id; baseHash:null means create-if-absent and the tombstone guard decides a genuine re-create. When both sides exist, hydrate (record→cache via saveDomainProjection) only if the record is provably newer. A cache-ahead body is retained and reported as recovery data, not written over canon: a newer timestamp does not prove it descended from the current record. Equal clocks with different content remain deliberately untouched for the same reason.
  • Aux + asset stores: symmetric, keys-only probe first — if the record is empty and the cache has it, push; if the cache is empty and the record has it, pull (idbPutMany(…,{mirror:false})); if both have it, leave it (write-through keeps it current). Never a bidirectional field merge — one direction per store.

When boot reconciliation moves document, store, or file payload, the return-to-app gate reports the landed category breakdown, payload bytes saved to / restored from the record, and elapsed time. Payload bytes exclude HTTP framing and failed attempts: they describe data successfully moved, not an estimate of metered network traffic. A no-op boot and a successful stale-key / tombstone cleanup stay silent; cleanup still converges in the background, while any degraded pass remains visible.

The member registry has a stricter pre-auth read because stale identity is not an acceptable steady state. Before the access gate decides who may sign in, it calls hydrateAuxFromRecord("instanceMembers") even when the browser projection is non-empty. That targeted read replaces local values from canon and removes locally cached members canon no longer contains. This closes the multi-browser boot case where another browser changed People after this browser last projected the registry; an empty record still falls back to the local first-run projection so it can seed canon normally.

After a clean reconciliation the cache is fully in agreement with the record; steady-state writes (§3–§4) keep it there. A degraded pass reports the failed direction and remains resumable. Existing-id cache-ahead divergence intentionally stays degraded until an explicit, provenance-aware recovery choice; automatic boot work never fabricates a base and overwrites canon.

Reconcile keeps failed outcomes split by direction. A failed hydrate leaves this browser behind canon, so the record-wins Force sync is a safe repair. A failed adopt leaves this browser ahead; Force sync would overwrite the unlanded local value, so the completion surface instead asks the user to keep the window open and save again. Mixed or otherwise unclassified failures fail safe as browser-ahead and never recommend a destructive pull.

Asset seed fallback also retains the record route's structured content rejection. A declared MIME that disagrees with recognisable bytes is corrected to the sniffed safe type and retried. Unsupported media is removed from the browser projection only when the complete readable ref-bearing cache set (domains, Constellations, onboarding documents, and the People registry) proves it is unreferenced; a referenced asset, or any unreadable encrypted record in that set, is retained and reported as an adopt failure. The removal is projection-only and never emits a canonical delete.


Part B — replication (master ↔ client over a tunnel)