Extraction pipeline — text → domain [Flow]

# Extraction pipeline — text → domain `[Flow]`

How a raw corpus becomes a measured domain: chunk the source, extract scene structure concurrently, reconcile and assemble it, then enrich the assembled Domain through resumable specialist phases. The imperative shell (a singleton runner) orchestrates; extraction calls pass through the one model boundary.

```mermaid
flowchart TB
    Source(["raw corpus text"])
    Source --> Chunk

    Chunk["**chunkTextsFromSource**<br/>split into scene-sized chunks"]
    Chunk -->|"schedule (concurrency = EXTRACTION_CONCURRENCY)"| Extract

    subgraph Extract["per-chunk structure — parallel, streamed, checkpointed"]
        direction TB
        LLM["extraction-llm → callGenerate('extract')<br/>+ strict parseJson"]
        Out["scenes · entities · System nodes"]
        LLM --> Out
    end

    Extract -->|"adaptive result checkpoints"| Organize["organization fan-out<br/>arcs · reconciliation · embeddings<br/>one shared concurrency budget"]

    Organize --> Re{"specialist rereads<br/>(optional, layer on top)"}
    Re -->|"fate-reextraction"| Fate["recover Fate/threads<br/>from resolved scenes"]
    Re -->|"system-reextraction"| Sys["recover System graph<br/>from propositions"]
    Fate & Sys --> Assemble["assemble Domain<br/>world summaries · metadata"]
    Assemble --> DomainPhases["Domain enrichment<br/>topic tree · spatial · game theory · script"]
    DomainPhases -->|"stable phase snapshots<br/>batched immutable merge + durable checkpoint"| Domain(["measured Domain + source assets"])
```

**Invariants**

- **Chunking sets the scene boundaries.** `chunkTextsFromSource` decides where scenes split before any LLM runs — the extraction is chunk-local, so the chunker is load-bearing for coherence.
- **Concurrent, streamed, and incrementally persisted.** Up to `EXTRACTION_CONCURRENCY` chunks extract at once; per-chunk output streams to listeners; result-bearing phases persist frequently for ordinary jobs and on a bounded adaptive cadence for very large jobs rather than writing after every result. Large result arrays live in deterministic `texts/` JSON shards, keeping the extraction document below its PUT boundary while changed-shard comparison avoids rewriting stable parts. Resume checkpoints report absolute completed-source progress, and specialist passes that mutate only the assembled Domain do not rewrite the unchanged extraction job.
- **Parallel work reads stable phase input.** Independent workers do not observe sibling completions. Their private outputs join at a checkpoint boundary, where the affected registry is copied once and the resulting Domain snapshot becomes the resumable artifact. On retry, completed units are discovered from durable result shards, per-unit progress, or enriched scene fields and only missing work is scheduled.
- **Re-extraction is additive, not a redo.** Fate and System re-extraction *layer on top* of the initial pass, recovering thread structure and the system graph from what the first pass produced. They never re-parse prose to overwrite canonical state — they read the measured result.
- **Same AI discipline as everywhere.** Extraction calls go through `callGenerate('extract')` with strict `parseJson` plus explicit validation; malformed output fails the run. The source text is stored once as a content-addressed asset (`contentRef`) and referenced, not re-embedded.

**Where it lives:** `src/lib/engine/extraction/` — `extraction-runner.ts` (singleton orchestrator) · `text-extraction.ts` · `extraction-llm.ts` · `fate-reextraction.ts` · `system-reextraction.ts` · `text-reconciliation-llm.ts`. Chunking: `src/lib/core/io/scene-splitting.ts`. The domain it populates: [domain-spine](../concepts/domain-spine.md). Extraction jobs UI: [extraction-jobs](../ui/extraction-jobs.md).

Raw Markdown source