Extraction pipeline — text → domain [Flow]

How a raw corpus becomes a measured domain: chunk the source, extract scene structure concurrently, reconcile and assemble it, then enrich the assembled Domain through resumable specialist phases. The imperative shell (a singleton runner) orchestrates; extraction calls pass through the one model boundary.

flowchart TB
    Source(["raw corpus text"])
    Source --> Chunk

    Chunk["**chunkTextsFromSource**<br/>split into scene-sized chunks"]
    Chunk -->|"schedule (concurrency = EXTRACTION_CONCURRENCY)"| Extract

    subgraph Extract["per-chunk structure — parallel, streamed, checkpointed"]
        direction TB
        LLM["extraction-llm → callGenerate('extract')<br/>+ strict parseJson"]
        Out["scenes · entities · System nodes"]
        LLM --> Out
    end

    Extract -->|"adaptive result checkpoints"| Organize["organization fan-out<br/>arcs · reconciliation · embeddings<br/>one shared concurrency budget"]

    Organize --> Re{"specialist rereads<br/>(optional, layer on top)"}
    Re -->|"fate-reextraction"| Fate["recover Fate/threads<br/>from resolved scenes"]
    Re -->|"system-reextraction"| Sys["recover System graph<br/>from propositions"]
    Fate & Sys --> Assemble["assemble Domain<br/>world summaries · metadata"]
    Assemble --> DomainPhases["Domain enrichment<br/>topic tree · spatial · game theory · script"]
    DomainPhases -->|"stable phase snapshots<br/>batched immutable merge + durable checkpoint"| Domain(["measured Domain + source assets"])

Invariants

• Chunking sets the scene boundaries. chunkTextsFromSource decides where scenes split before any LLM runs — the extraction is chunk-local, so the chunker is load-bearing for coherence.
• Concurrent, streamed, and incrementally persisted. Up to EXTRACTION_CONCURRENCY chunks extract at once; per-chunk output streams to listeners; result-bearing phases persist frequently for ordinary jobs and on a bounded adaptive cadence for very large jobs rather than writing after every result. Large result arrays live in deterministic texts/ JSON shards, keeping the extraction document below its PUT boundary while changed-shard comparison avoids rewriting stable parts. Resume checkpoints report absolute completed-source progress, and specialist passes that mutate only the assembled Domain do not rewrite the unchanged extraction job.
• Parallel work reads stable phase input. Independent workers do not observe sibling completions. Their private outputs join at a checkpoint boundary, where the affected registry is copied once and the resulting Domain snapshot becomes the resumable artifact. On retry, completed units are discovered from durable result shards, per-unit progress, or enriched scene fields and only missing work is scheduled.
• Re-extraction is additive, not a redo. Fate and System re-extraction layer on top of the initial pass, recovering thread structure and the system graph from what the first pass produced. They never re-parse prose to overwrite canonical state — they read the measured result.
• Same AI discipline as everywhere. Extraction calls go through callGenerate('extract') with strict parseJson plus explicit validation; malformed output fails the run. The source text is stored once as a content-addressed asset (contentRef) and referenced, not re-embedded.

Where it lives: src/lib/engine/extraction/ — extraction-runner.ts (singleton orchestrator) · text-extraction.ts · extraction-llm.ts · fate-reextraction.ts · system-reextraction.ts · text-reconciliation-llm.ts. Chunking: src/lib/core/io/scene-splitting.ts. The domain it populates: domain-spine. Extraction jobs UI: extraction-jobs.