Research pipeline — sources → Firecrawl → the Research pass [Flow]

The Research task's evidence engine, now held in one canonical driver — runResearchPipeline (src/lib/engine/research/run-pipeline.ts) — called by both shells (the headless research-run op and the interactive ResearchRunDashboard) through a ResearchPort. The shells adapt runtime concerns (commit vs dispatch, fs asset store vs browser cache, ctx.emit vs React state); the pipeline itself lands once. This is the internals of the maintenance-run converter.

flowchart TB
    Op["**research-run op** (headless)<br/>load domain · resolve targets/budgets ·<br/>mint runId · START/FINISH bracket"]
    Dash["**ResearchRunDashboard** (interactive)<br/>browser port · live progress"]
    Query["**ad-hoc Query** (manual)<br/>a search front end over the same Firecrawl service"]

    Op -->|"server ResearchPort"| Driver
    Dash -->|"browser ResearchPort"| Driver

    subgraph Driver["runResearchPipeline — the ONE Research pass (browser-safe · deterministic via the port)"]
        direction TB
        Catalog["**research catalog**<br/>sites (trusted sources) · topics"]
        Catalog --> Mode{"cycle mode"}
        Mode -->|"discover / track"| Search["**search** — 1 query per topic<br/>buildSearchQuery + recency · includeDomains"]
        Mode -->|"monitor"| Scrape["**monitor** — 1 scrape per site<br/>change-tracking + markdown"]
        Search --> FC
        Scrape --> FC
        FC["**Firecrawl v2** — request built pure, then fetched"]
        FC --> Articles["**Articles upserted** — every finding lands in the<br/>durable article repo (op AND dashboard alike)"]
        Articles --> Route["route findings → Streams<br/>(open questions · new streams)"]
        Route --> Apply["score + apply → **StreamPriors**<br/>(evidential weight; articleId citation)"]
        Apply --> Brief["research **brief**"] --> Refine["editorial **refine**"]
        Refine --> Monitor["**projection monitor** — window checks re-price live reads"]
        Monitor --> Tutor["**merge proposals** staged"]
    end

    Query -->|"SERP → open/scrape"| FC

    Tutor --> Receipt(["LOG_RESEARCH_RUN — findings · duration · credits ·<br/>partial-status preserved · projection-only runs logged"])

Invariants

• One pipeline, two shells, one port. The op and the dashboard call the same driver; each supplies a thin ResearchPort. Before the extraction the UI copy had drifted (no Articles, no brief, no refine/monitor/tutor) — the driver is what makes "the dashboard runs the real pass" true by construction. The driver is browser-safe (engine + core only) and deterministic: time and ids come from the port, and runId is minted shell-side so the op keeps its minute-bucketed collapse id.
• Two modes, two request shapes. Search (discover/track) issues one Google-syntax query per topic with a recency window; sites scope it via includeDomains. Monitor issues one scrape per site with change-tracking. Credits ≈ requests × breadth — the request shape is the spend model.
• Every finding becomes an Article. The durable article repo is the capture; a prior cites articleId (persistent) over {runId, findingIndex} (transient, tied to the fetch). The run receipt is the audit, and a partial or projection-only run still logs honestly.
• Findings never author the spine directly. They become StreamPriors — evidence scored by evidential weight into outcome reads (see LANGUAGE §Stream: outcomes are the universal frame) — which the reducer folds into Streams; the monitor's window checks re-price Positions. The crawl is evidence; belief movement is still a reduced, attributed mutation.
• Query is a front end, not a fourth mode. The ad-hoc Query surface drives the same Firecrawl service for hand-directed search (SERP → open/scrape), with starters seeded from the domain's own cycle topics; whatever the operator keeps enters through the same ingest path.

Where it lives: src/lib/engine/research/ — run-pipeline.ts (the driver + ResearchPort) · firecrawl.ts (pure request/normalize) · firecrawl-service.ts (HTTP) · search-query.ts · apply-stream.ts · ingest.ts · merge-to-arc.ts; the op shell: src/lib/server/ops/handlers/research-run.ts; the interactive shells: src/features/research/. The scheduled run that drives it: scheduler → operation-lifecycle.