Research pipeline — sources → Firecrawl → the Research pass [Flow]

# Research pipeline — sources → Firecrawl → the Research pass `[Flow]`

The Research task's evidence engine, now held in **one canonical driver** — `runResearchPipeline`
(`src/lib/engine/research/run-pipeline.ts`) — called by **both shells** (the headless `research-run` op
and the interactive ResearchRunDashboard) through a `ResearchPort`. The shells adapt runtime concerns
(commit vs dispatch, fs asset store vs browser cache, `ctx.emit` vs React state); the pipeline itself
lands once. This is the internals of the [maintenance-run](maintenance-run.md) converter.

```mermaid
flowchart TB
    Op["**research-run op** (headless)<br/>load domain · resolve targets/budgets ·<br/>mint runId · START/FINISH bracket"]
    Dash["**ResearchRunDashboard** (interactive)<br/>browser port · live progress"]
    Query["**ad-hoc Query** (manual)<br/>a search front end over the same Firecrawl service"]

    Op -->|"server ResearchPort"| Driver
    Dash -->|"browser ResearchPort"| Driver

    subgraph Driver["runResearchPipeline — the ONE Research pass (browser-safe · deterministic via the port)"]
        direction TB
        Catalog["**research catalog**<br/>sites (trusted sources) · topics"]
        Catalog --> Mode{"cycle mode"}
        Mode -->|"discover / track"| Search["**search** — 1 query per topic<br/>buildSearchQuery + recency · includeDomains"]
        Mode -->|"monitor"| Scrape["**monitor** — 1 scrape per site<br/>change-tracking + markdown"]
        Search --> FC
        Scrape --> FC
        FC["**Firecrawl v2** — request built pure, then fetched"]
        FC --> Articles["**Articles upserted** — every finding lands in the<br/>durable article repo (op AND dashboard alike)"]
        Articles --> Route["route findings → Streams<br/>(open questions · new streams)"]
        Route --> Apply["score + apply → **StreamPriors**<br/>(evidential weight; articleId citation)"]
        Apply --> Brief["research **brief**"] --> Refine["editorial **refine**"]
        Refine --> Monitor["**projection monitor** — window checks re-price live reads"]
        Monitor --> Tutor["**merge proposals** staged"]
    end

    Query -->|"SERP → open/scrape"| FC

    Tutor --> Receipt(["LOG_RESEARCH_RUN — findings · duration · credits ·<br/>partial-status preserved · projection-only runs logged"])
```

**Invariants**

- **One pipeline, two shells, one port.** The op and the dashboard call the same driver; each supplies a
  thin `ResearchPort`. Before the extraction the UI copy had drifted (no Articles, no brief, no
  refine/monitor/tutor) — the driver is what makes "the dashboard runs the real pass" true by
  construction. The driver is **browser-safe** (engine + core only) and **deterministic**: time and ids
  come from the port, and `runId` is minted shell-side so the op keeps its minute-bucketed collapse id.
- **Two modes, two request shapes.** *Search* (discover/track) issues one Google-syntax query per topic
  with a recency window; sites scope it via `includeDomains`. *Monitor* issues one scrape per site with
  change-tracking. Credits ≈ requests × breadth — the request shape *is* the spend model.
- **Every finding becomes an Article.** The durable article repo is the capture; a prior cites
  `articleId` (persistent) over `{runId, findingIndex}` (transient, tied to the fetch). The run receipt
  is the audit, and a partial or projection-only run still logs honestly.
- **Findings never author the spine directly.** They become StreamPriors — evidence scored by
  **evidential weight** into outcome reads (see LANGUAGE §Stream: outcomes are the universal frame) —
  which the reducer folds into Streams; the monitor's window checks re-price Positions. The crawl is
  evidence; belief movement is still a reduced, attributed mutation.
- **Query is a front end, not a fourth mode.** The ad-hoc Query surface drives the same Firecrawl
  service for hand-directed search (SERP → open/scrape), with starters seeded from the domain's own
  cycle topics; whatever the operator keeps enters through the same ingest path.

**Where it lives:** `src/lib/engine/research/` — `run-pipeline.ts` (**the driver + ResearchPort**) ·
`firecrawl.ts` (pure request/normalize) · `firecrawl-service.ts` (HTTP) · `search-query.ts` ·
`apply-stream.ts` · `ingest.ts` · `merge-to-arc.ts`; the op shell: `src/lib/server/ops/handlers/research-run.ts`;
the interactive shells: `src/features/research/`. The scheduled run that drives it:
[scheduler](scheduler.md) → [operation-lifecycle](operation-lifecycle.md).

Raw Markdown source