The synthesis pipeline — combining documents, surfacing contradictions
The multi-document synthesis pipeline takes a set of an archive's pages/documents and produces combined wiki pages: each synthesized page unites the sources and surfaces where the sources disagree (contradiction detection), respects the per-run budgets o…
The multi-document synthesis pipeline takes a set of an archive's pages/documents and produces combined wiki pages: each synthesized page unites the sources and surfaces where the sources disagree (contradiction detection), respects the per-run budgets of tokens/calls/pages, and touches the filesystem only after selective approval of what gets written.
Community feature (the pipeline runs on the same infrastructure as the chat orchestrator; per-run budgets). Sources:docs/SERVICES.md(the Synthesis domain),docs/USAGE.md,README.md:33. Provenance rows:FACT-CENSUS.mdF-SYNTH-1.
How the pipeline runs
The pipeline is orchestrated by runSynthesisPipeline (the synthesisService.ts facade; the body in synthesis/synthesisStages.ts — runPipelineStages, callSynthesisLLMStage, …):
| Step | What it does |
|---|---|
| 1. Entity extraction | over the selected corpus (synthesisStages.ts:569) |
| 2. Per-archive page | generatePageSummary per page (:282) |
| 3-4. Bottom and decision | the final blocks with the system prompt (the wiki block's injection: Pass 4 only on the SYSTEM role, synthesisStages.ts:784-788) |
| Contradictions | synthesisContradictionService.ts: the candidate pairing via Jaccard (the threshold 0.15) + the LLM judge per pair (judgePairContradiction) + the contradiction markers on the pages |
| Orphans and broken wikilinks | synthesisOrphanService.ts (detectOrphanPages, detectBrokenWikilinks) |
| Writing | synthesisPageWriter.ts: applyApprovedChanges persists the approved modifications; the source frontmatter (buildSourceFrontmatter) carries the provenance |
Budget tracking
BudgetTracker (synthesisBudgetService.ts) bounds tokens / LLM calls / pages per run (config via loadBudgetConfig): the pipeline stops when the budget exhausts — the already-written output stays valid. The budget watchdog parallels the chat orchestrator's.
Selective approval
The synthesis's proposed changes do NOT land automatically: they pass through the approval flow (applyApprovedChanges persists only what the user/approver confirmed). The deterministic preview (the same machinery as wikiWriteService) precedes every write; the proposed pages carry the source frontmatter.
Contradiction detection
Candidate pairs form by similarity (Jaccard, the threshold 0.15) and the suspicious pairs pass to the LLM judge (judgePairContradiction): the result = the contradiction markers on the involved pages (buildContradictionMarker) + the claim summary (extractClaimSummary). The process's fidelity samples weekly (runFidelitySample, Sundays 03:00 — synthesisFidelityService.ts).
Triggers and the job queue
- Post-OCR: the
synthesisTriggerServicequeue (onOcrJobCompleted) enqueues synthesis runs after OCR completes, with a cooldown (isCooldownActive). - The reaper: the
synthesisReaperJobscheduler (every 15 minutes) flips orphanedPROCESSINGruns toFAILED.
This guide covers the operator surface for the synthesis pipeline; the engineering deep dive lives in the twin post:
Provenance
| Section | Source |
|---|---|
| The facade / stages / budget / contradictions | docs/SERVICES.md (§ Synthesis + the synthesis* modules) — F-SYNTH-1 |
| Triggers / the reaper | docs/SERVICES.md (§ Schedulers & jobs — synthesisTriggerService, synthesisReaperJob) |
| The README line | README.md:33 (the synthesis line) — F-SYNTH-1 |
| The wiki injection / Pass 4 | docs/WIKI_SCHEMA_LAYER.md (§ Injection seams — synthesisStages.ts:784-788) |
---
Operator reference — Simmetric Chat v0.28.1 (docs refreshed 2026-10-02, census SHA `aca5b8ff`).