Document DLP pipeline — operator guide

When DLP document scanning is enabled for a workspace, every uploaded document passes through an async pg-boss scan after the ingest completes:

An optional Enterprise feature enableable per workspace (workspaces.dlpDocumentScanEnabled, default off; enabling requires the eval gate to pass). Source: docs/DLP_DOCUMENT_PIPELINE.md (Phase 192, DLP-01..06 — v0.28.1). Provenance rows: FACT-CENSUS.md F-DLP-1, F-DLP-2, F-AUDIT-1.

1. Detect — a 3-tier pipeline: admin-configurable regex patterns (dlpPatternService), checksum validation for the fixed-format Italian identifiers (Codice Fiscale incl. omocodia, P.IVA, IBAN — dlpChecksum), and LLM contextual NER for names/addresses (the existing Ollama client).
2. Mask — detected entities become document-wide numbered placeholders ([PERSON_1], [GOV_ID_2], …). Masking is idempotent (mask∘mask = mask) — the placeholder syntax can never re-match.
3. Re-embed masked — document_chunks.chunkText is rewritten with the masked text, the FTS searchVector/searchVectorMulti recomputed, and vectors re-embedded via the collector's /api/ingest/reembed endpoint. No unredacted vector remains searchable.
4. Encrypt originals — the original text + every entity value are stored AES-256-GCM encrypted (encryptionService, the ENCRYPTION_KEY contract). Masking is one-way: the encrypted originals are the only clean copy.

Admin controls

The per-workspace toggle (DLP-01)

Settings → Workspaces → the DLP document scan switch (the workspaces.dlpDocumentScanEnabled column, default off).

• The toggle is disabled with a gate-blocked helper until the eval gate (below) passes — DISABLING is never blocked.
• The state flips optimistically and reverts on a save error.

The eval gate (DLP-05)

The detection-quality eval must pass before enablement:

• POST /api/system/dlp/eval/run (admin) — runs the repeatable runner over the committed Italian-PII fixture corpus (packages/server/src/__tests__/fixtures/dlp-eval/). Persists the result to the SystemConfig key DLP_EVAL_LAST_RUN (not admin-editable).
• GET /api/system/dlp/eval/result (admin) — reads the last run. Never-run returns { passed: false, noRun: true } — never a silent pass.
• The gate metric: zero checksum-suppressed false positives on the GOV_ID/FINANCIAL classes. PERSON/ADDRESS recall is documented-only (nerMode marks stub vs live).

The legacy corpus backfill (DLP-06)

POST /api/system/dlp/backfill (admin, AlertDialog-confirmed in the UI):

• Eval-gated: refuses with 409 { error, gate: "eval-not-passed" } when the last eval result is absent or failed.
• Idempotent + resumable: the dlpScannedAt marker — a re-run enqueues only the documents lacking the marker.
• Rate-limited: queue concurrency 1 (below the live-scan queue).
• Convergence is pinned by the integration twin (real Postgres): masked chunks persisted, tsvectors recomputed, the reembed payload byte-equal to the stored masked text, the zero-remaining-PII audit query.

The four pattern classes (the European/Italian identifiers)

The checksum classes (GOV_ID/FINANCIAL) are deterministic; the contextual classes (PERSON/ADDRESS) pass through the LLM NER — their recall is the eval result's documented value.

Chat re-composition (DLP-03)

Masked placeholders reach the LLM; at stream end, exactly once, the server re-composes placeholders in the terminal message for users holding dlp:unmask on the citing document's workspace. The additive content field on the done SSE payload carries the re-composed text; widget visitors NEVER receive re-composed text (server-enforced; for the UI it is only a control-hiding matter).

Preview unmask (DLP-04)

GET /api/documents/:documentId/text serves masked text by default. ?unmask=true re-composes only through the gate ladder: a valid unmask param → the workspace toggle on → resolveWorkspaceRole + the dlp:unmask permission. Every miss arm returns masked text (200) — the endpoint is never an entitlement oracle. The UI toggle renders only for permitted users on documents with entities (DOM-absence), is per-view, and is never persisted.

The audit matrix (SC-4)

Every read path is pinned by a behavioral test (18 rows: the text view, the OCR preview, synthesis, citations, widget RAG, MCP tools, FTS, exports, the KB copy-from-doc, the admin re-embed). Widget/citations/MCP carry hard-never negative pins (masked, never re-composed).

This guide covers the operator surface for the document DLP pipeline; the engineering deep dive lives in the twin post:

• The engineering deep dive

Provenance

---

Operator reference — Simmetric Chat v0.28.1 (docs refreshed 2026-10-02, census SHA `aca5b8ff`).