The document DLP pipeline: masks that hold

The document enters once; the placeholder stays

An AI workspace's knowledge base ingests real documents - contracts, HR letters, invoices. Some contain the data nobody should casually read, and an LLM should never quote them. Simmetric Chat's answer for enabled workspaces: a document-scanning pipeline that turns the sensitive entities into document-wide numbered placeholders and keeps the only clean copy encrypted. The design point is the last clause - the placeholder stays: masking is one-way by construction, so "the document" and "the searchable corpus" become different objects by design.

This article walks the pipeline as the operator meets it: the async scan, the three detection tiers, what a placeholder guarantees, what happens at re-embedding, how the originals are stored, and the two unmask gates (chat re-composition, preview unmask) that make the clean copy reachable only on purpose.

An optional Enterprise feature, enableable per workspace and default off (the workspace toggle + the eval gate). Facts per FACT-CENSUS.md F-DLP-1.

The scan: async, queued, after ingest

When DLP document scan is enabled for a workspace, every uploaded document passes through an async pg-boss scan after the ingest completes - the upload path stays responsive, and the scan behaves like every other background job of the platform: queued, retried, observable. A legacy corpus has its own backfill (eval-gated and explicitly confirmed in the UI), rate-limited below the live queue and resumable: a marker column (dlpScannedAt) bounds each re-run to the documents that never got scanned.

Detect: three tiers

The detection is a 3-tier pipeline, each tier for what it is:

1. Admin-configurable regex patterns - the pattern service the admin curates (enable/disable any pattern, test previews).
2. Checksum validation for the fixed-format Italian/European identifiers - Codice Fiscale (including the omocodia validation), P.IVA, IBAN: these classes are deterministic; a string that fails its checksum is not an identifier, so ordinary numbers never trip them.
3. LLM contextual detection for names/addresses - the contextual classes pass through the model; their recall is a documented, measured value rather than a promise.

Two named classes are deterministic (the checksums) and two are contextual (the LLM classes) - and the eval gate measures exactly this split: the quality bar for enablement = zero suppressed false positives on the deterministic classes, with the contextual classes' recall documented per run.

The one number this post will never print: a count of "how many PII types exist". The census contract is explicit - untraceable numbers do not ship; the classes are named, the code says what it detects.

Mask: numbered placeholders that cannot re-match

Detected entities become document-wide numbered placeholders ([PERSON_1], [GOV_ID_2], ...). Three properties make that meaningful:

• Document-wide coherence - the same entity is the same placeholder across the whole document; the LLM keeps reading a consistent document.
• Idempotence - masking is its own fixpoint: the placeholder syntax can never re-match, so a re-scan never produces [PERSON_1_1]-class artifacts.
• The rewrite is total - the chunks are rewritten with the masked text, the FTS vectors recomputed, and the embeddings re-generated for the masked text. No unredacted vector remains searchable.

Encrypt: the only clean copy is AES-256-GCM

The original text and every entity value land encrypted (AES-256-GCM under the platform's key contract). That is what makes the pipeline honest: masking is not "hiding in a view layer" - the unmasked bytes simply do not exist unencrypted in the store. The encrypted originals are the only clean copy.

The two unmask gates

The clean copy is reachable through exactly two gates, both explicit:

1. Chat re-composition - at stream end, exactly once, the server re-composes placeholders in the terminal message only for users holding the dlp:unmask permission in the citing document's workspace. The re-composed text rides the SSE done payload; widget visitors never receive re-composed text (enforced server-side, not by hiding a control).
2. Preview unmask - the document-text endpoint serves masked text by default; an unmask param works only through the gate ladder (the param present, the workspace toggle on, the role resolving the permission). Every miss arm still returns masked text with a 200 - the endpoint is never an entitlement oracle.

The audit matrix

Every read path is pinned by a behavioral test (the text view, the OCR preview, synthesis, citations, the widget, MCP tools, full-text search, exports, the admin re-embed). The three hardest pins are the "never" arms: widget visitors, citation surfaces, and MCP tools receive masked text and never a re-composition. The audit-log class (the platform's immutable event log, F-AUDIT-1) records the operations that touch the surface.

Reading on

• Request a demo

The guide carries the complete operator walkthrough - the toggle, the eval runner, the backfill, every endpoint - at /en/docs/document-dlp-pipeline:

• Document DLP pipeline - the guide
• The plain-language version
• Facts in this post: FACT-CENSUS.md rows F-DLP-1 (the pipeline), F-DLP-2 (the untraced-count decision), F-AUDIT-1 (the event log)

Provenance: Simmetric Chat docs/DLP_DOCUMENT_PIPELINE.md (v0.28.1, Phase 192 DLP-01..06), packages/server/src/services/ (dlpPatternService/dlpChecksum/quota services), encryptionService - at the census's pinned docs state (SHA aca5b8ff).