Feature Guide

A guide to the platform features with internals, data flows, and code references. For each feature it documents how it works, how to use it, and how to extend it.

Hybrid RAG (Vector + PostgreSQL tsvector, RRF)

The hybrid search is split across two processes: the collector owns only the vector component (LanceDB/Qdrant); the server owns the FTS (PostgreSQL tsvector) and the RRF that fuses both.

Flow (packages/server/src/services/hybridSearchService.ts:104):

1. The agent invokes the rag_search skill (agent/builtinSkills.ts:27) with query, workspaceId, archiveId.
2. hybridSearch - Guard D-01: compares EMBEDDING_MODEL with the embeddingModel of the workspace Documents; on mismatch → the vector leg is skipped (FTS-only, no error).
3. Promise.all([vectorSearchViaCollector(), ftsSearch()]) - vector via the collector's POST /api/ingest/query; FTS via prisma.$queryRaw on document_chunks.searchVector (GIN index, ts_rank).
4. RRF fusion (k=60): for each chunk score += 1/(60+rank+1); the sources Set distinguishes vector/fts/both. Deterministic tiebreaker: score DESC → documentId ASC → chunkIndex ASC.
5. rag_search formats the chunks as [Source: name (match: semantic+keyword|vector|fts, score: ...)] and returns SourceCitation[].
6. Archive fallback D-07: if 0 results and archiveId is present, it re-runs on archive:${archiveId} and re-tags source: "archive".
7. Citations propagate to the SSE citations event (routes/chat.ts:508).

Metadata filtering (v0.22): rag_search accepts documentTypes filters (pdf/md/txt/csv/docx/xlsx) and dateFrom/dateTo (ISO; a date-only upper bound includes the whole day). The filters thread through both legs of the hybrid search: the vector leg reaches the collector as true pre-filters on stamped ingest metadata (pgvector JSONB predicates + Qdrant payload must-clauses; LanceDB and Chroma ignore the new keys with a logged warn), and the FTS leg gains parameterized SQL predicates on the documents table. A server-side backstop against the authoritative documents table after retrieval guarantees filtered correctness on every provider. Documents ingested from now on are always filterable; legacy vectors (pre-stamping) are excluded while a filter is active and regain filterability via admin re-embed - no schema change, no re-embedding of existing documents, unchanged behavior without filters.

Grounded citations (v0.22.0): after deduplication, a grounding filter caps citations per document at the top-2 by relevance score and drops chunk citations whose text shares no meaningful lexical overlap with the final answer (conservative gate - wiki/archive pages, web/memory results, citations without chunk text, and very short answers are always kept). The "No supporting evidence" disclaimer is judged on the filtered set (both the streaming and non-streaming paths).

Chunking (collector/services/chunker.ts:16): RecursiveCharacterTextSplitter, chunkSize=1000, overlap=200, separators ["\n\n","\n",". "," ",""]. Metadata: documentId, paragraph, charStart.

Embedding (collector/services/embeddings.ts): LocalEmbeddingProvider (Xenova all-MiniLM-L6-v2, 384-dim, air-gap) by default; OpenAIEmbeddingProvider (text-embedding-3-small, 1536-dim). Singleton cached per providerType:modelName.

Vector store (collector/services/vectorStore.ts): LanceDBProvider (default, path ${STORAGE_PATH}/vectors/lancedb, per-workspace table ws_<name>_<shortId>); QdrantProvider (remote, VECTOR_DB_PROVIDER=qdrant; the default stays lancedb).

FTS pass-through: the collector returns chunks[].chunkText to the server, which writes the DocumentChunk rows with searchVector = to_tsvector('simple', text). The collector never opens Prisma.

Extending: a new embedding provider → implement EmbeddingProvider + the getEmbeddingProvider factory + dimensions in KNOWN_MODEL_DIMS. RRF tuning → RRF_K in hybridSearchService.ts:15. Embedding model change → reindex via Settings → Maintenance (POST /api/system/reindex-documents).

Agent Orchestrator (ReAct)

packages/server/src/agent/orchestrator.ts - entry points runAgent (non-stream, :78) and runAgentStreaming (:416). Single-call ReAct pattern: one `streamLLM` per iteration is the decision call - streamResult.toolCall determines the tool-vs-answer branch.

Loop:

1. Setup: load WorkspaceAgentConfig, parse enabledSkills, resolve templates (resolveSystemPrompt), prepend user.customInstructions.
2. Resolve provider: resolveProviderConfig(providerId, model) → fallback buildFallbackConfig.
3. Constraint templates: localLLMOnly (blocks non-ollama providers), hybridSearchForced, citationRequired.
4. When ragContext (widget) is present → inject into the system prompt and remove rag_search from activeSkills.
5. Plan mode (agentConfig.planMode): synchronous planning phase → SSE plan event → injected into the execute prompt.
6. ReAct while(true):
   • Wallclock + token budget watchdog.
   • streamLLM with buffered replay D-02: tool-call iteration → discard the buffer; final answer → replay the tokens to the real onToken (progressive DLP flush).
   • !toolCall → try resolveImplicitToolCall (XML-ish tag recovery) → final answer, break.
   • Skill not found → circuit breaker BOT-03: trips at 3 consecutive / 5 total → abortReason: "unknown_tool_breaker".
   • Loop detection, per-skill timeout, truncateToolOutput, context cap.
7. finally: release the concurrency slot + persist WorkspaceTokenUsage on every exit path.

The v0.28.1 custom skills (/slug) are prompt-template-only (server-side arbitration across the personal/global/workspace scopes) - see 15 - Custom Skills.

Builtin skills (agent/builtinSkills.ts): rag_search, workspace_memory (KV on SystemConfig with the key ws_memory:<ws>:<key>), document_temp_process (ad-hoc upload to the collector), wiki_query (FTS over archive_pages + vector fallback + BFS wikilinks, max 10 pages depth 3), wiki_write (preview + SSE wiki_edit, RBAC archive:write).

Budget tracker (services/agentBudgetService.ts:122): watchdog for wallclock/tokens/context/loop-detection/per-skill-timeout/per-user-concurrency (CHAT_MAX_CONCURRENT_PER_USER). AbortReason: none|wallclock|token_budget|context_overflow|loop_detected|aborted|done|unknown_tool_breaker. truncateContextToByteBudget keeps sysMsg + pinned user msg + last N.

Model selection + fallback: priority per-chat → workspace default → global default → ENV (providerService.resolveProviderConfig). deriveCapabilities tags local-only/fastest/smartest/reasoning. 30s polling refreshModels → Provider.isAvailable. Frontend 3-tier fallback in useChat.ts with an Undo toast. Server fallback buildFallbackConfig (orchestrator.ts:879). The SSE done event includes modelUsed, providerUsed.

Extending: a new skill → registerSkill({...}) in builtinSkills.ts. Per-chat model → PATCH /workspaces/:ws/chats/:chatId/model. Plan mode → enable WorkspaceAgentConfig.planMode. Budget tuning → env AGENT_* (see 03 - Configuration).

Chat streaming (SSE)

POST /api/workspaces/:id/chat/stream → SSE. Events: token, status, citations ({ sources }), done ({ chatId, messageId, modelUsed, providerUsed, mcpSources }), error, plan, wiki_edit. Frontend: useChat.ts (@microsoft/fetch-event-source), abort via AbortController. Slash /model, Cmd+K palette, comparison Cmd+Shift+M. Effective-model persistence via the loadChat PATCH (stale default skipped, see frontend/CLAUDE.md).

Documents, ingestion, OCR

Ingestion (collector/routes/ingest.ts:219): multer (100 MB, MIME allowlist) → parseFile → chunkText → embed → vectorStore.addDocuments → notifyServerStatus (PUT /api/documents/:id/status with X-Collector-Secret). The response includes chunks[].chunkText for server-side FTS.

Parser (collector/services/parser.ts): PDF (pdf-parse, <100 chars → OCR skip D-04, vision is server-side), DOCX (officeparser → mammoth fallback), PPTX (officeparser), XLSX (node-xlsx), TXT/MD/CSV (UTF-8), YouTube (youtube-transcript-plus, 422 when no transcript). ocrMode (auto/vision/skip) in IngestUploadBodySchema.

Server-side vision-model OCR (the thing's name: server-side vision-model OCR, F-OCR-1) (services/ragOcrService.ts:59): pdfjs-dist loads the PDF (cap 50 pages) → renders PNG → Ollama vision model → strips grounding tags → concatenates. The air-gapped arm: tesseract.js 7.0.0 in-process (Phase 217, the air-gap arm - OCR-BENCH) - a factual statement, never the «no third-party OCR libraries» claim (F-OCR-2). Job service ocrJobService.ts with an approve/reject workflow, qualityScore, credibilityScore. Routes routes/ocr.ts. Frontend SettingsOcr (model catalog, job list, preview, custom instructions).

DLP filter (services/dlpFilter.ts): scanContent(text) → {hasMatch, matches, redactedText}. Since v0.22 the patterns are 10 DB-backed built-ins (the dlp_patterns table, up to 50 custom patterns with a live test preview): besides the original 6 (email, credit_card, ssn, api_key sk-..., aws_key AKIA..., private_key PEM) 4 EU/IT patterns were added - it_vat_iva (label-anchored Italian VAT number), it_codice_fiscale (16 chars, uppercase-only), iban (country prefix + length ≥ 15, ReDoS-safe), eu_phone (IT/EU phone, seeded DISABLED for admin review). Redaction [REDACTED]. Gate DLP_ENABLED (default `"true"` in CONFIG_DEFAULTS, permissions.ts:173); the DLP_BYPASS_ROLES setting (JSON array of role names, default []) exempts selected roles from every scan/redaction with a dlp.bypassed audit event. An admin "Show/Hide DLP texts" toggle reveals/re-hides the redacted texts (admin-only, session-scoped). Every dlp.* audit event tags the source surface (source: "chat" | "widget") derived from the X-Widget-Id header, with filters in the DLP Match History panel. Fallback to the built-in set when the DB is unreachable (5-minute TTL cache); the token-by-token streaming flush keeps the built-in patterns. Integrated in routes/chat.ts (input + output); SSE done → dlp_warning: true; the widget shows PIIWarningPrompt. See 16 - Document DLP Pipeline.

Archives, Wiki, Synthesis

Archive: workspace-scoped multi-page knowledge base, markdown pages with slug paths and [[Title]] wikilinks. 14 archive services (archivePageService, archiveGraphService, archiveIndexService, archiveBacklinkService, ...). Routes archive*.ts (20+ files): CRUD, pages, config, search (hybrid RRF), graph, export, index, import (copy-from-doc), schema templates.

Wiki embedding (services/wikiEmbeddingService.ts:8): indexWikiPage → chunk 800/100 → collector POST /api/ingest/wiki-pages (wiki_pages table, workspaceId = archive:<id>). Skills wiki_query (FTS + BFS wikilinks) and wiki_write (preview/dry-run + wiki_edit SSE). See 14 - Wiki Knowledge Engine.

Synthesis (services/synthesisService.ts): 5-pass pipeline: entity_extraction → summary_generation → candidate_search (BM25 tsvector) → llm_decision (CREATE/UPDATE/SKIP/FLAG_CONTRADICTION) → write_overview. PHI gate: when ArchiveConfig.localLLMOnly=true and the provider is not ollama → aborts pre-egress. Admin workflow: trigger → preview → approve/reject (routes/synthesis.ts). Contradiction detection is F-SYNTH-1 (the system «brings the documents together and flags where they disagree»). See 18 - Synthesis Pipeline.

Quotas & costs

Token quotas per-user/group with an hourly reset sweep (Phase 207); an unset quota is unlimited (fail-open, F-QUOTA-1). Cost quotas run through the hold-and-settle engine (QuotaBalance/QuotaHold) settling €-in-cents at the usage seam + per-model cost accounting with per-currency endpoints (Phase 203/209). See 17 - Quotas & Costs.

Backup

Bree scheduler (services/backupSchedulerService.ts): @mintplex-labs/bree, closeWorkerAfterMs:0/timeout:0 (long-running). bootstrapJobs at startup (cleanStaleLocks → loads the enabled BackupJobs). DB-level mutex via BackupLog.status="running"; cleanStaleLocks → running >2h = failed. Cron helper buildCronExpression(frequency, time, ...).

Pipeline (services/backupService.ts:282): dumpDatabase (pg_dump spawn no-shell) → collectFiles (storage dirs, path-traversal guard) → createArchive (archiver zlib level 9) → computeChecksum (streaming SHA-256) → uploadWithRetry (3 retries backoff) → sendFailureNotification (SMTP). Safety backup pre-restore (runSafetyBackup).

Encryption (services/encryptionService.ts): AES-256-GCM. Key: ENCRYPTION_KEY (base64 32 bytes) or scryptSync(JWT_SECRET, salt). Rotation-safe decrypt chain: [current, ...LEGACY_PREVIOUS_ENCRYPTION_KEYS, scryptLegacy?].

Destinations (services/backup/providers/): LocalProvider, S3Provider, S3CompatibleProvider, DropboxProvider, GoogleDriveProvider, FtpProvider, SftpProvider, EmailProvider (interface IBackupDestinationProvider). License backup_enabled + max_backup_destinations (Community 1, Enterprise ∞).

Restore (backupService.ts:617): restoreDatabaseWithPsql (psql --single-transaction --set ON_ERROR_STOP=on, all-or-nothing atomicity). Routes restore.ts + POST /api/backups/trigger (manual).

Retention (backupRetentionService.ts): per-job retentionDays (default 30); cleanup of successful BackupLog rows beyond the cutoff + hard delete. Individual errors do not block.

Push (VAPID Web Push)

routes/push.ts: initVapid reads the VAPID_* env vars (auto-generated in dev). GET /vapid-key (public), POST/DELETE /subscribe (license push_notifications), POST /test (admin). sendPushNotification(title, body) export for event triggers; removes subscriptions on 410 Gone. Frontend: service worker API.

Webhook

services/webhookService.ts: dispatchWebhookEvent(event, payload) fire-and-forget in parallel. deliverWebhook: body {event, timestamp, data}, header X-Webhook-Signature: sha256=HMAC-SHA256(secret, body). 3 retries (1s/2s/4s); 10 consecutive failures → auto-disable. Events: document.processed, chat.completed, user.created, workspace.updated, mcp.installed/uninstalled, ... Triggered by eventLogService.logEvent (async post-DB). Routes webhooks.ts (license webhooks).

Chat connectors (Enterprise)

Inbound channels connecting Telegram/Discord/Slack/WhatsApp to the same knowledge base - archive binding, welcome/fallback messages, rate and session limits; admin management with the EDIT arm (PUT /api/connectors/:id, F-CONN-2) and platform webhooks on a signed public surface (no RBAC). See 13 - Connectors.

v0.28.1 Provenance

Operational reference - Simmetric Chat v0.28.1 (docs updated 2026-10-02, census SHA `aca5b8ff`).

Cross-link

• Widget embed: 06 - Widget Integration
• MCP: 07 - MCP Marketplace
• Endpoints: 04 - API Reference
• Env/budget knobs: 03 - Configuration