Guide 05

Feature Guide

How the core features actually work.

Guida alle feature con internals, flusso dati e riferimenti file:line. Per ogni feature: come funziona, come si usa, come si estende.

RAG ibrido (Vector + PostgreSQL tsvector, RRF)

La search ibrida è dislocata su due processi: il collector possiede solo la componente vettoriale (LanceDB/Qdrant); il server possiede la FTS (PostgreSQL tsvector) e l'RRF che fonde entrambe.

Flusso (packages/server/src/services/hybridSearchService.ts:104):

L'agente invoca la skill rag_search (agent/builtinSkills.ts:27) con query, workspaceId, archiveId.
hybridSearch — Guard D-01: confronta EMBEDDING_MODEL con gli embeddingModel dei Document del workspace; se mismatch → salta il vector leg (FTS-only, no errore).
Promise.all([vectorSearchViaCollector(), ftsSearch()]) — vector via POST /api/ingest/query del collector; FTS via prisma.$queryRaw su document_chunks.searchVector (GIN index, ts_rank).
RRF fusion (k=60): per ogni chunk score += 1/(60+rank+1); sources Set distingue vector/fts/both. Tiebreaker deterministico: score DESC → documentId ASC → chunkIndex ASC.
rag_search formatta i chunk con [Source: name (match: semantic+keyword|vector|fts, score: ...)] e ritorna SourceCitation[].
Fallback archive D-07: se 0 risultati e archiveId presente, rilancia su archive:${archiveId} e ritagga source: "archive".
Citazioni propagate all'SSE citations (routes/chat.ts:508).

Chunking (collector/services/chunker.ts:16): RecursiveCharacterTextSplitter, chunkSize=1000, overlap=200, separator ["\n\n","\n",". "," ",""]. Metadata: documentId, paragraph, charStart.

Embedding (collector/services/embeddings.ts): LocalEmbeddingProvider (Xenova all-MiniLM-L6-v2, 384-dim, air-gap) di default; OpenAIEmbeddingProvider (text-embedding-3-small, 1536-dim). Singleton cached per providerType:modelName.

Vector store (collector/services/vectorStore.ts): LanceDBProvider (default, path ${STORAGE_PATH}/vectors/lancedb, tabella per-workspace ws_<name>_<shortId>); QdrantProvider (placeholder enterprise).

FTS pass-through: il collector ritorna chunks[].chunkText al server, che scrive DocumentChunk con searchVector = to_tsvector('simple', text). Il collector non apre mai Prisma.

Estendere: nuovo embedding provider → implementa EmbeddingProvider + factory getEmbeddingProvider + dimensioni in KNOWN_MODEL_DIMS. Tuning RRF → RRF_K in hybridSearchService.ts:15. Cambio modello embedding → reindex via Settings → Maintenance (POST /api/system/reindex-documents).

Agent Orchestrator (ReAct)

packages/server/src/agent/orchestrator.ts — entry runAgent (non-stream, :78) e runAgentStreaming (:416). Pattern ReAct single-call: una streamLLM per iterazione è la decision call — streamResult.toolCall determina branch tool-vs-risposta.

Loop:

Setup: carica WorkspaceAgentConfig, parse enabledSkills, risolve template (resolveSystemPrompt), prepend user.customInstructions.
Resolve provider: resolveProviderConfig(providerId, model) → fallback buildFallbackConfig.
Constraint template: localLLMOnly (blocca provider non-ollama), hybridSearchForced, citationRequired.
Se ragContext (widget) → inietta nel system prompt, rimuove rag_search da activeSkills.
Plan mode (agentConfig.planMode): fase planning sincrona → SSE plan event → iniezione nel prompt di execute.
ReAct while(true):
- Watchdog wallclock + token budget.
- streamLLM con buffered replay D-02: tool-call iteration → scarta buffer; final-answer → replay token al onToken reale (progressive DLP flush).
- !toolCall → prova resolveImplicitToolCall (recovery XML-ish tag) → final answer, break.
- Skill non trovata → circuit breaker BOT-03: trip a 3 consecutivi / 5 totali → abortReason: "unknown_tool_breaker".
- Loop detection, per-skill timeout, truncateToolOutput, context cap.
finally: rilascia slot concorrenza + persist WorkspaceTokenUsage su tutti gli exit path.

Skill builtin (agent/builtinSkills.ts): rag_search, workspace_memory (KV su SystemConfig con chiave ws_memory:<ws>:<key>), document_temp_process (upload ad-hoc al collector), wiki_query (FTS archive_pages + fallback vettoriale + BFS wikilinks, max 10 pagine depth 3), wiki_write (preview + SSE wiki_edit, RBAC archive:write).

Budget tracker (services/agentBudgetService.ts:122): watchdog wallclock/token/context/loop-detection/per-skill-timeout/per-user-concurrency (CHAT_MAX_CONCURRENT_PER_USER). AbortReason: none|wallclock|token_budget|context_overflow|loop_detected|aborted|done|unknown_tool_breaker. truncateContextToByteBudget keep sysMsg + pinned user msg + last N.

Selezione modello + fallback: priorità per-chat → workspace default → global default → ENV (providerService.resolveProviderConfig). deriveCapabilities tag local-only/fastest/smartest/reasoning. Polling 30s refreshModels → Provider.isAvailable. Frontend fallback 3-tier in useChat.ts con toast Undo. Server fallback buildFallbackConfig (orchestrator.ts:879). SSE done include modelUsed, providerUsed.

Estendere: nuova skill → registerSkill({...}) in builtinSkills.ts. Per-chat model → PATCH /workspaces/:ws/chats/:chatId/model. Plan mode → abilita WorkspaceAgentConfig.planMode. Tuning budget → env AGENT_* (vedi 03 — Configuration).

Chat streaming (SSE)

POST /api/workspaces/:id/chat/stream → SSE. Eventi: token, status, citations ({ sources }), done ({ chatId, messageId, modelUsed, providerUsed, mcpSources }), error, plan, wiki_edit. Frontend: useChat.ts (@microsoft/fetch-event-source), abort via AbortController. Slash /model, palette Cmd+K, confronto Cmd+Shift+M. Persistenza modello effettivo via loadChat PATCH (stale default skipped, vedi frontend/CLAUDE.md).

Documenti, ingestion, OCR

Ingestion (collector/routes/ingest.ts:219): multer (100 MB, allowlist MIME) → parseFile → chunkText → embed → vectorStore.addDocuments → notifyServerStatus (PUT /api/documents/:id/status con X-Collector-Secret). Risposta include chunks[].chunkText per FTS lato server.

Parser (collector/services/parser.ts): PDF (pdf-parse, <100 char → OCR skip D-04, vision è server-side), DOCX (officeparser → mammoth fallback), PPTX (officeparser), XLSX (node-xlsx), TXT/MD/CSV (UTF-8), YouTube (youtube-transcript-plus, 422 se no transcript). ocrMode (auto/vision/skip) in IngestUploadBodySchema.

OCR vision (server-side) (services/ragOcrService.ts:59): pdfjs-dist load PDF (cap 50 pagine) → render PNG → Ollama vision model → strip grounding tags → concatena. Job service ocrJobService.ts con workflow approve/reject, qualityScore, credibilityScore. Routes routes/ocr.ts. Frontend SettingsOcr (catalogo modelli, job list, preview, custom instructions).

DLP filter (services/dlpFilter.ts): scanContent(text) → {hasMatch, matches, redactedText}. 6 PII: email, credit_card, ssn, api_key (sk-...), aws_key (AKIA...), private_key (PEM). Redaction [REDACTED]. Integrato in routes/chat.ts (input + output), gate DLP_ENABLED (default false). SSE done → dlp_warning: true; widget mostra PIIWarningPrompt.

Archivi, Wiki, Synthesis

Archive: knowledge base multi-pagina workspace-scoped, pagine markdown con path slug e wikilink [[Title]]. 14 servizi archive (archivePageService, archiveGraphService, archiveIndexService, archiveBacklinkService, ...). Routes archive*.ts (20+ file): CRUD, pages, config, search (hybrid RRF), graph, export, index, import (copy-from-doc), schema templates.

Wiki embedding (services/wikiEmbeddingService.ts:8): indexWikiPage → chunk 800/100 → collector POST /api/ingest/wiki-pages (tabella wiki_pages, workspaceId = archive:<id>). Skill wiki_query (FTS + BFS wikilinks) e wiki_write (preview/dry-run + wiki_edit SSE).

Synthesis (services/synthesisService.ts): pipeline 5-pass: entity_extraction → summary_generation → candidate_search (BM25 tsvector) → llm_decision (CREATE/UPDATE/SKIP/FLAG_CONTRADICTION) → write_overview. PHI gate: se ArchiveConfig.localLLMOnly=true e provider non-ollama → abort pre-egress. Workflow admin: trigger → preview → approve/reject (routes/synthesis.ts).

Backup

Scheduler Bree (services/backupSchedulerService.ts): @mintplex-labs/bree, closeWorkerAfterMs:0/timeout:0 (long-running). bootstrapJobs a startup (cleanStaleLocks → carica BackupJob enabled). DB-level mutex via BackupLog.status="running"; cleanStaleLocks → running >2h = failed. Cron helper buildCronExpression(frequency, time, ...).

Pipeline (services/backupService.ts:282): dumpDatabase (pg_dump spawn no-shell) → collectFiles (storage dirs, path-traversal guard) → createArchive (archiver zlib level 9) → computeChecksum (SHA-256 streaming) → uploadWithRetry (3 retry backoff) → sendFailureNotification (SMTP). Safety backup pre-restore (runSafetyBackup).

Encryption (services/encryptionService.ts): AES-256-GCM. Key: ENCRYPTION_KEY (base64 32 byte) o scryptSync(JWT_SECRET, salt). Decrypt chain rotation-safe: [current, ...LEGACY_PREVIOUS_ENCRYPTION_KEYS, scryptLegacy?].

Destinazioni (services/backup/providers/): LocalProvider, S3Provider, S3CompatibleProvider, DropboxProvider, GoogleDriveProvider, FtpProvider, SftpProvider, EmailProvider (interfaccia IBackupDestinationProvider). License backup_enabled + max_backup_destinations (Community 1, Enterprise ∞).

Restore (backupService.ts:617): restoreDatabaseWithPsql (psql --single-transaction --set ON_ERROR_STOP=on, atomicity all-or-nothing). Routes restore.ts + POST /api/backups/trigger (manuale).

Retention (backupRetentionService.ts): per-job retentionDays (default 30); cleanup BackupLog success oltre cutoff + hard delete. Errori individuali non bloccano.

Push (VAPID Web Push)

routes/push.ts: initVapid legge VAPID_* env (auto-generate in dev). GET /vapid-key (pubblico), POST/DELETE /subscribe (license push_notifications), POST /test (admin). sendPushNotification(title, body) export per event-trigger; rimuove subscription 410 Gone. Frontend: service worker API.

Webhook

services/webhookService.ts: dispatchWebhookEvent(event, payload) fire-and-forget parallelo. deliverWebhook: body {event, timestamp, data}, header X-Webhook-Signature: sha256=HMAC-SHA256(secret, body). Retry 3 (1s/2s/4s); 10 fallimenti consecutivi → auto-disable. Eventi: document.processed, chat.completed, user.created, workspace.updated, mcp.installed/uninstalled, ... Triggerati da eventLogService.logEvent (async post-DB). Routes webhooks.ts (license webhooks).