Troubleshooting

Common symptoms, cause, resolution. File-line references and cross-links to the docs.

Startup & config

The server exits immediately with process.exit(1)

• Cause: env Zod validation failed (src/config/env.ts).
• Fix: check the logs - they name the invalid variable. Verify JWT_SECRET (req.), COLLECTOR_SECRET (req.), DATABASE_URL, ports. Compare against .env.example.

COLLECTOR_SECRET mismatch between server and collector

• Symptom: the collector cannot notify the document status; the upload stays "processing".
• Fix: set the same COLLECTOR_SECRET in the root .env (single config - the per-package .env layer was removed in v0.22). Constant-time comparison → silent 401 error.

WIDGET_API_KEY mismatch widget↔server

• Symptom: the widget fails to load config / chat 401 from the server.
• Fix: same key in the root .env (single config). Without the key, the cache-bust push is a no-op (5 min TTL).

Config changed from the UI has no effect

• Cause: ALWAYS_READONLY keys (JWT_SECRET, DATABASE_URL, ports, URLs) are rejected by the bulk settings; the others follow DB > ENV > Default.
• Fix: for readonly keys, edit .env + restart. PUT /api/system/settings returns { rejected: [...] } - check that array.

Database

Conflicting migrations / dev reset

• Fix: in dev with a throwaway DB, reset with pnpm db:migrate:reset:guard (requires PRISMA_MIGRATE_RESET_CONFIRM=yes or --force-accept-data-loss - never in production). Guarded deploy: pnpm db:migrate:guard (refuses destructive migrations without explicit consent). Verify PostgreSQL 16 and a correct DATABASE_URL.

Non-atomic backup restore / partial data

• Cause: the restore uses psql --single-transaction --set ON_ERROR_STOP=on; on failure it rolls back completely.
• Fix: check the psql logs; safety backup before restore (runSafetyBackup). Use /restore/:logId/dry-run before the real restore.

RAG & embedding

Vector search returns 0 results but FTS works

• Cause: D-01 guard - current EMBEDDING_MODEL ≠ the Documents' embeddingModel (e.g. the model changed after upload): the vector leg is skipped and search degrades to FTS-only.
• Fix: re-embed via POST /api/system/reembed-documents (admin, optional workspaceId/documentIds) to re-embed chunks with the current model; if you only need to rebuild the FTS index from the vector DB: POST /api/system/reindex-documents. Both are admin-only, also from Settings → Maintenance.

RAG degraded in the widget (status: rag-degraded)

• Cause: the RAG pre-search failed (collector down, empty workspace whitelist) or the linked-workspace search returned 0 results.
• Fix: check GET /api/health/rag; check the widget's WidgetWorkspace (GET /api/widgets/:id/workspaces); confirm the collector is up (GET :3210/api/health).

LanceDB table missing / query error

• Cause: the per-workspace ws_<name>_<shortId> table was not created or the storage moved.
• Fix: verify the collector's STORAGE_PATH is unchanged; reindex wiki/documents.

Agent & LLM

Agent abort: unknown_tool_breaker

• Cause: BOT-03 circuit breaker - 3 consecutive / 5 total unknown skills. Usually a model hallucinating unregistered tools.
• Fix: use a more capable model or reduce the enabled skills; verify the MCP connection is enabled and skills registered (mcp_<conn>_<tool>).

Agent abort: wallclock / token_budget / context_overflow

• Fix: raise env AGENT_WALLCLOCK_TIMEOUT_MS (default 600000), AGENT_MAX_TOTAL_TOKENS (200000), AGENT_MAX_CONTEXT_BYTES (500000). For simple tasks lower them to avoid stalls. See 03 - Configuration § Agent watchdog.

Model not responding / local LLM timeout

• Cause: Ollama down or model not pulled; LLM_TIMEOUT=0 = no timeout (slow local LLMs).
• Fix: ollama list / ollama pull gemma4:latest; verify OLLAMA_BASE_URL (http://ollama:11434 in compose). Provider fallback: check Provider.isAvailable (30s polling).

Provider fallback not triggered

• Cause: no alternative provider available (isAvailable=false) or no global default set.
• Fix: add a provider (POST /api/providers), set the default (PUT /:id/set-default), check the deriveCapabilities tags. Frontend 3-tier fallback with an Undo toast.

Widget

Widget does not load / empty iframe

• Cause: allowedOrigins does not include the host origin → dynamic CORS fails closed.
• Fix: add the origin to Widget.allowedOrigins (CSV string, max 2000 chars). Verify GET /api/config/:widgetId responds.

Lead capture 403

• Cause: config.leadCaptureEnabled=false for that widget.
• Fix: enable it on the Widget model (admin UI or API).

Widget rate limit 429

• Cause: limiter chain - widgetDailyMessageLimiter (5/24h per widget+IP, override sessionLimitPerDay), widgetChatLimiter (30/min per widgetId, override rateLimitPerMinute), widgetSessionLimiter (50 session creations/day per IP), widgetLeadLimiter (3/h per IP).
• Fix: consider the per-widget overrides (rateLimitPerMinute/sessionLimitPerDay); check which limiter triggered (the Retry-After header / error body).

MCP

Marketplace install 201 but tools unavailable

• Cause: connectMCPServer is fire-and-forget; the failure is logged but not propagated.
• Fix: POST /api/mcp-connections/:id/test (10s, returns {success, toolCount, error}). Verify the D-09 transport fallback; check activeConnections via GET /api/mcp-connections/statuses.

MCP disabled automatically

• Cause: webhook-style auto-disable? No - the MCP connection stays enabled; it is the webhook that auto-disables after 10 consecutive failures. For MCP, the 30-minute health check sets healthStatus (stale/down) but does not disable the connection.
• Fix: for webhooks, re-enable manually; for MCP, the health badge is informative.

MCP tool not used in a chat

• Cause: active per-chat pins excluding the connection, or the connection disabled.
• Fix: GET /api/chats/:chatId/pins; check resolveSkillsForChat (no pins → every enabled connection in the workspace).

License

402 Payment Required on a feature

• Cause: requireFeature(flag) or requireFeatureLimit gated it (Community).
• Fix: set an Enterprise LICENSE_KEY, or stay on Community with limits. The error body includes feature, tier, limit?, current?.

Enterprise expired at runtime

• Behavior: graceful degradation to Community (no restart); warning in the logs.
• Fix: renew LICENSE_KEY; getLicenseInfo() re-reads at runtime.

Security & audit

Audit log does not write

• Cause: the audit log is an Enterprise module - in Community logEvent is a no-op (the write delegate is not registered), so no event is recorded at all.
• Fix: Enterprise for the audit log + immutability (audit_log_immutable). Verify the correct entityType in the filters (GET /api/event-logs). Event webhooks still fire in Community (fire-and-forget).

DLP does not redact PII

• Cause: DLP_ENABLED=false via settings (the v0.22 default is "true" - someone turned it off), or a role in DLP_BYPASS_ROLES exempting it from the scan.
• Fix: verify/set DLP_ENABLED=true via settings. Redaction over 10 DB-backed built-in patterns: the 6 originals (email, credit_card, ssn, api_key, aws_key, private_key) plus the 4 EU/IT ones (it_vat_iva, it_codice_fiscale, iban, eu_phone - the latter seeded DISABLED for admin review); up to 50 custom patterns via the DLP Patterns panel. SSE done → dlp_matches (the frontend renders the DLPNotice under the message).

Backup

Backup job stuck "running" forever

• Cause: a dead worker without cleanup; DB mutex BackupLog.status="running".
• Fix: cleanStaleLocks marks running >2h as failed on the next bootstrap. Restart the server to trigger it, or update the status manually.

Checksum mismatch on restore

• Cause: corrupted archive / partial upload.
• Fix: streaming SHA-256 computed at upload; verify the destination file. Restore from a later log.

Known documentation drift

Quick health checks

curl localhost:3000/api/health        # { database, collector, disk }
curl localhost:3000/api/health/rag    # RAG status
curl localhost:3210/api/health        # collector
curl localhost:3211/health            # widget

v0.28.1 Provenance

Operational reference - Simmetric Chat v0.28.1 (docs updated 2026-10-02, census SHA `aca5b8ff`).

Cross-links

• Config/env: 03 - Configuration
• Internals: 05 - Feature Guide
• Deploy: 09 - Deployment