Multi-LLM Orchestration: Model Switching, Comparison, and Fallback

The one-model deployment is a decision you outgrow

Teams do not stay loyal to one LLM, and they should not. The fast local model that answers support questions perfectly fails at nuanced legal analysis. The frontier model that handles reasoning costs too much to answer "what's in this document?" A provider changes pricing, deprecates a model, or has an outage - and suddenly the whole product is down or renegotiating. A serious AI platform therefore treats "which model" as a runtime decision, not an installation decision.

Simmetric Chat treats it that way throughout: every provider boundary is a strategy, the model is selectable per chat, evaluation is built into the interface, and failure degrades gracefully instead of erroring. This article walks through the orchestration layer - the providers, the switching UX, the comparison mode, and the fallback chain.

Five first-class providers, twenty presets

Out of the box, the platform speaks to five first-class providers: Ollama (local, the privacy-first default), OpenAI, Anthropic, OpenRouter, and Gemini. Each gets a dedicated streaming parser - the wire formats differ enough that treating streaming as one generic problem produces subtle bugs - so token streaming, citations, and terminal events are parsed per provider rather than approximated.

Beyond the first-class five, twenty provider presets cover the OpenAI-compatible ecosystem: DeepSeek, Mistral, Kimi/Moonshot, NVIDIA NIM, Qwen, xAI, Z.AI/GLM, MiniMax, LM Studio, GitHub Copilot, and more. Because those services implement the OpenAI wire protocol, adding one is a configuration row - base URL, key, model list - not a code change. A deployment can therefore mix a local Ollama for sensitive work, a cloud frontier model for deep analysis, and a budget OpenAI-compatible provider for bulk summarization, all in one workspace.

Configuration follows the platform's precedence rules: environment variables establish infrastructure-level defaults, the admin panel edits take effect immediately, and settings resolve database over environment over default. The global default provider is exactly that - a default, overridable per workspace and per chat.

Switching models without leaving the conversation

Model selection lives where the conversation lives. The chat input carries a model badge showing what will answer the next message; clicking it - or pressing Cmd+K (Ctrl+K on Windows/Linux) - opens the quick-switch palette. The palette groups models by provider and annotates them with capability badges derived from model metadata:

• local-only - runs on-premise, no outbound data
• fastest - lowest latency in the current provider set
• smartest - highest quality available
• reasoning - chain-of-thought capable

Typing /model <name> in the input box switches instantly to a matching model, with a filtered palette as the fallback when the name is not an exact match. The command text is stripped from the message before sending, so the slash command never pollutes the chat history.

Two details make this ergonomic rather than merely possible. First, switching is mid-conversation: the same chat can move from a local model to a frontier model and back, with each message persisting which model actually served it - the audit trail of model usage survives the switch. Second, the palette is a keyboard-first surface: the Cmd+K pattern means model choice costs a keystroke, not a trip to a settings page, which is the difference between a feature people use and one they configure once and forget.

Side-by-side comparison

Choosing a model from a description is guesswork; choosing one from watching two answers to your real prompt is engineering. Pressing Cmd+Shift+M (Ctrl+Shift+M) opens side-by-side model comparison: two panes stream responses from different models simultaneously for the same prompt, token for token.

The workflow it enables is the one evaluation harnesses formalize, kept in the day-to-day surface:

• Ask once, watch both models reason - latency, phrasing, and grounding differences are immediately visible
• Keep one response and continue the conversation with that model
• Send to a single pane to merge the outputs back into one thread
• Close the comparison and carry on with either model

Because comparison uses the same streaming and citation infrastructure as normal chat, compared answers arrive with their RAG citations intact - you can evaluate not just how a model phrases an answer but whether it grounds the answer in the right sources. That distinction matters in a retrieval platform: a model that writes beautifully but cites the wrong document is worse than one that writes plainly and cites correctly.

The fallback chain

Models become unavailable: a provider outage, a deprecated model name, a quota exhausted mid-conversation. Simmetric Chat's answer is a deterministic fallback chain - if the selected model becomes unavailable, the system automatically falls back to the workspace default, then the global default, then any available model. A toast notifies you of the switch, with an Undo option.

The chain triggers on both periodic polling (every thirty seconds) and SSE errors during streaming, so a mid-answer provider failure re-routes without losing the conversation. The design goal is specific: a model outage should degrade your platform to its next-best model, not degrade it to an error page. Users see a notification rather than a stack trace, and the operator sees a platform that kept serving.

For multi-instance deployments the same design philosophy extends outward - the streaming fan-out and job queue that keep chat responsive under scale are externalized state, covered separately in the scaling article.

What the orchestration layer buys you

Pulling the threads together, the orchestration layer is what turns provider choice from a procurement decision into a set of runtime levers:

• Cost control. Route bulk work to a cheap OpenAI-compatible provider and reserve the frontier model for the prompts that need it, per chat rather than per deployment.
• Privacy zoning. Keep sensitive workspaces pinned to local-only models while letting general workspaces use cloud providers - the capability badges make the zone visible at selection time.
• Resilience. Provider outages become model switches, not downtime, with the fallback chain and the streaming re-route.
• Evaluation as a habit. Side-by-side comparison makes "which model should we standardize on?" a weekly two-minute check instead of a quarterly research project.

All of it sits on the same strategy-patterned provider boundary the rest of the platform uses, so adding a provider - first-class or OpenAI-compatible preset - never touches chat, RAG, or the agent loop.

Read the full guide in the docs →