LLM Configuration
The LLM providers your workspace uses are configured under Manage → AI:
| Page | Route | What it holds |
|---|---|---|
| LLM Providers | #/manage/ai/llm-providers |
Every provider, every model it offers, and which of those models the workspace may use. |
| Chat Settings | #/manage/ai/chat-settings |
The workspace-wide chat switches: chat on or off, how long chats are kept, and the copy disclaimer. |
| AI Indexing Models | #/manage/ai/vector-indexing |
Embedding models, configured separately — see sentence embeddings. |
This page covers the supported providers, the per-provider setup, and the policy decisions you make once a provider is wired up.
The Models screen
LLM Providers lists every provider and every model, with a per-model enable toggle and a detail strip that follows the pointer. Three behaviours are worth knowing:
- Enabling a model on a provider that is not set up yet asks for the API key and configures the provider in place — you do not have to add the provider first.
- Providers with no browsable catalog (a custom-code provider, a local server) show their own settings inline rather than an empty model list.
- Models the provider actually uses but the vendor catalog does not list — a newer release, an Azure deployment name, a model carried over from an earlier configuration — appear too, so they can be seen and switched off.
The enabled set is what users and agents can pick from: a single flat model picker is shared by the chat composer and the agent editor, so a chat or an agent can run on a specific model instead of the provider default. Leaving a model disabled removes it from both.
Existing provider configurations are carried forward automatically when a workspace upgrades, including on first use if the background migration has not reached them yet.
Supported providers
Twenty providers are configurable. Fifteen of them are plain API-key providers, configured the same way — paste a key, pick the models to enable:
| Provider | Default model |
|---|---|
| OpenAI | gpt-5-mini |
| Azure OpenAI | gpt-4o (the Host must point at the resource's OpenAI-compatible endpoint) |
| Google Gemini | gemini-2.5-flash |
| Groq | llama-3.3-70b-versatile |
| DeepSeek | deepseek-chat |
| xAI Grok | grok-4 |
| Perplexity | sonar-pro |
| Cohere | command-a-03-2025 |
| OpenRouter | openai/gpt-4o |
| DeepInfra | meta-llama/Llama-3.3-70B-Instruct |
| Moonshot AI | kimi-k2.5 |
| Z.AI | glm-4.6 |
| Alibaba Qwen | qwen3-max |
| MiniMax | MiniMax-M2 |
| Upstage | solar-pro2-251215 |
The other five have their own configuration shape:
| Provider | Notes |
|---|---|
| Anthropic (Claude) | Claude Opus / Sonnet / Haiku. No embedding service — pair with an OpenAI / Azure / local embedding model. |
| AWS Bedrock | Models hosted in your own AWS account, with IAM rather than an API key. |
| Ollama | A local Ollama server; the strongest data-residency story. |
| Mistral | Hosted Mistral models. |
| Custom (code) | A provider implemented as workspace code, for a gateway or a protocol none of the above speaks. |
Embeddings are configured separately from all of this: any OpenAI-compatible embedding endpoint, or one of the built-in CPU-friendly models (MiniLM, Arctic XS, Harrier) that need no external network. It is common to pair a hosted chat provider with local embeddings, so most of your text never leaves your network.
Using these providers from your own tools
Once a provider is configured, the workspace can relay completions to it through an OpenAI- or Anthropic-compatible API, so existing clients and SDKs can use it without ever holding the provider credentials.
Configuration goals
- Predictability — same prompt + same context → same answer, modulo provider-side variance.
- Safety — no unintended actions, no data leakage to the wrong provider.
- Cost control — explicit caps on tokens per response and tokens per turn.
- Traceability — every chat turn and every tool call logs which model produced what.
- Failure tolerance — degrade gracefully if the provider is unreachable.
Per-provider setup
OpenAI
- Create an API key from the OpenAI dashboard. Use a project key so you can revoke it independently.
- Manage → AI → LLM Providers → Add provider → OpenAI.
- Paste the API key. The workspace encrypts it with
MSK_GRAPH_MASTER_KEYbefore storing. - Pick a chat model (e.g.,
gpt-4o,gpt-4o-mini) and an embedding model (e.g.,text-embedding-3-small). - Click Test. The workspace makes a probe call to each model.
Limits to set: max output tokens (start at 1024), per-call timeout (30s).
Azure OpenAI
- Create an Azure OpenAI resource and a deployment for each model you want to use.
- Manage → AI → LLM Providers → Add provider → Azure OpenAI.
- Provide:
- Endpoint (e.g.,
https://my-resource.openai.azure.com/). - API key or Entra ID identity (recommended; uses the workspace's managed identity).
- Deployment name for chat and for embeddings.
- API version (use the latest stable).
- Click Test.
Notes: Azure deployments are per-region, per-quota. Set realistic timeouts (30–60s) for cold starts on lightly-used deployments.
Anthropic (Claude)
- Create a Claude API key from the Anthropic Console.
- Manage → AI → LLM Providers → Add provider → Anthropic.
- Paste the API key.
- Pick a chat model (e.g.,
claude-opus-4-7,claude-sonnet-4-6,claude-haiku-4-5). - Configure an embedding provider separately (OpenAI, Azure OpenAI, or local) — Anthropic does not currently offer an embedding service.
- Click Test. ===
Local OpenAI-compatible server
Useful for air-gapped or data-residency-sensitive deployments.
- Run a server that speaks the OpenAI HTTP protocol — examples: Ollama, vLLM, LM Studio, Text Generation Inference.
- Manage → AI → LLM Providers → Add provider → Custom (OpenAI-compatible).
- Provide:
- Base URL (e.g.,
http://ollama.internal:11434/v1). - An API key if the server requires one.
- The model identifier (e.g.,
llama3.1:70b).
- Click Test.
Notes: the local server runs on your own infrastructure — its capacity and latency are your problem. Size it to match your chat volume.
Local embeddings (built-in)
The workspace can produce embeddings locally without an external provider. Useful when:
- the corpus must never leave your network;
- the embedding cost from a hosted provider would be prohibitive;
- you want predictable latency under variable load.
- Manage → AI → AI Indexing Models.
- Pick a built-in model: MiniLM, Arctic XS, or Harrier.
- Trigger a rebuild from Settings → Maintenance → Rebuild embeddings.
Tradeoff: local embeddings are typically smaller and less accurate than hosted models, but on enterprise corpora the gap is usually narrow enough to not matter, especially when combined with text retrieval (hybrid search).
Picking the right chat model
| If you want… | Pick… |
|---|---|
| Lowest latency for tool-call-heavy chat | A small fast model (gpt-4o-mini, claude-haiku-4-5, a local 7B-class model) |
| Highest answer quality on long, dense context | A frontier model (gpt-4o, claude-opus-4-7) |
| Predictable cost at scale | A mid-tier model with a hard max_tokens cap |
| Air-gapped operation | A local 70B-class model on a GPU box |
| Strict data residency in EU/US/JP/… | A regional Azure OpenAI deployment |
Whatever you pick, set hard caps:
- Max output tokens per turn (start at 1024; raise for summarization workloads).
- Per-call timeout (start at 30s).
- Max tool calls per turn (start at 5; large values let the LLM thrash).
Prompt templates
Maintain a small set of reusable templates and version them with the rest of your code:
- Grounded Q&A — "Answer using ONLY the snippets below. Cite with
[1]." (Prompting Patterns) - Summarization — explicit length/structure constraint.
- Classification / extraction — strict JSON schema for output.
- Tool-using assistant — short, clear, list available tools.
Templates live under Manage → AI → Prompt Templates; export them to git for promotion.
Tooling and endpoint access
If your chat surface has tools available:
- Curate the catalog deliberately. Vague or overlapping tools cause unreliable selection. See AI Tools.
- Mark admin-only tools so they don't appear for regular users.
- Bound expensive tools with a hard
Take(...)and a cancellation check.
Fallback and degradation
Production deployments configure a fallback provider:
- If the primary chat provider returns
5xxor times out, the workspace transparently retries against the fallback. - If both fail, the workspace returns a
tool_invocation_failed(in tools) orexternal_provider_timeout(in endpoints) — see Error codes.
For embeddings, the workspace falls back to text retrieval only when the embedding provider is unavailable. New nodes won't get vector entries until the provider recovers; rebuild after recovery with Reindexing and re-embedding.
Cost guardrails
Every provider call is logged with token counts and (when known) cost. Operationalize this:
- Track external model consumption and cost under Manage → Operate → LLM Usage (
#/manage/operate/llm-usage), which also holds the model price list the cost figures come from. - Alert when daily spend exceeds a threshold.
- Monitor
/api/chatai/tools/metricsfor the tools that consume the most tokens — they often hide a prompt that's growing unbounded.
Validation
After provider config changes, walk this checklist:
- Test button succeeds for chat and embeddings.
- A search for a known phrase returns expected results (validates embeddings).
- A chat turn returns an answer with citations (validates chat + tool flow).
- Token counts appear in monitoring.
- Failure mode: temporarily block the provider's egress; verify the workspace returns a clear error rather than hanging.
Next steps
- The architecture this configuration plugs into: RAG and agent architecture.
- Prompt patterns: Prompting Patterns.
- Tool design: AI Tools.
- Embedding strategy: Vector Search, Embeddings.
- Data handling: Security → AI / model-provider data handling.