Read from the runtime, never assumed
The console lists the models the machine actually has, refreshed before every decision that depends on it.
The console does not keep a list of models it hopes are installed. It reads the catalog from the local runtime on the Refresh catalog button and automatically before every role probe, routing change and enable, so what you are choosing from is what the machine actually has.
The catalog lives in process memory. If the runtime is down the previous list is kept and the page badges that the refresh failed, rather than showing an empty catalog that would read as "no models installed".
The controls, verbatim
Enable Local AIUse deterministic rulesApply model routingTest active modelsRefresh catalogReset operations telemetry
Four roles, and only two are yours to set
Which model does what, where each is chosen, and what ships when nothing is.
| Role | Set where | Shipped fallback | Scope |
|---|---|---|---|
| Intent classification | /ai/routing, "Apply model routing" | qwen3:14b (env LOCAL_AI_MODEL) | Deployment, and no bot has a model of its own |
| Reply wording | /ai/routing | qwen3:14b | Deployment; the name is also fed to her prompt as a given fact |
| Embedder (knowledge base and search relevance) | not console-assignable | nomic-embed-text (768 dims) | Deployment |
| Bio generator (offline tooling only) | not a console role | qwen3.5:9b | Offline |
Roles, labels and fallbacks quoted from site material Block 3 section 2 · the catalog is read from Ollama GET /api/tags and held in process memory
Two of the four are console roles and two are not. The embedder is a code constant because changing it would invalidate every stored vector, and the bio generator is offline tooling that never touches a reply. The reply model's name is also fed to her prompt as a given fact, which is why she can answer what she runs on without being told separately: it is one of the identity lines on the system prompt page.
An unconfirmed model never becomes the live one
What happens when a model is missing, when activation fails, and when one dies mid-conversation.
Fail closed activation — Local AI becomes effective only after the selected role models are confirmed. Any failure keeps or returns the runtime to deterministic rules.
What is refused, and what is kept
Routing to a model that is not installed is refused rather than accepted and discovered later, and the previous route is kept. The environment gate is the outer switch: the effective state is the gate AND the stored preference, so a deployment that has not opted in cannot be enabled from the console at all.
Cannot route to an uninstalled model: {names}.
Local AI cannot be enabled because LOCAL_AI_ENABLED is false.
When it fails mid-request
At runtime any model failure falls back per request rather than per deployment: intent to the deterministic resolver, reply to the deterministic draft. A model that goes away mid-conversation costs the wording, never the decision.
The status badges state the runtime posture plainly, including that cloud fallback is disabled and telemetry is content-free. built
Check /en/security
What each window costs in VRAM, measured
The table the console renders, taken on one card, with the row that failed recorded as not measured rather than guessed.
| Model | Context | Total GB | VRAM GB | CPU GB | What was measured |
|---|---|---|---|---|---|
| qwen3:32b | 8192 | 22.11 | 22.11 | 0 | fully on GPU, but leaves under 1 GB free |
| qwen3:32b | 16384 | not measured | not measured | not measured | failed to load in two attempts; not measured rather than guessed |
| qwen3:32b | 32768 | 29.15 | 22.95 | 6.21 | SPILLED 6.21 GB to CPU |
| qwen3:14b | 8192 | 10.47 | 10.47 | 0 | fully on GPU, 11.1 GB free |
| qwen3:14b | 16384 | 11.83 | 11.83 | 0 | fully on GPU, 9.0 GB free |
| qwen3:14b | 24576 | 13.19 | 13.19 | 0 | SERVED: fully on GPU, 7.7 GB free with the embedder resident |
| qwen3:14b | 32768 | 14.55 | 14.55 | 0 | fully on GPU, 6.5 GB free |
| qwen3:14b | 40960 | 15.90 | 15.90 | 0 | fully on GPU at the model's maximum, 5.2 GB free |
Measured and shipped as constants: CONTEXT_MEASUREMENTS, src/interaction/reasoning.ts:106-126 · measured on the 24 GB card · 32B rows from CCB-S4-052, 14B rows from CCB-S5-045
How to read it
The served window is 24,576 tokens SERVED_CONTEXT_TOKENS, src/interaction/reasoning.ts:147 (D-231, measured) · host OLLAMA_CONTEXT_LENGTH, which is the row marked SERVED above. One row is deliberately not a number: a load that failed twice is recorded as not measured rather than guessed, which is the difference between a table you can plan against and a table that reads well.
The 6.21 GB spill that held this deployment at 8192 for a season is a fact about qwen3:32b, not about the window: that model costs 0.25 MiB of KV cache per token, so 32768 needs 8 GB of it on a card with about 2 GB spare. On qwen3:14b there is no spill at any window the model supports, right up to its 40960 maximum.
Off on every request, and the measurement behind it
Six times slower and three replies in five unusable is why the setting is off, and why it is a setting rather than a rule.
| Reasoning | Latency | Replies unusable |
|---|---|---|
| none | 2,782 ms | 0 of 5 unusable |
| low | 16,283 ms | 3 of 5 unusable |
| high | 16,854 ms | 3 of 5 |
| runtime default | 14,770 ms | 3 of 5 |
site material Block 3 section 2 · measured against qwen3:32b in the production request shape
Reasoning is off on every request, and the table is why: with it on, replies came back six times slower and three in five were unusable. That is a measurement in one production request shape against one model, not a claim about reasoning models in general, and it is the kind of decision worth re-running rather than inheriting.
A measurement is not a law. This one was taken against a specific model in a specific request shape; a different model, or a different shape, could move it, and the setting exists so it can be moved.