Local AI runtime

The model may classify and phrase. It never gets permission to act.

This is the load bearing rule of the whole runtime, and it lives in application code rather than in a prompt. A language model has exactly two jobs here. It may read a member message and say which entry of a closed list of intents it thinks was meant, and it may take a reply the application has already finished and word it more naturally. Everything else, the database reads, the consent writes, the publication decision, the actual sending of a message, happens in deterministic code that never asks a model for permission.

The intent catalog is closed and re-validated after the model has spoken. A result naming an intent outside the catalog, or one belonging to a plugin the operator has switched off, is treated exactly as if the model had said it did not understand. Confidence is clamped into range, slot values are truncated to fixed lengths, and a resolver that throws is caught and answered by the rules instead.

Consent gets a second, stricter gate. PUBLISH, UNPUBLISH and RESTORE are the intents that can increase what the public can see, and for those the model may only corroborate, never assert. UNDO is not gated the same way, because by construction it can only reverse an opt in and therefore only ever reduces exposure. The deterministic rule engine has to reach the same intent independently, and the model has to report at least 0.9 confidence. If either fails, the consent intent is dropped: the member gets whatever ordinary read-only intent the rules found, or UNKNOWN.

Consent is first person. If the classifier reports that the message was aimed at somebody else, that target lands in a slot whose mere presence means refuse. There is no admin path and no model path to opting another member in.

The reply module is built the same way. It receives a finished deterministic draft and has no database handle, no consent function, no tool call and no transport. In locked mode it may write one short opening sentence of at most 180 characters, and the application appends the protected text unchanged. In free mode it may rewrite the draft, but every required literal, every count, price and identifier, has to survive verbatim or the generated reply is discarded and the draft goes out as written.

The member message is untrusted text throughout. The classifier is told to treat it as material to classify and never as an instruction, and the reply model is told the same. What makes that hold is not the instruction but the wiring: neither model is connected to anything it could be talked into doing.

A model that invents an opt in cannot produce one: the deterministic rules have to independently agree before anything is published, and the only consent change a model can reach on its own takes content down.

Inference stays on infrastructure you control

The runtime talks to Ollama over an endpoint the operator sets, and the configuration loader refuses to accept a public one. The host in the base URL has to be written as localhost or a .localhost name, or as a literal address in 127.0.0.0/8, 10/8, 172.16 through 172.31, 192.168, ::1, or a link local or unique local IPv6 range. A URL carrying credentials, or a path, query or fragment, is rejected before the process starts.

The endpoint is an environment variable, not a form field. No admin control repoints inference at a different host, so somebody who reaches the console cannot quietly move member text somewhere else. No API key travels with the request, because there is none.

There is no cloud fallback, and that is not a setting somebody can flip. No client for a hosted model provider exists anywhere in the codebase. When the local endpoint is unreachable, the deterministic rules answer.

When the local endpoint is unreachable, nothing is sent anywhere else. The rules answer instead.

Two models, two jobs, chosen separately

Classifying and phrasing are different problems, so they get different model slots. A small, fast, obedient model can do intent classification while a larger one handles the language, or one model can serve both roles. The choice is stored per role, and each role gets its own request.

Model discovery comes from the Ollama catalog: the installed models with family, parameter size, quantization level, and file size on disk. The console shows the footprint of the two selected models, so what actually sits on the machine is visible rather than assumed.

The two calls are shaped differently on purpose. Classification runs at temperature zero against a strict JSON schema that admits only an active intent, a confidence between zero and one, a small fixed set of slots, and a language of en or de. Wording runs warmer, capped at 700 characters by default, and the generated text is cleaned before it can reach a member: code fences removed, control characters stripped, and any em dash, en dash or horizontal bar replaced with a plain hyphen.

Some values are not required but forbidden. The sender's display name never leaves the application: it is held back as blocked text and checked against what the model returns, and a generated reply containing it is discarded. Name prefixes are the application's business, not the model's.

Stored setting, effective state, and a probe in between

Three things can be true at once, and the console shows all three separately. Local AI can be available in the environment, requested in the stored settings, and effectively active in this process. It is only really running when the first two agree, and the name of the resolver actually installed sits next to them, so the claim is checkable rather than asserted.

Activation is fail closed. Switching local AI on runs a connection probe first: the catalog is fetched, and every model assigned to a role has to appear in it. If one is missing, activation fails with the missing name and the deterministic rules stay in charge. Routing changes go through the same check, so a role cannot be pointed at a model that is not installed.

Switching off works the other way round. The rules become active in the process immediately, and only then is the preference persisted. If that write fails, the message says exactly that: the rules are active, the preference did not save.

Turning local AI on runs a probe first. A missing model is a refusal, not a warning.

When the model fails, the answer is still deterministic

A timeout, an HTTP error, malformed JSON or an intent outside the catalog all land in the same place: the rule engine answers. A confidence below the threshold is treated as UNKNOWN, which is the same answer the model would have given had it said it did not understand. The member never sees an error text, and the failure is never converted into something that reads like a successful classification. Where the deterministic answer is 'not understood' and she is not confident she was even being addressed, she stays out of it rather than replying.

The reply path degrades the same way. If the wording model times out, or the generated text fails a guard, the deterministic draft goes out unchanged. The member reads a plainer sentence, which is the correct outcome and not a fault they need to hear about.

Every one of those events is counted. Fallbacks have their own counter, separate from failures, and so do guard overrides, the cases where the model proposed one intent and the deterministic gate produced a different final one. A fallback that could mask a fault appears as a number in the admin rather than only in a log file.

Telemetry with nothing in it to leak

The operations view answers operational questions: how many requests, how many succeeded, how many fell back, average and last latency, the time of the last success and the last failure, plus the last fifty events with role, outcome, model name, operation and latency.

What it does not contain is content. No member message, no prompt, no generated reply, no classified text. Every event in the operations buffer records an error as a category only: timeout, http-error, invalid-output, guard-rejection, unavailable, runtime-error. The runtime page additionally shows the last raw resolver error, which reports how the call failed and never carries member text. The runtime snapshot carries that fact as its own field rather than as a promise in a document.

The counters live in memory and are cleared by a restart or by an explicit reset. What is durable is the audit trail. Enabling or disabling local AI, changing role routing, and resetting telemetry are each written to the audit log with the acting administrator; the toggle and routing entries also record the models involved, and the reset entry records the counters it cleared.

No member message, no prompt and no generated reply is ever written to telemetry. Errors are recorded as a category.

What is not connected yet In development

GPU telemetry is not integrated. The hardware view reports the model inventory and the metadata Ollama supplies, and it marks utilization, VRAM, temperature, residency and queue depth as not integrated instead of filling them with plausible numbers.

Local AI generation of profile biographies and avatar images is not built. Today the local models serve exactly two callers, intent classification and reply wording, and nothing else in the system asks them for anything. Today the local models serve exactly two callers, intent classification and reply wording, and nothing else in the system asks them for anything.

Retrieval over the archive is prepared in the console and not configured in the code. And to leave no room for misreading, because it is the question that matters most: hash screening of media for known illegal material is a separate path from this runtime and is connected to no provider. Local AI does not screen media, and at present no other component does either.