LLM-Server/docs/architecture.md

Architecture

Source of truth: docker-compose.yml → each service below. This page explains why; flags, values and defaults live in the source. If they disagree, the source wins.

Terms as defined in CONTEXT.md.

Every workload calls the gateway (omniroute) with its own API key. The gateway forwards each call over a provider connection to one of two backends on the one GPU: llama-server (main model) or qwen-classifier (classifier model). qdrant sits beside them on the same Docker network. It isn't a model, so it isn't behind the gateway.

GPU access

Both backends share the one GPU, and both carry the same GPU settings in compose:

• group_add uses numeric GIDs, not group names. Docker resolves a named group_add entry against the container's own /etc/group, not the host's. That fails unpredictably when the image doesn't define the group, and it gets worse when several GPU services start at once. scripts/update.sh resolves the host's real GIDs into HOST_VIDEO_GID / HOST_RENDER_GID (see operations → Host-resolved values).
• GPU_MAX_HW_QUEUES works around ROCm/ROCm#5706: the card sits pinned at full load and boost clock whenever two HIP contexts touch it. With two backends running, there are always two.

Background: research/rocm-gpu-pin-and-render-group.md. Its service names are older than the current ones, but the findings still apply.

llama-server

The main model's backend. It is always running. Lazytainer used to idle-stop it, but was removed because its idle detection could never fire behind the gateway (research/lazytainer-omniroute-idle-stop.md).

• Model file (LLAMA_MODEL_FILE): the quant choice and its VRAM budget are in research/qwen3.8-27b-quant.md.
• LLAMA_GPU_LAYERS: the default puts every layer on the GPU. The model is dense, not Mixture-of-Experts, so this is the only RAM-offload knob. --n-cpu-moe, --cpu-moe and --override-tensor "exps" do nothing here. There is no SSD tier to enable either. llama.cpp mmaps the model file, so if GPU and RAM ever can't hold it, the OS pages the rest in from disk as a slow, implicit last resort. Explicit tiered offload is an open llama.cpp request.
• LLAMA_CTX_SIZE: set to the model's own maximum (max_position_embeddings). It only fits beside the weights because the KV cache is quantized (--cache-type-k/v). An fp16 KV cache at that size would not fit the card. The numbers are in the quant research doc.
• LLAMA_PARALLEL is the backend's number of slots. It's kept low on purpose: more slots meant more contended prefill, which pushed requests past the gateway's timeouts (gateway → Timeouts). Requests beyond the slot count wait in llama.cpp's own queue, which has no timeout. So the timeout that matters is the gateway's per-connection one.
• Per-slot context = LLAMA_CTX_SIZE / LLAMA_PARALLEL. llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions already fill a large share of it, so check that the per-slot figure stays comfortably above observed usage before lowering either value.
• --cache-reuse reuses cached KV for any matching prompt chunk instead of re-prefilling from scratch. That fixes the root cause behind the gateway's non-ping SSE timeout (research/omniroute-non-ping-sse-stream-timeout.md). It pairs with OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot so that there is a matching prefix to reuse.
• --jinja enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
• Tool-calling reliability of this model family on llama.cpp has open upstream parser bugs: research/qwen3.8-27b-tool-calling.md.
• No host port: see network access → Backends.

qwen-classifier

The classifier model's backend. It serves qwen-code's Auto Mode classifier (fastModel), which judges every tool call before it runs.

• Why a separate backend: when the classifier was an alias of the main model, every classification queued behind whatever heavy generation already held the main backend's slots, sometimes for minutes.
• Why GPU, not CPU: a CPU-only instance was tried first. Real classification calls carry a conversation transcript, and they ran past the gateway's timeout and retry-looped. It now runs on the GPU with a partial layer count (--n-gpu-layers), which leaves VRAM headroom for the main model.
• Context size: qwen-code hard-caps the classifier transcript in its own source (MAX_TRANSCRIPT_MESSAGES, MAX_HISTORICAL_ACTION_CHARS in packages/core/src/permissions/classifier-transcript.ts). --ctx-size gives margin over that worst case, and nothing more. The backend has a single slot, so its whole context is per-slot.
• VRAM: measured usage came out well above the weights+KV estimate. The cause was a missing --flash-attn: without it, the unfused attention buffers grow roughly quadratically with context. Batch size was the wrong lever. Check with rocm-smi after any change to the context size or the model.
• --reasoning off is a no-cost safety net against llama.cpp#20809. Some builds misdetect Qwen3-Instruct-2507 as a thinking model and route tool calls into reasoning_content. Nobody has confirmed whether the current image still needs it.
• Model choice: the smallest Qwen3 with long native context and no RoPE scaling. See research/fast-model-choice.md and research/classifier-context-window-scaling.md.
• Its provider connection needs specific gateway settings: see gateway → Provider connections.

omniroute

The gateway. Its runtime configuration (provider connections, API keys) lives in its own database and is managed through the dashboard, not in this repo: see gateway.md.

• OMNIROUTE_MEMORY_MB + mem_limit: the default heap is sized for dashboard-only use. OmniRoute's Docker guide recommends more for coding-agent traffic, and mem_limit is the matching container ceiling.
• stop_grace_period: SQLite needs time on shutdown to checkpoint its WAL back into the main database file. If this is shortened, recent writes (keys, provider connections) can be lost.
• Healthcheck is a TCP connect, not an HTTP probe. The image has only node (no curl, wget or python), and OmniRoute's Docker guide accepts a TCP probe as a liveness check.

qdrant

A vector store. It has no authentication, so its port is bound to the server's loopback interface only: see network access → qdrant.