chore: move code comments into docs, leave one-line pointers

Explanations from docker-compose.yml, .env.example and
scripts/update.sh now live in docs/architecture.md, gateway.md,
network-access.md and operations.md. Code keeps `# see docs/...`
pointers, one-line warnings (storage key, SSRF guard, WAL grace period,
git pull ordering) and "auto-filled — leave blank" markers.

Refs #55 (comment destination map #61)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-09-24 21:52:00 +02:00
1 parent a0e5e9704b
commit 05effe5c88
4 files changed
+31 -265

No files matched your search

+1 -1
View File
@@ -23,7 +23,7 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it,
- **`LLAMA_GPU_LAYERS`**: the default puts every layer on the GPU. The model is dense, not Mixture-of-Experts, so this is the only RAM-offload knob. `--n-cpu-moe`, `--cpu-moe` and `--override-tensor "exps"` do nothing here. There is no SSD tier to enable either. llama.cpp mmaps the model file, so if GPU and RAM ever can't hold it, the OS pages the rest in from disk as a slow, implicit last resort. Explicit tiered offload is an [open llama.cpp request](https://github.com/ggml-org/llama.cpp/discussions/12507).
- **`LLAMA_CTX_SIZE`**: set to the model's own maximum (`max_position_embeddings`). It only fits beside the weights because the KV cache is quantized (`--cache-type-k/v`). An fp16 KV cache at that size would not fit the card. The numbers are in the quant research doc.
- **`LLAMA_PARALLEL`** is the backend's number of **slots**. It's kept low on purpose: more slots meant more contended prefill, which pushed requests past the gateway's timeouts ([gateway → Timeouts](gateway.md#timeouts)). Requests beyond the slot count wait in llama.cpp's own queue, which has no timeout. So the timeout that matters is the gateway's per-connection one.
- **Per-slot context = `LLAMA_CTX_SIZE / LLAMA_PARALLEL`.** llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions have come close to it, so check that the per-slot figure stays above observed usage before lowering either value.
- **Per-slot context = `LLAMA_CTX_SIZE / LLAMA_PARALLEL`.** llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions already fill a large share of it, so check that the per-slot figure stays comfortably above observed usage before lowering either value.
- **`--cache-reuse`** reuses cached KV for any matching prompt chunk instead of re-prefilling from scratch. That fixes the root cause behind the gateway's non-ping SSE timeout ([`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md)). It pairs with OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot so that there is a matching prefix to reuse.
- **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
- **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).