From 2585f3eaabdb86ae9b9c08f912f1d38d673a8798 Mon Sep 17 00:00:00 2001 From: Haylan Date: Fri, 6 Nov 2026 19:24:28 +0100 Subject: [PATCH] feat(llm): re-enable RAM prompt cache (8192 MiB) with --no-cache-idle-slots --no-cache-idle-slots is the llama.cpp#27148 mitigation for --kv-unified, which was the cause of misspelled output with --cache-ram on. Co-Authored-By: Claude Opus 5.5 --- .env.example | 2 +- docker-compose.yml | 3 ++- docs/architecture.md | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.env.example b/.env.example index 29f5115..55d6462 100644 --- a/.env.example +++ b/.env.example @@ -5,7 +5,7 @@ LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf LLAMA_GPU_LAYERS=999 LLAMA_CTX_SIZE=262144 LLAMA_PARALLEL=2 -LLAMA_CACHE_RAM=0 +LLAMA_CACHE_RAM=8192 LLAMA_KV_CACHE_TYPE=q8_0 # Blank = speculative decoding off; see docs/architecture.md#llama-server diff --git a/docker-compose.yml b/docker-compose.yml index ac56d74..4dca92d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -27,7 +27,8 @@ services: --flash-attn on --cache-type-k ${LLAMA_KV_CACHE_TYPE:-q8_0} --cache-type-v ${LLAMA_KV_CACHE_TYPE:-q8_0} - --cache-ram ${LLAMA_CACHE_RAM:-24576} + --cache-ram ${LLAMA_CACHE_RAM:-8192} + --no-cache-idle-slots ${LLAMA_SPEC_ARGS:-} --jinja # see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port) diff --git a/docs/architecture.md b/docs/architecture.md index 873555b..ff2a9a2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -26,7 +26,7 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it, - **Per-slot context = `LLAMA_CTX_SIZE / LLAMA_PARALLEL`.** llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions already fill a large share of it, so check that the per-slot figure stays comfortably above observed usage before lowering either value. - **No `--cache-reuse`:** this model's context doesn't support it. llama-server logs "cache_reuse is not supported by this context, it will be disabled" at startup, so the flag was dropped. Re-prefill avoidance comes from `--cache-ram` (below) and OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot. The earlier fix for the gateway's non-ping SSE timeout ([`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md)) assumed `--cache-reuse` worked, so long re-prefills can still hit that timeout. - **Testing for token corruption** (typos, broken tool-call keys): flip one key at a time in the server's `.env`, then run `docker compose up -d llama-server`. Blank `LLAMA_SPEC_ARGS` rules speculative decoding in or out, `LLAMA_KV_CACHE_TYPE=f16` does the same for the quantized KV cache (lower `LLAMA_CTX_SIZE` so it fits), and `LLAMA_CACHE_RAM=0` turns off the RAM prompt cache. A drafter's quant can't cause wrong tokens: the target model verifies every draft token, so a worse drafter only lowers acceptance. -- **`LLAMA_CACHE_RAM`** (`--cache-ram`, MiB) is the host-RAM budget for saved prompt-cache entries. When a slot is reused, llama.cpp parks the old prompt's state in RAM so it can be restored instead of re-prefilled. The llama.cpp default is too small for ~40k-token agent prompts (entries are ~2 GiB each), so entries got evicted and requests re-prefilled in full. Raising it is the only place spare DDR RAM helps; it does not speed up decode. Keep it well below the host's free RAM. Evidence and the A/B method: [`research/qwen3.8-27b-throughput-tuning.md`](research/qwen3.8-27b-throughput-tuning.md). +- **`LLAMA_CACHE_RAM`** (`--cache-ram`, MiB) is the host-RAM budget for saved prompt-cache entries. When a slot is reused, llama.cpp parks the old prompt's state in RAM so it can be restored instead of re-prefilled. The llama.cpp default is too small for ~40k-token agent prompts (entries are ~2 GiB each), so entries got evicted and requests re-prefilled in full. Raising it is the only place spare DDR RAM helps; it does not speed up decode. Keep it well below the host's free RAM. `--no-cache-idle-slots` stays on while [llama.cpp#27148](https://github.com/ggml-org/llama.cpp/issues/27148) is open: with the RAM cache on, an LRU-picked slot can keep an unrelated conversation's state, which showed up as misspelled output. With `--kv-unified`, turning off idle-slot caching closes that path and keeps the RAM cache ([diagnosis](research/inference-engine-spec-decoding-multi-gpu.md)). Evidence and the A/B method: [`research/qwen3.8-27b-throughput-tuning.md`](research/qwen3.8-27b-throughput-tuning.md). - **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright. - **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md). - **Speculative decoding (DFlash2):** all its flags live in one key, `LLAMA_SPEC_ARGS`: `--spec-type draft-dflash`, the drafter GGUF (`-md`, fetched by the downloader from [`models.yml`](../models.yml)) the draft length (`--spec-draft-n-max`), and `--spec-draft-p-min`, which ends a draft early once DFlash2's own confidence in the next token drops below the threshold. Long thinking output at the recommended sampling temperature gets low acceptance, so short confident drafts waste less verify work than long ones ([`common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), DFlash2 lattice walk). One key means a single `.env` edit swaps the drafter or turns speculation off (leave it blank) for an A/B test. A blank server value differs from `.env.example`, so `update.sh` will ask about it on every run until it's restored ([operations → Env sync](operations.md#env-sync)). It replaced MTP (draft heads inside the target GGUF). Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low.