fix(llama-server): quantize KV cache, restore full 262144 context

A real session hit 'request (66192 tokens) exceeds the available
context size (65536 tokens)' — --parallel 2 was splitting the
131072 total into 65536/slot, too small for actual usage.

Add --cache-type-k/v q8_0 (roughly halves KV memory) so the model's
true max context (262144, already the documented .env.example goal)
fits in the same ~25.6GB/6GB-headroom footprint the old 131072 fp16
setting used, instead of shrinking per-slot context to fit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 12:00:07 +02:00
co-authored by Claude-Bot
parent 5767f548c3
commit d8736b6dd7
2 changed files with 11 additions and 10 deletions
+3 -1
View File
@@ -18,9 +18,11 @@ services:
--host 0.0.0.0
--port 8080
--n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
--ctx-size ${LLAMA_CTX_SIZE:-131072}
--ctx-size ${LLAMA_CTX_SIZE:-262144}
--parallel ${LLAMA_PARALLEL:-2}
--flash-attn on
--cache-type-k q8_0
--cache-type-v q8_0
--jinja
# No published host port: llama-server is reached only via the omniroute
# gateway on the ai-stack docker network now — see issue #15. Its