fix(llama-server): quantize KV cache, restore full 262144 context
A real session hit 'request (66192 tokens) exceeds the available context size (65536 tokens)' — --parallel 2 was splitting the 131072 total into 65536/slot, too small for actual usage. Add --cache-type-k/v q8_0 (roughly halves KV memory) so the model's true max context (262144, already the documented .env.example goal) fits in the same ~25.6GB/6GB-headroom footprint the old 131072 fp16 setting used, instead of shrinking per-slot context to fit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+3
-1
@@ -18,9 +18,11 @@ services:
|
||||
--host 0.0.0.0
|
||||
--port 8080
|
||||
--n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
|
||||
--ctx-size ${LLAMA_CTX_SIZE:-131072}
|
||||
--ctx-size ${LLAMA_CTX_SIZE:-262144}
|
||||
--parallel ${LLAMA_PARALLEL:-2}
|
||||
--flash-attn on
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--jinja
|
||||
# No published host port: llama-server is reached only via the omniroute
|
||||
# gateway on the ai-stack docker network now — see issue #15. Its
|
||||
|
||||
Reference in New Issue
Block a user