feat(llama.cpp): raise default context to 128K, document RAM/SSD offload knobs
LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700, ~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table. Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/ --cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that there's no explicit SSD offload tier to enable — llama.cpp's default mmap already falls back to disk implicitly if GPU+RAM run out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
This commit is contained in:
+1
-1
@@ -18,7 +18,7 @@ services:
|
||||
--host 0.0.0.0
|
||||
--port 8080
|
||||
--n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
|
||||
--ctx-size ${LLAMA_CTX_SIZE:-65536}
|
||||
--ctx-size ${LLAMA_CTX_SIZE:-131072}
|
||||
--jinja
|
||||
# No published host port: llama-server is reached only via the litellm
|
||||
# proxy on the ai-stack docker network now — see issue #15. Its
|
||||
|
||||
Reference in New Issue
Block a user