feat(llm): raise prompt cache RAM budget via --cache-ram
Log shows 253 prompt-cache evictions and 146 full ~42k re-prefills at the default budget. Add LLAMA_CACHE_RAM and the throughput-tuning research note. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
433ebc6597
commit
408ed3e586
5 files changed
+144
No files matched your search
@@ -28,6 +28,7 @@ services:
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--cache-reuse 256
|
||||
--cache-ram ${LLAMA_CACHE_RAM:-24576}
|
||||
--spec-type draft-dflash
|
||||
-md /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
|
||||
--spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}
|
||||
|
||||
Reference in new issue
Block a user