feat(llm): raise prompt cache RAM budget via --cache-ram

Log shows 253 prompt-cache evictions and 146 full ~42k re-prefills at the default budget.
Add LLAMA_CACHE_RAM and the throughput-tuning research note.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-10-20 21:05:27 +02:00
1 parent 433ebc6597
commit 408ed3e586
5 files changed
+144

No files matched your search

+1
View File
@@ -28,6 +28,7 @@ services:
--cache-type-k q8_0
--cache-type-v q8_0
--cache-reuse 256
--cache-ram ${LLAMA_CACHE_RAM:-24576}
--spec-type draft-dflash
-md /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
--spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}