haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm): raise prompt cache RAM budget via --cache-ram

Browse Source 
Log shows 253 prompt-cache evictions and 146 full ~42k re-prefills at the default budget.
Add LLAMA_CACHE_RAM and the throughput-tuning research note.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-10-20 21:05:27 +02:00
1 parent 433ebc6597
commit 408ed3e586
5 files changed
+144
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.env.example
+1
View File
@@ -5,6 +5,7 @@ LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
LLAMA_GPU_LAYERS=999
LLAMA_CTX_SIZE=262144
LLAMA_PARALLEL=2
LLAMA_CACHE_RAM=24576
LLAMA_DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
LLAMA_SPEC_N_MAX=7