feat(llm): re-enable RAM prompt cache (8192 MiB) with --no-cache-idle-slots
--no-cache-idle-slots is the llama.cpp#27148 mitigation for --kv-unified, which was the cause of misspelled output with --cache-ram on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
51684fb7a4
commit
2585f3eaab
3 files changed
+4
-3
No files matched your search
+2
-1
@@ -27,7 +27,8 @@ services:
|
||||
--flash-attn on
|
||||
--cache-type-k ${LLAMA_KV_CACHE_TYPE:-q8_0}
|
||||
--cache-type-v ${LLAMA_KV_CACHE_TYPE:-q8_0}
|
||||
--cache-ram ${LLAMA_CACHE_RAM:-24576}
|
||||
--cache-ram ${LLAMA_CACHE_RAM:-8192}
|
||||
--no-cache-idle-slots
|
||||
${LLAMA_SPEC_ARGS:-}
|
||||
--jinja
|
||||
# see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port)
|
||||
|
||||
Reference in new issue
Block a user