feat(llm): make spec decoding and KV cache type .env toggles
Move all speculative-decoding flags into LLAMA_SPEC_ARGS (blank = off) and the KV cache quant into LLAMA_KV_CACHE_TYPE, so token-corruption A/B tests are a single .env edit instead of a compose edit. Replaces LLAMA_DRAFT_MODEL_FILE and LLAMA_SPEC_N_MAX. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7d142ed59d
commit
8927e440aa
4 files changed
+10
-10
No files matched your search
+3
-5
@@ -25,12 +25,10 @@ services:
|
||||
--parallel ${LLAMA_PARALLEL:-2}
|
||||
--kv-unified
|
||||
--flash-attn on
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--cache-type-k ${LLAMA_KV_CACHE_TYPE:-q8_0}
|
||||
--cache-type-v ${LLAMA_KV_CACHE_TYPE:-q8_0}
|
||||
--cache-ram ${LLAMA_CACHE_RAM:-24576}
|
||||
--spec-type draft-dflash
|
||||
-md /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q8_0.gguf}
|
||||
--spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}
|
||||
${LLAMA_SPEC_ARGS:-}
|
||||
--jinja
|
||||
# see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port)
|
||||
expose:
|
||||
|
||||
Reference in new issue
Block a user