haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm): make spec decoding and KV cache type .env toggles

Browse Source 
Move all speculative-decoding flags into LLAMA_SPEC_ARGS (blank = off)
and the KV cache quant into LLAMA_KV_CACHE_TYPE, so token-corruption
A/B tests are a single .env edit instead of a compose edit.
Replaces LLAMA_DRAFT_MODEL_FILE and LLAMA_SPEC_N_MAX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-10-31 18:22:44 +01:00
1 parent 7d142ed59d
commit 8927e440aa
4 files changed
+10 -10
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.env.example
+3 -2
View File
@@ -6,9 +6,10 @@ LLAMA_GPU_LAYERS=999
LLAMA_CTX_SIZE=262144
LLAMA_PARALLEL=2
LLAMA_CACHE_RAM=24576
LLAMA_KV_CACHE_TYPE=q8_0
LLAMA_DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
LLAMA_SPEC_N_MAX=7
# Blank = speculative decoding off; see docs/architecture.md#llama-server
LLAMA_SPEC_ARGS=--spec-type draft-dflash -md /models/Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-draft-n-max 7
# --- SearXNG web search --- see docs/gateway.md#web-search
# Auto-filled by update.sh — leave blank.