feat(llm): switch to DFlash2 speculative decoding, drop qwen-classifier
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1e3ad169c0
commit
17747694bd
10 files changed
+30
-88
No files matched your search
+2
-2
@@ -6,8 +6,8 @@ LLAMA_GPU_LAYERS=999
|
||||
LLAMA_CTX_SIZE=262144
|
||||
LLAMA_PARALLEL=2
|
||||
|
||||
# see docs/architecture.md#qwen-classifier
|
||||
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
|
||||
LLAMA_DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
LLAMA_SPEC_N_MAX=7
|
||||
|
||||
# --- SearXNG web search --- see docs/gateway.md#web-search
|
||||
# Auto-filled by update.sh — leave blank.
|
||||
|
||||
Reference in new issue
Block a user