haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(llm): move qwen-classifier to GPU, right-size context

Browse Source 
CPU-only was too slow in practice: real classification calls blew past
OmniRoute's 60s timeout and retry-looped (504/499). Moved to GPU.

Also traced qwen-code's actual classifier transcript cap in its source
(MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000/message) —
worst case is ~40-50K tokens, not the 131072 originally set in
settings.json (copied from the main model's entry, not a real qwen-code
requirement). Dropped ctx-size to 65536 (~1.5x margin) so Q4_K_XL
weights + q4_0/q4_0 KV fit fully on GPU (~4.9GiB) inside the ~6.1GiB
free on the R9700, instead of needing partial CPU/GPU offload.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-09-09 16:59:02 +02:00
co-authored by Claude-Bot
parent 4353e5c0e8
commit 8e2650807c
2 changed files with 41 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+4 -4
View File
@@ -43,10 +43,10 @@ LLAMA_CTX_SIZE=262144
# comfortably above observed usage.
LLAMA_PARALLEL=2
# Dedicated CPU-only backend for qwen-code's tool-call harmfulness classifier
# (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
# qwen-classifier service comment for the why and the RAM math.
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
# Dedicated GPU-resident backend for qwen-code's tool-call harmfulness
# classifier (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
# qwen-classifier service comment for the why and the VRAM/context math.
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
# --- Lazytainer ---
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.