feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier

fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was
aliased onto llama-server's own 27B connection, so every tool-call safety
check queued behind whatever heavy generation was already running on that
model's 2 GPU slots.

Add qwen-classifier: a separate llama.cpp instance, CPU-only, running
Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context,
qwen-code's requirement, without lossy RoPE scaling). Structurally isolated
from llama-server's queue instead of sharing it. Sized for gameserver's
~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M-
class weights (~2.3GiB) fits comfortably, with better KV quality than the
q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-09 16:26:51 +02:00
co-authored by Claude-Bot
parent 16df051318
commit 828bd4c046
3 changed files with 69 additions and 10 deletions
+1
View File
@@ -233,6 +233,7 @@ docker compose build --pull
echo "==> ensuring models are downloaded (skips already-present files)"
docker compose --profile tools run --rm downloader
docker compose --profile tools run --rm downloader-fast
docker compose --profile tools run --rm downloader-classifier
docker compose --profile tools run --rm downloader-comfyui
echo "==> bringing up omniroute"