feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier

fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was
aliased onto llama-server's own 27B connection, so every tool-call safety
check queued behind whatever heavy generation was already running on that
model's 2 GPU slots.

Add qwen-classifier: a separate llama.cpp instance, CPU-only, running
Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context,
qwen-code's requirement, without lossy RoPE scaling). Structurally isolated
from llama-server's queue instead of sharing it. Sized for gameserver's
~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M-
class weights (~2.3GiB) fits comfortably, with better KV quality than the
q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-09 16:26:51 +02:00
co-authored by Claude-Bot
parent 16df051318
commit 828bd4c046
3 changed files with 69 additions and 10 deletions
+49
View File
@@ -46,6 +46,39 @@ services:
- "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
- "lazytainer.group.llamaserver.minPacketThreshold=2"
# Dedicated backend for qwen-code's tool-call harmfulness classifier
# (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
# 27B connection — every classification call then queued behind whatever
# heavy generation was already running on that model's 2 GPU slots (issue
# tracker: OmniRoute semaphore/pr-agent investigation). CPU-only, own
# process, own queue: structurally can't contend with llama-server for a
# GPU slot. Needs >=131072 ctx (qwen-code requirement); Qwen3-4B-Instruct-2507
# is the smallest Qwen3 that supports that natively (262144) without
# RoPE-scaling — the smaller 0.6B/1.7B/4B (non-2507) models only go to
# 40960. Sized for gameserver's ~17GiB free RAM: q8_0/q8_0 KV at full
# 131072 ctx is ~9.8GiB + ~2.3GiB Q4_K_M-class weights ≈ 12GiB, comfortable
# headroom, and better KV quality than the q4_0 that would've been needed
# to squeeze this onto the GPU's ~6GiB free VRAM instead.
qwen-classifier:
image: ghcr.io/ggml-org/llama.cpp:server
container_name: qwen-classifier
volumes:
- models:/models
command: >
-m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
--host 0.0.0.0
--port 8080
--n-gpu-layers 0
--ctx-size 131072
--parallel 1
--cache-type-k q8_0
--cache-type-v q8_0
--jinja
expose:
- "8080"
restart: unless-stopped
networks: [ai-stack]
# ponytail: one-off downloader, not a standing service — run via
# `docker compose --profile tools run --rm downloader`. Folded into
# scripts/update.sh, which runs this every time; the `test -f` guard is
@@ -67,6 +100,22 @@ services:
curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
# Same pattern as downloader above, separate service so this one small
# file doesn't get re-checked/re-pulled by the big model's job.
downloader-classifier:
image: curlimages/curl:latest
profiles: ["tools"]
user: root
volumes:
- models:/models
entrypoint: ["sh", "-c"]
command:
- >
test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
echo "already downloaded, skipping" ||
curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
# Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
# text encoder, VAE) — same test -f guard pattern as downloader above.
# See docs/research/image-generation-model-choice.md and issue #42.