feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier

fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was
aliased onto llama-server's own 27B connection, so every tool-call safety
check queued behind whatever heavy generation was already running on that
model's 2 GPU slots.

Add qwen-classifier: a separate llama.cpp instance, CPU-only, running
Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context,
qwen-code's requirement, without lossy RoPE scaling). Structurally isolated
from llama-server's queue instead of sharing it. Sized for gameserver's
~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M-
class weights (~2.3GiB) fits comfortably, with better KV quality than the
q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-09 16:26:51 +02:00
co-authored by Claude-Bot
parent 16df051318
commit 828bd4c046
3 changed files with 69 additions and 10 deletions
+19 -10
View File
@@ -29,18 +29,27 @@ LLAMA_GPU_LAYERS=999
# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
LLAMA_CTX_SIZE=262144
# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
# default) with no flag set — under concurrent subagent fan-out, 4 requests
# split the same GPU compute, so a large-context prefill can queue behind
# others long enough to blow past OmniRoute's stream-idle timeout, which then
# cancels the request (see issue-tracker notes on the timeout/cancel loop).
# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
# raise back toward 4 if throughput (not latency) becomes the bottleneck
# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
# checking that per-slot number stays comfortably above observed usage.
# Concurrent request slots — the real hardware ceiling for this GPU, not a
# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
# default; dropped to 2 because more contended prefill was blowing requests
# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
# the timeout that matters moved to OmniRoute's per-connection
# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
# comfortably above observed usage.
LLAMA_PARALLEL=2
# Dedicated CPU-only backend for qwen-code's tool-call harmfulness classifier
# (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
# qwen-classifier service comment for the why. 131072 ctx / q8_0 KV / Q4_K_M-
# class weights ≈ 12GiB, fits gameserver's ~17GiB free system RAM with room
# to spare.
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
# --- Lazytainer ---
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
LAZYTAINER_INACTIVE_TIMEOUT=900