feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier
fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was aliased onto llama-server's own 27B connection, so every tool-call safety check queued behind whatever heavy generation was already running on that model's 2 GPU slots. Add qwen-classifier: a separate llama.cpp instance, CPU-only, running Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context, qwen-code's requirement, without lossy RoPE scaling). Structurally isolated from llama-server's queue instead of sharing it. Sized for gameserver's ~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M- class weights (~2.3GiB) fits comfortably, with better KV quality than the q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+19
-10
@@ -29,18 +29,27 @@ LLAMA_GPU_LAYERS=999
|
||||
# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
|
||||
# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
|
||||
LLAMA_CTX_SIZE=262144
|
||||
# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
|
||||
# default) with no flag set — under concurrent subagent fan-out, 4 requests
|
||||
# split the same GPU compute, so a large-context prefill can queue behind
|
||||
# others long enough to blow past OmniRoute's stream-idle timeout, which then
|
||||
# cancels the request (see issue-tracker notes on the timeout/cancel loop).
|
||||
# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
|
||||
# raise back toward 4 if throughput (not latency) becomes the bottleneck
|
||||
# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
|
||||
# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
|
||||
# checking that per-slot number stays comfortably above observed usage.
|
||||
# Concurrent request slots — the real hardware ceiling for this GPU, not a
|
||||
# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
|
||||
# default; dropped to 2 because more contended prefill was blowing requests
|
||||
# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
|
||||
# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
|
||||
# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
|
||||
# the timeout that matters moved to OmniRoute's per-connection
|
||||
# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
|
||||
# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
|
||||
# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
|
||||
# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
|
||||
# comfortably above observed usage.
|
||||
LLAMA_PARALLEL=2
|
||||
|
||||
# Dedicated CPU-only backend for qwen-code's tool-call harmfulness classifier
|
||||
# (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
|
||||
# qwen-classifier service comment for the why. 131072 ctx / q8_0 KV / Q4_K_M-
|
||||
# class weights ≈ 12GiB, fits gameserver's ~17GiB free system RAM with room
|
||||
# to spare.
|
||||
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
|
||||
|
||||
# --- Lazytainer ---
|
||||
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
||||
LAZYTAINER_INACTIVE_TIMEOUT=900
|
||||
|
||||
Reference in New Issue
Block a user