feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier
fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was aliased onto llama-server's own 27B connection, so every tool-call safety check queued behind whatever heavy generation was already running on that model's 2 GPU slots. Add qwen-classifier: a separate llama.cpp instance, CPU-only, running Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context, qwen-code's requirement, without lossy RoPE scaling). Structurally isolated from llama-server's queue instead of sharing it. Sized for gameserver's ~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M- class weights (~2.3GiB) fits comfortably, with better KV quality than the q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -46,6 +46,39 @@ services:
|
||||
- "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
|
||||
- "lazytainer.group.llamaserver.minPacketThreshold=2"
|
||||
|
||||
# Dedicated backend for qwen-code's tool-call harmfulness classifier
|
||||
# (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
|
||||
# 27B connection — every classification call then queued behind whatever
|
||||
# heavy generation was already running on that model's 2 GPU slots (issue
|
||||
# tracker: OmniRoute semaphore/pr-agent investigation). CPU-only, own
|
||||
# process, own queue: structurally can't contend with llama-server for a
|
||||
# GPU slot. Needs >=131072 ctx (qwen-code requirement); Qwen3-4B-Instruct-2507
|
||||
# is the smallest Qwen3 that supports that natively (262144) without
|
||||
# RoPE-scaling — the smaller 0.6B/1.7B/4B (non-2507) models only go to
|
||||
# 40960. Sized for gameserver's ~17GiB free RAM: q8_0/q8_0 KV at full
|
||||
# 131072 ctx is ~9.8GiB + ~2.3GiB Q4_K_M-class weights ≈ 12GiB, comfortable
|
||||
# headroom, and better KV quality than the q4_0 that would've been needed
|
||||
# to squeeze this onto the GPU's ~6GiB free VRAM instead.
|
||||
qwen-classifier:
|
||||
image: ghcr.io/ggml-org/llama.cpp:server
|
||||
container_name: qwen-classifier
|
||||
volumes:
|
||||
- models:/models
|
||||
command: >
|
||||
-m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||
--host 0.0.0.0
|
||||
--port 8080
|
||||
--n-gpu-layers 0
|
||||
--ctx-size 131072
|
||||
--parallel 1
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--jinja
|
||||
expose:
|
||||
- "8080"
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
# ponytail: one-off downloader, not a standing service — run via
|
||||
# `docker compose --profile tools run --rm downloader`. Folded into
|
||||
# scripts/update.sh, which runs this every time; the `test -f` guard is
|
||||
@@ -67,6 +100,22 @@ services:
|
||||
curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
|
||||
# Same pattern as downloader above, separate service so this one small
|
||||
# file doesn't get re-checked/re-pulled by the big model's job.
|
||||
downloader-classifier:
|
||||
image: curlimages/curl:latest
|
||||
profiles: ["tools"]
|
||||
user: root
|
||||
volumes:
|
||||
- models:/models
|
||||
entrypoint: ["sh", "-c"]
|
||||
command:
|
||||
- >
|
||||
test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
|
||||
echo "already downloaded, skipping" ||
|
||||
curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||
https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||
|
||||
# Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
|
||||
# text encoder, VAE) — same test -f guard pattern as downloader above.
|
||||
# See docs/research/image-generation-model-choice.md and issue #42.
|
||||
|
||||
Reference in New Issue
Block a user