feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier
fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was aliased onto llama-server's own 27B connection, so every tool-call safety check queued behind whatever heavy generation was already running on that model's 2 GPU slots. Add qwen-classifier: a separate llama.cpp instance, CPU-only, running Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context, qwen-code's requirement, without lossy RoPE scaling). Structurally isolated from llama-server's queue instead of sharing it. Sized for gameserver's ~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M- class weights (~2.3GiB) fits comfortably, with better KV quality than the q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -233,6 +233,7 @@ docker compose build --pull
|
||||
echo "==> ensuring models are downloaded (skips already-present files)"
|
||||
docker compose --profile tools run --rm downloader
|
||||
docker compose --profile tools run --rm downloader-fast
|
||||
docker compose --profile tools run --rm downloader-classifier
|
||||
docker compose --profile tools run --rm downloader-comfyui
|
||||
|
||||
echo "==> bringing up omniroute"
|
||||
|
||||
Reference in New Issue
Block a user