docs(llm): document qwen-classifier reality, add --reasoning off safety net

- docker-compose.yml: add --reasoning off to qwen-classifier per
  fast-model-choice.md's own recommendation (ggml-org/llama.cpp#20809
  safety net) — missed in the original rollout, caught while writing
  this up.
- docs/coding-cli-setup/qwen-code.md: rewritten to match what's actually
  deployed (qwen-classifier, partial GPU offload, 65536 ctx, Q4_K_XL) —
  previously described an unimplemented llama-server-fast/8192-ctx plan.
  Documents the non-interactive MCP tool allow-list gap found live-testing.
- docs/coding-cli-setup/opencode.md: fix stale 65536 example that didn't
  match its own documented LLAMA_CTX_SIZE/LLAMA_PARALLEL formula (131072).
- docs/research/fast-model-choice.md: implementation note recording where
  the actual rollout diverged from this doc's original recommendations
  (service name, quant, context size, CPU-first-then-GPU path).
- docs/research/omniroute-account-semaphore-timeout.md: new — the
  hardcoded 30s per-connection semaphore timeout found during the
  pr-agent investigation, root-caused against OmniRoute's own source,
  and the maxConcurrent:null + providerSpecificData.timeoutMs fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-09 19:10:32 +02:00
co-authored by Claude-Bot
parent 76043e2c6f
commit 930e407053
6 changed files with 153 additions and 13 deletions
+33
View File
@@ -195,6 +195,39 @@ comfortably affords the higher-precision quant.
- [docs/research/qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) (this repo — cross-referenced
for the 27B model's own, still-open, tool-calling parser bugs)
## Implementation note (2026-09-09) — what actually shipped, and why it differs
The model pick (`Qwen3-4B-Instruct-2507`) held up and is what's deployed. Several sizing assumptions in
this doc didn't survive contact with the real deployment, though — worth recording so the next person
tuning this doesn't re-derive the same corrections from scratch:
- **Service name is `qwen-classifier`, not `llama-server-fast`** — this doc's proposed name never got
used. There's no `LLAMA_FAST_CTX_SIZE`/`LLAMA_FAST_PARALLEL` in `.env.example` either; the real config
lives inline in `docker-compose.yml`'s `qwen-classifier` command.
- **CPU-only was tried first and rejected** — this doc's VRAM budget analysis (§5) assumed GPU
residency from the start, but the actual rollout path tried CPU-only first (to sidestep VRAM
contention entirely) and found it too slow: real classification calls blew past OmniRoute's request
timeout and retry-looped. Moved to GPU after that, which is what §5's math was for all along.
- **Q4_K_XL weights, not Q8_0** — §5's "~2.4GB headroom" case assumed Q8_0 (4.28GB). In practice, fitting
the classifier onto the R9700 *alongside* the 27B model (not in an assumed-empty 7GB budget) left only
~6.1GB free VRAM total, and even Q4_K_XL (2.37GB) plus full-context KV cache didn't leave enough real
margin at full GPU offload — see the "measured live" numbers in `docker-compose.yml`'s `qwen-classifier`
comment block. Landed on **partial GPU offload (28/36 layers)** instead of full offload, which is not a
case this doc considered at all.
- **65536 context, not 8192** — §5 sized the context "in the low thousands," reasoning from qwen-code's
two-stage classifier description alone. Directly reading qwen-code's actual source
(`packages/core/src/permissions/classifier-transcript.ts`: `MAX_TRANSCRIPT_MESSAGES=40`,
`MAX_HISTORICAL_ACTION_CHARS=4000`/message) puts the real worst case at ~40-50K tokens — confirmed
live, a real classifier call during testing hit 15,116 prompt tokens. 8192 would have been undersized
for real usage; 65536 gives margin without the original setting.json value (131072, copied from the
main model's entry, not a real qwen-code requirement) wasting VRAM for no reason.
- **§4's `--reasoning off` recommendation was initially missed** in the first deployment pass and added
only once this doc was re-read while writing this note. It's now in `docker-compose.yml`'s
`qwen-classifier` command, per this doc's own "add it regardless, no-cost safety net" reasoning — still
unconfirmed whether the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build actually reproduces
#20809 (nothing in testing so far surfaced `reasoning_content` where `tool_calls` was expected, but
that wasn't specifically probed for either).
## Confidence/uncertainty summary
- **High confidence:** Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote);