feat: add llama-server-fast, a small non-thinking classifier model
Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for use as qwen-code CLI's Auto Mode classifier fastModel. Model choice researched in docs/research/fast-model-choice.md: architecturally non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added defensively per a known (closed) llama.cpp misdetection bug. - docker-compose.yml: llama-server-fast + downloader-fast services, omniroute depends_on updated - .env.example: LLAMA_FAST_* vars - scripts/update.sh: runs the new downloader profile Refs #44 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
This commit is contained in:
@@ -0,0 +1,221 @@
|
||||
# Which small model to run as the always-resident `fastModel` for qwen-code's Auto Mode classifier?
|
||||
|
||||
**Date:** 2026-09-06
|
||||
**Budget:** ≤7GB VRAM, resident concurrently alongside the existing Qwen3.8-27B instance on the single
|
||||
32GB R9700, via the same `llama.cpp:server-rocm` image already in `docker-compose.yml`.
|
||||
**Answer: Qwen3-4B-Instruct-2507, Q8_0 GGUF (~4.3GB weights).** The prior quick pass's tentative pick
|
||||
holds up under primary-source verification, for a more specific reason than "same tokenizer family":
|
||||
it is the only strong candidate in the shortlist that is *architecturally* non-thinking (no `<think>`
|
||||
code path exists at all, vs. models that are thinking-by-default and rely on a per-call
|
||||
`enable_thinking:false` toggle that llama.cpp does not cleanly expose). It does carry one directly
|
||||
relevant, documented llama.cpp bug — but that bug is closed, has a one-flag workaround, and is
|
||||
strictly less severe than the still-open Qwen3.5/Qwen3.8-lineage bugs already documented against the
|
||||
27B model in this repo.
|
||||
|
||||
## 1. What the classifier actually needs (grounding the requirement)
|
||||
|
||||
Per qwen-code's own docs, Auto Mode's permission gate is a two-stage LLM classifier:
|
||||
|
||||
- **Stage 1** — outputs only `{ shouldBlock: bool }`, ~300ms budget, thinking already disabled at the
|
||||
request level. If `shouldBlock` is `false`, the action proceeds immediately.
|
||||
- **Stage 2** — only runs when Stage 1 blocks; uses chain-of-thought review to downgrade false
|
||||
positives, ~3-5s budget.
|
||||
- Both stages use "your configured fast model (`/model --fast`)"; if none is configured, the full
|
||||
session model is used instead — which is the current, too-slow state this second model is meant to
|
||||
fix.
|
||||
|
||||
Source: [Qwen Code docs — Auto Mode](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/),
|
||||
[QwenLM/qwen-code docs/users/features/auto-mode.md](https://github.com/QwenLM/qwen-code/blob/main/docs/users/features/auto-mode.md).
|
||||
|
||||
A live qwen-code issue independently confirms the exact failure mode this repo already hit with the
|
||||
27B model — a model *thinking* inside the classifier path is a first-order latency problem, not a
|
||||
nice-to-have to tune later:
|
||||
|
||||
> "for a latency-sensitive permission gate, thinking should be disabled in every stage" — enabling it
|
||||
> "makes the review path slower and more expensive, which directly worsens the timeout problem."
|
||||
|
||||
That issue (timeouts tripping on slow inference) was closed by a PR that both loosened the stage
|
||||
timeout budgets *and* moved toward disabling thinking everywhere in the classifier.
|
||||
Source: [QwenLM/qwen-code issue #4676](https://github.com/QwenLM/qwen-code/issues/4676).
|
||||
|
||||
Takeaway for model selection: the request-level "don't think" instruction already exists in
|
||||
qwen-code's own classifier code. What matters is whether the **model + llama.cpp combination actually
|
||||
honors it reliably** — which is precisely where the 27B model failed (see
|
||||
[`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md)) and where several shortlist candidates
|
||||
have their own version of the same problem.
|
||||
|
||||
## 2. Candidates evaluated against primary sources
|
||||
|
||||
| Model | Params | GGUF size (quant) | Context | License | Thinking behavior | Tool-calling | Verdict |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **Qwen3-4B-Instruct-2507** | 4B | Q4_K_M 2.5GB / **Q8_0 4.28GB** | 262,144 native | Apache 2.0 | **Non-thinking only** — model card states it "does not generate `<think></think>` blocks in its output," full stop, no toggle needed | Yes, native `<tool_call>` format, BFCL-v3 61.9 | **Recommended** |
|
||||
| Qwen3-1.7B | 1.7B | ~1.1GB (Q4_K_M, typical) | 32,768 | Apache 2.0 | Thinking **on by default**; needs `enable_thinking:false` per call | Yes | Rejected — see §3 |
|
||||
| Qwen3-0.6B | 0.6B | ~0.4GB (Q4_K_M) | 32,768 | Apache 2.0 | Thinking on by default, same toggle issue as 1.7B | Yes, but weakest reasoning of the family | Rejected — undersized for reliability at this size, same toggle risk |
|
||||
| Llama-3.2-3B-Instruct | 3B | ~2GB (Q4_K_M, typical) | 128K | Llama 3.2 Community License — commercial use allowed, but text/EU carve-out language and an explicit >700M-MAU re-licensing clause | No thinking mode | Not natively documented on the model card fetched (no tool-call format called out) | Deprioritized — license has more fine print than Apache 2.0 for no clear benefit here |
|
||||
| Gemma-3-4b-it | 4B | Q4_K_M 2.49GB / Q8_0 4.13GB | 128K | Custom "Gemma" license (Google usage terms) | No documented thinking mode | Not documented on the model card fetched | Deprioritized — no confirmed native tool-calling story, non-Apache license |
|
||||
| Phi-4-mini-instruct | 3.8B | Q4_K_M 2.49GB / Q8_0 4.08GB | 128K | **MIT** | Not a reasoning model (that's the separate Phi-4-mini-**reasoning** model); no `<think>` tags by default | Yes — documented function-call format with dedicated tokens | Credible alternative — see §4 |
|
||||
| SmolLM3-3B | 3B | Q4_K_M ~1.9GB (typical) | 128K (64K trained + YaRN) | Apache 2.0 | **Thinking on by default** (`enable_thinking`), toggled via system-prompt flags | Yes (XML or Python-style tool calls) | Rejected — same thinking-by-default risk as Qwen3-1.7B |
|
||||
| Ministral-8B-Instruct-2410 | 8B | too large for budget at any useful quant with headroom | 128K | **Mistral Research License — commercial use requires contacting Mistral for a separate license** | Not documented as a reasoning model | Yes, documented function-calling with benchmark (31.6 vs Mistral-7B's 6.9) | Rejected — license restricts this repo's own dev-tooling use without contacting Mistral; also parameter count crowds the 7GB budget once Q8_0 + KV cache is counted |
|
||||
|
||||
Sources (fetched directly from each model's own HF card / GGUF repo unless noted):
|
||||
[Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507),
|
||||
[unsloth/Qwen3-4B-Instruct-2507-GGUF](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF),
|
||||
[Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B),
|
||||
[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B),
|
||||
[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct),
|
||||
[google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it),
|
||||
[bartowski/google_gemma-3-4b-it-GGUF](https://huggingface.co/bartowski/google_gemma-3-4b-it-GGUF),
|
||||
[microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct),
|
||||
[bartowski/microsoft_Phi-4-mini-instruct-GGUF](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF),
|
||||
[HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B),
|
||||
[mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410).
|
||||
|
||||
**Confidence note:** file sizes for Qwen3-1.7B/0.6B, Llama-3.2-3B, and SmolLM3-3B GGUF quants above are
|
||||
typical/approximate — those repos weren't individually re-verified against a specific GGUF file tree
|
||||
since all three families were eliminated on architectural grounds (§3) before size mattered. Sizes for
|
||||
the two models actually compared head-to-head (Qwen3-4B-Instruct-2507, Phi-4-mini-instruct) and
|
||||
Gemma-3-4b-it were pulled directly from each quantizer's own repo page.
|
||||
|
||||
## 3. Why "thinking-by-default + per-call toggle" is disqualifying, not just a minor ding
|
||||
|
||||
This is the deciding architectural distinction, and it's exactly the failure this second model exists
|
||||
to avoid. Qwen's own llama.cpp docs page states the toggle problem directly:
|
||||
|
||||
> "the hard switch implemented in the chat template is not exposed in llama.cpp" for controlling
|
||||
> `enable_thinking` — the documented workaround is to supply "a custom chat template equivalent to
|
||||
> always `enable_thinking=False`" via `--chat-template-file`.
|
||||
Source: [Qwen — Run with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html).
|
||||
|
||||
That means for Qwen3-1.7B, Qwen3-0.6B, and SmolLM3-3B — all thinking-on-by-default — reliably
|
||||
suppressing the reasoning phase in this llama.cpp/ROCm stack is not a request-body flag away; it needs
|
||||
a hand-maintained custom chat template file, which is exactly the kind of fragile, easy-to-silently-
|
||||
regress setup this task is trying to get away from (the 27B model's whole problem was reasoning_content
|
||||
being consumed before the answer). Qwen3-4B-Instruct-2507 has no such toggle to maintain in the first
|
||||
place — the model card states the non-thinking behavior as an unconditional property of the model, not
|
||||
a configurable default that has to be forced correctly on every request. This is a stronger claim than
|
||||
"same tokenizer family as the 27B" (the original quick-pass's reasoning) and is the actual basis for
|
||||
the recommendation.
|
||||
|
||||
## 4. The one documented risk specific to Qwen3-4B-Instruct-2507 — and why it doesn't change the pick
|
||||
|
||||
llama.cpp has its own closed, dated bug where server builds around **b8429** (March 2026)
|
||||
mis-detected Qwen3-Instruct-2507 models — the 4B included by name in the reporter's repro command — as
|
||||
thinking models, routing tool-call output into `reasoning_content` instead of `tool_calls`:
|
||||
|
||||
> "llama.cpp b8429 incorrectly detects Qwen3-Instruct-2507 models as thinking models (`thinking = 1`).
|
||||
> This causes tool calls to be captured as `reasoning_content` instead of being parsed into the
|
||||
> `tool_calls` array."
|
||||
|
||||
The documented, confirmed-working workaround is a single server flag:
|
||||
|
||||
```
|
||||
llama-server -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --jinja --port 8222 --reasoning off
|
||||
```
|
||||
|
||||
which restores `thinking = 0` and correct `finish_reason: tool_calls` output. The issue is **closed**.
|
||||
Source: [ggml-org/llama.cpp issue #20809](https://github.com/ggml-org/llama.cpp/issues/20809).
|
||||
|
||||
This is worth flagging honestly against the recommendation, but it's materially different from the
|
||||
open, only-partially-fixed Qwen3.5/Qwen3.8-lineage parser bugs already documented in this repo's
|
||||
[`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md) (issues #21158, #20837 — both open at time
|
||||
of that research): this is a llama.cpp *server-side misdetection* bug with a one-flag fix, not an
|
||||
unresolved upstream grammar/parser defect in the Qwen3.5 architecture family itself. Concretely: add
|
||||
`--reasoning off` to this second llama-server instance's command regardless — it's a no-cost safety net
|
||||
whether or not the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build still has the bug, and it
|
||||
directly targets the exact failure mode (reasoning_content eating the completion) that ruled out the
|
||||
27B model for this role in the first place.
|
||||
|
||||
## 5. VRAM math for the classifier role specifically
|
||||
|
||||
Qwen3-4B-Instruct-2507 is a plain (non-hybrid) transformer — every layer is standard GQA attention, so
|
||||
unlike the 27B model's Gated-DeltaNet hybrid, KV cache scales with *all* layers, not a fraction of them.
|
||||
From the model's own `config.json`:
|
||||
|
||||
- `num_hidden_layers`: 36, `num_key_value_heads`: 8, `head_dim`: 128
|
||||
Source: [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json).
|
||||
|
||||
Per-token KV cache (fp16, both K and V):
|
||||
`36 layers × 2 (K+V) × 8 kv_heads × 128 head_dim × 2 bytes = 144 KiB/token`
|
||||
|
||||
The classifier transcript is bounded by design — qwen-code's own two-stage design keeps Stage 1 to a
|
||||
`{shouldBlock}`-only judgment and Stage 2 to a chain-of-thought review of one blocked action, not an
|
||||
open-ended agent session — so a context window in the low thousands of tokens is generous headroom,
|
||||
not a tight fit:
|
||||
|
||||
| Context | KV cache (fp16) | KV cache (q8_0, `--cache-type-k/v q8_0`) | Weights (Q8_0) | Total (q8_0 KV) | Headroom under 7GB |
|
||||
|---|---|---|---|---|---|
|
||||
| 4,096 tokens | ~0.56 GB | ~0.28 GB | 4.28 GB | **~4.56 GB** | ~2.4 GB |
|
||||
| 8,192 tokens | ~1.13 GB | ~0.56 GB | 4.28 GB | **~4.84 GB** | ~2.2 GB |
|
||||
| 32,768 tokens (generous ceiling) | ~4.5 GB | ~2.25 GB | 4.28 GB | **~6.53 GB** | ~0.5 GB (tight) |
|
||||
|
||||
At any context length actually needed for a permission-gate classifier (thousands, not tens of
|
||||
thousands, of tokens), Q8_0 weights plus q8_0 KV cache comfortably clears the 7GB ceiling with headroom
|
||||
to spare for the compute buffer and batch overhead — matching the same `--cache-type-k q8_0
|
||||
--cache-type-v q8_0` pattern this repo already uses for the 27B instance. There's no need to drop to
|
||||
Q4_K_M (2.5GB) unless a much larger classifier context is anticipated later; Q8_0 is the better default
|
||||
here since it's a small model where quantization loss matters proportionally more, and the VRAM budget
|
||||
comfortably affords the higher-precision quant.
|
||||
|
||||
## 6. What would change the answer
|
||||
|
||||
- **If Phi-4-mini-instruct's MIT license matters more than matching the 27B model's tokenizer/template
|
||||
family**, it's a legitimate second choice: confirmed non-thinking by default, confirmed native
|
||||
function-calling format, comparable Q8_0 size (4.08GB), and a license with zero commercial-use fine
|
||||
print (vs. Apache 2.0's still-permissive but slightly more conditional terms). It wasn't picked
|
||||
because it has no llama.cpp-specific tool-calling track record verified in this pass (no equivalent
|
||||
to the issue #20809 workaround search done for it), so its actual reliability on this exact
|
||||
`llama.cpp:server-rocm` stack is less directly evidenced than Qwen3-4B-Instruct-2507's.
|
||||
- **If the classifier transcript ever needs to grow well past ~8K tokens routinely**, drop to Q4_K_M
|
||||
(2.5GB) to keep well clear of the 7GB ceiling — the KV-cache math in §5 shows the crossover point.
|
||||
- **If llama.cpp's #20809 misdetection turns out to still reproduce** on the exact
|
||||
`ghcr.io/ggml-org/llama.cpp:server-rocm` build this repo pulls, the fix is the one-flag
|
||||
`--reasoning off` workaround already confirmed in that issue — not a reason to pick a different
|
||||
model, since every thinking-capable alternative in this shortlist has an equal-or-worse version of
|
||||
the same class of bug with less clean workarounds (custom chat-template files, per §3).
|
||||
|
||||
## Sources
|
||||
|
||||
- [Qwen Code docs — Auto Mode](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/)
|
||||
- [QwenLM/qwen-code — docs/users/features/auto-mode.md](https://github.com/QwenLM/qwen-code/blob/main/docs/users/features/auto-mode.md)
|
||||
- [QwenLM/qwen-code issue #4676](https://github.com/QwenLM/qwen-code/issues/4676)
|
||||
- [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
|
||||
- [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
|
||||
- [unsloth/Qwen3-4B-Instruct-2507-GGUF](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF)
|
||||
- [Qwen — Run with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html)
|
||||
- [ggml-org/llama.cpp issue #20809](https://github.com/ggml-org/llama.cpp/issues/20809)
|
||||
- [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
|
||||
- [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
|
||||
- [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
|
||||
- [google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it)
|
||||
- [bartowski/google_gemma-3-4b-it-GGUF](https://huggingface.co/bartowski/google_gemma-3-4b-it-GGUF)
|
||||
- [microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct)
|
||||
- [bartowski/microsoft_Phi-4-mini-instruct-GGUF](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF)
|
||||
- [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B)
|
||||
- [mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410)
|
||||
- [docs/research/qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) (this repo — cross-referenced
|
||||
for the 27B model's own, still-open, tool-calling parser bugs)
|
||||
|
||||
## Confidence/uncertainty summary
|
||||
|
||||
- **High confidence:** Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote);
|
||||
qwen-code's two-stage classifier timing/design and its use of `/model --fast` (direct docs quote);
|
||||
the existence, exact symptom, and workaround of llama.cpp issue #20809 (direct issue quote); the
|
||||
KV-cache architecture math (computed directly from the model's own `config.json`, same method as the
|
||||
existing `qwen3.8-27b-quant.md` research in this repo).
|
||||
- **Medium confidence:** exact GGUF file sizes for Qwen3-1.7B, Qwen3-0.6B, Llama-3.2-3B-Instruct, and
|
||||
SmolLM3-3B — not individually re-verified against a specific quantizer's file tree since these were
|
||||
eliminated on architectural (thinking-toggle) grounds before size became the deciding factor; treat
|
||||
as typical/approximate, not exact.
|
||||
- **Low confidence / not independently verified:** whether the current
|
||||
`ghcr.io/ggml-org/llama.cpp:server-rocm` image (pulled fresh) still reproduces issue #20809's
|
||||
misdetection — the issue is closed but no changelog/PR diff was fetched to confirm the underlying
|
||||
detection logic was actually patched vs. the reporter simply adopting the `--reasoning off`
|
||||
workaround. Recommend a live smoke test (send one tool-calling request, confirm the response lands in
|
||||
`tool_calls` not `reasoning_content`, and time a trivial completion) before wiring this model in as
|
||||
the production `fastModel`, the same caveat this repo's `qwen3.8-27b-tool-calling.md` already flags
|
||||
for the 27B model.
|
||||
- Whether Phi-4-mini-instruct has a comparably clean llama.cpp tool-calling track record was **not**
|
||||
deep-dived (no issue-tracker search run against it) — it's flagged in §6 as a live alternative rather
|
||||
than fully evaluated, since Qwen3-4B-Instruct-2507's architectural non-thinking guarantee and
|
||||
same-family template consistency with the existing 27B deployment made it the clearer pick without
|
||||
needing that extra research pass.
|
||||
Reference in New Issue
Block a user