LLM-Server/docs/research/fast-model-choice.md

Which small model to run as the always-resident fastModel for qwen-code's Auto Mode classifier?

Date: 2026-09-06 Budget: ≤7GB VRAM, resident concurrently alongside the existing Qwen3.8-27B instance on the single 32GB R9700, via the same llama.cpp:server-rocm image already in docker-compose.yml. Answer: Qwen3-4B-Instruct-2507, Q8_0 GGUF (~4.3GB weights). The prior quick pass's tentative pick holds up under primary-source verification, for a more specific reason than "same tokenizer family": it is the only strong candidate in the shortlist that is architecturally non-thinking (no <think> code path exists at all, vs. models that are thinking-by-default and rely on a per-call enable_thinking:false toggle that llama.cpp does not cleanly expose). It does carry one directly relevant, documented llama.cpp bug — but that bug is closed, has a one-flag workaround, and is strictly less severe than the still-open Qwen3.5/Qwen3.8-lineage bugs already documented against the 27B model in this repo.

1. What the classifier actually needs (grounding the requirement)

Per qwen-code's own docs, Auto Mode's permission gate is a two-stage LLM classifier:

• Stage 1 — outputs only { shouldBlock: bool }, ~300ms budget, thinking already disabled at the request level. If shouldBlock is false, the action proceeds immediately.
• Stage 2 — only runs when Stage 1 blocks; uses chain-of-thought review to downgrade false positives, ~3-5s budget.
• Both stages use "your configured fast model (/model --fast)"; if none is configured, the full session model is used instead — which is the current, too-slow state this second model is meant to fix.

Source: Qwen Code docs — Auto Mode, QwenLM/qwen-code docs/users/features/auto-mode.md.

A live qwen-code issue independently confirms the exact failure mode this repo already hit with the 27B model — a model thinking inside the classifier path is a first-order latency problem, not a nice-to-have to tune later:

"for a latency-sensitive permission gate, thinking should be disabled in every stage" — enabling it "makes the review path slower and more expensive, which directly worsens the timeout problem."

That issue (timeouts tripping on slow inference) was closed by a PR that both loosened the stage timeout budgets and moved toward disabling thinking everywhere in the classifier. Source: QwenLM/qwen-code issue #4676.

Takeaway for model selection: the request-level "don't think" instruction already exists in qwen-code's own classifier code. What matters is whether the model + llama.cpp combination actually honors it reliably — which is precisely where the 27B model failed (see qwen3.8-27b-tool-calling.md) and where several shortlist candidates have their own version of the same problem.

2. Candidates evaluated against primary sources

Model	Params	GGUF size (quant)	Context	License	Thinking behavior	Tool-calling	Verdict
Qwen3-4B-Instruct-2507	4B	Q4_K_M 2.5GB / Q8_0 4.28GB	262,144 native	Apache 2.0	Non-thinking only — model card states it "does not generate <think></think> blocks in its output," full stop, no toggle needed	Yes, native <tool_call> format, BFCL-v3 61.9	Recommended
Qwen3-1.7B	1.7B	~1.1GB (Q4_K_M, typical)	32,768	Apache 2.0	Thinking on by default; needs enable_thinking:false per call	Yes	Rejected — see §3
Qwen3-0.6B	0.6B	~0.4GB (Q4_K_M)	32,768	Apache 2.0	Thinking on by default, same toggle issue as 1.7B	Yes, but weakest reasoning of the family	Rejected — undersized for reliability at this size, same toggle risk
Llama-3.2-3B-Instruct	3B	~2GB (Q4_K_M, typical)	128K	Llama 3.2 Community License — commercial use allowed, but text/EU carve-out language and an explicit >700M-MAU re-licensing clause	No thinking mode	Not natively documented on the model card fetched (no tool-call format called out)	Deprioritized — license has more fine print than Apache 2.0 for no clear benefit here
Gemma-3-4b-it	4B	Q4_K_M 2.49GB / Q8_0 4.13GB	128K	Custom "Gemma" license (Google usage terms)	No documented thinking mode	Not documented on the model card fetched	Deprioritized — no confirmed native tool-calling story, non-Apache license
Phi-4-mini-instruct	3.8B	Q4_K_M 2.49GB / Q8_0 4.08GB	128K	MIT	Not a reasoning model (that's the separate Phi-4-mini-reasoning model); no <think> tags by default	Yes — documented function-call format with dedicated tokens	Credible alternative — see §4
SmolLM3-3B	3B	Q4_K_M ~1.9GB (typical)	128K (64K trained + YaRN)	Apache 2.0	Thinking on by default (enable_thinking), toggled via system-prompt flags	Yes (XML or Python-style tool calls)	Rejected — same thinking-by-default risk as Qwen3-1.7B
Ministral-8B-Instruct-2410	8B	too large for budget at any useful quant with headroom	128K	Mistral Research License — commercial use requires contacting Mistral for a separate license	Not documented as a reasoning model	Yes, documented function-calling with benchmark (31.6 vs Mistral-7B's 6.9)	Rejected — license restricts this repo's own dev-tooling use without contacting Mistral; also parameter count crowds the 7GB budget once Q8_0 + KV cache is counted

Sources (fetched directly from each model's own HF card / GGUF repo unless noted): Qwen/Qwen3-4B-Instruct-2507, unsloth/Qwen3-4B-Instruct-2507-GGUF, Qwen/Qwen3-1.7B, Qwen/Qwen3-0.6B, meta-llama/Llama-3.2-3B-Instruct, google/gemma-3-4b-it, bartowski/google_gemma-3-4b-it-GGUF, microsoft/Phi-4-mini-instruct, bartowski/microsoft_Phi-4-mini-instruct-GGUF, HuggingFaceTB/SmolLM3-3B, mistralai/Ministral-8B-Instruct-2410.

Confidence note: file sizes for Qwen3-1.7B/0.6B, Llama-3.2-3B, and SmolLM3-3B GGUF quants above are typical/approximate — those repos weren't individually re-verified against a specific GGUF file tree since all three families were eliminated on architectural grounds (§3) before size mattered. Sizes for the two models actually compared head-to-head (Qwen3-4B-Instruct-2507, Phi-4-mini-instruct) and Gemma-3-4b-it were pulled directly from each quantizer's own repo page.

3. Why "thinking-by-default + per-call toggle" is disqualifying, not just a minor ding

This is the deciding architectural distinction, and it's exactly the failure this second model exists to avoid. Qwen's own llama.cpp docs page states the toggle problem directly:

"the hard switch implemented in the chat template is not exposed in llama.cpp" for controlling enable_thinking — the documented workaround is to supply "a custom chat template equivalent to always enable_thinking=False" via --chat-template-file. Source: Qwen — Run with llama.cpp.

That means for Qwen3-1.7B, Qwen3-0.6B, and SmolLM3-3B — all thinking-on-by-default — reliably suppressing the reasoning phase in this llama.cpp/ROCm stack is not a request-body flag away; it needs a hand-maintained custom chat template file, which is exactly the kind of fragile, easy-to-silently- regress setup this task is trying to get away from (the 27B model's whole problem was reasoning_content being consumed before the answer). Qwen3-4B-Instruct-2507 has no such toggle to maintain in the first place — the model card states the non-thinking behavior as an unconditional property of the model, not a configurable default that has to be forced correctly on every request. This is a stronger claim than "same tokenizer family as the 27B" (the original quick-pass's reasoning) and is the actual basis for the recommendation.

4. The one documented risk specific to Qwen3-4B-Instruct-2507 — and why it doesn't change the pick

llama.cpp has its own closed, dated bug where server builds around b8429 (March 2026) mis-detected Qwen3-Instruct-2507 models — the 4B included by name in the reporter's repro command — as thinking models, routing tool-call output into reasoning_content instead of tool_calls:

"llama.cpp b8429 incorrectly detects Qwen3-Instruct-2507 models as thinking models (thinking = 1). This causes tool calls to be captured as reasoning_content instead of being parsed into the tool_calls array."

The documented, confirmed-working workaround is a single server flag:

llama-server -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --jinja --port 8222 --reasoning off

which restores thinking = 0 and correct finish_reason: tool_calls output. The issue is closed. Source: ggml-org/llama.cpp issue #20809.

This is worth flagging honestly against the recommendation, but it's materially different from the open, only-partially-fixed Qwen3.5/Qwen3.8-lineage parser bugs already documented in this repo's qwen3.8-27b-tool-calling.md (issues #21158, #20837 — both open at time of that research): this is a llama.cpp server-side misdetection bug with a one-flag fix, not an unresolved upstream grammar/parser defect in the Qwen3.5 architecture family itself. Concretely: add --reasoning off to this second llama-server instance's command regardless — it's a no-cost safety net whether or not the current ghcr.io/ggml-org/llama.cpp:server-rocm build still has the bug, and it directly targets the exact failure mode (reasoning_content eating the completion) that ruled out the 27B model for this role in the first place.

5. VRAM math for the classifier role specifically

Qwen3-4B-Instruct-2507 is a plain (non-hybrid) transformer — every layer is standard GQA attention, so unlike the 27B model's Gated-DeltaNet hybrid, KV cache scales with all layers, not a fraction of them. From the model's own config.json:

• num_hidden_layers: 36, num_key_value_heads: 8, head_dim: 128 Source: Qwen/Qwen3-4B-Instruct-2507 config.json.

Per-token KV cache (fp16, both K and V): 36 layers × 2 (K+V) × 8 kv_heads × 128 head_dim × 2 bytes = 144 KiB/token

The classifier transcript is bounded by design — qwen-code's own two-stage design keeps Stage 1 to a {shouldBlock}-only judgment and Stage 2 to a chain-of-thought review of one blocked action, not an open-ended agent session — so a context window in the low thousands of tokens is generous headroom, not a tight fit:

Context	KV cache (fp16)	KV cache (q8_0, --cache-type-k/v q8_0)	Weights (Q8_0)	Total (q8_0 KV)	Headroom under 7GB
4,096 tokens	~0.56 GB	~0.28 GB	4.28 GB	~4.56 GB	~2.4 GB
8,192 tokens	~1.13 GB	~0.56 GB	4.28 GB	~4.84 GB	~2.2 GB
32,768 tokens (generous ceiling)	~4.5 GB	~2.25 GB	4.28 GB	~6.53 GB	~0.5 GB (tight)

At any context length actually needed for a permission-gate classifier (thousands, not tens of thousands, of tokens), Q8_0 weights plus q8_0 KV cache comfortably clears the 7GB ceiling with headroom to spare for the compute buffer and batch overhead — matching the same --cache-type-k q8_0 --cache-type-v q8_0 pattern this repo already uses for the 27B instance. There's no need to drop to Q4_K_M (2.5GB) unless a much larger classifier context is anticipated later; Q8_0 is the better default here since it's a small model where quantization loss matters proportionally more, and the VRAM budget comfortably affords the higher-precision quant.

6. What would change the answer

• If Phi-4-mini-instruct's MIT license matters more than matching the 27B model's tokenizer/template family, it's a legitimate second choice: confirmed non-thinking by default, confirmed native function-calling format, comparable Q8_0 size (4.08GB), and a license with zero commercial-use fine print (vs. Apache 2.0's still-permissive but slightly more conditional terms). It wasn't picked because it has no llama.cpp-specific tool-calling track record verified in this pass (no equivalent to the issue #20809 workaround search done for it), so its actual reliability on this exact llama.cpp:server-rocm stack is less directly evidenced than Qwen3-4B-Instruct-2507's.
• If the classifier transcript ever needs to grow well past ~8K tokens routinely, drop to Q4_K_M (2.5GB) to keep well clear of the 7GB ceiling — the KV-cache math in §5 shows the crossover point.
• If llama.cpp's #20809 misdetection turns out to still reproduce on the exact ghcr.io/ggml-org/llama.cpp:server-rocm build this repo pulls, the fix is the one-flag --reasoning off workaround already confirmed in that issue — not a reason to pick a different model, since every thinking-capable alternative in this shortlist has an equal-or-worse version of the same class of bug with less clean workarounds (custom chat-template files, per §3).

Sources

• Qwen Code docs — Auto Mode
• QwenLM/qwen-code — docs/users/features/auto-mode.md
• QwenLM/qwen-code issue #4676
• Qwen/Qwen3-4B-Instruct-2507
• Qwen/Qwen3-4B-Instruct-2507 config.json
• unsloth/Qwen3-4B-Instruct-2507-GGUF
• Qwen — Run with llama.cpp
• ggml-org/llama.cpp issue #20809
• Qwen/Qwen3-1.7B
• Qwen/Qwen3-0.6B
• meta-llama/Llama-3.2-3B-Instruct
• google/gemma-3-4b-it
• bartowski/google_gemma-3-4b-it-GGUF
• microsoft/Phi-4-mini-instruct
• bartowski/microsoft_Phi-4-mini-instruct-GGUF
• HuggingFaceTB/SmolLM3-3B
• mistralai/Ministral-8B-Instruct-2410
• docs/research/qwen3.8-27b-tool-calling.md (this repo — cross-referenced for the 27B model's own, still-open, tool-calling parser bugs)

Implementation note (2026-09-09) — what actually shipped, and why it differs

The model pick (Qwen3-4B-Instruct-2507) held up and is what's deployed. Several sizing assumptions in this doc didn't survive contact with the real deployment, though — worth recording so the next person tuning this doesn't re-derive the same corrections from scratch:

• Service name is qwen-classifier, not llama-server-fast — this doc's proposed name never got used. There's no LLAMA_FAST_CTX_SIZE/LLAMA_FAST_PARALLEL in .env.example either; the real config lives inline in docker-compose.yml's qwen-classifier command.
• CPU-only was tried first and rejected — this doc's VRAM budget analysis (§5) assumed GPU residency from the start, but the actual rollout path tried CPU-only first (to sidestep VRAM contention entirely) and found it too slow: real classification calls blew past OmniRoute's request timeout and retry-looped. Moved to GPU after that, which is what §5's math was for all along.
• Q4_K_XL weights, not Q8_0 — §5's "~2.4GB headroom" case assumed Q8_0 (4.28GB). In practice, fitting the classifier onto the R9700 alongside the 27B model (not in an assumed-empty 7GB budget) left only ~6.1GB free VRAM total, and even Q4_K_XL (2.37GB) plus full-context KV cache didn't leave enough real margin at full GPU offload — see the "measured live" numbers in docker-compose.yml's qwen-classifier comment block. Landed on partial GPU offload (28/36 layers) instead of full offload, which is not a case this doc considered at all.
• 65536 context, not 8192 — §5 sized the context "in the low thousands," reasoning from qwen-code's two-stage classifier description alone. Directly reading qwen-code's actual source (packages/core/src/permissions/classifier-transcript.ts: MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000/message) puts the real worst case at ~40-50K tokens — confirmed live, a real classifier call during testing hit 15,116 prompt tokens. 8192 would have been undersized for real usage; 65536 gives margin without the original setting.json value (131072, copied from the main model's entry, not a real qwen-code requirement) wasting VRAM for no reason.
• §4's --reasoning off recommendation was initially missed in the first deployment pass and added only once this doc was re-read while writing this note. It's now in docker-compose.yml's qwen-classifier command, per this doc's own "add it regardless, no-cost safety net" reasoning — still unconfirmed whether the current ghcr.io/ggml-org/llama.cpp:server-rocm build actually reproduces #20809 (nothing in testing so far surfaced reasoning_content where tool_calls was expected, but that wasn't specifically probed for either).

Confidence/uncertainty summary

• High confidence: Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote); qwen-code's two-stage classifier timing/design and its use of /model --fast (direct docs quote); the existence, exact symptom, and workaround of llama.cpp issue #20809 (direct issue quote); the KV-cache architecture math (computed directly from the model's own config.json, same method as the existing qwen3.8-27b-quant.md research in this repo).
• Medium confidence: exact GGUF file sizes for Qwen3-1.7B, Qwen3-0.6B, Llama-3.2-3B-Instruct, and SmolLM3-3B — not individually re-verified against a specific quantizer's file tree since these were eliminated on architectural (thinking-toggle) grounds before size became the deciding factor; treat as typical/approximate, not exact.
• Low confidence / not independently verified: whether the current ghcr.io/ggml-org/llama.cpp:server-rocm image (pulled fresh) still reproduces issue #20809's misdetection — the issue is closed but no changelog/PR diff was fetched to confirm the underlying detection logic was actually patched vs. the reporter simply adopting the --reasoning off workaround. Recommend a live smoke test (send one tool-calling request, confirm the response lands in tool_calls not reasoning_content, and time a trivial completion) before wiring this model in as the production fastModel, the same caveat this repo's qwen3.8-27b-tool-calling.md already flags for the 27B model.
• Whether Phi-4-mini-instruct has a comparably clean llama.cpp tool-calling track record was not deep-dived (no issue-tracker search run against it) — it's flagged in §6 as a live alternative rather than fully evaluated, since Qwen3-4B-Instruct-2507's architectural non-thinking guarantee and same-family template consistency with the existing 27B deployment made it the clearer pick without needing that extra research pass.