haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Confirm Qwen3.8-27B tool-calling compatibility with llama.cpp's Anthropic Messages API shim #3

New Issue
Closed
opened 2026-08-24 10:09:42 +00:00 by haylan · 1 comment
haylan commented 2026-08-24 10:09:42 +00:00
Owner
Copy Link
Copy Source

Part of #1

Question

Is Qwen3.8-27B's tool-call output format compatible with llama.cpp's native Anthropic Messages API shim (used to serve Claude Code CLI), or does it need a chat-template fix the way Qwen3-Coder did (see qwen-code#176, Unsloth GGUF discussion #10 referenced in prior research)? Check the model's card, any known chat-template issues, and whether its dense/hybrid-attention architecture (not a coding-specialist model) causes any known problems with structured tool_use/tool_result blocks in an agentic coding loop.

Part of #1 ## Question Is Qwen3.8-27B's tool-call output format compatible with llama.cpp's native Anthropic Messages API shim (used to serve Claude Code CLI), or does it need a chat-template fix the way Qwen3-Coder did (see qwen-code#176, Unsloth GGUF discussion #10 referenced in prior research)? Check the model's card, any known chat-template issues, and whether its dense/hybrid-attention architecture (not a coding-specialist model) causes any known problems with structured tool_use/tool_result blocks in an agentic coding loop.
haylan added the wayfinder:research label 2026-08-24 10:09:42 +00:00
haylan self-assigned this 2026-08-24 10:12:36 +00:00
haylan commented 2026-08-24 10:16:43 +00:00
Author
Owner
Copy Link
Copy Source

Answer: No — not confirmed compatible; likely needs a fix, and it may not be a simple chat-template swap this time.

Short version: llama.cpp's Anthropic Messages API shim (PR #17570) just converts Anthropic's wire format to the existing OpenAI-compatible tool-calling pipeline — it adds no independent risk, but also inherits every bug that pipeline has for the model served. For Qwen3.8-27B (built on the Qwen3.5 architectural lineage), that pipeline has documented, dated bugs:

  • llama.cpp issue #21158: Qwen3.5-27B tool-call parsing fails when the model emits any text before <tool_call>; a merged fix (PR #20424) only partially resolved it, folded into still-open issue #20260.
  • llama.cpp issue #20837 (open): Qwen3.5 9B emits tool calls as inert XML text inside/around thinking blocks and stops, when thinking is enabled. Workaround: enable_thinking:false.
  • Related lineage bugs: #20164 (tool calls break under long context w/ optional params), #20809 (tool calls misclassified as reasoning_content).
  • Separately, the Qwen3.5 chat template itself has already churned post-release in ways that broke naive Jinja engines (unsloth/Qwen3.5-35B-A3B-GGUF discussion #18).

Unlike the Qwen3-Coder precedent (qwen-code#176 / unsloth discussion #10), where a corrected chat_template.jinja fully fixed things, the Qwen3.5-lineage bug in #21158 survived a merged upstream PR — the fix likely requires llama.cpp parser/grammar-trigger changes, not just a template swap.

No primary source connects the hybrid linear-attention (Gated DeltaNet) architecture itself to tool-call reliability — flagging that as unconfirmed/untested. The one Qwen3.8-27B-specific bug found (llama.cpp discussion #27164) is a CUDA numerical-correctness bug in the DeltaNet path (stale library causing garbage output), not a tool-call format bug — relevant as general risk context but not causally tied to tool-calling.

Recommendation: keep "unverified, accepted knowingly" but sharpen it — this is no longer just "untested," there's direct evidence of open/partial upstream parser bugs against this exact lineage. Do a live smoke test (multi-turn tool use, thinking enabled) against the actual deployed llama.cpp build before relying on it unattended.

Full findings with citations: docs/research/qwen3.8-27b-tool-calling.md on branch research/qwen3.8-27b-tool-calling.

## Answer: No — not confirmed compatible; likely needs a fix, and it may not be a simple chat-template swap this time. **Short version:** llama.cpp's Anthropic Messages API shim (PR #17570) just converts Anthropic's wire format to the existing OpenAI-compatible tool-calling pipeline — it adds no independent risk, but also inherits every bug that pipeline has for the model served. For Qwen3.8-27B (built on the Qwen3.5 architectural lineage), that pipeline has **documented, dated bugs**: - llama.cpp issue #21158: Qwen3.5-27B tool-call parsing fails when the model emits any text before `<tool_call>`; a merged fix (PR #20424) only partially resolved it, folded into still-open issue #20260. - llama.cpp issue #20837 (open): Qwen3.5 9B emits tool calls as inert XML text inside/around thinking blocks and stops, when thinking is enabled. Workaround: `enable_thinking:false`. - Related lineage bugs: #20164 (tool calls break under long context w/ optional params), #20809 (tool calls misclassified as reasoning_content). - Separately, the Qwen3.5 chat template itself has already churned post-release in ways that broke naive Jinja engines (unsloth/Qwen3.5-35B-A3B-GGUF discussion #18). Unlike the Qwen3-Coder precedent (qwen-code#176 / unsloth discussion #10), where a corrected `chat_template.jinja` fully fixed things, the Qwen3.5-lineage bug in #21158 survived a merged upstream PR — the fix likely requires llama.cpp parser/grammar-trigger changes, not just a template swap. No primary source connects the hybrid linear-attention (Gated DeltaNet) architecture itself to tool-call reliability — flagging that as **unconfirmed/untested**. The one Qwen3.8-27B-specific bug found (llama.cpp discussion #27164) is a CUDA numerical-correctness bug in the DeltaNet path (stale library causing garbage output), not a tool-call format bug — relevant as general risk context but not causally tied to tool-calling. **Recommendation:** keep "unverified, accepted knowingly" but sharpen it — this is no longer just "untested," there's direct evidence of open/partial upstream parser bugs against this exact lineage. Do a live smoke test (multi-turn tool use, thinking enabled) against the actual deployed llama.cpp build before relying on it unattended. Full findings with citations: `docs/research/qwen3.8-27b-tool-calling.md` on branch `research/qwen3.8-27b-tool-calling`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Blocks
#4 Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer)
haylan/LLM-Server
#6 Write local-usage docs for Claude Code CLI, Kimi CLI, and OpenCode CLI against the local endpoint
haylan/LLM-Server
Reference: haylan/LLM-Server#3