LLM-Server/docs/research/qwen3.8-27b-quant.md

Research: Unsloth dynamic 4-bit GGUF for Qwen/Qwen3.8-27B

Question: Does a UD-Q4_K_XL (or equivalent Unsloth dynamic) GGUF quant of Qwen/Qwen3.8-27B exist? If not, what's the closest tool-call-safe alternative, and what's the VRAM footprint on a 32GB RDNA4 card (AMD Radeon AI PRO R9700, gfx1201) via llama.cpp ROCm/HIP, at 32K and 128K context?

Answer: Yes, it exists. unsloth/Qwen3.8-27B-GGUF on Hugging Face contains Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6 GB), Unsloth's Dynamic v3.0 4-bit "XL" quant that keeps higher precision on attention/critical layers. This is the recommended download — no need to fall back to AWQ or plain Q4_K_M.

Model identity (context)

Qwen/Qwen3.8-27B is a real, current model — released 2026-08-14 by Alibaba's Qwen team (Apache 2.0), the successor to Qwen3.6-27B. It's a dense 27B hybrid model: 64 layers, full_attention_interval: 4, meaning only 16 of 64 layers are standard full-attention (KV-caching) layers; the other 48 are Gated DeltaNet linear-attention layers whose state is fixed-size regardless of context length. hidden_size=5120, num_attention_heads=24, num_key_value_heads=4 (GQA), head_dim=256, native context 262,144 tokens (YaRN-extensible to 1M). It's natively multimodal (vision + video) via a separate mmproj file. Source: Qwen/Qwen3.8-27B model card, Qwen/Qwen3.8-27B config.json, QwenLM/Qwen3.8 GitHub repo.

Recommended repo / file

• Repo: unsloth/Qwen3.8-27B-GGUF
• File: Qwen3.8-27B-UD-Q4_K_XL.gguf — 17.6 GB
• Optional vision projector (only needed for image/video input): mmproj-BF16.gguf (931 MB) or mmproj-F16.gguf (928 MB)

Confirmed directly from the repo's file tree (primary source, fetched from the HF repo page itself): unsloth/Qwen3.8-27B-GGUF/tree/main. The repo also ships UD-Q4_K_M (16.5 GB), UD-Q4_K_S (15.4 GB), and other Dynamic v3.0 sizes (IQ1 through Q8_K_XL), but UD-Q4_K_XL is the one that matches the "dynamic 4-bit XL" spec in the ticket and gives the best accuracy/size tradeoff at 4-bit — Unsloth's own docs name it as the recommended default for this model. Source: Unsloth docs — Qwen3.8 how-to-run page ("Unsloth recommends UD-Q4_K_XL as the primary quantization... 4-bit quants work on 16-19GB VRAM").

Why prefer this over plain Q4_K_M/AWQ for tool-calling fidelity: Unsloth's Dynamic v3.0 quantization selectively keeps higher bit-width on attention/critical layers rather than uniformly quantizing everything, and Unsloth's own model-card notes call out improved "tool calling: improved parsing [of] nested objects to make tools succeed more" for this quant family. Since a dynamic XL quant is directly available, there was no need to fall back to evaluating AWQ alternatives.

Confidence note: The repo tree listing and file sizes were fetched directly from the Hugging Face page (primary source) and are high confidence. I did not download and independently checksum the file. The "improved tool calling" claim comes from Unsloth's own docs page (primary, first-party), not an independent benchmark — treat it as a vendor claim, not independently verified.

ModelScope / other sources checked

Search results also surfaced Qwen/Qwen3.8-27B mirrored patterns typical of ModelScope + HF dual-publishing (per the QwenLM GitHub repo, which states new releases go out on "Hugging Face and ModelScope" simultaneously), and a non-Unsloth GGUF from bartowski/Qwen3.8-27B-GGUF (imatrix-calibrated, includes tool-calling/reasoning conversations in the calibration corpus) as a secondary alternative if the Unsloth repo were ever unavailable. I did not deep-dive ModelScope's page directly (Unsloth's HF repo already answers the question), so treat the ModelScope-mirror claim as unconfirmed — inferred from the GitHub repo's release notes rather than fetched directly from modelscope.cn.

VRAM footprint on a 32GB RDNA4 card (R9700, gfx1201) via llama.cpp ROCm/HIP

Weights: 17.6 GB (UD-Q4_K_XL.gguf, fits entirely on-GPU on a 32GB card).

KV cache: Because only 16 of 64 layers are standard full-attention (GQA: 4 KV heads × 256 head_dim), the KV cache scales with just those 16 layers — the other 48 Gated-DeltaNet layers carry a fixed, context-length-independent recurrent state (order tens of MB total, negligible next to the attention KV cache).

Per-token KV cache (fp16, both K and V, 16 full-attn layers): 16 layers × 2 (K+V) × 4 kv_heads × 256 head_dim × 2 bytes = 64 KiB/token

Context	KV cache (fp16)	Weights	Weights + KV cache	Headroom on 32GB
32K tokens	~2.0 GB	17.6 GB	~19.6 GB	~12 GB free (compute buffer, batch, overhead)
128K tokens	~8.0 GB	17.6 GB	~25.6 GB	~6 GB free — comfortable but tighter

Both fit on a 32GB card with room to spare. If more headroom is wanted at 128K (e.g., for larger batch/ubatch or multiple parallel sequences), quantizing the KV cache to Q8_0 in llama.cpp (--cache-type-k q8_0 --cache-type-v q8_0) roughly halves the cache to ~4 GB, bringing the 128K total to ~21.6 GB.

Sanity check against a third-party source: a blog doing the same math independently states the full 262,144-token native context costs "about 17GB of KV cache" for this hybrid architecture. My formula gives 262,144 × 64 KiB ≈ 16.0 GB, consistent with that figure. Source: WebSearch summary citing "Qwen3.8-27B VRAM/KV-cache math" blog (secondary source, used only as a cross-check on the arithmetic, not as a primary claim).

ROCm/HIP + gfx1201 support: ROCm 7.2 added official gfx1201 support (some report 7.1.1 working via HSA_OVERRIDE_GFX_VERSION); llama.cpp builds with -DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1201. Note: community reports say the Vulkan backend is currently ~23% faster than ROCm/HIP on gfx1201 for token generation, and the HIP backend has a known issue keeping the GPU at elevated clocks/power after idle. These are performance/operational notes, not blockers for fitting the model in VRAM. Sources: tlee933/llama.cpp-rdna4-gfx1201 GitHub, ggml-org/llama.cpp discussion #20881 (R9700 ROCm), ggml-org/llama.cpp discussion #15021 (ROCm perf). Confidence note: these ROCm/gfx1201 specifics come from secondary (community) sources, not AMD's or ggml-org's official docs directly fetched — treat performance numbers as indicative, not guaranteed.

Bottom line for the wayfinder map

• Use unsloth/Qwen3.8-27B-GGUF, file Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6 GB).
• Fits comfortably on the 32GB R9700 at both 32K context (~19.6 GB total) and 128K context (~25.6 GB total, or ~21.6 GB with q8_0 KV cache).
• Build llama.cpp with ROCm/HIP targeting gfx1201; Vulkan is a viable faster alternative backend if HIP's idle-power quirk is a problem.