haylanandClaude-Bot
ce1ab05c3e
Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
129 lines
7.1 KiB
Markdown
129 lines
7.1 KiB
Markdown
# Research: Unsloth dynamic 4-bit GGUF for Qwen/Qwen3.8-27B
|
||
|
||
**Question:** Does a UD-Q4_K_XL (or equivalent Unsloth dynamic) GGUF quant of
|
||
`Qwen/Qwen3.8-27B` exist? If not, what's the closest tool-call-safe
|
||
alternative, and what's the VRAM footprint on a 32GB RDNA4 card
|
||
(AMD Radeon AI PRO R9700, gfx1201) via llama.cpp ROCm/HIP, at 32K and 128K
|
||
context?
|
||
|
||
**Answer: Yes, it exists.** `unsloth/Qwen3.8-27B-GGUF` on Hugging Face
|
||
contains `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB), Unsloth's Dynamic v3.0
|
||
4-bit "XL" quant that keeps higher precision on attention/critical layers.
|
||
This is the recommended download — no need to fall back to AWQ or plain
|
||
Q4_K_M.
|
||
|
||
## Model identity (context)
|
||
|
||
`Qwen/Qwen3.8-27B` is a real, current model — released 2026-08-14 by
|
||
Alibaba's Qwen team (Apache 2.0), the successor to Qwen3.6-27B. It's a dense
|
||
27B hybrid model: 64 layers, `full_attention_interval: 4`, meaning only 16
|
||
of 64 layers are standard full-attention (KV-caching) layers; the other 48
|
||
are Gated DeltaNet linear-attention layers whose state is fixed-size
|
||
regardless of context length. `hidden_size=5120`, `num_attention_heads=24`,
|
||
`num_key_value_heads=4` (GQA), `head_dim=256`, native context 262,144 tokens
|
||
(YaRN-extensible to 1M). It's natively multimodal (vision + video) via a
|
||
separate `mmproj` file.
|
||
Source: [Qwen/Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B),
|
||
[Qwen/Qwen3.8-27B config.json](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json),
|
||
[QwenLM/Qwen3.8 GitHub repo](https://github.com/QwenLM/Qwen3.8).
|
||
|
||
## Recommended repo / file
|
||
|
||
- **Repo:** `unsloth/Qwen3.8-27B-GGUF`
|
||
- **File:** `Qwen3.8-27B-UD-Q4_K_XL.gguf` — **17.6 GB**
|
||
- Optional vision projector (only needed for image/video input):
|
||
`mmproj-BF16.gguf` (931 MB) or `mmproj-F16.gguf` (928 MB)
|
||
|
||
Confirmed directly from the repo's file tree (primary source, fetched from
|
||
the HF repo page itself):
|
||
[unsloth/Qwen3.8-27B-GGUF/tree/main](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main).
|
||
The repo also ships `UD-Q4_K_M` (16.5 GB), `UD-Q4_K_S` (15.4 GB), and other
|
||
Dynamic v3.0 sizes (IQ1 through Q8_K_XL), but `UD-Q4_K_XL` is the one that
|
||
matches the "dynamic 4-bit XL" spec in the ticket and gives the best
|
||
accuracy/size tradeoff at 4-bit — Unsloth's own docs name it as the
|
||
recommended default for this model.
|
||
Source: [Unsloth docs — Qwen3.8 how-to-run page](https://unsloth.ai/docs/models/qwen3.8)
|
||
("Unsloth recommends UD-Q4_K_XL as the primary quantization... 4-bit quants
|
||
work on 16-19GB VRAM").
|
||
|
||
Why prefer this over plain `Q4_K_M`/AWQ for tool-calling fidelity: Unsloth's
|
||
Dynamic v3.0 quantization selectively keeps higher bit-width on
|
||
attention/critical layers rather than uniformly quantizing everything, and
|
||
Unsloth's own model-card notes call out improved "tool calling: improved
|
||
parsing [of] nested objects to make tools succeed more" for this quant
|
||
family. Since a dynamic XL quant is directly available, there was no need to
|
||
fall back to evaluating AWQ alternatives.
|
||
|
||
**Confidence note:** The repo tree listing and file sizes were fetched
|
||
directly from the Hugging Face page (primary source) and are high
|
||
confidence. I did not download and independently checksum the file. The
|
||
"improved tool calling" claim comes from Unsloth's own docs page (primary,
|
||
first-party), not an independent benchmark — treat it as a vendor claim, not
|
||
independently verified.
|
||
|
||
## ModelScope / other sources checked
|
||
|
||
Search results also surfaced `Qwen/Qwen3.8-27B` mirrored patterns typical of
|
||
ModelScope + HF dual-publishing (per the QwenLM GitHub repo, which states
|
||
new releases go out on "Hugging Face and ModelScope" simultaneously), and a
|
||
non-Unsloth GGUF from `bartowski/Qwen3.8-27B-GGUF` (imatrix-calibrated,
|
||
includes tool-calling/reasoning conversations in the calibration corpus) as
|
||
a secondary alternative if the Unsloth repo were ever unavailable. I did not
|
||
deep-dive ModelScope's page directly (Unsloth's HF repo already answers the
|
||
question), so treat the ModelScope-mirror claim as **unconfirmed** —
|
||
inferred from the GitHub repo's release notes rather than fetched directly
|
||
from modelscope.cn.
|
||
|
||
## VRAM footprint on a 32GB RDNA4 card (R9700, gfx1201) via llama.cpp ROCm/HIP
|
||
|
||
**Weights:** 17.6 GB (`UD-Q4_K_XL.gguf`, fits entirely on-GPU on a 32GB card).
|
||
|
||
**KV cache:** Because only 16 of 64 layers are standard full-attention
|
||
(GQA: 4 KV heads × 256 head_dim), the KV cache scales with just those 16
|
||
layers — the other 48 Gated-DeltaNet layers carry a fixed, context-length-independent
|
||
recurrent state (order tens of MB total, negligible next to the attention KV cache).
|
||
|
||
Per-token KV cache (fp16, both K and V, 16 full-attn layers):
|
||
`16 layers × 2 (K+V) × 4 kv_heads × 256 head_dim × 2 bytes = 64 KiB/token`
|
||
|
||
| Context | KV cache (fp16) | Weights | Weights + KV cache | Headroom on 32GB |
|
||
|---|---|---|---|---|
|
||
| 32K tokens | ~2.0 GB | 17.6 GB | **~19.6 GB** | ~12 GB free (compute buffer, batch, overhead) |
|
||
| 128K tokens | ~8.0 GB | 17.6 GB | **~25.6 GB** | ~6 GB free — comfortable but tighter |
|
||
|
||
Both fit on a 32GB card with room to spare. If more headroom is wanted at
|
||
128K (e.g., for larger batch/ubatch or multiple parallel sequences),
|
||
quantizing the KV cache to Q8_0 in llama.cpp (`--cache-type-k q8_0
|
||
--cache-type-v q8_0`) roughly halves the cache to ~4 GB, bringing the 128K
|
||
total to ~21.6 GB.
|
||
|
||
**Sanity check against a third-party source:** a blog doing the same math
|
||
independently states the full 262,144-token native context costs "about
|
||
17GB of KV cache" for this hybrid architecture. My formula gives
|
||
262,144 × 64 KiB ≈ 16.0 GB, consistent with that figure.
|
||
Source: [WebSearch summary citing "Qwen3.8-27B VRAM/KV-cache math" blog](https://umesh-malik.com/blog/qwen3-8-27b-vram-kv-cache-math)
|
||
(secondary source, used only as a cross-check on the arithmetic, not as a
|
||
primary claim).
|
||
|
||
**ROCm/HIP + gfx1201 support:** ROCm 7.2 added official `gfx1201` support
|
||
(some report 7.1.1 working via `HSA_OVERRIDE_GFX_VERSION`); llama.cpp builds
|
||
with `-DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1201`. Note: community
|
||
reports say the Vulkan backend is currently ~23% faster than ROCm/HIP on
|
||
gfx1201 for token generation, and the HIP backend has a known issue keeping
|
||
the GPU at elevated clocks/power after idle. These are performance/operational
|
||
notes, not blockers for fitting the model in VRAM.
|
||
Sources: [tlee933/llama.cpp-rdna4-gfx1201 GitHub](https://github.com/tlee933/llama.cpp-rdna4-gfx1201),
|
||
[ggml-org/llama.cpp discussion #20881 (R9700 ROCm)](https://github.com/ggml-org/llama.cpp/discussions/20881),
|
||
[ggml-org/llama.cpp discussion #15021 (ROCm perf)](https://github.com/ggml-org/llama.cpp/discussions/15021).
|
||
**Confidence note:** these ROCm/gfx1201 specifics come from secondary
|
||
(community) sources, not AMD's or ggml-org's official docs directly fetched
|
||
— treat performance numbers as indicative, not guaranteed.
|
||
|
||
## Bottom line for the wayfinder map
|
||
|
||
- Use **`unsloth/Qwen3.8-27B-GGUF`**, file **`Qwen3.8-27B-UD-Q4_K_XL.gguf`** (17.6 GB).
|
||
- Fits comfortably on the 32GB R9700 at both 32K context (~19.6 GB total)
|
||
and 128K context (~25.6 GB total, or ~21.6 GB with q8_0 KV cache).
|
||
- Build llama.cpp with ROCm/HIP targeting `gfx1201`; Vulkan is a viable
|
||
faster alternative backend if HIP's idle-power quirk is a problem.
|