From ce1ab05c3ef5ea15c1e89be5001d0e46e1f6b033 Mon Sep 17 00:00:00 2001 From: Haylan Date: Mon, 24 Aug 2026 12:16:42 +0200 Subject: [PATCH] docs: research Qwen3.8-27B Unsloth dynamic GGUF quant + VRAM budget Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP. Co-Authored-By: Claude Sonnet 5 --- docs/research/qwen3.8-27b-quant.md | 128 +++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 docs/research/qwen3.8-27b-quant.md diff --git a/docs/research/qwen3.8-27b-quant.md b/docs/research/qwen3.8-27b-quant.md new file mode 100644 index 0000000..da6c6a4 --- /dev/null +++ b/docs/research/qwen3.8-27b-quant.md @@ -0,0 +1,128 @@ +# Research: Unsloth dynamic 4-bit GGUF for Qwen/Qwen3.8-27B + +**Question:** Does a UD-Q4_K_XL (or equivalent Unsloth dynamic) GGUF quant of +`Qwen/Qwen3.8-27B` exist? If not, what's the closest tool-call-safe +alternative, and what's the VRAM footprint on a 32GB RDNA4 card +(AMD Radeon AI PRO R9700, gfx1201) via llama.cpp ROCm/HIP, at 32K and 128K +context? + +**Answer: Yes, it exists.** `unsloth/Qwen3.8-27B-GGUF` on Hugging Face +contains `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB), Unsloth's Dynamic v3.0 +4-bit "XL" quant that keeps higher precision on attention/critical layers. +This is the recommended download — no need to fall back to AWQ or plain +Q4_K_M. + +## Model identity (context) + +`Qwen/Qwen3.8-27B` is a real, current model — released 2026-08-14 by +Alibaba's Qwen team (Apache 2.0), the successor to Qwen3.6-27B. It's a dense +27B hybrid model: 64 layers, `full_attention_interval: 4`, meaning only 16 +of 64 layers are standard full-attention (KV-caching) layers; the other 48 +are Gated DeltaNet linear-attention layers whose state is fixed-size +regardless of context length. `hidden_size=5120`, `num_attention_heads=24`, +`num_key_value_heads=4` (GQA), `head_dim=256`, native context 262,144 tokens +(YaRN-extensible to 1M). It's natively multimodal (vision + video) via a +separate `mmproj` file. +Source: [Qwen/Qwen3.8-27B model card](https://huggingface.co/Qwen/Qwen3.8-27B), +[Qwen/Qwen3.8-27B config.json](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json), +[QwenLM/Qwen3.8 GitHub repo](https://github.com/QwenLM/Qwen3.8). + +## Recommended repo / file + +- **Repo:** `unsloth/Qwen3.8-27B-GGUF` +- **File:** `Qwen3.8-27B-UD-Q4_K_XL.gguf` — **17.6 GB** +- Optional vision projector (only needed for image/video input): + `mmproj-BF16.gguf` (931 MB) or `mmproj-F16.gguf` (928 MB) + +Confirmed directly from the repo's file tree (primary source, fetched from +the HF repo page itself): +[unsloth/Qwen3.8-27B-GGUF/tree/main](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main). +The repo also ships `UD-Q4_K_M` (16.5 GB), `UD-Q4_K_S` (15.4 GB), and other +Dynamic v3.0 sizes (IQ1 through Q8_K_XL), but `UD-Q4_K_XL` is the one that +matches the "dynamic 4-bit XL" spec in the ticket and gives the best +accuracy/size tradeoff at 4-bit — Unsloth's own docs name it as the +recommended default for this model. +Source: [Unsloth docs — Qwen3.8 how-to-run page](https://unsloth.ai/docs/models/qwen3.8) +("Unsloth recommends UD-Q4_K_XL as the primary quantization... 4-bit quants +work on 16-19GB VRAM"). + +Why prefer this over plain `Q4_K_M`/AWQ for tool-calling fidelity: Unsloth's +Dynamic v3.0 quantization selectively keeps higher bit-width on +attention/critical layers rather than uniformly quantizing everything, and +Unsloth's own model-card notes call out improved "tool calling: improved +parsing [of] nested objects to make tools succeed more" for this quant +family. Since a dynamic XL quant is directly available, there was no need to +fall back to evaluating AWQ alternatives. + +**Confidence note:** The repo tree listing and file sizes were fetched +directly from the Hugging Face page (primary source) and are high +confidence. I did not download and independently checksum the file. The +"improved tool calling" claim comes from Unsloth's own docs page (primary, +first-party), not an independent benchmark — treat it as a vendor claim, not +independently verified. + +## ModelScope / other sources checked + +Search results also surfaced `Qwen/Qwen3.8-27B` mirrored patterns typical of +ModelScope + HF dual-publishing (per the QwenLM GitHub repo, which states +new releases go out on "Hugging Face and ModelScope" simultaneously), and a +non-Unsloth GGUF from `bartowski/Qwen3.8-27B-GGUF` (imatrix-calibrated, +includes tool-calling/reasoning conversations in the calibration corpus) as +a secondary alternative if the Unsloth repo were ever unavailable. I did not +deep-dive ModelScope's page directly (Unsloth's HF repo already answers the +question), so treat the ModelScope-mirror claim as **unconfirmed** — +inferred from the GitHub repo's release notes rather than fetched directly +from modelscope.cn. + +## VRAM footprint on a 32GB RDNA4 card (R9700, gfx1201) via llama.cpp ROCm/HIP + +**Weights:** 17.6 GB (`UD-Q4_K_XL.gguf`, fits entirely on-GPU on a 32GB card). + +**KV cache:** Because only 16 of 64 layers are standard full-attention +(GQA: 4 KV heads × 256 head_dim), the KV cache scales with just those 16 +layers — the other 48 Gated-DeltaNet layers carry a fixed, context-length-independent +recurrent state (order tens of MB total, negligible next to the attention KV cache). + +Per-token KV cache (fp16, both K and V, 16 full-attn layers): +`16 layers × 2 (K+V) × 4 kv_heads × 256 head_dim × 2 bytes = 64 KiB/token` + +| Context | KV cache (fp16) | Weights | Weights + KV cache | Headroom on 32GB | +|---|---|---|---|---| +| 32K tokens | ~2.0 GB | 17.6 GB | **~19.6 GB** | ~12 GB free (compute buffer, batch, overhead) | +| 128K tokens | ~8.0 GB | 17.6 GB | **~25.6 GB** | ~6 GB free — comfortable but tighter | + +Both fit on a 32GB card with room to spare. If more headroom is wanted at +128K (e.g., for larger batch/ubatch or multiple parallel sequences), +quantizing the KV cache to Q8_0 in llama.cpp (`--cache-type-k q8_0 +--cache-type-v q8_0`) roughly halves the cache to ~4 GB, bringing the 128K +total to ~21.6 GB. + +**Sanity check against a third-party source:** a blog doing the same math +independently states the full 262,144-token native context costs "about +17GB of KV cache" for this hybrid architecture. My formula gives +262,144 × 64 KiB ≈ 16.0 GB, consistent with that figure. +Source: [WebSearch summary citing "Qwen3.8-27B VRAM/KV-cache math" blog](https://umesh-malik.com/blog/qwen3-8-27b-vram-kv-cache-math) +(secondary source, used only as a cross-check on the arithmetic, not as a +primary claim). + +**ROCm/HIP + gfx1201 support:** ROCm 7.2 added official `gfx1201` support +(some report 7.1.1 working via `HSA_OVERRIDE_GFX_VERSION`); llama.cpp builds +with `-DGGML_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1201`. Note: community +reports say the Vulkan backend is currently ~23% faster than ROCm/HIP on +gfx1201 for token generation, and the HIP backend has a known issue keeping +the GPU at elevated clocks/power after idle. These are performance/operational +notes, not blockers for fitting the model in VRAM. +Sources: [tlee933/llama.cpp-rdna4-gfx1201 GitHub](https://github.com/tlee933/llama.cpp-rdna4-gfx1201), +[ggml-org/llama.cpp discussion #20881 (R9700 ROCm)](https://github.com/ggml-org/llama.cpp/discussions/20881), +[ggml-org/llama.cpp discussion #15021 (ROCm perf)](https://github.com/ggml-org/llama.cpp/discussions/15021). +**Confidence note:** these ROCm/gfx1201 specifics come from secondary +(community) sources, not AMD's or ggml-org's official docs directly fetched +— treat performance numbers as indicative, not guaranteed. + +## Bottom line for the wayfinder map + +- Use **`unsloth/Qwen3.8-27B-GGUF`**, file **`Qwen3.8-27B-UD-Q4_K_XL.gguf`** (17.6 GB). +- Fits comfortably on the 32GB R9700 at both 32K context (~19.6 GB total) + and 128K context (~25.6 GB total, or ~21.6 GB with q8_0 KV cache). +- Build llama.cpp with ROCm/HIP targeting `gfx1201`; Vulkan is a viable + faster alternative backend if HIP's idle-power quirk is a problem.