haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Confirm quantization availability for Qwen3.8-27B #2

New Issue
Closed
opened 2026-08-24 10:09:41 +00:00 by haylan · 1 comment
haylan commented 2026-08-24 10:09:41 +00:00
Owner
Copy Link
Copy Source

Part of #1

Question

Does an Unsloth-style dynamic 4-bit GGUF quant ("UD-Q4_K_XL" or equivalent) of Qwen/Qwen3.8-27B exist (check Hugging Face, Unsloth's org, ModelScope)? If not, what is the closest available quantization that preserves tool-call/structured-output fidelity (favor dynamic/XL quants or AWQ over plain Q4_K_M), and what's its approximate VRAM footprint for weights + KV cache budget on a 32GB card at 32K-128K context? Report the exact repo/filename to use.

Part of #1 ## Question Does an Unsloth-style dynamic 4-bit GGUF quant ("UD-Q4_K_XL" or equivalent) of Qwen/Qwen3.8-27B exist (check Hugging Face, Unsloth's org, ModelScope)? If not, what is the closest available quantization that preserves tool-call/structured-output fidelity (favor dynamic/XL quants or AWQ over plain Q4_K_M), and what's its approximate VRAM footprint for weights + KV cache budget on a 32GB card at 32K-128K context? Report the exact repo/filename to use.
haylan added the wayfinder:research label 2026-08-24 10:09:41 +00:00
haylan self-assigned this 2026-08-24 10:12:35 +00:00
haylan commented 2026-08-24 10:17:41 +00:00
Author
Owner
Copy Link
Copy Source

Resolved: yes, a dynamic 4-bit XL quant exists.

Repo: unsloth/Qwen3.8-27B-GGUF
File: Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6 GB) — confirmed directly from the HF repo's file tree.

VRAM budget on a 32GB RDNA4 card (R9700/gfx1201) via llama.cpp ROCm/HIP:

  • 32K context: ~19.6 GB (17.6 GB weights + ~2.0 GB KV cache fp16)
  • 128K context: ~25.6 GB (17.6 GB weights + ~8.0 GB KV cache fp16; ~21.6 GB if KV cache quantized to q8_0)

Both fit comfortably. The model is a hybrid dense/linear-attention architecture (only 16 of 64 layers are standard full-attention/KV-caching layers, per full_attention_interval: 4), which keeps KV cache growth much smaller than a plain-attention 27B model would need.

Full writeup with citations (HF pages, Unsloth docs, QwenLM GitHub, llama.cpp ROCm/gfx1201 notes) and confidence/uncertainty notes: docs/research/qwen3.8-27b-quant.md, committed on branch research/qwen3.8-27b-quant (local-only — not pushed to the remote).

**Resolved: yes, a dynamic 4-bit XL quant exists.** Repo: `unsloth/Qwen3.8-27B-GGUF` File: `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB) — confirmed directly from the HF repo's file tree. VRAM budget on a 32GB RDNA4 card (R9700/gfx1201) via llama.cpp ROCm/HIP: - 32K context: ~19.6 GB (17.6 GB weights + ~2.0 GB KV cache fp16) - 128K context: ~25.6 GB (17.6 GB weights + ~8.0 GB KV cache fp16; ~21.6 GB if KV cache quantized to q8_0) Both fit comfortably. The model is a hybrid dense/linear-attention architecture (only 16 of 64 layers are standard full-attention/KV-caching layers, per `full_attention_interval: 4`), which keeps KV cache growth much smaller than a plain-attention 27B model would need. Full writeup with citations (HF pages, Unsloth docs, QwenLM GitHub, llama.cpp ROCm/gfx1201 notes) and confidence/uncertainty notes: `docs/research/qwen3.8-27b-quant.md`, committed on branch `research/qwen3.8-27b-quant` (local-only — not pushed to the remote).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Blocks
#4 Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer)
haylan/LLM-Server
Reference: haylan/LLM-Server#2