haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5b866b93ff725abf7bab4d3c1b1644465d4e10b5
LLM-Server/docs/research/dual-r9700-speed-and-offload.md
T

34 KiB
Raw Blame History

Research: fastest inference on 2x R9700 (27B Q8_K_XL vs Flash-Next, multi-GPU modes, RAM/SSD offload, Strata)

Date: 2026-10-29

Question: A second Radeon AI PRO R9700 arrives in about a week. What gives the fastest decode (and prefill) for agentic coding on this box: Qwen3.8-27B-UD-Q8_K_XL, Qwen3.8-Flash-Next-UD-Q4_K_XL, or something else? Which llama.cpp multi-GPU mode helps, does RAM/SSD offload work with 32 GB of RAM, and is Strata worth switching to?

Scope note: research only. Nothing in docker-compose.yml or .env.example was changed, so nothing needs deploying.

Hardware assumed: Ryzen 9 5950X, 32 GB DDR4 dual channel (reported ~94 % used, swap in use), ASRock B550M Pro4, 1 TB NVMe, Ubuntu 24.04, kernel 6.8, 1x R9700 now and 2x soon.

TL;DR

# Finding Confidence
1 The second card does not make the dense 27B decode faster with -sm layer. Layer split runs the cards one after the other, so one token still reads every weight at one card's bandwidth. It only adds VRAM. High (llama.cpp docs)
2 UD-Q8_K_XL is a quality move, not a speed move. It reads ~30.1 GB per token against ~16.8 GB for today's UD-Q4_K_XL. Estimated decode is ~15 t/s without speculation and ~25 t/s with DFlash2, against ~46 t/s measured today. Medium (estimate, math below)
3 -sm tensor is the only mode that can speed up one stream, and on ROCm it is immature. The one measured 2-card ROCm result for this exact model is 1.31x one card (2x 7900 XTX). DFlash2 crashes with -sm tensor (open bug), and MTP plus -sm tensor has lockup reports. Your second slot is PCIe 3.0 x4 behind the chipset, which hurts the per-layer all-reduces. Medium
4 Best use of the second card for 2 parallel agents: two independent single-GPU llama-server replicas, one per card, each --parallel 1. Each agent keeps today's ~46 t/s, the decode drop when the other slot prefills goes away, and each slot gets the full 262144 tokens instead of 131072. Needs a second backend connection in OmniRoute with per-conversation stickiness. Medium-high
5 Flash-Next UD-Q4_K_XL (111.3 GB) does not fit. Even with the 28.8 GB n-gram table left on SSD, 82.5 GB of weights remain for 64 GB of VRAM. The ~20-24 GB of experts left over would come from NVMe, because RAM is full. Estimated decode is then ~6-12 t/s. Only the all-in-VRAM quants (UD-Q2_K_XL, UD-IQ3_XXS) are realistic, at a large quality cost (mean KLD 0.22 / 0.17 vs 0.047). Medium
6 RAM is the binding constraint, and compose itself is a likely cause of the swap. --cache-ram is set to 24576 MiB (24 GiB) on a 32 GB host that also runs OmniRoute with mem_limit: 10g. High for the config, inferred for the swap
7 Strata: a real, fast-moving MIT engine built around Flash-Next expert tiering. It has R9700 HIP builds and a layer split for 2 cards, but its design wants all experts in RAM (~88 GB for UD-Q4_K_XL), it does not run the dense 27B at all, and its numbers are self-reported on hosts with 47-503 GB of RAM. Not worth it on 32 GB. Revisit after a RAM upgrade (board max 128 GB). Medium

1. Platform facts that drive everything

  • R9700: 32 GB GDDR6, 256-bit, 20 Gbps, 640 GB/s, PCIe 5.0 x16, 300 W (Puget Systems listing, Tom's Hardware launch). AMD's own product page timed out on fetch, so this is from third-party listings.
  • Effective bandwidth, calibrated from this repo's data: no-spec decode for the 27B Q4 on one R9700 is 25-30 t/s across three outside sources (qwen3.8-27b-throughput-tuning.md). 16.8 GB/token x ~26.4 t/s ≈ ~440 GB/s effective (~69 % of peak). All estimates below use 420-450 GB/s.
  • ASRock B550M Pro4 slots, from the ASRock manual (PDF) spec page with a Matisse/Vermeer CPU: "2 x PCI Express x16 Slots (PCIE1: Gen4x16 mode; PCIE3: Gen3 x4 mode)", plus one PCIe 3.0 x1 slot (PCIE2). M2_1 is "up to Gen4x4", M2_2 "up to Gen3 x2", and M2_2 shares lanes with SATA3_5_6. Memory is dual channel, max 128 GB. So the second R9700 runs at PCIe 3.0 x4 (~3.9 GB/s per direction).
  • PCIE3 is behind the chipset. The manual does not say so explicitly. It follows from the AM4 lane budget: the CPU gives x16 to the GPU, x4 to the CPU-attached M.2, and x4 as the chipset uplink. B550's own lanes and uplink are PCIe 3.0 (ComputerBase B550 test, Guru3D). Card 2 therefore shares the chipset's PCIe 3.0 x4 uplink with M2_2, SATA, USB and LAN. GPU-to-GPU traffic crosses CPU root complex → chipset → slot. Whether ROCm P2P works across that path was not verified. Assume it falls back to staging through host memory.
  • DDR4: dual-channel DDR4-3200 peaks at 2 x 8 B x 3200 MT/s = 51.2 GB/s. Plan for ~35-45 GB/s effective (estimate).
  • RAM budget today: docker-compose.yml sets --cache-ram ${LLAMA_CACHE_RAM:-24576} (24 GiB) and OmniRoute mem_limit: 10g (OMNIROUTE_MEMORY_MB=8192). That is 34 GiB of permitted use on a 32 GB host before the OS, qdrant and page cache. --cache-ram is a ceiling that fills up over time, which fits the reported 94 % plus swap. docs/architecture.md already says to "keep it well below the host's free RAM".
  • Physical and power, unverified: two 300 W cards plus a 142 W-PPT 5950X need a PSU of roughly 850 W or more. 500k-context-…roadmap.md already flagged PSU headroom as the limit. A dual-slot card in the bottom slot of a micro-ATX board can also need a fifth rear bracket in the case. Check both before the card arrives.

2. Qwen3.8-27B UD-Q8_K_XL

File facts (HF tree API plus the GGUF header, parsed from the first 12 MB of each file):

UD-Q4_K_XL (today) UD-Q8_K_XL
File size 17.56 GB 31.46 GB
Tensor types mixed IQ4/Q4_K-Q6_K, Q8_0 for small tensors Q8_0 FFN/attention, BF16 attn_q/k/v and output
token_embd (row lookup, not streamed) 0.72 GB 1.35 GB
Bytes streamed per token ~16.8 GB ~30.1 GB (1.79x)

Sources: HF tree API, file page. Both files carry the built-in MTP head (qwen35.block_count = 65, nextn_predict_layers = 1).

Architecture (unsloth config.json): dense, Qwen3_5ForConditionalGeneration, 64 layers, hybrid. 48 Gated DeltaNet (linear_attention) layers and 16 full_attention layers (full_attention_interval: 4), 24 Q heads, 4 KV heads, head_dim 256, 262144 native context. Every weight is read for every token.

Fit on 2x32 GB. KV is 16 layers x 2 x 4 heads x 256 x 1.0625 B (q8_0) ≈ 34.8 KB/token, so ~9.1 GB at 262144 tokens. 31.5 GB weights + 1.1 GB DFlash2 Q4_K_M (z-lab GGUF tree) + 9.1 GB KV + ~3-5 GB compute buffers per card comes to ~46-50 GB of 64 GB (estimate). It fits with room. It does not fit on one card.

Decode estimate (bandwidth-bound, ~440 GB/s per card):

Config Math Est. decode, no spec With speculation (x1.7, the ratio this repo measures for DFlash2: 45.9 / ~27)
Q4_K_XL, 1 card (today) 16.8 / 440 = 38 ms ~26 t/s (measured 25-30) 45.9 t/s measured
Q8_K_XL, -sm layer 30.1 / 440 = 68 ms, cards in series ~15 t/s ~25 t/s
Q8_K_XL, -sm tensor at 1.31x 15 x 1.31 ~19 t/s DFlash2 crashes here (see §3). MTP x1.5 gives ~29 t/s if stable
Q4_K_XL, -sm tensor at 1.31x 26 x 1.31 ~34 t/s MTP x1.5 gives ~50 t/s if stable

All of these except the measured row are estimates. Q8_K_XL roughly halves single-stream speed against today. Pick it only if Q4 quality is shown to be the bottleneck on your own tasks.

3. llama.cpp multi-GPU on ROCm

Source: docs/multi-gpu.md and common/arg.cpp on master (fetched 2026-10-29). --split-mode takes none, layer, row, tensor.

Mode What it does Decode on bandwidth-bound cards Status
layer (default) Pipeline: each GPU owns a contiguous block of layers and their KV No single-stream gain. One token walks GPU0 then GPU1, each reading only its half, so the total time is the same. Prefill can overlap ubatches across GPUs. The docs say: "priority is a fast prefill. Can tolerate slow interconnect speeds" Mature
row Old row-split of dense matmuls Docs: "Deprecated … Superseded by tensor". PR #19378: needs "a synchronization after every single operation", only worth it on old slow GPUs Avoid
tensor Meta-backend tensor parallelism: weights and KV split, all-reduce at sync points Both cards read their half of each layer at the same time, so it is the only mode with a decode win. Docs: "Performance should be good for multiple NVIDIA GPUs using the CUDA backend, no guarantees otherwise" EXPERIMENTAL. Merged in #19378 (2026-04-09)

Measured -sm tensor numbers that are relevant here:

  • Qwen3.8-27B Q4_K_XL, 2x RX 7900 XTX, ROCm: 47.06 t/s = 1.31x one card. Per-card DRAM utilization drops from 65.9 % to 43.0 % when split (#28863, closed without a fix). This is the best proxy for a 2x R9700 result.
  • The #19378 author on ROCm (RX 6800 + MI50, RX 9060 XT + MI100): "the performance is bad vs. the -sm layer baseline". On 2x RTX 4090, the dense Gemma4 31B decoded 1.36x faster, while pp2048 fell to 0.61x and the MoE model got slower.
  • #27825 (merged 2026-09-15) turned on the internal AllReduce for HIP. On an RX 9070 at x16 plus an RX 6800 XT at PCIe 4.0 x4, Gemma4 31B Q6_K: tg 23.62 → 24.15 (+2 %), pp +16 %. RCCL is opt-in at build time (-DGGML_HIP_RCCL=ON), "disabled because … it was not universally beneficial" (multi-gpu.md). Whether the official server-rocm image builds with RCCL was not checked.
  • 2x R9700 user, Gemma4 26B-A4B Q8_0: -sm tensor f16 KV 29.7 t/s vs -sm layer 19 t/s (#23567). Same card, a different (MoE) model.
  • A 5950X + dual-R9700 user runs "Qwen 3.6 27B Q8 … with tensor split" and reports trouble mixing tensor-split and layer-split processes on ROCm (#28582, open).

Constraints and bugs that apply to this stack:

  • Architecture support: qwen35 and qwen4exp are not on the deny list in llm_arch_supports_sm_tensor (src/llama-arch.cpp, master). -sm tensor was re-enabled for qwen4exp in #28569.
  • -sm tensor needs flash-attn on (llama-context.cpp). Quantized KV is supported since #23792 (2026-06-01). llama.cpp's docs/multi-gpu.md still says it errors, so the doc is stale there. --fit is "not supported with tensor", so set --ctx-size by hand.
  • DFlash2 + -sm tensor crashes: GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) (#28777, open). The ROCm variant (#27829) was closed as stale, and the fix PR #27858 is open. Its tester still hit a different assert, and a commenter says "this is already a problem for ROCm". The workaround in #28777 is -sm layer.
  • MTP + -sm tensor: reproducible lockups on CUDA (#27122, closed as stale). One ROCm report (2x V620, Q8_K_XL) saw one hard abort in ~67 h.
  • gfx1201 flash-attn garbage at some context lengths for Qwen3.8-27B (head_dim 256): immediate EOS in bands, reproduced on 2x R9700, Linux, --tensor-split 1.7,1 (#30884, open). Another commenter could not reproduce it through /v1/chat/completions. Run a needle test on any new config.

PCIe x4 chipset slot, estimated effects:

  • -sm layer decode: one hidden-state hop per token (5120 x 4 B ≈ 20 KB). Negligible. Prefill: ~10 MB per 512-token ubatch hop, ~3 ms at 3.9 GB/s. Small.
  • -sm tensor decode: ~2 all-reduces per layer x 64 layers ≈ 128 small reductions per token. These are latency-bound, not size-bound. At an assumed 20-50 µs each through host memory, that is 2.5-6.4 ms per token, 10-25 % of a 25 ms token (estimate). This is the likely reason #28863 lands at 1.31x and not ~1.9x, and an x4-through-chipset link will not do better.
  • -sm tensor prefill: ~2 x tokens x 5120 x 2 B per layer. For a 2048-token ubatch that is ~2.7 GB over 64 layers, ~0.7 s at 3.9 GB/s, so prefill is capped at roughly 3000 t/s plus compute (estimate). Expect prefill to get worse than layer split, as in the #19378 tables.

Flags:

  • -ts/--tensor-split sets proportions. With layer the default split follows free memory. With tensor the default is an even split.
  • -mg/--main-gpu applies only to -sm none (pick the card) or row (intermediate results). It does nothing for layer/tensor.
  • Draft placement: --spec-draft-device/-devd (default: follows --device), -ngld, -otd (common/arg.cpp). Under layer, the output head sits on the last GPU. Place the drafter on one card and A/B both cards. No source documents which placement is better for DFlash2, which reads hidden states from several target layers (n_extract=5) spread over both cards.

4. Qwen3.8-Flash-Next UD-Q4_K_XL

Model facts (unsloth model card, GGUF header of shard 1, and shard headers 2-4):

  • general.architecture = qwen4exp, described as "A Preview of the Qwen4 Architecture". MoE: 125B total, 6B activated, plus a 51B n-gram embedding and a 4B MTP head. 48 layers, hidden size 2560, layout 12 x (3 x Gated DeltaNet + 1 x Qwen Sparse Attention), each layer followed by an MoE block. 512 experts, 10 routed + 1 shared, expert FFN dim 640. Native context 262144. License qwen-community-1.0. The 27B is Apache 2.0 per qwen3.8-27b-quant.md.
  • UD-Q4_K_XL = 111.33 GB in 4 shards (0.01 + 49.86 + 49.38 + 12.09 GB, HF tree API). It splits into:
    • per_layer_token_embd (n-gram table): 28.80 GB IQ4_NL, read a few rows per token. The same size in every UD quant.
    • routed experts ffn_{gate,up,down}_exps: 77.02 GB
    • everything else (attention/DeltaNet, shared experts, routers, hyper-connections, output head): ~4.8 GB, mostly Q8_0
  • Bytes per token: 4.8 + 77.02 x 10/512 (1.50) ≈ ~6.3 GB, 2.7x less than the 27B Q4. The all-in-VRAM ceiling is 6.3 / 440 → ~70 t/s. Batch-1 MoE kernels gather scattered experts and reach less of peak, so the estimate is ~40-60 t/s if it fit in VRAM. For scale, Strata reports 63 (prose) / 87 (code) t/s on one R9700 with this file and its own engine (§6).
  • Quality (Qwen card, BF16): SWE-bench Pro 62.5 vs 61.7 for the 27B, SWE-bench Multilingual 81.0 vs 73.8, DeepSWE 58.7 vs 42.2, Toolathlon 73.5 vs 67.1. Unsloth's quant table (run guide) gives mean KLD: UD-Q2_K_XL 0.225, UD-IQ3_XXS 0.165, UD-Q3_K_XL 0.107, UD-IQ4_XS 0.084, UD-Q4_K_XL 0.047.

Fit in 64 GB VRAM (non-PLE = total − 28.8 GB, the PLE left on SSD via --lazy-mode, §5):

Quant Total Non-PLE (needs VRAM) Result on 2x32 GB
UD-Q4_K_XL 111.3 82.5 ~20-24 GB of experts must live off-GPU. Does not fit
UD-IQ4_XS 93.7 64.9 ~6-9 GB off-GPU after KV and buffers
UD-Q3_K_XL 90.0 61.2 Borderline, a few GB off-GPU
UD-IQ3_XXS 82.0 53.2 Fits, ~10 GB headroom
UD-Q2_K_XL 78.9 50.1 Fits, ~13 GB headroom

KV is small. Only the 12 QSA layers have one: 12 x 2 KV heads x 256 x 2 (K+V) x 2 B ≈ 24.6 KB/token f16, ~6.4 GB at 262144 tokens, ~3.4 GB at q8_0 (estimate; ignores indexer keys).

llama.cpp support status:

  • Merged: model support #27742 (2026-08-27), MTP #29761 (2026-10-01), -sm tensor re-enabled #28569, many QSA fixes through 2026-10-27 (e.g. #31243).

  • Open bugs that hit this setup:

    • #29149: HIP livelock during load on an R9700, with both UD-IQ3_XXS and UD-Q4_K_XL. The root-cause comment places it in the mmap upload path; "the non-mmap path stages through pinned buffers instead". Try --load-mode none.
    • #29811 and #30860: MTP asserts at startup.
    • #30887: the DFlash2 drafter crashes on the first prompt (R9700, ROCm).
    • #30815: QSA chunking, still open.

    Usable, but rough on HIP.

5. Offloading to DDR RAM and SSD

What llama.cpp offers (master common/arg.cpp):

  • -ot/--override-tensor <regex>=<buft> places matching tensors, e.g. blk\.[0-9]+\.ffn_.*_exps=CPU. -cmoe/--cpu-moe puts all experts on the CPU. -ncmoe/--n-cpu-moe N puts the experts of the first N layers on the CPU. These are MoE-only, so they do nothing for the dense 27B (already noted in docs/architecture.md).
  • --moe-cache-mib N: an LRU cache in VRAM for host-resident experts, merged in #29887 (2026-10-07). On Flash-Next Q4_0 the PR measured 4090 25.0 → 39.4 t/s and 5090 30.8 → 67.8 t/s, on an EPYC host with RAM for all experts. "Small batches only (<= 32 tokens)".
  • -lzm/--lazy-mode {on,auto,off}: "read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)". auto applies to tensors over 4 GiB, which covers the 28.8 GB PLE table by default (renamed in #27969). Per #30772 (closed unmerged), mapped PLE pages still accumulate in RSS and single-threaded madvise can bottleneck NVMe.
  • -lm/--load-mode {auto,none,mmap,…}. With mmap, a tensor that fits nowhere is paged from disk implicitly.
  • Pipeline parallelism turns off as soon as any tensor override is used (!has_tensor_overrides() in llama-context.cpp, as described in the open PR #29963). So -ot/-ncmoe plus two GPUs also loses the layer-split prefill overlap.

Numbers for this box:

  • Dense 27B: no useful offload. Every weight is read every token, and DDR4 is ~51 GB/s against ~440 GB/s VRAM. Each GB moved to RAM costs ~20-25 ms/token on the CPU (estimate). With two cards the Q8_K_XL fits in VRAM anyway. The only RAM tier that helps is --cache-ram (prompt cache), as recorded in qwen3.8-27b-throughput-tuning.md.
  • Flash-Next UD-Q4_K_XL, if RAM were free (hypothetical): ~22 GB of experts on CPU = 29 % of experts, ~0.43 GB read per token from DDR4 at ~40 GB/s ≈ 11 ms. The GPU part ≈ (4.8 + 1.07) / 440 ≈ 13 ms. Layers are sequential, so the parts add up: ~24 ms ≈ ~40 t/s, more with --moe-cache-mib (estimate). Prefill uploads the host experts once per ubatch: 22 GB over PCIe 4.0 x16 at ~25 GB/s ≈ 0.9 s per ubatch, so ~18 s of transfer for a 40k-token prompt at -ub 2048 (estimate).
  • Flash-Next UD-Q4_K_XL, on this box: RAM is full (§1). Even with --cache-ram cut to near zero, 32 GB minus OS, OmniRoute (up to 10 GB) and qdrant leaves maybe ~15 GB (estimate), less than the ~22 GB needed. The rest is paged from the CPU-attached Gen4 x4 NVMe. At ~0.43 GB/token and an assumed 3-6 GB/s for ~1-3 MB expert reads, that is ~70-140 ms/token, ~6-12 t/s. Prefill re-reads ~22 GB per ubatch from disk, so minutes per 40k-token prompt. The page cache will also compete with the swap that is already happening. An outside data point: an RX 9060 XT 16 GB with 64 GB DDR4-3200 got "8 t/s on short context. Longer context 3 t/s" on llama.cpp ROCm (#29149 comment). Not viable.
  • PLE table on SSD (any Flash-Next quant): 32 rows per token (#30772) of 90 B each (Strata AMD_HIP.md), each on its own 4 KB page, so ≤128 KB of random IO per token. At 50 t/s that is a few thousand IOPS, trivial. Prefill touches ~32 pages per token, ~1.3 M random reads for 40k tokens. On NVMe at high queue depth that is a few seconds, but single-threaded madvise (#30772) can make it much slower. Strata measured +22.5 % prefill with the table in RAM against direct reads (§6). Decode impact: low. Prefill: measure it.

6. Strata

Identity. github.com/Niko1221/Strata: "Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost". C++, MIT (LICENSE). Its docs/HOW_IT_WORKS.md says it is "Built with parts of llama.cpp / ggml (MIT)". It supports Qwen3.8-Flash-Next (ISTA-DASLab GSQ-RCO quants recommended, Unsloth UD-IQ4_XS…Q8_0 supported), Qwen3.6-35B-A3B and DeepSeek V4.1-Flash. There is no dense Qwen3.8-27B support, so it cannot replace today's model.

How it tiers (docs/HOW_IT_WORKS.md, docs/MODELS.md):

  • VRAM: attention/DeltaNet, routers, shared experts, output head, MTP, KV ("from 64K: only its most-read part, the rest streams from RAM"), plus an expert cache that fills the rest of VRAM with the most-used experts and adapts at runtime.
  • RAM: "all 24,576 experts, pinned". The CPU computes cache misses in place, concurrently with the GPU (AVX-512/AVX2). --pcie-frac splits misses between PCIe fetch and CPU compute. On gfx12 it defaults to 0.15 after a measured +9.9 % to +25.6 % decode on an R9700 (docs/AMD_HIP.md).
  • SSD: the 28.8 GB n-gram table, "read a few rows per token through the OS cache". From 0.2.7 it keeps the table in RAM when there is room for it plus 8 GiB.
  • Low-RAM mode: "setup maps them from the model files and keeps in RAM only what the card does not hold … With a small card most experts come from the SSD and it is much slower".
  • Multi-GPU (docs/MULTI_GPU.md): pipeline (layer) split, "not tensor parallelism". "No NVLink or peer-to-peer access is needed, cards on x4 or x1 slots work". Each card caches the experts of its own layers, so two cards hold about twice as many experts. docs/AMD_HIP.md documents HIP layer split for gfx1201 pairs and adds "A split is worth it when no single card holds the model's experts".

AMD/ROCm: the R9700 (gfx1201) is "Maintainer-tested", with Linux hip-rx9070 engine packages in every recent release (releases).

RAM requirement for the requested file: docs/MODELS.md lists UD-Q4_K_XL as "111 GB download, 72 GiB experts, RAM for all experts ~88 GB". With less RAM it "keeps as many as fit and reads the rest from the SSD (much slower)".

Claimed vs verifiable:

Claim Source Host RAM Verifiable?
UD-Q4_K_XL on one R9700: 63 t/s prose, 87 code, 32K prompt 1,110-1,320 t/s docs/MODELS.md, "engines before 0.1.45.3" not stated there Self-reported
IQ3_S on one R9700: 87.6-101.9 t/s across 1K-128K community R9700 report ~503 GiB, 2x EPYC 9334 Has per-request JSON and hashes. "Single GPU still uses CPU experts, system RAM and PCIe; these rates do not establish desktop-CPU speed or minimum RAM"
R9700 + 9070 XT, 47 GB RAM, Coder IQ1_M: 45.5 / 48.3 t/s decode docs/AMD_HIP.md 47 GB Self-reported
"125B+ MoE on a single 12GB GPU", "25-38 tok/s" stratallm.org n/a The site calls itself "Unofficial Community Documentation". Its speed figures contradict each other and link no method. Ignore it

None of these numbers were rerun for this doc. The project says its benchmark protocol "is still a draft" (docs/COMMUNITY_BENCHMARKS.md).

Maturity: repo created 2026-09-24. ~4,690 commits, ~3,594 of them by the one maintainer. ~36k stars, 895 open issues, a release roughly daily (0.1.40 → 0.2.7 in three weeks). Installation is curl … | sh, an AppImage, or Docker (docs/DOCKER.md, not read). It is young, single-maintainer and churning, and it would need its own container and gateway wiring here.

Verdict for this box: not worth it now. The 32 GB of RAM defeats its core design (all experts pinned in RAM). Low-RAM mode on 2x32 GB VRAM would still read ~20 GB of experts from NVMe for UD-Q4_K_XL. It cannot run the dense 27B that this stack is tuned around. Revisit if RAM goes to 96-128 GB (the board takes 128 GB). On a big-RAM host its self-reported R9700 Flash-Next numbers (60-100 t/s) are well above llama.cpp's 27B, so it would be worth a real A/B then.

7. Recommendation

Primary: keep Qwen3.8-27B-UD-Q4_K_XL + DFlash2 and run two single-GPU replicas, one per card.

Why:

  • (1) Layer split gives a dense model no decode gain.
  • (2) Tensor split is experimental on ROCm, measured at 1.31x, crashes with DFlash2, and would run over a PCIe 3.0 x4 chipset link.
  • (3) Q8_K_XL roughly halves decode.
  • (4) Flash-Next at a usable quant does not fit, and RAM offload pages from NVMe.
  • (5) The workload is 2 parallel agents. The log already shows decode falling to p10 15 t/s while the other slot prefills (qwen3.8-27b-throughput-tuning.md). Two replicas remove that interference and double each slot's context.

Per-instance flags (two services, identical except the device). Each is today's command with these changes:

--device ROCm0               # ROCm1 for the second service; confirm names with --list-devices
--n-gpu-layers 999
--ctx-size 262144 --parallel 1   # full 262144 per agent instead of 131072
--flash-attn on --cache-type-k q8_0 --cache-type-v q8_0
--cache-ram 6144             # per instance; both together must stay well under free RAM (§1)
--spec-type draft-dflash -md /models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-n-max 7
--jinja

GPU_MAX_HW_QUEUES=1 stays on both services (docs/architecture.md#gpu-access). Gateway: register both backends in OmniRoute and make sure a conversation stays on one backend (prompt-cache affinity). Otherwise every switch is a full re-prefill. Whether OmniRoute can balance two connections per conversation was not checked.

Independently of the second card: lower LLAMA_CACHE_RAM. 24 GiB on a 32 GB host with a 10 GB OmniRoute limit oversubscribes RAM.

Alternatives to A/B, in order:

  1. Single fast stream: Q4_K_XL with -sm tensor -ts 1,1 -fa on --fit off --ctx-size <set by hand> --spec-type draft-mtp --spec-draft-n-max 2 (the built-in MTP head; DFlash2 crashes with tensor). Estimate ~34 t/s no spec, ~50 with MTP if stable. Watch for #27122-style lockups.
  2. Quality: Q8_K_XL (or UD-Q6_K_XL at 25.3 GB) with -sm layer, DFlash2 placed with -devd, ~25 t/s estimated. Only if Q4 quality is shown to fail on your tasks.
  3. Flash-Next experiment: UD-IQ3_XXS or UD-Q2_K_XL, all in VRAM: -sm layer -ngl 999 --lazy-mode auto --load-mode none -fa on -c 131072, MTP via -md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp once #29811/#30860 are fixed. Judge it on your own agent tasks. The KLD at these sizes is 3.5-5x that of UD-Q4_K_XL.

8. What to benchmark once the card arrives

  1. Topology: sudo lspci -vv LnkSta for both cards (expect 16 GT/s x16 and 8 GT/s x4). Run rocm-smi --showtopo and rocm-bandwidth-test for host↔GPU and GPU↔GPU (does P2P work through the chipset?). Check the PSU and the case slot fit first.
  2. Per-card baseline: llama-bench -m Qwen3.8-27B-UD-Q4_K_XL.gguf -dev ROCm0 and then -dev ROCm1, -fa 1 -ctk q8_0 -ctv q8_0 -p 512,8192 -n 128 -d 0,32768. Decode should match. Load time and prompt-cache restores are where the x4 link shows.
  3. Split modes: the same bench with -sm layer vs -sm tensor -ts 1,1 for Q4_K_XL and Q8_K_XL. Compare tg against #28863's 1.31x and pp against layer.
  4. Server replay (fixed ~40k and ~2k prompts, production temperature, concurrency 1 and 2) for: (a) two replicas, (b) one -sm layer --parallel 2, (c) -sm tensor + MTP. Record timings.predicted_per_second, prompt_per_second, draft acceptance, and the p10 decode under concurrency.
  5. Correctness: a needle test at ~6-10k and ~60-90k tokens on each config, because #30884 reproduces on 2x R9700 with --tensor-split. Also check that DFlash2 output under -sm layer matches single-card output on a fixed prompt.
  6. RAM: free -h and vmstat 5 after an hour of agent traffic, before and after lowering --cache-ram. Swap-in should be ~0.
  7. Optional Flash-Next: that UD-IQ3_XXS/UD-Q2_K_XL loads at all on HIP (#29149, try --load-mode none), tg/pp at 4k/32k, and prefill time with the PLE table on NVMe.

Open questions

  • Does OmniRoute support two backends with sticky routing per conversation?
  • Does the ghcr.io/ggml-org/llama.cpp:server-rocm image build with GGML_HIP_RCCL? It only matters for -sm tensor.
  • Does ROCm P2P work between a CPU-attached slot and a chipset-attached slot on B550? It is unverified, and the answer probably does not matter outside tensor mode.
  • Is there a B550/X570 board with CPU x8/x8 bifurcation that fits this chassis? It would give both cards Gen4 x8. It is only worth it if step 3 shows tensor scaling well. No specific board was checked.

Sources

Reference in new issue View Git Blame Copy Permalink