haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Feat/dual gpu' (#67) from feat/dual-gpu into main

Browse Source 
Reviewed-on: #67
This commit was merged in pull request #67.
This commit is contained in:
haylan committed 2026-11-04 06:23:16 +00:00
2 parents 1484907ff8 baf54002cd
commit 9fbdf3e664
2 files changed
+241
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/research/README.md
+1
View File
@@ -11,6 +11,7 @@ Each file here is a **dated snapshot**: research or a decision record as it stoo
| [colibri-ai-stack-evaluation.md](colibri-ai-stack-evaluation.md) | Current | Rejected option ("watch, don't integrate"). It lists ComfyUI, which has since been removed |
| [connecting-cloud-providers.md](connecting-cloud-providers.md) | Superseded by the OmniRoute migration | Written for LiteLLM. It mentions Open WebUI, which has been removed |
| [dashscope-websearch-selfhost-options.md](dashscope-websearch-selfhost-options.md) | Current | Why DashScope web search isn't emulated |
| [dual-r9700-speed-and-offload.md](dual-r9700-speed-and-offload.md) | Current | Second R9700: split modes, 27B Q8_K_XL vs Flash-Next, RAM/SSD offload, Strata. Proposals only, nothing applied |
| [fast-model-choice.md](fast-model-choice.md) | Current (drift) | Classifier model choice. Its addendum records the service name (`qwen-classifier`) and quant drift |
| [langchain-pgvector-vs-litellm-pgvector.md](langchain-pgvector-vs-litellm-pgvector.md) | Superseded by the OmniRoute migration | LiteLLM-era RAG comparison |
| [lazytainer-omniroute-idle-stop.md](lazytainer-omniroute-idle-stop.md) | Superseded: Lazytainer and `switch-model.sh` removed | Root cause of why Lazytainer's idle-stop never fired behind the gateway, which is the reason it was removed |
docs/research/dual-r9700-speed-and-offload.md
+240
View File
@@ -0,0 +1,240 @@
# Research: fastest inference on 2x R9700 (27B Q8_K_XL vs Flash-Next, multi-GPU modes, RAM/SSD offload, Strata)
**Date:** 2026-10-29
**Question:** A second Radeon AI PRO R9700 arrives in about a week. What gives the fastest decode (and prefill) for agentic coding on this box: `Qwen3.8-27B-UD-Q8_K_XL`, `Qwen3.8-Flash-Next-UD-Q4_K_XL`, or something else? Which llama.cpp multi-GPU mode helps, does RAM/SSD offload work with 32 GB of RAM, and is Strata worth switching to?
**Scope note:** research only. Nothing in `docker-compose.yml` or `.env.example` was changed, so nothing needs deploying.
**Hardware assumed:** Ryzen 9 5950X, 32 GB DDR4 dual channel (reported ~94 % used, swap in use), ASRock B550M Pro4, 1 TB NVMe, Ubuntu 24.04, kernel 6.8, 1x R9700 now and 2x soon.
## TL;DR
| # | Finding | Confidence |
|---|---|---|
| 1 | **The second card does not make the dense 27B decode faster with `-sm layer`.** Layer split runs the cards one after the other, so one token still reads every weight at one card's bandwidth. It only adds VRAM. | High (llama.cpp docs) |
| 2 | **`UD-Q8_K_XL` is a quality move, not a speed move.** It reads ~30.1 GB per token against ~16.8 GB for today's `UD-Q4_K_XL`. Estimated decode is ~15 t/s without speculation and ~25 t/s with DFlash2, against ~46 t/s measured today. | Medium (estimate, math below) |
| 3 | **`-sm tensor` is the only mode that can speed up one stream, and on ROCm it is immature.** The one measured 2-card ROCm result for this exact model is 1.31x one card (2x 7900 XTX). DFlash2 crashes with `-sm tensor` (open bug), and MTP plus `-sm tensor` has lockup reports. Your second slot is PCIe 3.0 x4 behind the chipset, which hurts the per-layer all-reduces. | Medium |
| 4 | **Best use of the second card for 2 parallel agents: two independent single-GPU `llama-server` replicas**, one per card, each `--parallel 1`. Each agent keeps today's ~46 t/s, the decode drop when the other slot prefills goes away, and each slot gets the full 262144 tokens instead of 131072. Needs a second backend connection in OmniRoute with per-conversation stickiness. | Medium-high |
| 5 | **Flash-Next UD-Q4_K_XL (111.3 GB) does not fit.** Even with the 28.8 GB n-gram table left on SSD, 82.5 GB of weights remain for 64 GB of VRAM. The ~20-24 GB of experts left over would come from NVMe, because RAM is full. Estimated decode is then ~6-12 t/s. Only the all-in-VRAM quants (`UD-Q2_K_XL`, `UD-IQ3_XXS`) are realistic, at a large quality cost (mean KLD 0.22 / 0.17 vs 0.047). | Medium |
| 6 | **RAM is the binding constraint, and compose itself is a likely cause of the swap.** `--cache-ram` is set to 24576 MiB (24 GiB) on a 32 GB host that also runs OmniRoute with `mem_limit: 10g`. | High for the config, inferred for the swap |
| 7 | **Strata:** a real, fast-moving MIT engine built around Flash-Next expert tiering. It has R9700 HIP builds and a layer split for 2 cards, but its design wants all experts in RAM (~88 GB for UD-Q4_K_XL), it does not run the dense 27B at all, and its numbers are self-reported on hosts with 47-503 GB of RAM. Not worth it on 32 GB. Revisit after a RAM upgrade (board max 128 GB). | Medium |
## 1. Platform facts that drive everything
- **R9700:** 32 GB GDDR6, 256-bit, 20 Gbps, **640 GB/s**, PCIe 5.0 x16, 300 W ([Puget Systems listing](https://www.pugetsystems.com/parts/Video-Card/AMD-Radeon-AI-PRO-R9700-32GB-16908/), [Tom's Hardware launch](https://www.tomshardware.com/pc-components/gpus/amd-launches-radeon-ai-pro-r9700-to-challenge-nvidias-ai-market-dominance)). AMD's own product page timed out on fetch, so this is from third-party listings.
- **Effective bandwidth, calibrated from this repo's data:** no-spec decode for the 27B Q4 on one R9700 is 25-30 t/s across three outside sources ([`qwen3.8-27b-throughput-tuning.md`](qwen3.8-27b-throughput-tuning.md)). 16.8 GB/token x ~26.4 t/s ≈ **~440 GB/s effective (~69 % of peak)**. All estimates below use 420-450 GB/s.
- **ASRock B550M Pro4 slots**, from the [ASRock manual (PDF)](https://download.asrock.com/Manual/B550M%20Pro4.pdf) spec page with a Matisse/Vermeer CPU: "2 x PCI Express x16 Slots (PCIE1: Gen4x16 mode; PCIE3: Gen3 x4 mode)", plus one PCIe 3.0 x1 slot (PCIE2). M2_1 is "up to Gen4x4", M2_2 "up to Gen3 x2", and M2_2 shares lanes with SATA3_5_6. Memory is dual channel, max 128 GB. So **the second R9700 runs at PCIe 3.0 x4 (~3.9 GB/s per direction)**.
- **PCIE3 is behind the chipset.** The manual does not say so explicitly. It follows from the AM4 lane budget: the CPU gives x16 to the GPU, x4 to the CPU-attached M.2, and x4 as the chipset uplink. B550's own lanes and uplink are PCIe 3.0 ([ComputerBase B550 test](https://www.computerbase.de/2020-06/amd-b550-chipsatz-mainboard-test/), [Guru3D](https://www.guru3d.com/review/amd-ryzen-3-3100-and-3300x-review/page-3/)). Card 2 therefore shares the chipset's PCIe 3.0 x4 uplink with M2_2, SATA, USB and LAN. GPU-to-GPU traffic crosses CPU root complex → chipset → slot. Whether ROCm P2P works across that path was not verified. Assume it falls back to staging through host memory.
- **DDR4:** dual-channel DDR4-3200 peaks at 2 x 8 B x 3200 MT/s = **51.2 GB/s**. Plan for ~35-45 GB/s effective (estimate).
- **RAM budget today:** `docker-compose.yml` sets `--cache-ram ${LLAMA_CACHE_RAM:-24576}` (24 GiB) and OmniRoute `mem_limit: 10g` (`OMNIROUTE_MEMORY_MB=8192`). That is 34 GiB of permitted use on a 32 GB host before the OS, qdrant and page cache. `--cache-ram` is a ceiling that fills up over time, which fits the reported 94 % plus swap. `docs/architecture.md` already says to "keep it well below the host's free RAM".
- **Physical and power, unverified:** two 300 W cards plus a 142 W-PPT 5950X need a PSU of roughly 850 W or more. [`500k-context-…roadmap.md`](500k-context-2-parallel-agents-gpu-roadmap.md) already flagged PSU headroom as the limit. A dual-slot card in the bottom slot of a micro-ATX board can also need a fifth rear bracket in the case. Check both before the card arrives.
## 2. Qwen3.8-27B `UD-Q8_K_XL`
**File facts** (HF tree API plus the GGUF header, parsed from the first 12 MB of each file):
| | UD-Q4_K_XL (today) | UD-Q8_K_XL |
|---|---|---|
| File size | 17.56 GB | **31.46 GB** |
| Tensor types | mixed IQ4/Q4_K-Q6_K, Q8_0 for small tensors | Q8_0 FFN/attention, **BF16** `attn_q/k/v` and `output` |
| `token_embd` (row lookup, not streamed) | 0.72 GB | 1.35 GB |
| **Bytes streamed per token** | **~16.8 GB** | **~30.1 GB (1.79x)** |
Sources: [HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-27B-GGUF/tree/main), [file page](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_info=Qwen3.8-27B-UD-Q8_K_XL.gguf). Both files carry the built-in MTP head (`qwen35.block_count = 65`, `nextn_predict_layers = 1`).
**Architecture** ([unsloth config.json](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/raw/main/config.json)): dense, `Qwen3_5ForConditionalGeneration`, 64 layers, hybrid. 48 Gated DeltaNet (`linear_attention`) layers and 16 `full_attention` layers (`full_attention_interval: 4`), 24 Q heads, 4 KV heads, head_dim 256, 262144 native context. Every weight is read for every token.
**Fit on 2x32 GB.** KV is 16 layers x 2 x 4 heads x 256 x 1.0625 B (q8_0) ≈ 34.8 KB/token, so ~9.1 GB at 262144 tokens. 31.5 GB weights + 1.1 GB DFlash2 Q4_K_M ([z-lab GGUF tree](https://huggingface.co/api/models/z-lab/Qwen3.8-27B-DFlash2-GGUF/tree/main)) + 9.1 GB KV + ~3-5 GB compute buffers per card comes to **~46-50 GB of 64 GB** (estimate). It fits with room. It does not fit on one card.
**Decode estimate** (bandwidth-bound, ~440 GB/s per card):
| Config | Math | Est. decode, no spec | With speculation (x1.7, the ratio this repo measures for DFlash2: 45.9 / ~27) |
|---|---|---|---|
| Q4_K_XL, 1 card (today) | 16.8 / 440 = 38 ms | ~26 t/s (measured 25-30) | **45.9 t/s measured** |
| Q8_K_XL, `-sm layer` | 30.1 / 440 = 68 ms, cards in series | **~15 t/s** | **~25 t/s** |
| Q8_K_XL, `-sm tensor` at 1.31x | 15 x 1.31 | ~19 t/s | DFlash2 crashes here (see §3). MTP x1.5 gives ~29 t/s if stable |
| Q4_K_XL, `-sm tensor` at 1.31x | 26 x 1.31 | ~34 t/s | MTP x1.5 gives ~50 t/s if stable |
All of these except the measured row are estimates. Q8_K_XL roughly halves single-stream speed against today. Pick it only if Q4 quality is shown to be the bottleneck on your own tasks.
## 3. llama.cpp multi-GPU on ROCm
Source: [`docs/multi-gpu.md`](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md) and `common/arg.cpp` on master (fetched 2026-10-29). `--split-mode` takes `none, layer, row, tensor`.
| Mode | What it does | Decode on bandwidth-bound cards | Status |
|---|---|---|---|
| `layer` (default) | Pipeline: each GPU owns a contiguous block of layers and their KV | **No single-stream gain.** One token walks GPU0 then GPU1, each reading only its half, so the total time is the same. Prefill can overlap ubatches across GPUs. The docs say: "priority is a fast prefill. Can tolerate slow interconnect speeds" | Mature |
| `row` | Old row-split of dense matmuls | Docs: "**Deprecated** … Superseded by `tensor`". [PR #19378](https://github.com/ggml-org/llama.cpp/pull/19378): needs "a synchronization after every single operation", only worth it on old slow GPUs | Avoid |
| `tensor` | Meta-backend tensor parallelism: weights **and** KV split, all-reduce at sync points | Both cards read their half of each layer at the same time, so it is the only mode with a decode win. Docs: "Performance should be good for multiple NVIDIA GPUs using the CUDA backend, no guarantees otherwise" | **EXPERIMENTAL**. Merged in [#19378](https://github.com/ggml-org/llama.cpp/pull/19378) (2026-04-09) |
**Measured `-sm tensor` numbers that are relevant here:**
- **Qwen3.8-27B Q4_K_XL, 2x RX 7900 XTX, ROCm: 47.06 t/s = 1.31x one card.** Per-card DRAM utilization drops from 65.9 % to 43.0 % when split ([#28863](https://github.com/ggml-org/llama.cpp/issues/28863), closed without a fix). This is the best proxy for a 2x R9700 result.
- The #19378 author on ROCm (RX 6800 + MI50, RX 9060 XT + MI100): "the performance is bad vs. the `-sm layer` baseline". On 2x RTX 4090, the dense Gemma4 31B decoded **1.36x** faster, while pp2048 fell to 0.61x and the MoE model got slower.
- [#27825](https://github.com/ggml-org/llama.cpp/pull/27825) (merged 2026-09-15) turned on the internal AllReduce for HIP. On an RX 9070 at x16 plus an RX 6800 XT at **PCIe 4.0 x4**, Gemma4 31B Q6_K: tg 23.62 → 24.15 (+2 %), pp +16 %. RCCL is opt-in at build time (`-DGGML_HIP_RCCL=ON`), "disabled because … it was not universally beneficial" (multi-gpu.md). Whether the official `server-rocm` image builds with RCCL was not checked.
- 2x R9700 user, Gemma4 26B-A4B Q8_0: `-sm tensor` f16 KV 29.7 t/s vs `-sm layer` 19 t/s ([#23567](https://github.com/ggml-org/llama.cpp/issues/23567)). Same card, a different (MoE) model.
- A 5950X + dual-R9700 user runs "Qwen 3.6 27B Q8 … with tensor split" and reports trouble mixing tensor-split and layer-split processes on ROCm ([#28582](https://github.com/ggml-org/llama.cpp/issues/28582), open).
**Constraints and bugs** that apply to this stack:
- Architecture support: `qwen35` and `qwen4exp` are **not** on the deny list in `llm_arch_supports_sm_tensor` (`src/llama-arch.cpp`, master). `-sm tensor` was re-enabled for qwen4exp in [#28569](https://github.com/ggml-org/llama.cpp/pull/28569).
- `-sm tensor` needs flash-attn on (`llama-context.cpp`). Quantized KV is supported since [#23792](https://github.com/ggml-org/llama.cpp/pull/23792) (2026-06-01). llama.cpp's `docs/multi-gpu.md` still says it errors, so the doc is stale there. `--fit` is "not supported with `tensor`", so set `--ctx-size` by hand.
- **DFlash2 + `-sm tensor` crashes:** `GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0)` ([#28777](https://github.com/ggml-org/llama.cpp/issues/28777), open). The ROCm variant ([#27829](https://github.com/ggml-org/llama.cpp/issues/27829)) was closed as stale, and the fix PR [#27858](https://github.com/ggml-org/llama.cpp/pull/27858) is open. Its tester still hit a different assert, and a commenter says "this is already a problem for ROCm". The workaround in #28777 is `-sm layer`.
- **MTP + `-sm tensor`:** reproducible lockups on CUDA ([#27122](https://github.com/ggml-org/llama.cpp/issues/27122), closed as stale). One ROCm report (2x V620, Q8_K_XL) saw one hard abort in ~67 h.
- **gfx1201 flash-attn garbage at some context lengths for Qwen3.8-27B** (head_dim 256): immediate EOS in bands, reproduced on **2x R9700, Linux, `--tensor-split 1.7,1`** ([#30884](https://github.com/ggml-org/llama.cpp/issues/30884), open). Another commenter could not reproduce it through `/v1/chat/completions`. Run a needle test on any new config.
**PCIe x4 chipset slot, estimated effects:**
- `-sm layer` decode: one hidden-state hop per token (5120 x 4 B ≈ 20 KB). Negligible. Prefill: ~10 MB per 512-token ubatch hop, ~3 ms at 3.9 GB/s. Small.
- `-sm tensor` decode: ~2 all-reduces per layer x 64 layers ≈ 128 small reductions per token. These are latency-bound, not size-bound. At an assumed 20-50 µs each through host memory, that is 2.5-6.4 ms per token, 10-25 % of a 25 ms token (estimate). This is the likely reason #28863 lands at 1.31x and not ~1.9x, and an x4-through-chipset link will not do better.
- `-sm tensor` prefill: ~2 x tokens x 5120 x 2 B per layer. For a 2048-token ubatch that is ~2.7 GB over 64 layers, ~0.7 s at 3.9 GB/s, so prefill is capped at roughly 3000 t/s plus compute (estimate). Expect prefill to get worse than layer split, as in the #19378 tables.
**Flags:**
- `-ts/--tensor-split` sets proportions. With `layer` the default split follows free memory. With `tensor` the default is an even split.
- `-mg/--main-gpu` applies only to `-sm none` (pick the card) or `row` (intermediate results). It does nothing for `layer`/`tensor`.
- Draft placement: `--spec-draft-device/-devd` (default: follows `--device`), `-ngld`, `-otd` (`common/arg.cpp`). Under `layer`, the output head sits on the last GPU. Place the drafter on one card and A/B both cards. No source documents which placement is better for DFlash2, which reads hidden states from several target layers (`n_extract=5`) spread over both cards.
## 4. Qwen3.8-Flash-Next `UD-Q4_K_XL`
**Model facts** ([unsloth model card](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), GGUF header of shard 1, and shard headers 2-4):
- `general.architecture = qwen4exp`, described as "A Preview of the Qwen4 Architecture". **MoE: 125B total, 6B activated, plus a 51B n-gram embedding and a 4B MTP head.** 48 layers, hidden size 2560, layout 12 x (3 x Gated DeltaNet + 1 x Qwen Sparse Attention), each layer followed by an MoE block. **512 experts, 10 routed + 1 shared**, expert FFN dim 640. Native context 262144. License `qwen-community-1.0`. The 27B is Apache 2.0 per [`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md).
- **UD-Q4_K_XL = 111.33 GB** in 4 shards (0.01 + 49.86 + 49.38 + 12.09 GB, [HF tree API](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main?recursive=true)). It splits into:
- `per_layer_token_embd` (n-gram table): **28.80 GB** IQ4_NL, read a few rows per token. The same size in every UD quant.
- routed experts `ffn_{gate,up,down}_exps`: **77.02 GB**
- everything else (attention/DeltaNet, shared experts, routers, hyper-connections, output head): **~4.8 GB**, mostly Q8_0
- **Bytes per token:** 4.8 + 77.02 x 10/512 (1.50) ≈ **~6.3 GB**, 2.7x less than the 27B Q4. The all-in-VRAM ceiling is 6.3 / 440 → ~70 t/s. Batch-1 MoE kernels gather scattered experts and reach less of peak, so the estimate is **~40-60 t/s if it fit in VRAM**. For scale, Strata reports 63 (prose) / 87 (code) t/s on one R9700 with this file and its own engine (§6).
- **Quality** (Qwen card, BF16): SWE-bench Pro 62.5 vs 61.7 for the 27B, SWE-bench Multilingual 81.0 vs 73.8, DeepSWE 58.7 vs 42.2, Toolathlon 73.5 vs 67.1. Unsloth's quant table ([run guide](https://unsloth.ai/docs/models/qwen3.8-next)) gives mean KLD: UD-Q2_K_XL 0.225, UD-IQ3_XXS 0.165, UD-Q3_K_XL 0.107, UD-IQ4_XS 0.084, **UD-Q4_K_XL 0.047**.
**Fit in 64 GB VRAM** (non-PLE = total − 28.8 GB, the PLE left on SSD via `--lazy-mode`, §5):
| Quant | Total | Non-PLE (needs VRAM) | Result on 2x32 GB |
|---|---|---|---|
| UD-Q4_K_XL | 111.3 | **82.5** | ~20-24 GB of experts must live off-GPU. **Does not fit** |
| UD-IQ4_XS | 93.7 | 64.9 | ~6-9 GB off-GPU after KV and buffers |
| UD-Q3_K_XL | 90.0 | 61.2 | Borderline, a few GB off-GPU |
| UD-IQ3_XXS | 82.0 | 53.2 | **Fits**, ~10 GB headroom |
| UD-Q2_K_XL | 78.9 | 50.1 | **Fits**, ~13 GB headroom |
KV is small. Only the 12 QSA layers have one: 12 x 2 KV heads x 256 x 2 (K+V) x 2 B ≈ 24.6 KB/token f16, ~6.4 GB at 262144 tokens, ~3.4 GB at q8_0 (estimate; ignores indexer keys).
**llama.cpp support status:**
- Merged: model support [#27742](https://github.com/ggml-org/llama.cpp/pull/27742) (2026-08-27), MTP [#29761](https://github.com/ggml-org/llama.cpp/pull/29761) (2026-10-01), `-sm tensor` re-enabled [#28569](https://github.com/ggml-org/llama.cpp/pull/28569), many QSA fixes through 2026-10-27 (e.g. [#31243](https://github.com/ggml-org/llama.cpp/pull/31243)).
- **Open bugs that hit this setup:**
- [#29149](https://github.com/ggml-org/llama.cpp/issues/29149): **HIP livelock during load on an R9700**, with both UD-IQ3_XXS and UD-Q4_K_XL. The root-cause comment places it in the mmap upload path; "the non-mmap path stages through pinned buffers instead". Try `--load-mode none`.
- [#29811](https://github.com/ggml-org/llama.cpp/issues/29811) and [#30860](https://github.com/ggml-org/llama.cpp/issues/30860): MTP asserts at startup.
- [#30887](https://github.com/ggml-org/llama.cpp/issues/30887): the DFlash2 drafter crashes on the first prompt (R9700, ROCm).
- [#30815](https://github.com/ggml-org/llama.cpp/pull/30815): QSA chunking, still open.
Usable, but rough on HIP.
## 5. Offloading to DDR RAM and SSD
**What llama.cpp offers** (master `common/arg.cpp`):
- `-ot/--override-tensor <regex>=<buft>` places matching tensors, e.g. `blk\.[0-9]+\.ffn_.*_exps=CPU`. `-cmoe/--cpu-moe` puts all experts on the CPU. `-ncmoe/--n-cpu-moe N` puts the experts of the first N layers on the CPU. These are MoE-only, so they do nothing for the dense 27B (already noted in `docs/architecture.md`).
- `--moe-cache-mib N`: an LRU cache in VRAM for host-resident experts, merged in [#29887](https://github.com/ggml-org/llama.cpp/pull/29887) (2026-10-07). On Flash-Next Q4_0 the PR measured 4090 25.0 → 39.4 t/s and 5090 30.8 → 67.8 t/s, on an EPYC host with RAM for all experts. "Small batches only (<= 32 tokens)".
- `-lzm/--lazy-mode {on,auto,off}`: "read the rows of such tensors from disk on demand instead of keeping them resident (requires mmap)". `auto` applies to tensors over 4 GiB, which **covers the 28.8 GB PLE table by default** (renamed in [#27969](https://github.com/ggml-org/llama.cpp/pull/27969)). Per [#30772](https://github.com/ggml-org/llama.cpp/pull/30772) (closed unmerged), mapped PLE pages still accumulate in RSS and single-threaded `madvise` can bottleneck NVMe.
- `-lm/--load-mode {auto,none,mmap,…}`. With mmap, a tensor that fits nowhere is paged from disk implicitly.
- **Pipeline parallelism turns off as soon as any tensor override is used** (`!has_tensor_overrides()` in `llama-context.cpp`, as described in the open PR [#29963](https://github.com/ggml-org/llama.cpp/pull/29963)). So `-ot`/`-ncmoe` plus two GPUs also loses the layer-split prefill overlap.
**Numbers for this box:**
- **Dense 27B:** no useful offload. Every weight is read every token, and DDR4 is ~51 GB/s against ~440 GB/s VRAM. Each GB moved to RAM costs ~20-25 ms/token on the CPU (estimate). With two cards the Q8_K_XL fits in VRAM anyway. The only RAM tier that helps is `--cache-ram` (prompt cache), as recorded in [`qwen3.8-27b-throughput-tuning.md`](qwen3.8-27b-throughput-tuning.md).
- **Flash-Next UD-Q4_K_XL, if RAM were free (hypothetical):** ~22 GB of experts on CPU = 29 % of experts, ~0.43 GB read per token from DDR4 at ~40 GB/s ≈ 11 ms. The GPU part ≈ (4.8 + 1.07) / 440 ≈ 13 ms. Layers are sequential, so the parts add up: **~24 ms ≈ ~40 t/s**, more with `--moe-cache-mib` (estimate). Prefill uploads the host experts once per ubatch: 22 GB over PCIe 4.0 x16 at ~25 GB/s ≈ 0.9 s per ubatch, so ~18 s of transfer for a 40k-token prompt at `-ub 2048` (estimate).
- **Flash-Next UD-Q4_K_XL, on this box:** RAM is full (§1). Even with `--cache-ram` cut to near zero, 32 GB minus OS, OmniRoute (up to 10 GB) and qdrant leaves maybe ~15 GB (estimate), less than the ~22 GB needed. The rest is paged from the CPU-attached Gen4 x4 NVMe. At ~0.43 GB/token and an assumed 3-6 GB/s for ~1-3 MB expert reads, that is **~70-140 ms/token, ~6-12 t/s**. Prefill re-reads ~22 GB per ubatch from disk, so **minutes per 40k-token prompt**. The page cache will also compete with the swap that is already happening. An outside data point: an RX 9060 XT 16 GB with 64 GB DDR4-3200 got "8 t/s on short context. Longer context 3 t/s" on llama.cpp ROCm ([#29149 comment](https://github.com/ggml-org/llama.cpp/issues/29149)). **Not viable.**
- **PLE table on SSD (any Flash-Next quant):** 32 rows per token (#30772) of 90 B each (Strata `AMD_HIP.md`), each on its own 4 KB page, so ≤128 KB of random IO per token. At 50 t/s that is a few thousand IOPS, trivial. Prefill touches ~32 pages per token, ~1.3 M random reads for 40k tokens. On NVMe at high queue depth that is a few seconds, but single-threaded `madvise` (#30772) can make it much slower. Strata measured +22.5 % prefill with the table in RAM against direct reads (§6). Decode impact: low. Prefill: measure it.
## 6. Strata
**Identity.** [github.com/Niko1221/Strata](https://github.com/Niko1221/Strata): "Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost". C++, **MIT** ([LICENSE](https://github.com/Niko1221/Strata/blob/main/LICENSE)). Its `docs/HOW_IT_WORKS.md` says it is "Built with parts of llama.cpp / ggml (MIT)". It supports Qwen3.8-Flash-Next (ISTA-DASLab GSQ-RCO quants recommended, Unsloth UD-IQ4_XS…Q8_0 supported), Qwen3.6-35B-A3B and DeepSeek V4.1-Flash. **There is no dense Qwen3.8-27B support**, so it cannot replace today's model.
**How it tiers** ([`docs/HOW_IT_WORKS.md`](https://github.com/Niko1221/Strata/blob/main/docs/HOW_IT_WORKS.md), [`docs/MODELS.md`](https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md)):
- **VRAM:** attention/DeltaNet, routers, shared experts, output head, MTP, KV ("from 64K: only its most-read part, the rest streams from RAM"), plus an **expert cache** that fills the rest of VRAM with the most-used experts and adapts at runtime.
- **RAM:** "all 24,576 experts, pinned". The CPU computes cache misses in place, concurrently with the GPU (AVX-512/AVX2). `--pcie-frac` splits misses between PCIe fetch and CPU compute. On gfx12 it defaults to 0.15 after a measured +9.9 % to +25.6 % decode on an R9700 (`docs/AMD_HIP.md`).
- **SSD:** the 28.8 GB n-gram table, "read a few rows per token through the OS cache". From 0.2.7 it keeps the table in RAM when there is room for it plus 8 GiB.
- **Low-RAM mode:** "setup maps them from the model files and keeps in RAM only what the card does not hold … With a small card most experts come from the SSD and it is much slower".
- **Multi-GPU** ([`docs/MULTI_GPU.md`](https://github.com/Niko1221/Strata/blob/main/docs/MULTI_GPU.md)): pipeline (layer) split, "not tensor parallelism". "No NVLink or peer-to-peer access is needed, cards on x4 or x1 slots work". Each card caches the experts of its own layers, so two cards hold about twice as many experts. `docs/AMD_HIP.md` documents HIP layer split for gfx1201 pairs and adds "A split is worth it when no single card holds the model's experts".
**AMD/ROCm:** the R9700 (gfx1201) is "Maintainer-tested", with Linux `hip-rx9070` engine packages in every recent release ([releases](https://github.com/Niko1221/Strata/releases)).
**RAM requirement for the requested file:** `docs/MODELS.md` lists UD-Q4_K_XL as "111 GB download, 72 GiB experts, **RAM for all experts ~88 GB**". With less RAM it "keeps as many as fit and reads the rest from the SSD (much slower)".
**Claimed vs verifiable:**
| Claim | Source | Host RAM | Verifiable? |
|---|---|---|---|
| UD-Q4_K_XL on one R9700: 63 t/s prose, 87 code, 32K prompt 1,110-1,320 t/s | `docs/MODELS.md`, "engines before 0.1.45.3" | not stated there | Self-reported |
| IQ3_S on one R9700: 87.6-101.9 t/s across 1K-128K | [community R9700 report](https://github.com/Niko1221/Strata/blob/main/bench/results/2026-10-08-community-r9700-linux/README.md) | **~503 GiB**, 2x EPYC 9334 | Has per-request JSON and hashes. "Single GPU still uses CPU experts, system RAM and PCIe; these rates do not establish desktop-CPU speed or minimum RAM" |
| R9700 + 9070 XT, 47 GB RAM, Coder IQ1_M: 45.5 / 48.3 t/s decode | `docs/AMD_HIP.md` | 47 GB | Self-reported |
| "125B+ MoE on a single 12GB GPU", "25-38 tok/s" | [stratallm.org](https://stratallm.org/) | n/a | The site calls itself "Unofficial Community Documentation". Its speed figures contradict each other and link no method. Ignore it |
None of these numbers were rerun for this doc. The project says its benchmark protocol "is still a draft" (`docs/COMMUNITY_BENCHMARKS.md`).
**Maturity:** repo created 2026-09-24. ~4,690 commits, ~3,594 of them by the one maintainer. ~36k stars, 895 open issues, a release roughly daily (0.1.40 → 0.2.7 in three weeks). Installation is `curl … | sh`, an AppImage, or Docker (`docs/DOCKER.md`, not read). It is young, single-maintainer and churning, and it would need its own container and gateway wiring here.
**Verdict for this box:** not worth it now. The 32 GB of RAM defeats its core design (all experts pinned in RAM). Low-RAM mode on 2x32 GB VRAM would still read ~20 GB of experts from NVMe for UD-Q4_K_XL. It cannot run the dense 27B that this stack is tuned around. **Revisit if RAM goes to 96-128 GB** (the board takes 128 GB). On a big-RAM host its self-reported R9700 Flash-Next numbers (60-100 t/s) are well above llama.cpp's 27B, so it would be worth a real A/B then.
## 7. Recommendation
**Primary: keep `Qwen3.8-27B-UD-Q4_K_XL` + DFlash2 and run two single-GPU replicas, one per card.**
Why:
- (1) Layer split gives a dense model no decode gain.
- (2) Tensor split is experimental on ROCm, measured at 1.31x, crashes with DFlash2, and would run over a PCIe 3.0 x4 chipset link.
- (3) Q8_K_XL roughly halves decode.
- (4) Flash-Next at a usable quant does not fit, and RAM offload pages from NVMe.
- (5) The workload is 2 parallel agents. The log already shows decode falling to p10 15 t/s while the other slot prefills ([`qwen3.8-27b-throughput-tuning.md`](qwen3.8-27b-throughput-tuning.md)). Two replicas remove that interference and double each slot's context.
Per-instance flags (two services, identical except the device). Each is today's command with these changes:
```text
--device ROCm0 # ROCm1 for the second service; confirm names with --list-devices
--n-gpu-layers 999
--ctx-size 262144 --parallel 1 # full 262144 per agent instead of 131072
--flash-attn on --cache-type-k q8_0 --cache-type-v q8_0
--cache-ram 6144 # per instance; both together must stay well under free RAM (§1)
--spec-type draft-dflash -md /models/Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-n-max 7
--jinja
```
`GPU_MAX_HW_QUEUES=1` stays on both services (`docs/architecture.md#gpu-access`). Gateway: register both backends in OmniRoute and make sure a conversation stays on one backend (prompt-cache affinity). Otherwise every switch is a full re-prefill. Whether OmniRoute can balance two connections per conversation was not checked.
**Independently of the second card: lower `LLAMA_CACHE_RAM`.** 24 GiB on a 32 GB host with a 10 GB OmniRoute limit oversubscribes RAM.
**Alternatives to A/B, in order:**
1. **Single fast stream:** Q4_K_XL with `-sm tensor -ts 1,1 -fa on --fit off --ctx-size <set by hand> --spec-type draft-mtp --spec-draft-n-max 2` (the built-in MTP head; DFlash2 crashes with `tensor`). Estimate ~34 t/s no spec, ~50 with MTP if stable. Watch for #27122-style lockups.
2. **Quality:** Q8_K_XL (or UD-Q6_K_XL at 25.3 GB) with `-sm layer`, DFlash2 placed with `-devd`, ~25 t/s estimated. Only if Q4 quality is shown to fail on your tasks.
3. **Flash-Next experiment:** UD-IQ3_XXS or UD-Q2_K_XL, all in VRAM: `-sm layer -ngl 999 --lazy-mode auto --load-mode none -fa on -c 131072`, MTP via `-md MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --spec-type draft-mtp` once #29811/#30860 are fixed. Judge it on your own agent tasks. The KLD at these sizes is 3.5-5x that of UD-Q4_K_XL.
## 8. What to benchmark once the card arrives
1. **Topology:** `sudo lspci -vv` LnkSta for both cards (expect 16 GT/s x16 and 8 GT/s x4). Run `rocm-smi --showtopo` and `rocm-bandwidth-test` for host↔GPU and GPU↔GPU (does P2P work through the chipset?). Check the PSU and the case slot fit first.
2. **Per-card baseline:** `llama-bench -m Qwen3.8-27B-UD-Q4_K_XL.gguf -dev ROCm0` and then `-dev ROCm1`, `-fa 1 -ctk q8_0 -ctv q8_0 -p 512,8192 -n 128 -d 0,32768`. Decode should match. Load time and prompt-cache restores are where the x4 link shows.
3. **Split modes:** the same bench with `-sm layer` vs `-sm tensor -ts 1,1` for Q4_K_XL and Q8_K_XL. Compare tg against #28863's 1.31x and pp against layer.
4. **Server replay** (fixed ~40k and ~2k prompts, production temperature, concurrency 1 and 2) for: (a) two replicas, (b) one `-sm layer --parallel 2`, (c) `-sm tensor` + MTP. Record `timings.predicted_per_second`, `prompt_per_second`, draft acceptance, and the p10 decode under concurrency.
5. **Correctness:** a needle test at ~6-10k and ~60-90k tokens on each config, because #30884 reproduces on 2x R9700 with `--tensor-split`. Also check that DFlash2 output under `-sm layer` matches single-card output on a fixed prompt.
6. **RAM:** `free -h` and `vmstat 5` after an hour of agent traffic, before and after lowering `--cache-ram`. Swap-in should be ~0.
7. **Optional Flash-Next:** that UD-IQ3_XXS/UD-Q2_K_XL loads at all on HIP (#29149, try `--load-mode none`), tg/pp at 4k/32k, and prefill time with the PLE table on NVMe.
## Open questions
- Does OmniRoute support two backends with sticky routing per conversation?
- Does the `ghcr.io/ggml-org/llama.cpp:server-rocm` image build with `GGML_HIP_RCCL`? It only matters for `-sm tensor`.
- Does ROCm P2P work between a CPU-attached slot and a chipset-attached slot on B550? It is unverified, and the answer probably does not matter outside `tensor` mode.
- Is there a B550/X570 board with CPU x8/x8 bifurcation that fits this chassis? It would give both cards Gen4 x8. It is only worth it if step 3 shows `tensor` scaling well. No specific board was checked.
## Sources
- Hugging Face: [unsloth/Qwen3.8-27B-GGUF tree](https://huggingface.co/api/models/unsloth/Qwen3.8-27B-GGUF/tree/main), [its config.json](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/raw/main/config.json), [UD-Q8_K_XL file](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_info=Qwen3.8-27B-UD-Q8_K_XL.gguf), [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) and its [tree](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main?recursive=true), [UD-Q4_K_XL shard 1](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF?show_file_info=UD-Q4_K_XL%2FQwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf), [z-lab DFlash2 GGUF](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF), [incoai Flash-Next DFlash2 GGUF](https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2-GGUF). Tensor sizes come from parsing the GGUF headers (range requests) with a throwaway script.
- Unsloth: [Qwen3.8-Flash-Next run guide](https://unsloth.ai/docs/models/qwen3.8-next) (sizes, KLD table, MTP flags).
- llama.cpp master (fetched 2026-10-29): [`docs/multi-gpu.md`](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), [`common/arg.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp), [`src/llama-arch.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp), [`src/llama-context.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-context.cpp). PRs: [#19378](https://github.com/ggml-org/llama.cpp/pull/19378), [#23792](https://github.com/ggml-org/llama.cpp/pull/23792), [#27342](https://github.com/ggml-org/llama.cpp/pull/27342), [#27742](https://github.com/ggml-org/llama.cpp/pull/27742), [#27825](https://github.com/ggml-org/llama.cpp/pull/27825), [#27858](https://github.com/ggml-org/llama.cpp/pull/27858), [#27861](https://github.com/ggml-org/llama.cpp/pull/27861), [#27969](https://github.com/ggml-org/llama.cpp/pull/27969), [#28569](https://github.com/ggml-org/llama.cpp/pull/28569), [#29761](https://github.com/ggml-org/llama.cpp/pull/29761), [#29887](https://github.com/ggml-org/llama.cpp/pull/29887), [#29963](https://github.com/ggml-org/llama.cpp/pull/29963), [#30772](https://github.com/ggml-org/llama.cpp/pull/30772), [#30815](https://github.com/ggml-org/llama.cpp/pull/30815), [#31243](https://github.com/ggml-org/llama.cpp/pull/31243). Issues: [#23567](https://github.com/ggml-org/llama.cpp/issues/23567), [#27122](https://github.com/ggml-org/llama.cpp/issues/27122), [#27829](https://github.com/ggml-org/llama.cpp/issues/27829), [#28582](https://github.com/ggml-org/llama.cpp/issues/28582), [#28777](https://github.com/ggml-org/llama.cpp/issues/28777), [#28863](https://github.com/ggml-org/llama.cpp/issues/28863), [#29149](https://github.com/ggml-org/llama.cpp/issues/29149), [#29811](https://github.com/ggml-org/llama.cpp/issues/29811), [#30860](https://github.com/ggml-org/llama.cpp/issues/30860), [#30884](https://github.com/ggml-org/llama.cpp/issues/30884), [#30887](https://github.com/ggml-org/llama.cpp/issues/30887).
- Hardware: [ASRock B550M Pro4 manual](https://download.asrock.com/Manual/B550M%20Pro4.pdf) (asrock.com's spec page returned no content to the fetcher), [ComputerBase B550](https://www.computerbase.de/2020-06/amd-b550-chipsatz-mainboard-test/), [Guru3D B550 PCIe](https://www.guru3d.com/review/amd-ryzen-3-3100-and-3300x-review/page-3/), [Puget R9700](https://www.pugetsystems.com/parts/Video-Card/AMD-Radeon-AI-PRO-R9700-32GB-16908/), [Tom's Hardware R9700](https://www.tomshardware.com/pc-components/gpus/amd-launches-radeon-ai-pro-r9700-to-challenge-nvidias-ai-market-dominance).
- Strata: [repo](https://github.com/Niko1221/Strata) (README, LICENSE, `docs/HOW_IT_WORKS.md`, `MODELS.md`, `MULTI_GPU.md`, `SECOND_GPU.md`, `AMD_HIP.md`, `COMMUNITY_BENCHMARKS.md`, read via the GitHub API on 2026-10-29), [releases](https://github.com/Niko1221/Strata/releases), [stratallm.org](https://stratallm.org/).
- This repo: `docker-compose.yml`, `.env.example`, `docs/architecture.md`, [`qwen3.8-27b-throughput-tuning.md`](qwen3.8-27b-throughput-tuning.md), [`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md), [`500k-context-2-parallel-agents-gpu-roadmap.md`](500k-context-2-parallel-agents-gpu-roadmap.md).