LLM-Server/docs/research/qwen3.8-27b-decode-speed.md

Research: decode tokens/s on the R9700 (llama-server, Qwen3.8-27B + DFlash2)

Date: 2026-10-20

Question: Why is decode at ~52k context about plain-decode speed (26-28 t/s) when an older, shorter-context log averaged 45.9 t/s, and what can be changed to raise decode (and prefill where relevant)? Evidence: llmserver.log (re-analysed per request) plus llama.cpp source.

Scope note: research only. Nothing in compose or .env.example was changed and nothing was deployed. This note builds on qwen3.8-27b-throughput-tuning.md (cache/prefill ranking, DDR tier, five outside configs) and qwen3.8-27b-quant.md (quant choice, VRAM budget). It does not repeat them, and it corrects one premise of the first (see finding 1).

Labels. [V] verified from source (URL pinned to a commit, or a repo file:line). [L] computed from llmserver.log by me. [R] reported by a third party (a page or thread, not re-run). [I] inferred, not measured. WebFetch summaries of web pages were produced by a small model, so every number taken from a page is [R] even when the page is first-party.

Source pin. llama.cpp master at 839a572a4cdecdb80d211297b8e7d7323a6e99d2 (fetched 2026-10-20). Links below point at that commit. The running image (server-rocm) may be older than this commit. Its build string is not in the logs I have, so every "[V]" is "true on that commit".

TL;DR

#	Finding	Label
1	The draft length has been 5, not 7, in the old log too. All 306 draft acceptance summaries have generated divisible by 5 and mean len = 1 + accepted / (generated / 5). So n_max=5 in the new startup log is the same setting, not a new clamp. The earlier note's "n_max is 7" came from compose and was wrong	[L] + [V]
2	master clamps n_max to block_size - 1 (= 7 for block_size 8). 5 is below that, so no clamp fired. 5 must have been requested: most likely LLAMA_SPEC_N_MAX=5 in the server's .env (the repo .env.example default differs, and .env is not in the repo)	[V] + [I]
3	Decode speed tracks acceptance almost linearly (r = 0.97 over 256 requests) and acceptance falls with depth: 0.75 at <8k, 0.57 at 8-20k, 0.42 at 20-40k, 0.40 at 40-60k, while decode falls 64.6 to 31.9 t/s. The 45.9 t/s average was dominated by short-context requests	[L]
4	Bandwidth ceiling for plain decode: 17.6 GB / 640 GB/s = 27.5 ms = 36.4 t/s (33.0 t/s at 52k depth with q8_0 KV). At the 79-81 % efficiency best seen for dense models on this card that is 26-27 t/s, which is exactly the measured "plain" rate	[I] on [R] inputs
5	A spec step costs 74-94 ms in the old log, against a ~28-30 ms weight-streaming floor. About 45 ms per step is fixed overhead (drafter, 6-wide verify, sync, sampling), so speculation only pays when acceptance is high	[L] + [I]
6	DFlash2 draws its draft greedily and gives the server no draft distribution, so at temp > 0 a draft token is accepted only if the target's own sampled token equals it. Sampler settings on the client therefore change acceptance directly	[V]
7	GPU_MAX_HW_QUEUES=1 is free for default llama.cpp: multi-stream graph execution is opt-in (GGML_CUDA_GRAPH_OPT=1)	[V]

Top 3 experiments (details in "Experiment plan"): A sweep --spec-draft-n-max 3, 4, 5, 7 and add --spec-draft-p-min; B measure acceptance versus client sampler settings with a fixed prompt (temp, top-p/top-k, penalties); C decompose the 74-94 ms step with llama-bench (pp6 at depth vs tg1) and rocm-smi clocks, before touching KV type, slots or quant.

1. DFlash2: why acceptance and speedup vary

1.1 The n_max 5 vs 7 discrepancy

• The init line prints n_max=%d from this->params.n_max before the clamp: common/speculative.cpp#L1086. The clamp is #L1090-L1099: n_draft_max = (is_dspark && sample_from_anchor) ? block_size : block_size - 1, with the warning requested draft size (n_max=%d, n_min=%d) exceeds the trained block size %d -- clamping to %d. [V]
• For this drafter (block_size=8, DFlash2 is not DSpark) the limit is 7. So a requested 7 passes unclamped and a requested 5 prints 5. The init line therefore shows the requested value; no clamp explains 5. [V] (on that commit)
• n_extract=5 in the log is target_layer_ids_n, the number of target layers whose features feed the drafter (#L1040-L1044). It is not used in the clamp, so the 5 = 5 match is a coincidence on this commit. [V] An older build could clamp differently; I could not check history for this. [I]
• Log proof that 5 is the real draft length. The server prints mean len = 1 + accepted / verif_steps (tools/server/server-context.cpp#L704-L725). With verif_steps = generated / 5 this reproduces the printed mean len in all 306 summaries; with 7 it matches none. Examples: llmserver.log:3 (47 accepted / 105 generated, 21 steps, 47/21 + 1 = 3.24), :36 (64/170, 34 steps, 2.88), :114 (828/2050, 410 steps, 3.02). [L]
• To confirm on the server: grep -n SPEC_N_MAX .env and docker logs llama-server | grep -E "n_max|exceeds the trained". [I], not run.

1.2 What the drafter does per step

• The drafter input is [id_last, <mask> * n_max] and one forward pass denoises the whole block. DFlash2 then greedily walks a candidate lattice (selector_top_k candidates per position, argmax per step): #L1296-L1310 and #L1336-L1366. The block size in the drafter batch is n_max + 1, so a smaller n_max shrinks the drafter batch too. [V]
• --spec-draft-p-min on DFlash2: stop extending the draft at the first position where softmax(selector scores) at the argmax is below p_min (#L1349-L1358). It is a per-position stop, default 0.00 (common/common.h#L333, common/arg.cpp#L4248). The git log shows DFlash2-specific p_min arrived with the DFlash2 PR (#27342, commit titles "Add p_min in DFlash2" per the PR page [R]). [V] for the code. The generated multiples of 5 in the log confirm p_min was 0 in the old run. [L]
• n_min (--spec-draft-n-min, default 0): a draft shorter than n_min is discarded (#L1362-L1364). [V]
• sample_from_anchor=true only changes the layout for DSpark drafters; for DFlash2 the block is n_max + 1 (#L1298). [V]
• --spec-draft-sampling probabilistic (arg.cpp#L4264) is documented as sampling the draft and verifying by rejection. The DFlash2 branch never fills the draft distribution (result_q), and the server then falls back to plain sample-and-match: use_rejection = use_spec_rejection() && !spec_draft_q.empty() (server-context.cpp#L4537). So that flag does nothing for DFlash2. [V] (read of the code path; not run).

1.3 Why acceptance varies

1. Sampling. With the DFlash2 drafter, a draft token is accepted iff the target's sampled token equals it (common/sampling.cpp#L690-L720, draft[i] != id breaks). At temp 0 the target token is its argmax, the best case for matching a greedy draft. At temp > 0 it falls with flatness of the target distribution and with anything that reshapes it (top-p/top-k, repetition or presence penalties). [V] for the mechanism. The size of the effect for this workload is unmeasured. [I]
   • Contradiction to resolve: the earlier note cites a third-party claim that greedy sampling "collapses" DFlash2 acceptance. The source reading says the opposite for the server path. Treat that claim as unverified until experiment B settles it.
   • Server-side the temp > 0 test is task->params.sampling.temp > 0 (server-context.cpp#L508). The drafter itself ignores temperature. [V]
2. Context depth. In the old log, acceptance halves from <8k to >40k tokens (0.75 to 0.40, table below). A PR commenter also reported code prompts decaying faster with context length than prose (PR #27342 comments) [R]. The model card gives no context length for the drafter. [R]
3. Content type. Not separable in this log (no prompt/response classes). The authors' acceptance lengths (mean tokens per verify step, 7 drafted, block size 8, temp 1.0 / top-p 0.95 / top-k 20, SGLang on an H200) are GSM8K 5.46, MATH-500 5.28, HumanEval 4.39, MBPP 4.79 (z-lab/Qwen3.8-27B-DFlash2, via a search summary) [R]. Your mean len of 2.9-4.9 (max possible 6 at n_max 5) is below that, partly because those are short prompts. [I]
4. Concurrency. Issue #27117 (gfx1151, closed "not planned"): acceptance collapses as the number of concurrently drafting sequences grows (0.51-0.66 at 4 active, 0.06-0.49 at 16), and --spec-draft-n-max 1 recovered it. [R] Your two slots are far below that regime, but the bimodal per-slot acceptance in that issue is worth watching if both slots decode at once. [I]

1.4 Authors' recommended values

• Model card quick start: --spec-type draft-dflash --spec-draft-n-max 7 (z-lab/Qwen3.8-27B-DFlash2-GGUF) [R]. No block size, temperature or p_min advice is given there. The card says the output is lossless (greedy identical, sampling distribution preserved).
• A PR commenter on an RTX 3090 used --spec-draft-n-max 4 and reported that 7 was slower: DFlash2 at n=4 averaged 62.21 t/s at 0.6153 acceptance, MTP 57.79 t/s at 0.6317 (PR #27342 comments) [R].
• Drafter quant does not matter for acceptance: acceptance length 5.28 (BF16), 5.13 (Q8_0), 5.39 (Q4_K_M) on 8 GSM8K prompts; the card itself implies Q4_K_M above BF16 is noise. Q4_K_M is ~1.1 GB, so it also costs the least bandwidth. [R]

1.5 What the old log shows (re-analysis)

Per-request join of stop processing: n_tokens (final depth), eval time t/s and draft acceptance, requests with >= 200 generated tokens, 256 requests (script: ad-hoc Python over llmserver.log, not committed). Draft length 5, so tokens per verify step is 1 + 5 x acceptance.

Final depth	n	Decode t/s (mean)	Acceptance (mean)	Tokens/step	Step time (tokens/step / t/s)
< 8k	125	64.6	0.75	4.75	~74 ms
8-20k	28	44.2	0.57	3.85	~87 ms
20-40k	17	34.6	0.42	3.10	~90 ms
40-60k	86	31.9	0.40	3.00	~94 ms

• Correlation of per-request decode t/s with acceptance: r = 0.97, slope ~90 t/s per 1.0 acceptance. [L]
• Step time rises only ~20 ms from 0 to 50k depth, while tokens per step drops 4.75 to 3.0. So about 80 % of the decode loss with depth is acceptance, not attention cost. [L] (derived, mean-of-ratios caveat).
• The new run (26-28 t/s at ~52k, tg_3s 20-33) is at or just below the 40-60k bucket (31.9), consistent with the same acceptance regime. [L]

2. Decode speed versus context depth

2.1 What the attention costs here

• Only 16 of 64 layers keep a KV cache (qwen3.8-27b-quant.md "Model identity"). KV bytes per token: 16 layers x 2 x 4 heads x 256 x element size = 65,536 B (f16), 34,816 B (q8_0, 1.0625 B/element), 18,432 B (q4_0, 0.5625 B/element). At 52k tokens: 3.4 GB (f16), 1.81 GB (q8_0), 0.96 GB (q4_0). Against 17.6 GB of weights read per token, q8_0 KV adds ~10 % bytes at 52k, f16 ~19 %, q4_0 ~5 %. [I] (arithmetic from the quant note's architecture facts.)
• So KV precision can move plain decode at 52k by at most a few percent. Going to q4_0 would buy ~1 GB less read (~5 %) at an unmeasured quality cost. Do not expect it to fix the 2x gap. [I]

2.2 Which flash-attention kernel runs on gfx1201

From ggml/src/ggml-cuda/fattn.cu#L540-L722 (ggml_cuda_get_best_fattn_kernel), applied to head_dim 256, 24 query heads, 4 KV heads (gqa_ratio 6, effective ratio 2), K/V q8_0:

Query batch (tokens)	Kernel on gfx1201	Why (source)
1 (plain decode)	VEC	WMMA needs Q->ne[1] * gqa_ratio_eff > 16 for head_dim > 128 (#L698-L702), so 1 x 2 fails. For quantized K/V the vector kernel is chosen when Q->ne[1] <= 2 (#L714-L720). Reads q8_0 directly.
2	VEC (quantized)	same rule
3 to 8 (verify batch of a spec step)	TILE	8 x 2 = 16 is not > 16, so no WMMA; quantized KV is not eligible for VEC above 2 tokens.

• TILE and MMA kernels "need f16 K/V" (#L735-L740): with a quantized cache the K and V views are converted to f16 scratch before the kernel runs. So a verify step at 52k converts ~100M q8_0 elements per layer (K plus V) to f16 on 16 layers. Rough extra traffic: read 1.8 GB plus write 3.4 GB plus read 3.4 GB per step, ~8.6 GB, ~13 ms at 640 GB/s. [I] (the conversion and kernel choice are [V]; the traffic estimate and its cost are inferred, not profiled). That is consistent with the ~20 ms step-time rise from 0 to 50k in 1.5. With f16 KV the conversion disappears (TILE reads f16 directly) but the cache is 1.9x larger, so the net is not obvious. [I]
• No kernel-level bug report for gfx1201 + quantized KV turned up in the sources I read. The only RDNA4 data point is [R]: on RADV (Vulkan) q8_0 vs q4_0 KV was within +-0.5 % (discussion #21043). That is a different backend and says nothing about the HIP path.

2.3 --parallel 2 and --kv-unified

• With --kv-unified there is one KV stream. get_n_kv pads cells.used_max_p1(), the highest occupied cell across all sequences (src/llama-kv-cache.cpp#L1282-L1292). The flash-attention kernels only skip fully masked tiles at the end of the range (flash_attn_mask_to_KV_max, fattn-common.cuh#L671-L723). So when the other slot holds a long prompt in higher cells, slot 0's decode may read and mask-skip past tens of thousands of foreign cells. [V] for both pieces, [I] for the combined cost (not profiled; which cells each slot occupies is not logged).
• This matches the pattern in the older log (requests alternate slot 0 / slot 1, each ~40-50k tokens, throughput note "Pattern at :380-:470"), but no measurement separates it. Experiment D below tests it.
• --kv-unified-per-slot N exists (arg.cpp#L1652) and --no-kv-unified gives one stream per slot (arg.cpp#L1727-L1734). Non-unified mode splits --ctx-size evenly, so one slot could no longer use more than half the context. [V]

2.4 Checkpoints and recurrent state during decode

• The target is qwen35 architecture, which llm_arch_supports_rs_rollback lists (src/llama-arch.cpp#L1256-L1262). I am assuming Qwen3.8 loads as that arch; the name was not verified. [I]
• For DFlash/DSpark the server requests n_rs_seq = draft.n_max recurrent-state snapshots per sequence (common/common.h#L401-L407; src/llama-context.cpp#L191-L195). The server creates a speculative checkpoint only if draft.size() > llama_n_rs_seq(ctx_tgt) (server-context.cpp#L3541-L3558), and restores one only if n_rollback > n_rs_seq (#L4559). With n_rs_seq == n_max that never happens, so a rejection is a snapshot-index rollback and not a state copy. [V]
• If the build is older or the arch is not on the rollback list, the COMMON_CONTEXT_SEQ_RM_TYPE_FULL path copies a checkpoint every step (use_ckpt_tgt, #L3463); the startup line speculative decoding will use checkpoints (#L1388-L1390) would show it. Check the startup log for that line. [V] for the code, [I] for which case applies.
• --ctx-checkpoints / --checkpoint-min-step (defaults 32 and 8192, common.h#L639-L642) are for prompt/prefix reuse. Nothing I read ties them to the decode loop, so they should not cost decode time. [I]
• The "cache_reuse is not supported by this context" startup line is the server's llama_memory_can_shift check (server-context.cpp#L1328-L1336). It confirms the open question in the throughput note. [V]

3. Backend, HIP flags, clocks

3.1 ROCm versus Vulkan (RADV)

Everything here is [R] unless noted; the thread posts are self-reported and mixed in setup.

Source	Setup	Numbers
discussion #21043 (JohnTDI-cpu, build 8554)	R9700, RADV, Qwen3.5-27B Q4_K_M	tg128 29.07 t/s (RADV), 29.01 (AMDVLK); pp512 798 (RADV). Computed there as ~456 GB/s, 71 % of 640 GB/s; "best dense ~79-81 %"
same	R9700, ROCm, 27B Q8_0, q4_0 KV, 64k ctx	pp 960 (3.7k) to 708 (58.7k) t/s; tg 42.8 to 28.0 t/s over the same range (MTP use not stated, and 42.8 t/s is above a 29 GiB Q8_0 plain-decode ceiling of ~22 t/s, so it is almost certainly with speculation)
same	Windows, HIP, gfx1201	pp512 1106-1211; tg128 29.07; "Vulkan decode still wins"
same	R9700, Qwen3.6-27B UD-Q4_K_XL, q4_0 KV, MTP (build b9203)	tg128 28.2 baseline, 47.4 with mtp-3
discussion #20881	R9700 and others	MoE Qwen3-Coder-30B-A3B Q4_0 ~196 t/s on RADV; one ROCm segfault report on Qwen3.5-35B-A3B
throughput note (cat5edopeHA)	R9700, MTP n3	Vulkan 68.5 / ROCm 55.1 short, 50.5 / 40.2 at 32k, 40.9 / 31.2 at 64k

• A search summary also cites a blog (not fetched) with ROCm ahead for dense 27B decode (42.8 vs 29.1), the opposite of cat5edopeHA. The sources conflict, so only a local A/B decides. [R]
• DFlash2 on Vulkan: not verified. The DFlash graph and feature extraction live in common code and the model graph, so it may work, but I did not check that a Vulkan build runs draft-dflash. [I]
• Prefill: HIP uses MMQ on RDNA4 unconditionally (ggml_cuda_should_use_mmq returns true for RDNA4, mmq.cu#L468-L471), so GGML_CUDA_FORCE_MMQ changes nothing on gfx1201. [V]

3.2 Decode kernels for the verify batch

• Weights are Q4_K-family. Matrix multiplies with up to 8 columns go to MMVQ (MMVQ_MAX_BATCH_SIZE 8, mmvq.cuh). RDNA4 has no per-type override, so ne11 <= 8 uses MMVQ (mmvq.cu#L318-L430). The source comment says k-quants "cost more to decode and mvq redoes that per column". [V] So the verify batch of 6 (n_max 5) or 8 (n_max 7) pays dequant ALU work per column; cost grows with batch width even though weight bytes are read once. [I] (the per-column cost on gfx1201 was not measured). This is the mechanism behind "lower n_max can win": it shrinks both the MMVQ columns and the drafter batch.
• Batch 9 or more would fall to MMQ, which is not reachable: n_max is capped at 7 so the batch is at most 8. [V]

3.3 HIP flags and environment variables

Item	Verdict	Source
GGML_HIP_GRAPHS	Already ON in the CMake default and the official ROCm Dockerfile does not turn it off. Variable graph shapes force a re-warmup: any change in node properties resets warmup_complete (ggml-cuda.cu#L4806-L4845). With p_min > 0 the verify batch width varies, so graphs may be re-captured more often. Debug with GGML_CUDA_GRAPH_DEBUG=1; disable with GGML_CUDA_DISABLE_GRAPHS=1.	[V] (ggml/CMakeLists.txt#L217, rocm.Dockerfile); effect on speed [I]
GGML_CUDA_FORCE_MMQ	No effect on RDNA4 (see 3.1)	[V]
GGML_CUDA_GRAPH_OPT=1	Opt-in multi-stream concurrent branches (ggml-cuda.cu#L5016-L5030). Off by default	[V]
GPU_MAX_HW_QUEUES=1	HIP caps hardware queues per process per device (default 4, HIP env docs). llama.cpp issues all kernels on one stream unless GGML_CUDA_GRAPH_OPT=1, so one queue should cost nothing for decode. The repo set it for ROCm/ROCm#5706 (docs/architecture.md#gpu-access). Do not combine GGML_CUDA_GRAPH_OPT=1 with it without a test	[R] for the doc, [V] for the llama.cpp side, [I] for "no decode cost"
-bs, --backend-sampling	Experimental, default off (arg.cpp#L2310). Moves target sampling onto the GPU, which would remove the CPU sampling of 6 logit rows per step. Interplay with speculation is unverified	[V] flag exists; benefit [I]
rocm-smi perf level / compute profile	In #21043: "ROCm rocm-smi high vs auto: auto was faster" on a single card; a LACT 330 W cap with COMPUTE profile gave tg 102 to 110 on a dual-card MoE run; power_dpm_force_performance_level=high gave +14 % tg on one Vulkan run. Mixed, hardware and backend differ	[R]
PCIe ASPM pcie_aspm.policy=performance	Same thread: dense decode on RADV 29.30 to 32.46 (+10.8 %), one user, not reproduced on eGPU or dual-slot. It is a runtime sysfs switch, so it is cheap to try	[R]
MCLK not reaching max under load	Same thread, kernel 6.17, RADV	[R]; check with rocm-smi during decode, [I]

3.4 Threads, mmap, mlock, poll

• With every layer on the GPU the CPU is only a scheduler. #21043 tested -t, -b, -mmp 0, --prio 2, CPU pinning, hugepages and nice on RADV and found all within +-0.3 % [R]. The server default --poll level is 50 (common.h#L77) and -t was -1 (16 here). I found no evidence these matter for a GPU-resident run, so do not spend experiments on them. [I]
• One exception worth knowing: speculative drafting has its own thread/poll flags (--spec-draft-threads, --spec-draft-poll, arg.cpp), but the drafter is on the GPU too, so they should be equally inert. [I]

4. Quant and model options

Inputs: GGUF sizes from qwen3.8-27b-quant.md (UD-Q4_K_XL 17.6 GB, UD-Q4_K_M 16.5, UD-Q4_K_S 15.4) and the throughput note (UD-IQ4_XS 13.3 GB). Bandwidth 640 GB/s is the card spec as quoted in #21043 [R]. Weights only; 52k q8_0 KV adds 1.81 GB (2.1).

Quant	Bytes read/token	Ceiling at 640 GB/s	At 80 % efficiency	Same with 52k q8_0 KV (ceiling / 80 %)
UD-Q4_K_XL (running)	17.6 GB	36.4 t/s (27.5 ms)	29.1	33.0 / 26.4
UD-Q4_K_M	16.5 GB	38.8	31.0	35.0 / 28.0
UD-Q4_K_S	15.4 GB	41.6	33.2	37.2 / 29.8
UD-IQ4_XS	13.3 GB	48.1	38.5	43.5 / 34.8

• All [I] (arithmetic). The 80 % factor is the "best dense ~79-81 %" figure in #21043 [R]. The measured plain-decode rate at depth, 26-28 t/s, sits on the 52k column for Q4_K_XL (26.4), so the card is already near its bandwidth limit for plain decode. There is no hidden 2x in the non-speculative path.
• A smaller quant gains at most ~13 % (Q4_K_M) to ~32 % (IQ4_XS) on the plain-decode ceiling and also shrinks the spec verify weight read. Quality cost is not measured here. The IQ4 family dequantizes slower than Q4_K, so the real gain is below the ceiling ratio. [I] Test with KL divergence or perplexity on a small fixed corpus (llama-perplexity) before any switch.
• Drafter quant: Q4_K_M at ~1.1 GB is already the smallest tested, with acceptance as good as BF16 (1.4). Nothing to gain. [R]
• Per spec step the drafter adds ~1.1 GB of reads (~1.7 ms at the ceiling), small against the ~45 ms fixed overhead found in 1.5. [I]

5. Where the step time goes (open)

Old-log step times are 74 ms at <8k and 94 ms at 40-60k (1.5). The weight-streaming floor at 80 % efficiency is ~34 ms. The remaining ~40-60 ms per step is not explained by any single item I could verify. Candidate contributors, none measured:

1. 6-column MMVQ with k-quant dequant per column (3.2).
2. Drafter forward over n_max + 1 positions plus injecting 5 layers of target features per accepted token (n_extract=5, speculative.cpp#L1040), including the encoder fused into KV injection (commit 662a0b0).
3. CPU sampling of 6 full-vocab logit rows plus device-to-host copies per step (-bs is off).
4. Host synchronisation between drafter and target per step (two contexts, no overlap).
5. Quantized-to-f16 KV conversion for the TILE kernel at depth (2.2).

Experiment C isolates 1 and 5 without the server; the rest need a profile (rocprofv3), which is out of scope here.

6. Experiment plan

Rules: one change per run, cold slot, same prompt, restart the server between configs with ./scripts/update.sh on the server (this box cannot deploy). All numbers come from the server log lines prompt eval time, eval time and draft acceptance (llmserver.log:32-36 show the format).

Benchmark protocol (minimal, comparable)

1. Fixed prompts (kept in the repo's scratch area, not committed): P-code = ~8k tokens of source plus the instruction "continue the file" and P-code-52k = the same material padded to ~52k tokens; P-reason = a reasoning question over the same 52k context. Same bytes every run.
2. Fixed request: n_predict=512 (400+ tokens so the 3-s tg window averages out), seed fixed, cache_prompt false for the first request so prefill is cold, then the same request again with the prompt cache warm only if you want decode in isolation.
3. Sampler: run each prompt twice, once with the production client settings (record them from the request body) and once with temperature=0. Record the sampler fields in the run name.
4. Repeats: N = 5 per cell, both slots idle except the request under test; discard the first.
5. Read from the log for each request: prompt tokens/s (prompt eval time), decode t/s (eval time), draft acceptance and mean len. Report median and min/max, not the mean.
6. Decode-only control: llama-bench inside a throwaway container (stop llama-server first, one GPU): llama-bench -m <model> -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 0 -n 256 -d 0,16384,52000 -r 3. -n 256 avoids the "implicit ubatch 128 on hybrid models" artifact reported with -n 128 in #21043 [R]. llama-bench does not use speculation or the prompt cache.
7. Environment capture per run: rocm-smi --showclocks --showpower --showtemp once during the decode (SCLK, MCLK), plus the full startup log (build string, n_rs_seq, checkpoint line).

Experiments, in priority order

#	Change (exact)	Expected effect	How to measure	Risk
A	LLAMA_SPEC_N_MAX = 3, 4, 5, 7 (one run each); then 5 with --spec-draft-p-min 0.3 and 0.5	Fewer wasted drafted tokens and a narrower verify batch when acceptance is ~0.4; could be neutral or negative when acceptance is high. Direction at 52k is the point; short context favors the larger value. No estimate of size without data	Decode t/s, mean len, generated (must be a multiple of the n_max, or not if p_min is on), on P-code and P-code-52k	Low. p_min makes batch widths vary, which can trigger graph re-capture (3.3); if so, compare with GGML_CUDA_DISABLE_GRAPHS=1
B	No server change: send P-code-52k / P-reason with (1) production sampler, (2) temperature=0, (3) production minus penalties, (4) production minus top-p/top-k	Isolates how much acceptance the sampler costs (6, 1.3). A large gap says the fix is in the client/gateway sampler defaults, not llama.cpp	draft acceptance and mean len per cell; the full per-position acc per pos appears at trace level only	None
C	Stop llama-server; in a container with the same image: llama-bench ... -p 1,2,4,6,8 -n 0 -d 0,52000 and -p 0 -n 256 -d 0,52000; rocm-smi clocks during a server decode	Splits step time into verify-batch cost (pp6 time at depth) vs plain tg1. Shows whether verify at depth already costs ~60+ ms	llama-bench t/s table; compare pp6 time with measured step time	Low. Needs a short outage of the only GPU consumer
D	--parallel 1 (then --no-kv-unified with --parallel 2) on the same P-code-52k	Tells whether the second slot or the unified cache costs single-stream decode (2.3). If decode at 52k jumps, slot layout is the cost	Decode t/s on a request with the other slot idle; then with the other slot holding a 50k context	Medium. --parallel 1 serialises agents; non-unified halves per-slot max context
E	echo performance | sudo tee /sys/module/pcie_aspm/parameters/policy, and check MCLK/SCLK under load; try rocm-smi --setperflevel high only if MCLK sags	Reported +10.8 % on one RADV dense run; unknown for ROCm. Free to try and revert	Decode t/s on P-code (plain control from C)	Low; revert with the previous policy value
F	-bs (--backend-sampling)	Removes CPU sampling of the verify rows if that is a real share of the step	A/B on P-code	Medium: experimental, speculation interplay unverified
G	GGML_CUDA_DISABLE_GRAPHS=1 and, separately, GGML_CUDA_GRAPH_DEBUG=1 to see re-warmup counts	Tells whether graph churn from the variable shapes is a cost	A/B on P-code	Low
H	-ctk q4_0 -ctv q4_0 at 52k	Saves ~0.85 GB read per token (~4-5 % on plain decode) and removes some f16 conversion traffic; quality risk	A/B on P-code-52k plus a spot-check of output quality	Medium (quality)
I	-ctk f16 -ctv f16 --ctx-size 131072	Removes q8_0 to f16 conversion for the TILE verify kernel (2.2); cache is 1.9x larger. VRAM: f16 KV for 131072 tokens is ~8 GB	A/B on P-code-52k	Medium (halves context, VRAM)
J	Smaller quant (UD-Q4_K_M or UD-IQ4_XS) from the same repo	Up to +13 % / +32 % on the plain ceiling (4); quality unmeasured	A/B plus KLD/perplexity	Medium (quality, download)
K	Vulkan (RADV) image, if draft-dflash runs on it	Reported decode lead on some setups, but conflicting sources and prefill lag at short context	A/B with the same protocol	High (new image, unverified DFlash2 support)
L	--spec-type draft-dflash,ngram-mod (list is accepted, arg.cpp#L4314-L4322)	Extra drafts on repeated text (tool JSON, code). Interaction with the DFlash drafter is unchecked	A/B on P-code	Medium

Not worth an experiment: -t, --poll, --no-mmap, --mlock, GGML_CUDA_FORCE_MMQ, drafter quant (3.4, 3.1, 1.4).

Open questions

• What does the server .env say for LLAMA_SPEC_N_MAX, and what does the new startup log print for speculative decoding will use checkpoints and n_rs_seq?
• What temperature, top-p/top-k and penalties do OmniRoute clients actually send? If they differ per client, acceptance will differ per client.
• Is the ~45 ms fixed overhead per spec step in the target verify, the drafter, or the CPU? Experiment C and a rocprofv3 trace would answer it.
• Does a Vulkan build run draft-dflash at all?
• Which exact llama.cpp build is in the image, and is it before or after the n_rs_seq snapshot change?

Sources

• Repo: llmserver.log (lines cited inline), docker-compose.yml, .env.example, docs/architecture.md#llama-server, docs/research/qwen3.8-27b-throughput-tuning.md, docs/research/qwen3.8-27b-quant.md.
• llama.cpp at 839a572a4cdecdb80d211297b8e7d7323a6e99d2: common/speculative.cpp, common/arg.cpp, common/common.h, common/sampling.cpp, tools/server/server-context.cpp, src/llama-kv-cache.cpp, src/llama-arch.cpp, src/llama-context.cpp, src/llama-memory-recurrent.cpp, ggml/src/ggml-cuda/{fattn.cu,fattn-common.cuh,mmq.cu,mmvq.cu,mmvq.cuh,ggml-cuda.cu}, ggml/CMakeLists.txt, .devops/rocm.Dockerfile. Line numbers were read from raw.githubusercontent.com/.../master/... on the day of fetch; if the link anchors drift, search by the quoted identifier.
• PR #27342 (DFlash2), issue #27117, discussion #21043, discussion #20881.
• z-lab/Qwen3.8-27B-DFlash2-GGUF, z-lab/Qwen3.8-27B-DFlash2 (via a search summary), HIP environment variables.
• Not fetched: the Phoronix R9700 ROCm vs Vulkan review (HTTP 403), AMD ROCm RDNA4 pages beyond the HIP env page.