LLM-Server/docs/research/qwen3.8-27b-throughput-tuning.md

Research: faster tokens/s on the R9700 (llama-server, Qwen3.8-27B + DFlash2)

Date: 2026-10-20

Question: Can llama-server's configuration be tuned for more tokens per second, and can system DDR RAM act as a slower cache tier? Evidence: llmserver.log (about 3 h of traffic) plus five outside reference configs.

Scope note: research only. Nothing in compose or .env.example was changed.

TL;DR, ranked

#	Change	Expected gain	Risk	Confidence
1	Find out why ~half of all requests are full ~40-50k-token re-prefills (log shows selected slot by LRU and no prefix reuse), then fix the prefix instability or give the cache room: --cache-ram up (e.g. 24576), keep --ctx-checkpoints at its default	Largest available. Prefill is 2.06 h of the ~4.7 h of summed slot time in the log. Every avoided 45k prefill saves ~47 s	Low (host RAM only)	Medium: the cause is not visible in the log (see Open questions)
2	Sweep --spec-draft-n-max (try 3, 4, 5 vs 7) and add --spec-draft-p-min (e.g. 0.5)	Maybe +5 to +15 % decode, unmeasured. Measured acceptance is only 50 % at n_max 7	Low	Low-medium: needs an A/B
3	A/B -ub 1024 or 2048 (with -b 2048+) for prefill	Maybe +5 to +20 % prefill, unmeasured. One reference saw no gain on Vulkan, another uses 8192/8192	Low-medium (compute buffer VRAM; ~6 GB headroom)	Low
4	A/B the Vulkan (RADV) build against ROCm	A reference measured +23 to +31 % decode with MTP, and faster prefill at 32k+ depth. ROCm was faster at short prompts	Medium (new image, DFlash2 support on Vulkan is unchecked)	Medium (single third-party repo)
5	Do NOT expect gains from: --no-kv-offload, --n-cpu-moe/--cpu-moe, --mlock, RAM layer offload	None or negative	n/a	High

Decode (45.9 t/s weighted) is already well above plain decode (~25-30 t/s for this model on this card per the references), so DFlash2 works. The remaining headroom is in prefill and in cache hits, not raw decode.

Measured baseline (from llmserver.log)

The log is a mid-run tail. It has no startup block. There is no build string, n_ctx, KV buffer sizes, ubatch or device memory line, so the llama.cpp version and the values below come from compose, not from the log. Grep for build:, load_tensors, KV buffer, context checkpoints enabled and cache_reuse is not supported returned nothing. Get those lines with docker logs llama-server | head -200 on the server after the next restart. The timestamp format is MMMMM.SS.mmm.uuu (minutes of uptime), so the 10917 to 11098 range is about 181 min.

All figures below were computed over the whole file (10000 lines).

• Tasks: 360 launched (launch_slot_), 305 completed with a timing summary, 56 cancel task lines (first at llmserver.log:2113).
• Generation: weighted average 45.9 t/s over 380k generated tokens. Per request (those with >=200 tokens): average 49.4, max 76.9, min 4.1 t/s. Running tg (the live counter, llmserver.log:98-100): p10 15.0, p50 33.0, p90 56.6 t/s. The low end is decode competing with the other slot's prefill, not a different config.
• Prefill: 933-980 t/s on 38-50k-token prompts (:32, :65). Rate falls with depth, from 1282 t/s at the first 6k tokens to ~945 t/s at 50k (:8-:31). Bucket averages: <500 tokens 417 t/s, 0.5-5k 839, 5-20k 952, >20k 904.
• Speculation: draft acceptance 50.1 % overall (n=306 summaries), mean draft length 3.86. Individual requests range 0.38 (:36) to 0.71 (:69). n_max is 7, so a lot of drafted tokens are rejected.
• Totals: summed prefill time 7404 s, summed decode time 9503 s. Prefill is not the minority.
• Request shape: 146 of 305 requests were >20k-token prefills (average 42.5k tokens, ~47 s each). Typical output on those is small (:33 99 tokens, :66 78 tokens). release n_tokens: min 1097, median 8395, max 57073.
• Cache behaviour: --cache-reuse is evidently not what saves these requests. 200 slot selections were by LRU, 160 by LCP similarity (average f_sim_best 0.60, e.g. 0.997 at :2279). 253 making room for prompt cache entry, removing oldest entry (size ~1.8-2.3 GiB) warnings (:7, :40, :73) show the host-RAM prompt cache is active and full. The upstream default is 8192 MiB (see sources), so it holds only ~3-4 such entries.
• Pattern at :380-:470: requests alternate slot 0 / slot 1 and each does a full ~38-46k prefill by LRU. The prefix is not shared with the previous request on either slot.

Configured (compose, docker-compose.yml): --ctx-size 262144, --parallel 2 (131072 per slot), --kv-unified, --flash-attn on, --cache-type-k/v q8_0, --cache-reuse 256, --spec-type draft-dflash, -md ...DFlash2-Q4_K_M.gguf, --spec-draft-n-max 7, --n-gpu-layers 999, GPU_MAX_HW_QUEUES=1. Not set, so defaults: -b 2048, -ub 512, threads, --cache-ram (8192), --ctx-checkpoints (32), --spec-draft-p-min (0.00), sampling for the drafter. Prior measured VRAM: ~81 % (25.8 of 31.9 GB) at 262144 q8_0 (project memory note llama-server-vram-baseline).

Flag verification (primary: llama.cpp master)

Sources: tools/server/README.md and common/arg.cpp and tools/server/server-context.cpp on ggml-org/llama.cpp master, fetched 2026-10-20. Master is not necessarily the build in the log (version unknown, see above), so check llama-server --help inside the container before using any flag.

Flag	Exists on master	Default / note
-cram, --cache-ram N	yes	default 8192 MiB, -1 no limit, 0 disable. Env LLAMA_ARG_CACHE_RAM
-ctxcp, --ctx-checkpoints N (alias --swa-checkpoints)	yes	default 32 per slot
-cms, --checkpoint-min-step N	yes (server README)	default 8192 tokens, 0 = no minimum
--cache-reuse N	yes	"min chunk size to attempt reusing from the cache via KV shifting", default 0. server-context.cpp disables it with the log line cache_reuse is not supported by this context, it will be disabled if llama_memory_can_shift is false
--cache-idle-slots	yes	default enabled, requires cache-ram
-sps, --slot-prompt-similarity	yes	default 0.10 (matches the 0.100 thold in the log)
-b, -ub	yes	2048 / 512
--spec-type, --spec-draft-n-max/-n-min/-p-min	yes	n_max default 3, n_min 0, p_min 0.00. The README says the old --draft-max "has been removed"
-kvo/-nkvo, --kv-offload	yes	default enabled
-lm, --load-mode (mmap, mlock, ...)	yes	default auto. Newer master has no standalone --mlock/--no-mmap in the README text I read; older builds do
-cmoe, -ncmoe	yes	MoE only. Irrelevant for a dense model
-t, --threads	yes	-1

Not verified: that the draft-dflash value of --spec-type is in the log's build (the stack evidently runs it, so it is). Not verified: that hybrid (Gated DeltaNet + attention) memory reports can_shift = false. I did not read llama-memory-hybrid.cpp. If it does, --cache-reuse 256 is silently disabled for this model and the real mechanism is checkpoints plus the prompt cache. Check the startup log for the cache_reuse is not supported line. Also not verified: the exact condition under which a checkpoint is created (the fetched server-context.cpp was cut off before it), so the checkpoint advice below is a lead.

Recommendation details

1. Prompt cache / re-prefill (rank 1). The cost is in the log, the cause is not. Three candidates, in order of likelihood:
   • (a) the client or gateway changes bytes early in the prompt on each request (timestamps, session ids, memory injection, tool list order), so LCP is near zero. Fix at the source: diff two consecutive prompts from a long gap.
   • (b) the model is hybrid, a recurrent state cannot be rolled back to an arbitrary token, so partial-prefix reuse needs a checkpoint at the right position (--ctx-checkpoints, --checkpoint-min-step). With -cms 8192, a checkpoint sits at most every 8192 tokens, so some prefix is lost on each divergence. -cms 2048 is a candidate, at ~tens of MiB each (unmeasured).
   • (c) the cache is too small. 2.3 GiB per 50k-token entry against 8192 MiB means ~3 entries for 2 slots plus history. Raise --cache-ram (host RAM). This is the DDR RAM tier, see below.
2. Speculation (rank 2). Acceptance 50 % at n_max 7 means expected accepted length ~3.9. The z-lab/lucebox write-up notes DFlash2 was trained at block 8 and that --spec-draft-n-max 7 is clamped to it, so 7 is already the maximum useful value. Lower n_max cuts wasted drafter and verify work at the cost of fewer tokens per step. --spec-draft-p-min makes the drafter stop early on low-confidence tokens. Both are standard knobs, but no reference tuned them for DFlash2 on llama.cpp. Treat as an A/B, not a recommendation. Greedy sampling collapses DFlash2 acceptance (cat5edopeHA ISSUES.md #8), so if clients send temperature 0 for some workloads, acceptance will be bad there. Check per-request temperature against the 0.38-0.71 spread.
3. Batch (rank 3). Prefill rate is 933-980 t/s at 40-50k depth. References on the same card: cat5edopeHA nospec ROCm 541 t/s at 32k and 298 at 64k (Q4_K_M, -b/-ub default), rahlquist 727 at 40k and 564 at 65k with 8192/8192 (f16 KV, llama-bench). This repo's 945 at 46k is already faster than either at that depth, so do not expect large batch gains. Only one controlled -ub data point exists (cat5edopeHA, Vulkan, decode got slightly worse with 1024/1024).
4. KV quant (neutral). q8_0 keeps VRAM headroom. lucebox measured f16 KV slightly slower than q8_0 on its own engine (202.5 vs 208.1 t/s). Do not drop to q4_0 for speed. rahlquist serves q4_0 but gives no quality data.
5. Threads, mmap, mlock. The whole model and KV sit in VRAM, so CPU thread count affects only the host-side sampling/scheduling. No gain expected. --mlock would pin the 17.6 GB GGUF in RAM for no benefit once it is on the GPU.

DDR RAM as a slower tier

Idea	Helps t/s?	Why
Layer offload to CPU (--n-gpu-layers below 999)	No, slower	Dense decode reads all weights per token. VRAM bandwidth is several times DDR. Every layer moved to RAM adds PCIe/DDR-bound time. Weights (17.6 GB) already fit
--cpu-moe, --n-cpu-moe	No	Model is dense. Flag text is "keep the Mixture of Experts (MoE) weights ... in the CPU"
--no-kv-offload (KV in RAM)	No, much slower	Attention reads KV every step. Keeps VRAM but moves the hot path to DDR
--cache-ram (host-RAM prompt cache)	Yes, for agent workloads	It is exactly this feature. It stores whole-slot states in host RAM and reloads them instead of re-prefilling (get_available_slot saves a slot when f_keep < 0.5, per server-context.cpp). The log shows it already churning at the 8192 MiB default
--slot-save-path	Maybe, manual	Persists slots to disk, restore via API. Needs client orchestration
mmap	Only a load-time effect	Model is on GPU after load

Rough sizing: entries are 1.8-2.3 GiB for 38-50k tokens (log :7, :40, :73). 32 GiB of cache would hold ~14 such entries. Host RAM size on the server is not known to me. Restoring an entry copies about 2 GiB over PCIe, on the order of a second, against ~47 s of prefill. That timing is an estimate, not measured.

Comparison with reference configs

Legend: V = independently verified against llama.cpp source or reproduced in this repo's own data. P = reported only by the page, treated as a lead.

Source	Quant / engine	Ctx, KV	Spec	Batch	R9700 numbers	Status
This repo (log)	UD-Q4_K_XL, llama.cpp ROCm	262144, q8_0, -np 2	DFlash2 n_max 7	default 2048/512	decode 45.9 t/s avg, prefill 933-980 t/s at 40-50k	V (log)
cat5edopeHA/qwen38-27b-ai1 (first-party repo)	Q4_K_M + baked MTP, llama.cpp d222767, HIP and Vulkan	262144, q8_0, -np 1, -fa on, -ngl 99	draft-mtp n_max 3 (best on both; Vulkan 4-5 wins at >=8k)	defaults; -b 1024 -ub 1024 no gain	mtp3 decode Vulkan/ROCm: 68.5/55.1 short, 50.5/40.2 at 32k, 40.9/31.2 at 64k. nospec 25-30. Prefill ROCm 1047 short, 541 at 32k, 298 at 64k (Vulkan 377 at 64k)	P for numbers (not rerun). Its flags exist on master (V)
rahlquist/qwen38-r9700 (first-party repo)	Q4_K_M, ROCm build 10438	65536, f16 bench, q4_0 in serving	MTP draft did not load (VRAM)	8192/8192 (ubatch 512 at 128k)	prefill 1080 at 2.5k, 727 at 40k, 564 at 65k. Decode 26.4 to 22.5 (no spec)	P. Its decode matches nospec numbers elsewhere
lucebox blog (secondary)	UD-IQ4_XS, own dflash_server engine (not llama.cpp); llama.cpp + PR #27342 for comparison	131072, q8_0	DFlash2 --draft-block-size 16; llama.cpp run used n_max 7	not stated	own engine 208 decode (HumanEval, greedy). llama.cpp + DFlash2: 54.6 decode, 59.7 at 1.5k, ~31-35 at 36-70k	P. The llama.cpp row is the fair comparison to this repo: log average 45.9 sits inside it
aifactum.de (secondary)	llama.cpp Vulkan, Windows (AMD figure, preliminary)	not stated	MTP, 2 draft tokens	not stated	up to 51.8 t/s; advises disabling mmap in LM Studio	P. Vendor number, no config
willitrunai.com (secondary)	Q4_K_M, estimator	168K "safe"	none	none	decode 16.3, prefill 688 (all estimates)	P. A calculator, contradicts itself (22.7 vs 16.9 GB) and is below measured data. Ignore

Take-aways:

• Same card, no-spec decode is ~25-30 t/s in three independent sources. This repo's 45.9 t/s with DFlash2 is in line with the lucebox llama.cpp + DFlash2 row (54.6 on short HumanEval prompts, ~31-36 at deep context). No reference shows a materially faster llama.cpp ROCm decode than this repo already has. The 200+ t/s figure is a different engine (lucebox) with a 16-wide block and greedy short prompts, not a flag change here.
• The only reference-backed decode lever is the backend: Vulkan decode beat ROCm by 23-31 % in one repo, but ROCm was faster at prefill for short prompts, and Vulkan faster at >=32k. This workload is both prefill-heavy and deep, so the net is unclear. Needs a local llama-bench on both builds. The repo's docs/research/qwen3.8-27b-quant.md already mentions the same Vulkan claim from a community source.
• Differences that do not obviously matter: this repo's quant (UD-Q4_K_XL, 17.6 GB) is bigger than UD-IQ4_XS (13.3 GB) and Q4_K_M (16.5 GB). A smaller quant reads fewer bytes per token and would decode a little faster, at a quality cost not measured here. lucebox's --pure requant claim is about their own engine and not applicable.
• cat5edopeHA used -np 1. This repo uses 2, and the log shows decode slowing while the other slot prefills (running tg p10 15 t/s vs p90 57 t/s). -np 1 removes that interference but serializes agents and risks the gateway timeouts documented in docs/gateway.md.

Proposed diffs (not applied)

All via .env/compose, A/B one at a time. Compose is the source of truth, so these would need doc updates per docs/agents/writing-docs.md rule 4.

 # docker-compose.yml, llama-server command
+      --cache-ram ${LLAMA_CACHE_RAM:-24576}      # rank 1, host RAM
+      --checkpoint-min-step ${LLAMA_CHECKPOINT_MIN_STEP:-2048}   # rank 1 lead, verify flag name on the running build
-      --spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}
+      --spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}  # sweep 3,4,5,7
+      --spec-draft-p-min 0.5                     # rank 2, A/B
+      -ub 1024                                   # rank 3, A/B (-b stays 2048)

Before any of that, confirm VRAM headroom: ~6 GB was left at 81 % (memory note). A larger -ub raises the compute buffer.

How to benchmark before and after

1. Baseline from the server's own timings. Replay a fixed ~40k-token prompt and a fixed ~2k-token prompt, temperature as production uses (not 0), 3 repeats, one slot idle. Read timings.prompt_per_second, predicted_per_second, draft_n, draft_n_accepted from the response (the field names are used by cat5edopeHA's harness; confirm on this build).
2. llama-bench for batch sweeps, inside the container (build must match): llama-bench -m <model> -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 512,8192,40000 -n 128 -ub 512,1024,2048. llama-bench does not exercise speculation or the prompt cache, so it only informs rank 3 and the backend question.
3. Cache hit rate. After changes, count selected slot by LCP similarity vs by LRU and making room for prompt cache entry per hour in the log, and sum prompt eval time. Target: sum of prefill seconds per hour drops. This is the metric for rank 1.
4. Change one variable per run. Restart llama-server with ./scripts/update.sh on the server (this box cannot deploy).

Open questions

• Why do consecutive ~45k-token requests share no prefix? Capture two raw request bodies and diff the first 2k tokens. Check whether OmniRoute's slot affinity is active (log shows LRU selection 200 times).
• What llama.cpp build is running, and does the startup log say cache_reuse is not supported by this context? If yes, --cache-reuse 256 can be removed and docs/architecture.md is wrong about it.
• How much host RAM does the server have free for --cache-ram?
• Do clients send temperature 0 (collapses DFlash2 acceptance per cat5edopeHA ISSUES.md #8)? Acceptance varies 0.38-0.71 per request.
• Is DFlash2 supported on a Vulkan llama.cpp build (the reference used a fork for it)? Unchecked.
• Which cancellations (56 cancel task) are client timeouts mid-prefill, and do they waste prefill work?

Sources

• Log facts: g:\_DEV\repos\LLM-Server\llmserver.log (line numbers above).
• Compose and docs: docker-compose.yml, .env.example, docs/architecture.md#llama-server, docs/research/omniroute-non-ping-sse-stream-timeout.md, docs/research/qwen3.8-27b-quant.md.
• llama.cpp primary: tools/server/README.md, common/arg.cpp, tools/server/server-context.cpp, PR #27342 (cited by the repo's own docs, not re-read here).
• References: cat5edopeHA/qwen38-27b-ai1 (README, RECIPES.md, ISSUES.md, BENCHMARKS.md), rahlquist/qwen38-r9700, lucebox, aifactum.de, willitrunai.com. The WebFetch summaries of these pages were produced by a small model; numbers were not re-checked against the raw pages.