# Research: faster tokens/s on the R9700 (llama-server, Qwen3.8-27B + DFlash2)

**Date:** 2026-10-20

**Question:** Can `llama-server`'s configuration be tuned for more tokens per second, and can system DDR RAM act as a slower cache tier? Evidence: `llmserver.log` (about 3 h of traffic) plus five outside reference configs.

**Scope note:** research only. Nothing in compose or `.env.example` was changed.

## TL;DR, ranked

| # | Change | Expected gain | Risk | Confidence |
|---|---|---|---|---|
| 1 | Find out why ~half of all requests are full ~40-50k-token re-prefills (log shows `selected slot by LRU` and no prefix reuse), then fix the prefix instability or give the cache room: `--cache-ram` up (e.g. 24576), keep `--ctx-checkpoints` at its default | Largest available. Prefill is 2.06 h of the ~4.7 h of summed slot time in the log. Every avoided 45k prefill saves ~47 s | Low (host RAM only) | Medium: the cause is not visible in the log (see Open questions) |
| 2 | Sweep `--spec-draft-n-max` (try 3, 4, 5 vs 7) and add `--spec-draft-p-min` (e.g. 0.5) | Maybe +5 to +15 % decode, unmeasured. Measured acceptance is only 50 % at n_max 7 | Low | Low-medium: needs an A/B |
| 3 | A/B `-ub 1024` or `2048` (with `-b 2048`+) for prefill | Maybe +5 to +20 % prefill, unmeasured. One reference saw no gain on Vulkan, another uses 8192/8192 | Low-medium (compute buffer VRAM; ~6 GB headroom) | Low |
| 4 | A/B the Vulkan (RADV) build against ROCm | A reference measured +23 to +31 % decode with MTP, and faster prefill at 32k+ depth. ROCm was faster at short prompts | Medium (new image, DFlash2 support on Vulkan is unchecked) | Medium (single third-party repo) |
| 5 | Do NOT expect gains from: `--no-kv-offload`, `--n-cpu-moe`/`--cpu-moe`, `--mlock`, RAM layer offload | None or negative | n/a | High |

Decode (45.9 t/s weighted) is already well above plain decode (~25-30 t/s for this model on this card per the references), so DFlash2 works. The remaining headroom is in prefill and in cache hits, not raw decode.

## Measured baseline (from `llmserver.log`)

**The log is a mid-run tail. It has no startup block.** There is no build string, `n_ctx`, KV buffer sizes, ubatch or device memory line, so the llama.cpp version and the values below come from compose, not from the log. Grep for `build:`, `load_tensors`, `KV buffer`, `context checkpoints enabled` and `cache_reuse is not supported` returned nothing. Get those lines with `docker logs llama-server | head -200` on the server after the next restart. The timestamp format is `MMMMM.SS.mmm.uuu` (minutes of uptime), so the 10917 to 11098 range is about 181 min.

All figures below were computed over the whole file (10000 lines).

- Tasks: 360 launched (`launch_slot_`), 305 completed with a timing summary, 56 `cancel task` lines (first at `llmserver.log:2113`).
- **Generation:** weighted average 45.9 t/s over 380k generated tokens. Per request (those with >=200 tokens): average 49.4, max 76.9, min 4.1 t/s. Running `tg` (the live counter, `llmserver.log:98`-`100`): p10 15.0, p50 33.0, p90 56.6 t/s. The low end is decode competing with the other slot's prefill, not a different config.
- **Prefill:** 933-980 t/s on 38-50k-token prompts (`:32`, `:65`). Rate falls with depth, from 1282 t/s at the first 6k tokens to ~945 t/s at 50k (`:8`-`:31`). Bucket averages: <500 tokens 417 t/s, 0.5-5k 839, 5-20k 952, >20k 904.
- **Speculation:** draft acceptance 50.1 % overall (n=306 summaries), mean draft length 3.86. Individual requests range 0.38 (`:36`) to 0.71 (`:69`). n_max is 7, so a lot of drafted tokens are rejected.
- **Totals:** summed prefill time 7404 s, summed decode time 9503 s. Prefill is not the minority.
- **Request shape:** 146 of 305 requests were >20k-token prefills (average 42.5k tokens, ~47 s each). Typical output on those is small (`:33` 99 tokens, `:66` 78 tokens). `release` n_tokens: min 1097, median 8395, max 57073.
- **Cache behaviour:** `--cache-reuse` is evidently not what saves these requests. 200 slot selections were `by LRU`, 160 `by LCP similarity` (average f_sim_best 0.60, e.g. 0.997 at `:2279`). 253 `making room for prompt cache entry, removing oldest entry (size ~1.8-2.3 GiB)` warnings (`:7`, `:40`, `:73`) show the host-RAM prompt cache is active and full. The upstream default is 8192 MiB (see sources), so it holds only ~3-4 such entries.
- **Pattern at `:380`-`:470`:** requests alternate slot 0 / slot 1 and each does a full ~38-46k prefill by LRU. The prefix is not shared with the previous request on either slot.

Configured (compose, `docker-compose.yml`): `--ctx-size 262144`, `--parallel 2` (131072 per slot), `--kv-unified`, `--flash-attn on`, `--cache-type-k/v q8_0`, `--cache-reuse 256`, `--spec-type draft-dflash`, `-md ...DFlash2-Q4_K_M.gguf`, `--spec-draft-n-max 7`, `--n-gpu-layers 999`, `GPU_MAX_HW_QUEUES=1`. Not set, so defaults: `-b 2048`, `-ub 512`, threads, `--cache-ram` (8192), `--ctx-checkpoints` (32), `--spec-draft-p-min` (0.00), sampling for the drafter. Prior measured VRAM: ~81 % (25.8 of 31.9 GB) at 262144 q8_0 (project memory note `llama-server-vram-baseline`).

## Flag verification (primary: llama.cpp master)

Sources: `tools/server/README.md` and `common/arg.cpp` and `tools/server/server-context.cpp` on `ggml-org/llama.cpp` master, fetched 2026-10-20. Master is not necessarily the build in the log (version unknown, see above), so check `llama-server --help` inside the container before using any flag.

| Flag | Exists on master | Default / note |
|---|---|---|
| `-cram, --cache-ram N` | yes | default 8192 MiB, -1 no limit, 0 disable. Env `LLAMA_ARG_CACHE_RAM` |
| `-ctxcp, --ctx-checkpoints N` (alias `--swa-checkpoints`) | yes | default 32 per slot |
| `-cms, --checkpoint-min-step N` | yes (server README) | default 8192 tokens, 0 = no minimum |
| `--cache-reuse N` | yes | "min chunk size to attempt reusing from the cache via KV shifting", default 0. `server-context.cpp` disables it with the log line `cache_reuse is not supported by this context, it will be disabled` if `llama_memory_can_shift` is false |
| `--cache-idle-slots` | yes | default enabled, requires cache-ram |
| `-sps, --slot-prompt-similarity` | yes | default 0.10 (matches the `0.100 thold` in the log) |
| `-b`, `-ub` | yes | 2048 / 512 |
| `--spec-type`, `--spec-draft-n-max/-n-min/-p-min` | yes | n_max default 3, n_min 0, p_min 0.00. The README says the old `--draft-max` "has been removed" |
| `-kvo/-nkvo, --kv-offload` | yes | default enabled |
| `-lm, --load-mode` (mmap, mlock, ...) | yes | default auto. Newer master has no standalone `--mlock`/`--no-mmap` in the README text I read; older builds do |
| `-cmoe`, `-ncmoe` | yes | MoE only. Irrelevant for a dense model |
| `-t, --threads` | yes | -1 |

Not verified: that the `draft-dflash` value of `--spec-type` is in the log's build (the stack evidently runs it, so it is). Not verified: that hybrid (Gated DeltaNet + attention) memory reports `can_shift = false`. I did not read `llama-memory-hybrid.cpp`. If it does, `--cache-reuse 256` is silently disabled for this model and the real mechanism is checkpoints plus the prompt cache. Check the startup log for the `cache_reuse is not supported` line. Also not verified: the exact condition under which a checkpoint is created (the fetched `server-context.cpp` was cut off before it), so the checkpoint advice below is a lead.

### Recommendation details

1. **Prompt cache / re-prefill (rank 1).** The cost is in the log, the cause is not. Three candidates, in order of likelihood:
   - (a) the client or gateway changes bytes early in the prompt on each request (timestamps, session ids, memory injection, tool list order), so LCP is near zero. Fix at the source: diff two consecutive prompts from a long gap.
   - (b) the model is hybrid, a recurrent state cannot be rolled back to an arbitrary token, so partial-prefix reuse needs a checkpoint at the right position (`--ctx-checkpoints`, `--checkpoint-min-step`). With `-cms` 8192, a checkpoint sits at most every 8192 tokens, so some prefix is lost on each divergence. `-cms 2048` is a candidate, at ~tens of MiB each (unmeasured).
   - (c) the cache is too small. 2.3 GiB per 50k-token entry against 8192 MiB means ~3 entries for 2 slots plus history. Raise `--cache-ram` (host RAM). This is the DDR RAM tier, see below.
2. **Speculation (rank 2).** Acceptance 50 % at n_max 7 means expected accepted length ~3.9. The z-lab/lucebox write-up notes DFlash2 was trained at block 8 and that `--spec-draft-n-max 7` is clamped to it, so 7 is already the maximum useful value. Lower n_max cuts wasted drafter and verify work at the cost of fewer tokens per step. `--spec-draft-p-min` makes the drafter stop early on low-confidence tokens. Both are standard knobs, but no reference tuned them for DFlash2 on llama.cpp. Treat as an A/B, not a recommendation. Greedy sampling collapses DFlash2 acceptance (cat5edopeHA `ISSUES.md` #8), so if clients send temperature 0 for some workloads, acceptance will be bad there. Check per-request temperature against the 0.38-0.71 spread.
3. **Batch (rank 3).** Prefill rate is 933-980 t/s at 40-50k depth. References on the same card: cat5edopeHA nospec ROCm 541 t/s at 32k and 298 at 64k (Q4_K_M, `-b/-ub` default), rahlquist 727 at 40k and 564 at 65k with 8192/8192 (f16 KV, `llama-bench`). This repo's 945 at 46k is already faster than either at that depth, so do not expect large batch gains. Only one controlled `-ub` data point exists (cat5edopeHA, Vulkan, decode got slightly worse with 1024/1024).
4. **KV quant (neutral).** q8_0 keeps VRAM headroom. lucebox measured f16 KV slightly slower than q8_0 on its own engine (202.5 vs 208.1 t/s). Do not drop to q4_0 for speed. rahlquist serves q4_0 but gives no quality data.
5. **Threads, mmap, mlock.** The whole model and KV sit in VRAM, so CPU thread count affects only the host-side sampling/scheduling. No gain expected. `--mlock` would pin the 17.6 GB GGUF in RAM for no benefit once it is on the GPU.

## DDR RAM as a slower tier

| Idea | Helps t/s? | Why |
|---|---|---|
| Layer offload to CPU (`--n-gpu-layers` below 999) | No, slower | Dense decode reads all weights per token. VRAM bandwidth is several times DDR. Every layer moved to RAM adds PCIe/DDR-bound time. Weights (17.6 GB) already fit |
| `--cpu-moe`, `--n-cpu-moe` | No | Model is dense. Flag text is "keep the Mixture of Experts (MoE) weights ... in the CPU" |
| `--no-kv-offload` (KV in RAM) | No, much slower | Attention reads KV every step. Keeps VRAM but moves the hot path to DDR |
| **`--cache-ram` (host-RAM prompt cache)** | **Yes, for agent workloads** | It is exactly this feature. It stores whole-slot states in host RAM and reloads them instead of re-prefilling (`get_available_slot` saves a slot when `f_keep < 0.5`, per `server-context.cpp`). The log shows it already churning at the 8192 MiB default |
| `--slot-save-path` | Maybe, manual | Persists slots to disk, restore via API. Needs client orchestration |
| mmap | Only a load-time effect | Model is on GPU after load |

Rough sizing: entries are 1.8-2.3 GiB for 38-50k tokens (log `:7`, `:40`, `:73`). 32 GiB of cache would hold ~14 such entries. Host RAM size on the server is not known to me. Restoring an entry copies about 2 GiB over PCIe, on the order of a second, against ~47 s of prefill. That timing is an estimate, not measured.

## Comparison with reference configs

Legend: **V** = independently verified against llama.cpp source or reproduced in this repo's own data. **P** = reported only by the page, treated as a lead.

| Source | Quant / engine | Ctx, KV | Spec | Batch | R9700 numbers | Status |
|---|---|---|---|---|---|---|
| **This repo (log)** | UD-Q4_K_XL, llama.cpp ROCm | 262144, q8_0, `-np 2` | DFlash2 n_max 7 | default 2048/512 | decode 45.9 t/s avg, prefill 933-980 t/s at 40-50k | V (log) |
| [cat5edopeHA/qwen38-27b-ai1](https://github.com/cat5edopeHA/qwen38-27b-ai1) (first-party repo) | Q4_K_M + baked MTP, llama.cpp `d222767`, HIP and Vulkan | 262144, q8_0, `-np 1`, `-fa on`, `-ngl 99` | `draft-mtp` n_max 3 (best on both; Vulkan 4-5 wins at >=8k) | defaults; `-b 1024 -ub 1024` no gain | mtp3 decode Vulkan/ROCm: 68.5/55.1 short, 50.5/40.2 at 32k, 40.9/31.2 at 64k. nospec 25-30. Prefill ROCm 1047 short, 541 at 32k, 298 at 64k (Vulkan 377 at 64k) | P for numbers (not rerun). Its flags exist on master (V) |
| [rahlquist/qwen38-r9700](https://github.com/rahlquist/qwen38-r9700) (first-party repo) | Q4_K_M, ROCm build 10438 | 65536, f16 bench, q4_0 in serving | MTP draft did not load (VRAM) | 8192/8192 (ubatch 512 at 128k) | prefill 1080 at 2.5k, 727 at 40k, 564 at 65k. Decode 26.4 to 22.5 (no spec) | P. Its decode matches nospec numbers elsewhere |
| [lucebox blog](https://www.lucebox.com/blog/qwen38-r9700) (secondary) | UD-IQ4_XS, own `dflash_server` engine (not llama.cpp); llama.cpp + PR #27342 for comparison | 131072, q8_0 | DFlash2 `--draft-block-size 16`; llama.cpp run used n_max 7 | not stated | own engine 208 decode (HumanEval, greedy). **llama.cpp + DFlash2: 54.6 decode**, 59.7 at 1.5k, ~31-35 at 36-70k | P. The llama.cpp row is the fair comparison to this repo: log average 45.9 sits inside it |
| [aifactum.de](https://aifactum.de/ki-hardware/qwen3-8-27b-lokal-auf-amd-r9700/) (secondary) | llama.cpp Vulkan, Windows (AMD figure, preliminary) | not stated | MTP, 2 draft tokens | not stated | up to 51.8 t/s; advises disabling mmap in LM Studio | P. Vendor number, no config |
| [willitrunai.com](https://willitrunai.com/can-run/qwen-3.8-27b-on-radeon-ai-pro-r9700-32gb) (secondary) | Q4_K_M, estimator | 168K "safe" | none | none | decode 16.3, prefill 688 (all estimates) | P. A calculator, contradicts itself (22.7 vs 16.9 GB) and is below measured data. Ignore |

Take-aways:

- Same card, no-spec decode is ~25-30 t/s in three independent sources. This repo's 45.9 t/s with DFlash2 is in line with the lucebox llama.cpp + DFlash2 row (54.6 on short HumanEval prompts, ~31-36 at deep context). **No reference shows a materially faster llama.cpp ROCm decode than this repo already has.** The 200+ t/s figure is a different engine (lucebox) with a 16-wide block and greedy short prompts, not a flag change here.
- The only reference-backed decode lever is the **backend**: Vulkan decode beat ROCm by 23-31 % in one repo, but ROCm was faster at prefill for short prompts, and Vulkan faster at >=32k. This workload is both prefill-heavy and deep, so the net is unclear. Needs a local `llama-bench` on both builds. The repo's `docs/research/qwen3.8-27b-quant.md` already mentions the same Vulkan claim from a community source.
- Differences that do not obviously matter: this repo's quant (UD-Q4_K_XL, 17.6 GB) is bigger than UD-IQ4_XS (13.3 GB) and Q4_K_M (16.5 GB). A smaller quant reads fewer bytes per token and would decode a little faster, at a quality cost not measured here. lucebox's `--pure` requant claim is about their own engine and not applicable.
- cat5edopeHA used `-np 1`. This repo uses 2, and the log shows decode slowing while the other slot prefills (running tg p10 15 t/s vs p90 57 t/s). `-np 1` removes that interference but serializes agents and risks the gateway timeouts documented in `docs/gateway.md`.

## Proposed diffs (not applied)

All via `.env`/compose, A/B one at a time. Compose is the source of truth, so these would need doc updates per `docs/agents/writing-docs.md` rule 4.

```diff
 # docker-compose.yml, llama-server command
+      --cache-ram ${LLAMA_CACHE_RAM:-24576}      # rank 1, host RAM
+      --checkpoint-min-step ${LLAMA_CHECKPOINT_MIN_STEP:-2048}   # rank 1 lead, verify flag name on the running build
-      --spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}
+      --spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}  # sweep 3,4,5,7
+      --spec-draft-p-min 0.5                     # rank 2, A/B
+      -ub 1024                                   # rank 3, A/B (-b stays 2048)
```

Before any of that, confirm VRAM headroom: ~6 GB was left at 81 % (memory note). A larger `-ub` raises the compute buffer.

## How to benchmark before and after

1. **Baseline from the server's own timings.** Replay a fixed ~40k-token prompt and a fixed ~2k-token prompt, `temperature` as production uses (not 0), 3 repeats, one slot idle. Read `timings.prompt_per_second`, `predicted_per_second`, `draft_n`, `draft_n_accepted` from the response (the field names are used by cat5edopeHA's harness; confirm on this build).
2. **`llama-bench`** for batch sweeps, inside the container (build must match): `llama-bench -m <model> -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -p 512,8192,40000 -n 128 -ub 512,1024,2048`. `llama-bench` does not exercise speculation or the prompt cache, so it only informs rank 3 and the backend question.
3. **Cache hit rate.** After changes, count `selected slot by LCP similarity` vs `by LRU` and `making room for prompt cache entry` per hour in the log, and sum `prompt eval time`. Target: sum of prefill seconds per hour drops. This is the metric for rank 1.
4. Change one variable per run. Restart `llama-server` with `./scripts/update.sh` on the server (this box cannot deploy).

## Open questions

- Why do consecutive ~45k-token requests share no prefix? Capture two raw request bodies and diff the first 2k tokens. Check whether OmniRoute's slot affinity is active (log shows LRU selection 200 times).
- What llama.cpp build is running, and does the startup log say `cache_reuse is not supported by this context`? If yes, `--cache-reuse 256` can be removed and `docs/architecture.md` is wrong about it.
- How much host RAM does the server have free for `--cache-ram`?
- Do clients send temperature 0 (collapses DFlash2 acceptance per cat5edopeHA `ISSUES.md` #8)? Acceptance varies 0.38-0.71 per request.
- Is DFlash2 supported on a Vulkan llama.cpp build (the reference used a fork for it)? Unchecked.
- Which cancellations (56 `cancel task`) are client timeouts mid-prefill, and do they waste prefill work?

## Sources

- Log facts: `g:\_DEV\repos\LLM-Server\llmserver.log` (line numbers above).
- Compose and docs: `docker-compose.yml`, `.env.example`, `docs/architecture.md#llama-server`, `docs/research/omniroute-non-ping-sse-stream-timeout.md`, `docs/research/qwen3.8-27b-quant.md`.
- llama.cpp primary: [tools/server/README.md](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md), [common/arg.cpp](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp), [tools/server/server-context.cpp](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/server-context.cpp), [PR #27342](https://github.com/ggml-org/llama.cpp/pull/27342) (cited by the repo's own docs, not re-read here).
- References: [cat5edopeHA/qwen38-27b-ai1](https://github.com/cat5edopeHA/qwen38-27b-ai1) (README, RECIPES.md, ISSUES.md, BENCHMARKS.md), [rahlquist/qwen38-r9700](https://github.com/rahlquist/qwen38-r9700), [lucebox](https://www.lucebox.com/blog/qwen38-r9700), [aifactum.de](https://aifactum.de/ki-hardware/qwen3-8-27b-lokal-auf-amd-r9700/), [willitrunai.com](https://willitrunai.com/can-run/qwen-3.8-27b-on-radeon-ai-pro-r9700-32gb). The WebFetch summaries of these pages were produced by a small model; numbers were not re-checked against the raw pages.