feat(bench): add speed-bench tool service for drafter benchmarks

Runs llama.cpp's SPEED-Bench client (pinned commit) against
llama-server over the compose network, in the tools profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.gitignore

@@ -5,4 +5,6 @@
 data/
 .leankg/
 .cache/
-.qwen/tmp
+.qwen/tmp
+# speed-bench output, see docs/operations.md#speed-bench
+bench-results/

README.md

@@ -22,7 +22,7 @@ It creates and syncs `.env`, generates secrets, downloads the models, and brings
 | [`docs/architecture.md`](docs/architecture.md) | Each service, why it exists, and why it's configured the way it is |
 | [`docs/gateway.md`](docs/gateway.md) | OmniRoute: dashboard, provider connections, API keys, timeouts, web search |
 | [`docs/network-access.md`](docs/network-access.md) | Who can reach which port, and why |
-| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads |
+| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads; speed bench |
 | [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md) | Pointing Claude Code, Kimi CLI, OpenCode and Qwen Code at the gateway |
 | [`docs/research/`](docs/research/README.md) | Dated research and decision records, each marked current or superseded |
 | [`docs/server-planing.md`](docs/server-planing.md) | Background research: a 4× GPU rack build. Not an active plan |

docker-compose.yml

@@ -67,6 +67,16 @@ services:
         curl -L --fail --create-dirs -o /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
         https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
 
+  # see docs/operations.md#speed-bench
+  speed-bench:
+    profiles: ["tools"]
+    build: ./speed-bench
+    entrypoint: ["python", "/app/speed_bench.py", "--url", "http://llama-server:8080"]
+    volumes:
+      - hf-cache:/root/.cache/huggingface
+      - ./bench-results:/results
+    networks: [ai-stack]
+
   # see docs/architecture.md#omniroute; runtime config lives in its dashboard, see docs/gateway.md
   omniroute:
     image: diegosouzapw/omniroute:latest
@@ -127,3 +137,4 @@ volumes:
   models:
   omniroute-data:
   qdrant-data:
+  hf-cache:

docs/operations.md

@@ -1,6 +1,6 @@
 # Operations
 
-> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh) and [`.env.example`](../.env.example). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins.
+> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins.
 
 The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed.
 
@@ -40,3 +40,23 @@ The conflict screen uses [gum](https://github.com/charmbracelet/gum), a single s
 ## Model downloads
 
 Each model file has its own one-off downloader service in the `tools` compose profile. `update.sh` runs them every time. A `test -f` guard skips files that already exist, so re-runs don't re-download. Files live in the named `models` volume, not a host bind-mount. The downloaders run as root because the named volume is created root-owned. There's one service per model, so the small model's job never re-checks the large one.
+
+## Speed bench
+
+`speed-bench` runs llama.cpp's [SPEED-Bench client](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/bench/speed-bench/README.md) against the running `llama-server`. It measures decode speed (`avg_pred_t/s`), latency and draft acceptance per category. Use it to compare drafters and their flags with numbers instead of gut feel. It's in the `tools` profile, so `update.sh` never builds or runs it. The first run builds the image.
+
+- **The scripts are pinned** to one llama.cpp commit (`LLAMA_CPP_REF` in the Dockerfile). The server image is not pinned, so bump the ref if the client stops matching the server's response format.
+- **It reaches the backend over the compose network**, so llama-server needs no host port. The dataset (`nvidia/SPEED-Bench`, not gated) downloads on first run and is cached in the `hf-cache` volume. Results land in `bench-results/` (gitignored).
+- **It measures speed, not correctness.** The output text isn't saved, so it won't catch a drafter that corrupts tokens.
+
+Run it on the server. Pass `--concurrency` equal to `LLAMA_PARALLEL` for throughput, or leave it at 1 for the single-session speed one CLI sees:
+
+```sh
+docker compose --profile tools run --rm speed-bench --category coding --limit 8 --output spec.json
+```
+
+To get a speedup figure, also run a **baseline** without speculative decoding. Temporarily remove the `--spec-*` and `-md` lines from the llama-server command, run `docker compose up -d llama-server`, then run the same bench with `--output baseline.json`. Keep `--bench`, `--category`, `--osl` and `--limit` identical. Restore the lines with `git checkout docker-compose.yml`, then compare:
+
+```sh
+docker compose --profile tools run --rm --entrypoint python speed-bench /app/speed_bench_compare.py --baseline baseline.json --speculative spec.json
+```

speed-bench/Dockerfile

@@ -0,0 +1,9 @@
+# see docs/operations.md#speed-bench
+FROM python:3.12-slim
+ARG LLAMA_CPP_REF=0ec4027a2a0183de87c007069adddc06e665f7de
+ADD https://raw.githubusercontent.com/ggml-org/llama.cpp/${LLAMA_CPP_REF}/tools/server/bench/speed-bench/requirements.txt \
+    https://raw.githubusercontent.com/ggml-org/llama.cpp/${LLAMA_CPP_REF}/tools/server/bench/speed-bench/speed_bench.py \
+    https://raw.githubusercontent.com/ggml-org/llama.cpp/${LLAMA_CPP_REF}/tools/server/bench/speed-bench/speed_bench_compare.py \
+    /app/
+RUN pip install --no-cache-dir -r /app/requirements.txt
+WORKDIR /results