From 68348aad726a5148f498cd81cdcdc552101c424f Mon Sep 17 00:00:00 2001 From: Haylan Date: Sat, 31 Oct 2026 16:43:06 +0100 Subject: [PATCH] feat(bench): add speed-bench tool service for drafter benchmarks Runs llama.cpp's SPEED-Bench client (pinned commit) against llama-server over the compose network, in the tools profile. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 4 +++- README.md | 2 +- docker-compose.yml | 11 +++++++++++ docs/operations.md | 22 +++++++++++++++++++++- speed-bench/Dockerfile | 9 +++++++++ 5 files changed, 45 insertions(+), 3 deletions(-) create mode 100644 speed-bench/Dockerfile diff --git a/.gitignore b/.gitignore index ce3001f..ba4954b 100644 --- a/.gitignore +++ b/.gitignore @@ -5,4 +5,6 @@ data/ .leankg/ .cache/ -.qwen/tmp \ No newline at end of file +.qwen/tmp +# speed-bench output, see docs/operations.md#speed-bench +bench-results/ diff --git a/README.md b/README.md index 1c8ae18..f3a777f 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ It creates and syncs `.env`, generates secrets, downloads the models, and brings | [`docs/architecture.md`](docs/architecture.md) | Each service, why it exists, and why it's configured the way it is | | [`docs/gateway.md`](docs/gateway.md) | OmniRoute: dashboard, provider connections, API keys, timeouts, web search | | [`docs/network-access.md`](docs/network-access.md) | Who can reach which port, and why | -| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads | +| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads; speed bench | | [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md) | Pointing Claude Code, Kimi CLI, OpenCode and Qwen Code at the gateway | | [`docs/research/`](docs/research/README.md) | Dated research and decision records, each marked current or superseded | | [`docs/server-planing.md`](docs/server-planing.md) | Background research: a 4× GPU rack build. Not an active plan | diff --git a/docker-compose.yml b/docker-compose.yml index 4f7e978..d82158b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -67,6 +67,16 @@ services: curl -L --fail --create-dirs -o /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} + # see docs/operations.md#speed-bench + speed-bench: + profiles: ["tools"] + build: ./speed-bench + entrypoint: ["python", "/app/speed_bench.py", "--url", "http://llama-server:8080"] + volumes: + - hf-cache:/root/.cache/huggingface + - ./bench-results:/results + networks: [ai-stack] + # see docs/architecture.md#omniroute; runtime config lives in its dashboard, see docs/gateway.md omniroute: image: diegosouzapw/omniroute:latest @@ -127,3 +137,4 @@ volumes: models: omniroute-data: qdrant-data: + hf-cache: diff --git a/docs/operations.md b/docs/operations.md index b40bede..97a3576 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,6 +1,6 @@ # Operations -> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh) and [`.env.example`](../.env.example). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins. +> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins. The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed. @@ -40,3 +40,23 @@ The conflict screen uses [gum](https://github.com/charmbracelet/gum), a single s ## Model downloads Each model file has its own one-off downloader service in the `tools` compose profile. `update.sh` runs them every time. A `test -f` guard skips files that already exist, so re-runs don't re-download. Files live in the named `models` volume, not a host bind-mount. The downloaders run as root because the named volume is created root-owned. There's one service per model, so the small model's job never re-checks the large one. + +## Speed bench + +`speed-bench` runs llama.cpp's [SPEED-Bench client](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/bench/speed-bench/README.md) against the running `llama-server`. It measures decode speed (`avg_pred_t/s`), latency and draft acceptance per category. Use it to compare drafters and their flags with numbers instead of gut feel. It's in the `tools` profile, so `update.sh` never builds or runs it. The first run builds the image. + +- **The scripts are pinned** to one llama.cpp commit (`LLAMA_CPP_REF` in the Dockerfile). The server image is not pinned, so bump the ref if the client stops matching the server's response format. +- **It reaches the backend over the compose network**, so llama-server needs no host port. The dataset (`nvidia/SPEED-Bench`, not gated) downloads on first run and is cached in the `hf-cache` volume. Results land in `bench-results/` (gitignored). +- **It measures speed, not correctness.** The output text isn't saved, so it won't catch a drafter that corrupts tokens. + +Run it on the server. Pass `--concurrency` equal to `LLAMA_PARALLEL` for throughput, or leave it at 1 for the single-session speed one CLI sees: + +```sh +docker compose --profile tools run --rm speed-bench --category coding --limit 8 --output spec.json +``` + +To get a speedup figure, also run a **baseline** without speculative decoding. Temporarily remove the `--spec-*` and `-md` lines from the llama-server command, run `docker compose up -d llama-server`, then run the same bench with `--output baseline.json`. Keep `--bench`, `--category`, `--osl` and `--limit` identical. Restore the lines with `git checkout docker-compose.yml`, then compare: + +```sh +docker compose --profile tools run --rm --entrypoint python speed-bench /app/speed_bench_compare.py --baseline baseline.json --speculative spec.json +``` diff --git a/speed-bench/Dockerfile b/speed-bench/Dockerfile new file mode 100644 index 0000000..331bff2 --- /dev/null +++ b/speed-bench/Dockerfile @@ -0,0 +1,9 @@ +# see docs/operations.md#speed-bench +FROM python:3.12-slim +ARG LLAMA_CPP_REF=0ec4027a2a0183de87c007069adddc06e665f7de +ADD https://raw.githubusercontent.com/ggml-org/llama.cpp/${LLAMA_CPP_REF}/tools/server/bench/speed-bench/requirements.txt \ + https://raw.githubusercontent.com/ggml-org/llama.cpp/${LLAMA_CPP_REF}/tools/server/bench/speed-bench/speed_bench.py \ + https://raw.githubusercontent.com/ggml-org/llama.cpp/${LLAMA_CPP_REF}/tools/server/bench/speed-bench/speed_bench_compare.py \ + /app/ +RUN pip install --no-cache-dir -r /app/requirements.txt +WORKDIR /results