diff --git a/docs/operations.md b/docs/operations.md index 97a3576..759a448 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,6 +1,6 @@ # Operations -> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins. +> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile) and [`scripts/bench.sh`](../scripts/bench.sh). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins. The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed. @@ -49,14 +49,14 @@ Each model file has its own one-off downloader service in the `tools` compose pr - **It reaches the backend over the compose network**, so llama-server needs no host port. The dataset (`nvidia/SPEED-Bench`, not gated) downloads on first run and is cached in the `hf-cache` volume. Results land in `bench-results/` (gitignored). - **It measures speed, not correctness.** The output text isn't saved, so it won't catch a drafter that corrupts tokens. -Run it on the server. Pass `--concurrency` equal to `LLAMA_PARALLEL` for throughput, or leave it at 1 for the single-session speed one CLI sees: +Run it on the server with `./scripts/bench.sh`. It wraps the compose calls and creates `bench-results/` first, so Docker doesn't create it root-owned. Extra arguments go to the SPEED-Bench client. Pass `--concurrency` equal to `LLAMA_PARALLEL` for throughput, or leave it at 1 for the single-session speed one CLI sees: ```sh -docker compose --profile tools run --rm speed-bench --category coding --limit 8 --output spec.json +./scripts/bench.sh run spec --category coding --limit 8 ``` -To get a speedup figure, also run a **baseline** without speculative decoding. Temporarily remove the `--spec-*` and `-md` lines from the llama-server command, run `docker compose up -d llama-server`, then run the same bench with `--output baseline.json`. Keep `--bench`, `--category`, `--osl` and `--limit` identical. Restore the lines with `git checkout docker-compose.yml`, then compare: +To get a speedup figure, also run a **baseline** without speculative decoding. Temporarily remove the `--spec-*` and `-md` lines from the llama-server command, run `docker compose up -d llama-server`, then run `./scripts/bench.sh run baseline` with the same arguments. Keep `--bench`, `--category`, `--osl` and `--limit` identical. Restore the lines with `git checkout docker-compose.yml` and `docker compose up -d llama-server`, then compare: ```sh -docker compose --profile tools run --rm --entrypoint python speed-bench /app/speed_bench_compare.py --baseline baseline.json --speculative spec.json +./scripts/bench.sh compare baseline spec ``` diff --git a/scripts/bench.sh b/scripts/bench.sh new file mode 100755 index 0000000..e54554f --- /dev/null +++ b/scripts/bench.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Benchmarks llama-server with SPEED-Bench; see docs/operations.md#speed-bench +set -euo pipefail +cd "$(dirname "$0")/.." + +usage() { + echo "Usage: $0 run [speed_bench args...] # writes bench-results/.json" >&2 + echo " $0 compare # names from earlier runs" >&2 + exit 1 +} + +case "${1:-}" in + run) + [ $# -ge 2 ] || usage + name=$2; shift 2 + mkdir -p bench-results + docker compose --profile tools run --rm speed-bench --output "$name.json" "$@" + ;; + compare) + [ $# -eq 3 ] || usage + docker compose --profile tools run --rm --entrypoint python speed-bench \ + /app/speed_bench_compare.py --baseline "$2.json" --speculative "$3.json" + ;; + *) usage ;; +esac