63 lines
5.4 KiB
Markdown
63 lines
5.4 KiB
Markdown
# Operations
|
|
|
|
> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile) and [`scripts/bench.sh`](../scripts/bench.sh). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins.
|
|
|
|
The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed.
|
|
|
|
## Self-update
|
|
|
|
`git pull` runs before anything else. A script that updates itself isn't atomic against its own file changing mid-run. Pulling later once executed a mix of old and new script and compose content in a single pass. Keep the pull first.
|
|
|
|
## Env sync
|
|
|
|
`.env` is created from `.env.example` if it's missing. After that:
|
|
|
|
- **Keys with a real default in `.env.example`** are tracked config, synced every run. A key missing from `.env` is added. A key whose value matches is left alone. A key whose value **differs** is a conflict:
|
|
- Interactively, all conflicts appear on one screen ([gum](#gum)). Selected keys take the new value, and unselected keys keep the server's value.
|
|
- Without a TTY (cron, CI, a pipe), any conflict is an error unless `--force` is passed, which accepts every new value.
|
|
|
|
To change a default for good, change `.env.example`, not the server's `.env`.
|
|
- **Keys left blank in `.env.example`** ([secrets](#secrets) and [host-resolved values](#host-resolved-values)) are never synced. They're only filled when blank.
|
|
- Keys removed from `.env.example` stay in the server's `.env`: the script adds keys and never prunes them. They're harmless, and you can delete them by hand.
|
|
|
|
There's no rollback. If a bad config lands, `git revert` and re-run the script.
|
|
|
|
## Secrets
|
|
|
|
Random secrets are generated with `openssl` when blank and never overwritten after that. **`OMNIROUTE_STORAGE_ENCRYPTION_KEY` must never change after the first run**: it encrypts OmniRoute's whole database, and changing it makes the existing data unreadable. `OMNIROUTE_WS_BRIDGE_SECRET` is required by OmniRoute in production (its `docs/reference/ENVIRONMENT.md`). The per-workload API keys aren't here at all: see [gateway → API keys](gateway.md#api-keys).
|
|
|
|
## Host-resolved values
|
|
|
|
These are resolved on the server itself. Set them by hand only if resolution fails:
|
|
|
|
- `SEARXNG_LAN_IP`, from `search.home`: see [gateway → Web search](gateway.md#web-search).
|
|
- `HOST_VIDEO_GID` / `HOST_RENDER_GID`, from the host's `video` / `render` groups: see [architecture → GPU access](architecture.md#gpu-access).
|
|
|
|
## Gum
|
|
|
|
The conflict screen uses [gum](https://github.com/charmbracelet/gum), a single static binary. It's vendored as a release tarball under `scripts/vendor/`, because the server has no outbound internet access. Other architectures fall back to downloading it, and if that fails, to a plain-bash prompt. The extracted binary is cached under `.cache/gum/` (gitignored).
|
|
|
|
## Model downloads
|
|
|
|
Each model file has its own one-off downloader service in the `tools` compose profile. `update.sh` runs them every time. A `test -f` guard skips files that already exist, so re-runs don't re-download. Files live in the named `models` volume, not a host bind-mount. The downloaders run as root because the named volume is created root-owned. There's one service per model, so the small model's job never re-checks the large one.
|
|
|
|
## Speed bench
|
|
|
|
`speed-bench` runs llama.cpp's [SPEED-Bench client](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/bench/speed-bench/README.md) against the running `llama-server`. It measures decode speed (`avg_pred_t/s`), latency and draft acceptance per category. Use it to compare drafters and their flags with numbers instead of gut feel. It's in the `tools` profile, so `update.sh` never builds or runs it. The first run builds the image.
|
|
|
|
- **The scripts are pinned** to one llama.cpp commit (`LLAMA_CPP_REF` in the Dockerfile). The server image is not pinned, so bump the ref if the client stops matching the server's response format.
|
|
- **It reaches the backend over the compose network**, so llama-server needs no host port. The dataset (`nvidia/SPEED-Bench`, not gated) downloads on first run and is cached in the `hf-cache` volume. Results land in `bench-results/` (gitignored).
|
|
- **It measures speed, not correctness.** The output text isn't saved, so it won't catch a drafter that corrupts tokens.
|
|
|
|
Run it on the server with `./scripts/bench.sh`. It wraps the compose calls and creates `bench-results/` first, so Docker doesn't create it root-owned. Extra arguments go to the SPEED-Bench client. Pass `--concurrency` equal to `LLAMA_PARALLEL` for throughput, or leave it at 1 for the single-session speed one CLI sees:
|
|
|
|
```sh
|
|
./scripts/bench.sh run spec --category coding --limit 8
|
|
```
|
|
|
|
To get a speedup figure, also run a **baseline** without speculative decoding. Temporarily remove the `--spec-*` and `-md` lines from the llama-server command, run `docker compose up -d llama-server`, then run `./scripts/bench.sh run baseline` with the same arguments. Keep `--bench`, `--category`, `--osl` and `--limit` identical. Restore the lines with `git checkout docker-compose.yml` and `docker compose up -d llama-server`, then compare:
|
|
|
|
```sh
|
|
./scripts/bench.sh compare baseline spec
|
|
```
|