haylanandClaude Bot
5b866b93ff
The downloader's test -f guard never picks up an upstream re-upload of an existing filename. The new script moves the old file aside, re-runs the compose downloader, restores on failure, and restarts llama-server. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
74 lines
6.7 KiB
Markdown
74 lines
6.7 KiB
Markdown
# Operations
|
|
|
|
> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`scripts/redownload-model.sh`](../scripts/redownload-model.sh), [`.env.example`](../.env.example), [`models.yml`](../models.yml), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile) and [`scripts/bench.sh`](../scripts/bench.sh). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins.
|
|
|
|
The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed.
|
|
|
|
## Self-update
|
|
|
|
`git pull` runs before anything else. A script that updates itself isn't atomic against its own file changing mid-run. Pulling later once executed a mix of old and new script and compose content in a single pass. Keep the pull first.
|
|
|
|
## Env sync
|
|
|
|
`.env` is created from `.env.example` if it's missing. After that:
|
|
|
|
- **Keys with a real default in `.env.example`** are tracked config, synced every run. A key missing from `.env` is added. A key whose value matches is left alone. A key whose value **differs** is a conflict:
|
|
- Interactively, all conflicts appear on one screen ([gum](#gum)). Selected keys take the new value, and unselected keys keep the server's value.
|
|
- Without a TTY (cron, CI, a pipe), any conflict is an error unless `--force` is passed, which accepts every new value.
|
|
|
|
To change a default for good, change `.env.example`, not the server's `.env`.
|
|
- **Keys left blank in `.env.example`** ([secrets](#secrets) and [host-resolved values](#host-resolved-values)) are never synced. They're only filled when blank.
|
|
- Keys removed from `.env.example` stay in the server's `.env`: the script adds keys and never prunes them. They're harmless, and you can delete them by hand.
|
|
|
|
There's no rollback. If a bad config lands, `git revert` and re-run the script.
|
|
|
|
## Secrets
|
|
|
|
Random secrets are generated with `openssl` when blank and never overwritten after that. **`OMNIROUTE_STORAGE_ENCRYPTION_KEY` must never change after the first run**: it encrypts OmniRoute's whole database, and changing it makes the existing data unreadable. `OMNIROUTE_WS_BRIDGE_SECRET` is required by OmniRoute in production (its `docs/reference/ENVIRONMENT.md`). The per-workload API keys aren't here at all: see [gateway → API keys](gateway.md#api-keys).
|
|
|
|
## Host-resolved values
|
|
|
|
These are resolved on the server itself. Set them by hand only if resolution fails:
|
|
|
|
- `SEARXNG_LAN_IP`, from `search.home`: see [gateway → Web search](gateway.md#web-search).
|
|
- `HOST_VIDEO_GID` / `HOST_RENDER_GID`, from the host's `video` / `render` groups: see [architecture → GPU access](architecture.md#gpu-access).
|
|
|
|
## Gum
|
|
|
|
The conflict screen uses [gum](https://github.com/charmbracelet/gum), a single static binary. It's vendored as a release tarball under `scripts/vendor/`, because the server has no outbound internet access. Other architectures fall back to downloading it, and if that fails, to a plain-bash prompt. The extracted binary is cached under `.cache/gum/` (gitignored).
|
|
|
|
## Model downloads
|
|
|
|
One `downloader` service in the `tools` compose profile fetches every file listed in [`models.yml`](../models.yml) (`file` + `url` per entry). `update.sh` runs it every time. Files that already exist are skipped, so re-runs don't re-download. To add a model, add an entry there; to load it, point the matching `LLAMA_*_FILE` key in `.env` at the same `file` name. The two are not linked, so a typo means llama-server can't find the file.
|
|
|
|
- **Partial downloads** go to `<file>.part` and are only renamed once complete. Otherwise an interrupted download would leave a truncated file that the skip check treats as done.
|
|
- **Files live in the named `models` volume**, not a host bind-mount. The downloader runs as root because the named volume is created root-owned.
|
|
- **It needs internet on each run**: it installs `curl` and `yq` from Alpine's package mirror before reading the list.
|
|
|
|
### Forcing a model re-download
|
|
|
|
The skip check means `update.sh` never picks up an upstream change to a file that's already on disk (for example, a re-uploaded GGUF with the same name). [`scripts/redownload-model.sh`](../scripts/redownload-model.sh) forces it: with no argument it refreshes `LLAMA_MODEL_FILE` from `.env`, or pass a `file` name from `models.yml` (for example the drafter). It reuses the compose downloader, so the URL still lives only in `models.yml`.
|
|
|
|
- The old file is renamed to `<file>.old` and deleted only after the new download succeeds. If the download fails or is interrupted, the old file is put back. This needs room for both copies in the `models` volume at once.
|
|
- It ends by restarting `llama-server`, because the running process keeps the old file mapped until it restarts.
|
|
|
|
## Speed bench
|
|
|
|
`speed-bench` runs llama.cpp's [SPEED-Bench client](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/bench/speed-bench/README.md) against the running `llama-server`. It measures decode speed (`avg_pred_t/s`), latency and draft acceptance per category. Use it to compare drafters and their flags with numbers instead of gut feel. It's in the `tools` profile, so `update.sh` never builds or runs it. The first run builds the image.
|
|
|
|
- **The scripts are pinned** to one llama.cpp commit (`LLAMA_CPP_REF` in the Dockerfile). The server image is not pinned, so bump the ref if the client stops matching the server's response format.
|
|
- **It reaches the backend over the compose network**, so llama-server needs no host port. The dataset (`nvidia/SPEED-Bench`, not gated) downloads on first run and is cached in the `hf-cache` volume. Results land in `bench-results/` (gitignored).
|
|
- **It measures speed, not correctness.** The output text isn't saved, so it won't catch a drafter that corrupts tokens.
|
|
|
|
Run it on the server with `./scripts/bench.sh`. It wraps the compose calls and creates `bench-results/` first, so Docker doesn't create it root-owned. Extra arguments go to the SPEED-Bench client. Pass `--concurrency` equal to `LLAMA_PARALLEL` for throughput, or leave it at 1 for the single-session speed one CLI sees:
|
|
|
|
```sh
|
|
./scripts/bench.sh run spec --category coding --limit 8
|
|
```
|
|
|
|
To get a speedup figure, also run a **baseline** without speculative decoding. Temporarily remove the `--spec-*` and `-md` lines from the llama-server command, run `docker compose up -d llama-server`, then run `./scripts/bench.sh run baseline` with the same arguments. Keep `--bench`, `--category`, `--osl` and `--limit` identical. Restore the lines with `git checkout docker-compose.yml` and `docker compose up -d llama-server`, then compare:
|
|
|
|
```sh
|
|
./scripts/bench.sh compare baseline spec
|
|
```
|