LLM-Server/docs/operations.md

Operations

Source of truth: scripts/update.sh, scripts/redownload-model.sh, .env.example, models.yml, and the speed-bench service in docker-compose.yml with its speed-bench/Dockerfile and scripts/bench.sh. This page explains what the script does and why; the steps and defaults live in the source. If they disagree, the source wins.

The stack runs on the server. After any change to the repo, run ./scripts/update.sh there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed.

Self-update

git pull runs before anything else. A script that updates itself isn't atomic against its own file changing mid-run. Pulling later once executed a mix of old and new script and compose content in a single pass. Keep the pull first.

Env sync

.env is created from .env.example if it's missing. After that:

• Keys with a real default in .env.example are tracked config, synced every run. A key missing from .env is added. A key whose value matches is left alone. A key whose value differs is a conflict:

  • Interactively, all conflicts appear on one screen (gum). Selected keys take the new value, and unselected keys keep the server's value.
  • Without a TTY (cron, CI, a pipe), any conflict is an error unless --force is passed, which accepts every new value.

  To change a default for good, change .env.example, not the server's .env.

• Keys left blank in .env.example (secrets and host-resolved values) are never synced. They're only filled when blank.

• Keys removed from .env.example stay in the server's .env: the script adds keys and never prunes them. They're harmless, and you can delete them by hand.

There's no rollback. If a bad config lands, git revert and re-run the script.

Secrets

Random secrets are generated with openssl when blank and never overwritten after that. OMNIROUTE_STORAGE_ENCRYPTION_KEY must never change after the first run: it encrypts OmniRoute's whole database, and changing it makes the existing data unreadable. OMNIROUTE_WS_BRIDGE_SECRET is required by OmniRoute in production (its docs/reference/ENVIRONMENT.md). The per-workload API keys aren't here at all: see gateway → API keys.

Host-resolved values

These are resolved on the server itself. Set them by hand only if resolution fails:

• SEARXNG_LAN_IP, from search.home: see gateway → Web search.
• HOST_VIDEO_GID / HOST_RENDER_GID, from the host's video / render groups: see architecture → GPU access.

Gum

The conflict screen uses gum, a single static binary. It's vendored as a release tarball under scripts/vendor/, because the server has no outbound internet access. Other architectures fall back to downloading it, and if that fails, to a plain-bash prompt. The extracted binary is cached under .cache/gum/ (gitignored).

Model downloads

One downloader service in the tools compose profile fetches every file listed in models.yml (file + url per entry). update.sh runs it every time. Files that already exist are skipped, so re-runs don't re-download. To add a model, add an entry there; to load it, point the matching LLAMA_*_FILE key in .env at the same file name. The two are not linked, so a typo means llama-server can't find the file.

• Partial downloads go to <file>.part and are only renamed once complete. Otherwise an interrupted download would leave a truncated file that the skip check treats as done.
• Files live in the named models volume, not a host bind-mount. The downloader runs as root because the named volume is created root-owned.
• It needs internet on each run: it installs curl and yq from Alpine's package mirror before reading the list.

Forcing a model re-download

The skip check means update.sh never picks up an upstream change to a file that's already on disk (for example, a re-uploaded GGUF with the same name). scripts/redownload-model.sh forces it: with no argument it refreshes LLAMA_MODEL_FILE from .env, or pass a file name from models.yml (for example the drafter). It reuses the compose downloader, so the URL still lives only in models.yml.

• The old file is renamed to <file>.old and deleted only after the new download succeeds. If the download fails or is interrupted, the old file is put back. This needs room for both copies in the models volume at once.
• It ends by restarting llama-server, because the running process keeps the old file mapped until it restarts.

Speed bench

speed-bench runs llama.cpp's SPEED-Bench client against the running llama-server. It measures decode speed (avg_pred_t/s), latency and draft acceptance per category. Use it to compare drafters and their flags with numbers instead of gut feel. It's in the tools profile, so update.sh never builds or runs it. The first run builds the image.

• The scripts are pinned to one llama.cpp commit (LLAMA_CPP_REF in the Dockerfile). The server image is not pinned, so bump the ref if the client stops matching the server's response format.
• It reaches the backend over the compose network, so llama-server needs no host port. The dataset (nvidia/SPEED-Bench, not gated) downloads on first run and is cached in the hf-cache volume. Results land in bench-results/ (gitignored).
• It measures speed, not correctness. The output text isn't saved, so it won't catch a drafter that corrupts tokens.

Run it on the server with ./scripts/bench.sh. It wraps the compose calls and creates bench-results/ first, so Docker doesn't create it root-owned. Extra arguments go to the SPEED-Bench client. Pass --concurrency equal to LLAMA_PARALLEL for throughput, or leave it at 1 for the single-session speed one CLI sees:

./scripts/bench.sh run spec --category coding --limit 8

To get a speedup figure, also run a baseline without speculative decoding. Temporarily remove the --spec-* and -md lines from the llama-server command, run docker compose up -d llama-server, then run ./scripts/bench.sh run baseline with the same arguments. Keep --bench, --category, --osl and --limit identical. Restore the lines with git checkout docker-compose.yml and docker compose up -d llama-server, then compare:

./scripts/bench.sh compare baseline spec