LLM-Server/docs/operations.md

Operations

Source of truth: scripts/update.sh, .env.example, and the speed-bench service in docker-compose.yml with its speed-bench/Dockerfile and scripts/bench.sh. This page explains what the script does and why; the steps and defaults live in the source. If they disagree, the source wins.

The stack runs on the server. After any change to the repo, run ./scripts/update.sh there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed.

Self-update

git pull runs before anything else. A script that updates itself isn't atomic against its own file changing mid-run. Pulling later once executed a mix of old and new script and compose content in a single pass. Keep the pull first.

Env sync

.env is created from .env.example if it's missing. After that:

• Keys with a real default in .env.example are tracked config, synced every run. A key missing from .env is added. A key whose value matches is left alone. A key whose value differs is a conflict:

  • Interactively, all conflicts appear on one screen (gum). Selected keys take the new value, and unselected keys keep the server's value.
  • Without a TTY (cron, CI, a pipe), any conflict is an error unless --force is passed, which accepts every new value.

  To change a default for good, change .env.example, not the server's .env.

• Keys left blank in .env.example (secrets and host-resolved values) are never synced. They're only filled when blank.

• Keys removed from .env.example stay in the server's .env: the script adds keys and never prunes them. They're harmless, and you can delete them by hand.

There's no rollback. If a bad config lands, git revert and re-run the script.

Secrets

Random secrets are generated with openssl when blank and never overwritten after that. OMNIROUTE_STORAGE_ENCRYPTION_KEY must never change after the first run: it encrypts OmniRoute's whole database, and changing it makes the existing data unreadable. OMNIROUTE_WS_BRIDGE_SECRET is required by OmniRoute in production (its docs/reference/ENVIRONMENT.md). The per-workload API keys aren't here at all: see gateway → API keys.

Host-resolved values

These are resolved on the server itself. Set them by hand only if resolution fails:

• SEARXNG_LAN_IP, from search.home: see gateway → Web search.
• HOST_VIDEO_GID / HOST_RENDER_GID, from the host's video / render groups: see architecture → GPU access.

Gum

The conflict screen uses gum, a single static binary. It's vendored as a release tarball under scripts/vendor/, because the server has no outbound internet access. Other architectures fall back to downloading it, and if that fails, to a plain-bash prompt. The extracted binary is cached under .cache/gum/ (gitignored).

Model downloads

Each model file has its own one-off downloader service in the tools compose profile. update.sh runs them every time. A test -f guard skips files that already exist, so re-runs don't re-download. Files live in the named models volume, not a host bind-mount. The downloaders run as root because the named volume is created root-owned. There's one service per model, so the small model's job never re-checks the large one.

Speed bench

speed-bench runs llama.cpp's SPEED-Bench client against the running llama-server. It measures decode speed (avg_pred_t/s), latency and draft acceptance per category. Use it to compare drafters and their flags with numbers instead of gut feel. It's in the tools profile, so update.sh never builds or runs it. The first run builds the image.

• The scripts are pinned to one llama.cpp commit (LLAMA_CPP_REF in the Dockerfile). The server image is not pinned, so bump the ref if the client stops matching the server's response format.
• It reaches the backend over the compose network, so llama-server needs no host port. The dataset (nvidia/SPEED-Bench, not gated) downloads on first run and is cached in the hf-cache volume. Results land in bench-results/ (gitignored).
• It measures speed, not correctness. The output text isn't saved, so it won't catch a drafter that corrupts tokens.

Run it on the server with ./scripts/bench.sh. It wraps the compose calls and creates bench-results/ first, so Docker doesn't create it root-owned. Extra arguments go to the SPEED-Bench client. Pass --concurrency equal to LLAMA_PARALLEL for throughput, or leave it at 1 for the single-session speed one CLI sees:

./scripts/bench.sh run spec --category coding --limit 8

To get a speedup figure, also run a baseline without speculative decoding. Temporarily remove the --spec-* and -md lines from the llama-server command, run docker compose up -d llama-server, then run ./scripts/bench.sh run baseline with the same arguments. Keep --bench, --category, --osl and --limit identical. Restore the lines with git checkout docker-compose.yml and docker compose up -d llama-server, then compare:

./scripts/bench.sh compare baseline spec