From 814428694629430684ec575359f36dd27098ddfb Mon Sep 17 00:00:00 2001 From: Haylan Date: Sat, 31 Oct 2026 16:48:58 +0100 Subject: [PATCH] refactor(models): merge downloaders into one, driven by models.yml Replace the per-model downloader services with a single downloader that reads file/url pairs from models.yml. Downloads now go to a .part file first, so an interrupted run can't leave a truncated model that later runs skip. Co-Authored-By: Claude Opus 5.5 --- docker-compose.yml | 34 ++++++++++++---------------------- docs/architecture.md | 2 +- docs/operations.md | 8 ++++++-- models.yml | 7 +++++++ scripts/update.sh | 1 - 5 files changed, 26 insertions(+), 26 deletions(-) create mode 100644 models.yml diff --git a/docker-compose.yml b/docker-compose.yml index d82158b..8dd3ea4 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -38,34 +38,24 @@ services: restart: unless-stopped networks: [ai-stack] - # see docs/operations.md#model-downloads (also covers downloader-classifier) + # see docs/operations.md#model-downloads; the list of files is models.yml downloader: - image: curlimages/curl:latest + image: alpine:3 profiles: ["tools"] user: root volumes: - models:/models - entrypoint: ["sh", "-c"] + - ./models.yml:/models.yml:ro + entrypoint: ["sh", "-euo", "pipefail", "-c"] command: - - > - test -f /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} && - echo "already downloaded, skipping" || - curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} - https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} - - downloader-draft: - image: curlimages/curl:latest - profiles: ["tools"] - user: root - volumes: - - models:/models - entrypoint: ["sh", "-c"] - command: - - > - test -f /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} && - echo "already downloaded, skipping" || - curl -L --fail --create-dirs -o /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} - https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} + - | + apk add --no-cache -q curl yq-go + yq -r '.models[] | (.file // "") + " " + (.url // "")' /models.yml | while read -r file url; do + if [ -z "$$url" ]; then echo "models.yml: entry '$$file' needs both file and url" >&2; exit 1; fi + if [ -f "/models/$$file" ]; then echo "$$file: already downloaded, skipping"; continue; fi + curl -L --fail -o "/models/$$file.part" "$$url" + mv "/models/$$file.part" "/models/$$file" + done # see docs/operations.md#speed-bench speed-bench: diff --git a/docs/architecture.md b/docs/architecture.md index ae21b99..5f90139 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -28,7 +28,7 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it, - **`LLAMA_CACHE_RAM`** (`--cache-ram`, MiB) is the host-RAM budget for saved prompt-cache entries. When a slot is reused, llama.cpp parks the old prompt's state in RAM so it can be restored instead of re-prefilled. The llama.cpp default is too small for ~40k-token agent prompts (entries are ~2 GiB each), so entries got evicted and requests re-prefilled in full. Raising it is the only place spare DDR RAM helps; it does not speed up decode. Keep it well below the host's free RAM. Evidence and the A/B method: [`research/qwen3.8-27b-throughput-tuning.md`](research/qwen3.8-27b-throughput-tuning.md). - **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright. - **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md). -- **Speculative decoding (DFlash2):** `--spec-type draft-dflash` with a separate drafter GGUF (`LLAMA_DRAFT_MODEL_FILE`), fetched by `downloader-draft`. It replaced MTP (draft heads inside the target GGUF). `LLAMA_SPEC_N_MAX` is the draft length. Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low. +- **Speculative decoding (DFlash2):** `--spec-type draft-dflash` with a separate drafter GGUF (`LLAMA_DRAFT_MODEL_FILE`), fetched by the downloader from [`models.yml`](../models.yml). It replaced MTP (draft heads inside the target GGUF). `LLAMA_SPEC_N_MAX` is the draft length. Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low. - **No classifier backend:** the separate `qwen-classifier` was removed to free VRAM for the drafter. Anything that needs a small "fast" model must use the main model. - **No host port**: see [network access → Backends](network-access.md#backends). diff --git a/docs/operations.md b/docs/operations.md index 759a448..7e216ee 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,6 +1,6 @@ # Operations -> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile) and [`scripts/bench.sh`](../scripts/bench.sh). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins. +> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh), [`.env.example`](../.env.example), [`models.yml`](../models.yml), and the `speed-bench` service in [`docker-compose.yml`](../docker-compose.yml) with its [`speed-bench/Dockerfile`](../speed-bench/Dockerfile) and [`scripts/bench.sh`](../scripts/bench.sh). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins. The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed. @@ -39,7 +39,11 @@ The conflict screen uses [gum](https://github.com/charmbracelet/gum), a single s ## Model downloads -Each model file has its own one-off downloader service in the `tools` compose profile. `update.sh` runs them every time. A `test -f` guard skips files that already exist, so re-runs don't re-download. Files live in the named `models` volume, not a host bind-mount. The downloaders run as root because the named volume is created root-owned. There's one service per model, so the small model's job never re-checks the large one. +One `downloader` service in the `tools` compose profile fetches every file listed in [`models.yml`](../models.yml) (`file` + `url` per entry). `update.sh` runs it every time. Files that already exist are skipped, so re-runs don't re-download. To add a model, add an entry there; to load it, point the matching `LLAMA_*_FILE` key in `.env` at the same `file` name. The two are not linked, so a typo means llama-server can't find the file. + +- **Partial downloads** go to `.part` and are only renamed once complete. Otherwise an interrupted download would leave a truncated file that the skip check treats as done. +- **Files live in the named `models` volume**, not a host bind-mount. The downloader runs as root because the named volume is created root-owned. +- **It needs internet on each run**: it installs `curl` and `yq` from Alpine's package mirror before reading the list. ## Speed bench diff --git a/models.yml b/models.yml new file mode 100644 index 0000000..c4df725 --- /dev/null +++ b/models.yml @@ -0,0 +1,7 @@ +# Model files the downloader fetches into the models volume; see docs/operations.md#model-downloads +# `file` must match the LLAMA_*_FILE value in .env that loads it. +models: + - file: Qwen3.8-27B-UD-Q4_K_XL.gguf + url: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-UD-Q4_K_XL.gguf + - file: Qwen3.8-27B-DFlash2-Q4_K_M.gguf + url: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/Qwen3.8-27B-DFlash2-Q4_K_M.gguf diff --git a/scripts/update.sh b/scripts/update.sh index 5fdd374..3b14191 100755 --- a/scripts/update.sh +++ b/scripts/update.sh @@ -175,7 +175,6 @@ docker compose build --pull echo "==> ensuring models are downloaded (skips already-present files)" docker compose --profile tools run --rm downloader -docker compose --profile tools run --rm downloader-draft echo "==> bringing up omniroute" docker compose up -d --wait omniroute