refactor(models): merge downloaders into one, driven by models.yml
Replace the per-model downloader services with a single downloader that reads file/url pairs from models.yml. Downloads now go to a .part file first, so an interrupted run can't leave a truncated model that later runs skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
28c102e0d9
commit
8144286946
5 files changed
+26
-26
No files matched your search
@@ -28,7 +28,7 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it,
|
||||
- **`LLAMA_CACHE_RAM`** (`--cache-ram`, MiB) is the host-RAM budget for saved prompt-cache entries. When a slot is reused, llama.cpp parks the old prompt's state in RAM so it can be restored instead of re-prefilled. The llama.cpp default is too small for ~40k-token agent prompts (entries are ~2 GiB each), so entries got evicted and requests re-prefilled in full. Raising it is the only place spare DDR RAM helps; it does not speed up decode. Keep it well below the host's free RAM. Evidence and the A/B method: [`research/qwen3.8-27b-throughput-tuning.md`](research/qwen3.8-27b-throughput-tuning.md).
|
||||
- **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
|
||||
- **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).
|
||||
- **Speculative decoding (DFlash2):** `--spec-type draft-dflash` with a separate drafter GGUF (`LLAMA_DRAFT_MODEL_FILE`), fetched by `downloader-draft`. It replaced MTP (draft heads inside the target GGUF). `LLAMA_SPEC_N_MAX` is the draft length. Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low.
|
||||
- **Speculative decoding (DFlash2):** `--spec-type draft-dflash` with a separate drafter GGUF (`LLAMA_DRAFT_MODEL_FILE`), fetched by the downloader from [`models.yml`](../models.yml). It replaced MTP (draft heads inside the target GGUF). `LLAMA_SPEC_N_MAX` is the draft length. Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low.
|
||||
- **No classifier backend:** the separate `qwen-classifier` was removed to free VRAM for the drafter. Anything that needs a small "fast" model must use the main model.
|
||||
- **No host port**: see [network access → Backends](network-access.md#backends).
|
||||
|
||||
|
||||
Reference in new issue
Block a user