feat(llm): add --spec-draft-p-min 0.5 to DFlash2 spec args
DFlash2 truncates its draft when selector confidence drops below p_min, cutting wasted verify work on long, low-acceptance thinking output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6ce527741b
commit
51684fb7a4
2 files changed
+2
-2
No files matched your search
+1
-1
@@ -9,7 +9,7 @@ LLAMA_CACHE_RAM=0
|
|||||||
LLAMA_KV_CACHE_TYPE=q8_0
|
LLAMA_KV_CACHE_TYPE=q8_0
|
||||||
|
|
||||||
# Blank = speculative decoding off; see docs/architecture.md#llama-server
|
# Blank = speculative decoding off; see docs/architecture.md#llama-server
|
||||||
LLAMA_SPEC_ARGS=--spec-type draft-dflash -md /models/Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-draft-n-max 3
|
LLAMA_SPEC_ARGS=--spec-type draft-dflash -md /models/Qwen3.8-27B-DFlash2-Q8_0.gguf --spec-draft-n-max 3 --spec-draft-p-min 0.5
|
||||||
|
|
||||||
# --- SearXNG web search --- see docs/gateway.md#web-search
|
# --- SearXNG web search --- see docs/gateway.md#web-search
|
||||||
# Auto-filled by update.sh — leave blank.
|
# Auto-filled by update.sh — leave blank.
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it,
|
|||||||
- **`LLAMA_CACHE_RAM`** (`--cache-ram`, MiB) is the host-RAM budget for saved prompt-cache entries. When a slot is reused, llama.cpp parks the old prompt's state in RAM so it can be restored instead of re-prefilled. The llama.cpp default is too small for ~40k-token agent prompts (entries are ~2 GiB each), so entries got evicted and requests re-prefilled in full. Raising it is the only place spare DDR RAM helps; it does not speed up decode. Keep it well below the host's free RAM. Evidence and the A/B method: [`research/qwen3.8-27b-throughput-tuning.md`](research/qwen3.8-27b-throughput-tuning.md).
|
- **`LLAMA_CACHE_RAM`** (`--cache-ram`, MiB) is the host-RAM budget for saved prompt-cache entries. When a slot is reused, llama.cpp parks the old prompt's state in RAM so it can be restored instead of re-prefilled. The llama.cpp default is too small for ~40k-token agent prompts (entries are ~2 GiB each), so entries got evicted and requests re-prefilled in full. Raising it is the only place spare DDR RAM helps; it does not speed up decode. Keep it well below the host's free RAM. Evidence and the A/B method: [`research/qwen3.8-27b-throughput-tuning.md`](research/qwen3.8-27b-throughput-tuning.md).
|
||||||
- **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
|
- **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
|
||||||
- **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).
|
- **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).
|
||||||
- **Speculative decoding (DFlash2):** all its flags live in one key, `LLAMA_SPEC_ARGS`: `--spec-type draft-dflash`, the drafter GGUF (`-md`, fetched by the downloader from [`models.yml`](../models.yml)) and the draft length (`--spec-draft-n-max`). One key means a single `.env` edit swaps the drafter or turns speculation off (leave it blank) for an A/B test. A blank server value differs from `.env.example`, so `update.sh` will ask about it on every run until it's restored ([operations → Env sync](operations.md#env-sync)). It replaced MTP (draft heads inside the target GGUF). Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low.
|
- **Speculative decoding (DFlash2):** all its flags live in one key, `LLAMA_SPEC_ARGS`: `--spec-type draft-dflash`, the drafter GGUF (`-md`, fetched by the downloader from [`models.yml`](../models.yml)) the draft length (`--spec-draft-n-max`), and `--spec-draft-p-min`, which ends a draft early once DFlash2's own confidence in the next token drops below the threshold. Long thinking output at the recommended sampling temperature gets low acceptance, so short confident drafts waste less verify work than long ones ([`common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), DFlash2 lattice walk). One key means a single `.env` edit swaps the drafter or turns speculation off (leave it blank) for an A/B test. A blank server value differs from `.env.example`, so `update.sh` will ask about it on every run until it's restored ([operations → Env sync](operations.md#env-sync)). It replaced MTP (draft heads inside the target GGUF). Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low.
|
||||||
- **No classifier backend:** the separate `qwen-classifier` was removed to free VRAM for the drafter. Anything that needs a small "fast" model must use the main model.
|
- **No classifier backend:** the separate `qwen-classifier` was removed to free VRAM for the drafter. Anything that needs a small "fast" model must use the main model.
|
||||||
- **No host port**: see [network access → Backends](network-access.md#backends).
|
- **No host port**: see [network access → Backends](network-access.md#backends).
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user