haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm): switch to DFlash2 speculative decoding, drop qwen-classifier

Browse Source 
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF.
Remove the qwen-classifier backend to free VRAM (GPU was at 100%).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-10-05 21:28:55 +02:00
1 parent 1e3ad169c0
commit 17747694bd
10 files changed
+30 -88
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+1 -1
View File
@@ -2,7 +2,7 @@
> **Source of truth:** [`docker-compose.yml`](docker-compose.yml) (what runs) and [`scripts/update.sh`](scripts/update.sh) (how it's deployed). The docs explain *why*; if they disagree with the source, the source wins.
A self-hosted inference stack on one AMD GPU server. Two llama.cpp **backends**, the main model and the classifier model, sit behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches them through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
A self-hosted inference stack on one AMD GPU server. A llama.cpp **backend** serving the main model sits behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches it through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
## Quickstart