feat(llm): switch to DFlash2 speculative decoding, drop qwen-classifier
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1e3ad169c0
commit
17747694bd
10 files changed
+30
-88
No files matched your search
+1
-1
@@ -13,7 +13,7 @@ Browse to `http://<server>:<OMNIROUTE_DASHBOARD_PORT>` from the LAN. The dashboa
|
||||
Each backend is registered once in the dashboard (Providers → add a **llama.cpp** provider) or via `POST /api/providers`. The base URL is `http://<service>:<container port>` on the stack network, using the service name and port from compose. None of this is checked into the repo. These docs assume the main model is registered as `qwen3.8-27b-local`.
|
||||
|
||||
- **`OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` / `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`** have to be on, because backend URLs are private addresses. They relax OmniRoute's SSRF guard, which normally blocks registering a provider at a private or local address. Anyone with dashboard access can therefore point the gateway at any host on the stack network or the LAN.
|
||||
- **Classifier connection:** turn OmniRoute's memory-tool injection off for it. Injected tools break the classifier's output ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)). Also give it a generous per-connection timeout (see [Timeouts](#timeouts)).
|
||||
- **Stale classifier connection:** the `qwen-classifier` backend no longer exists. Delete its provider connection in the dashboard, or repoint it at the main model. If repointed, turn OmniRoute's memory-tool injection off for it ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)).
|
||||
- Background on registering local backends: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md).
|
||||
|
||||
## API keys
|
||||
|
||||
Reference in new issue
Block a user