feat(llm): switch to DFlash2 speculative decoding, drop qwen-classifier
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1e3ad169c0
commit
17747694bd
10 files changed
+30
-88
No files matched your search
+4
-4
@@ -17,7 +17,7 @@ _Avoid_: using it as a synonym for Server
|
||||
### Serving models
|
||||
|
||||
**Backend**:
|
||||
A model server process that serves one model to the gateway (`llama-server`, `qwen-classifier`).
|
||||
A model server process that serves one model to the gateway (`llama-server`).
|
||||
_Avoid_: provider, model server
|
||||
|
||||
**Slot**:
|
||||
@@ -28,9 +28,9 @@ _Avoid_: parallel request, lane
|
||||
The large model that does the actual work for workloads.
|
||||
_Avoid_: chat model
|
||||
|
||||
**Classifier model**:
|
||||
The small model that judges a coding CLI's tool calls before they run.
|
||||
_Avoid_: fast model (`fastModel` only when quoting qwen-code's config key)
|
||||
**Draft model**:
|
||||
The small speculative-decoding drafter that runs beside the main model inside `llama-server` (DFlash2).
|
||||
_Avoid_: classifier model (the separate classifier backend was removed)
|
||||
|
||||
### Reaching the models
|
||||
|
||||
|
||||
Reference in new issue
Block a user