feat(llm): switch to DFlash2 speculative decoding, drop qwen-classifier
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed
1 parent
1e3ad169c0
commit
17747694bd
10 files changed
+30
-88
No files matched your search
@@ -24,6 +24,6 @@ Terms as defined in [`CONTEXT.md`](../../CONTEXT.md). Every coding CLI is a work
|
||||
| [Claude Code](claude-code.md) | Anthropic Messages | `ANTHROPIC_BASE_URL` env var |
|
||||
| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `config.toml` provider block |
|
||||
| [OpenCode](opencode.md) | OpenAI Chat Completions | `opencode.json` provider block |
|
||||
| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model + classifier model) | `~/.qwen/settings.json` `modelProviders.openai` |
|
||||
| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model) | `~/.qwen/settings.json` `modelProviders.openai` |
|
||||
|
||||
If requests fail with a semaphore timeout under concurrent load, see [gateway → Timeouts](../gateway.md#timeouts).
|
||||
@@ -4,11 +4,11 @@
|
||||
|
||||
> **Source of truth:** [Qwen Code's repo](https://github.com/QwenLM/qwen-code) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win.
|
||||
|
||||
Qwen Code speaks plain **OpenAI Chat Completions**. Unlike the other CLIs, it needs *two* models: the main model, and the classifier model as its `fastModel` for Auto Mode's action classifier. Each has its own provider connection in the gateway, and both are reached through the same gateway URL.
|
||||
Qwen Code speaks plain **OpenAI Chat Completions**. It uses the main model only. There is no separate classifier backend any more ([architecture → llama-server](../architecture.md#llama-server)).
|
||||
|
||||
## Why a second model exists
|
||||
## Auto Mode's classifier (`fastModel`)
|
||||
|
||||
Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call: it decides whether to auto-approve or block a shell command or tool call before it runs. It runs on its own backend, `qwen-classifier`. Why that backend exists and how it's sized: [architecture → qwen-classifier](../architecture.md#qwen-classifier).
|
||||
Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call. It calls `fastModel`. With no classifier backend, either point `fastModel` at the main model (it then queues behind the main backend's slots, the reason the separate backend once existed), or leave `fastModel` unset and don't use Auto Mode.
|
||||
|
||||
## `~/.qwen/settings.json`
|
||||
|
||||
@@ -22,16 +22,6 @@ Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gat
|
||||
"envKey": "OMNIROUTE_API_KEY",
|
||||
"baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1",
|
||||
"generationConfig": { "contextWindowSize": <LLAMA_CTX_SIZE / LLAMA_PARALLEL> }
|
||||
},
|
||||
{
|
||||
"id": "<classifier-provider-id-in-omniroute>",
|
||||
"name": "qwen3-4b-classifier",
|
||||
"envKey": "OMNIROUTE_API_KEY",
|
||||
"baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1",
|
||||
"generationConfig": {
|
||||
"contextWindowSize": <qwen-classifier --ctx-size>,
|
||||
"extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
@@ -40,14 +30,12 @@ Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gat
|
||||
"name": "<main-model-provider-id-in-omniroute>",
|
||||
"baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1"
|
||||
},
|
||||
"fastModel": "<classifier-provider-id-in-omniroute>"
|
||||
"fastModel": "<main-model-provider-id-in-omniroute>"
|
||||
}
|
||||
```
|
||||
|
||||
- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=<qwen-code-cli API key>` before launching. Both providers can share one API key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
|
||||
- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=<qwen-code-cli API key>` before launching.
|
||||
- **The main model's `contextWindowSize` is the per-slot context**, not `LLAMA_CTX_SIZE` itself ([overview](index.md)).
|
||||
- **The classifier's `contextWindowSize` is its whole `--ctx-size`.** It has a single slot, so there's no per-slot division ([architecture → qwen-classifier](../architecture.md#qwen-classifier)).
|
||||
- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see [`research/fast-model-choice.md`](../research/fast-model-choice.md) §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
|
||||
- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
|
||||
|
||||
## Web search via OmniRoute
|
||||
@@ -107,7 +95,7 @@ It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above —
|
||||
|
||||
## Auto Mode tuning
|
||||
|
||||
Auto Mode's action classifier calls the classifier model above. Even on its own GPU-resident backend, give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call:
|
||||
Auto Mode's action classifier calls `fastModel`, now the main model, so it can queue behind a heavy generation. Give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -122,6 +110,6 @@ Auto Mode's action classifier calls the classifier model above. Even on its own
|
||||
|
||||
`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each.
|
||||
|
||||
Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see [gateway → Dashboard](../gateway.md#dashboard)). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.
|
||||
The main model's provider connection already carries the long per-connection timeout (`providerSpecificData.timeoutMs`, see [gateway → Timeouts](../gateway.md#timeouts)), which now covers classifier calls too.
|
||||
|
||||
Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.
|
||||
Reference in new issue
Block a user