feat(llm): switch to DFlash2 speculative decoding, drop qwen-classifier

Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF.
Remove the qwen-classifier backend to free VRAM (GPU was at 100%).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

.env.example

@@ -6,8 +6,8 @@ LLAMA_GPU_LAYERS=999
 LLAMA_CTX_SIZE=262144
 LLAMA_PARALLEL=2
 
-# see docs/architecture.md#qwen-classifier
+LLAMA_DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
-LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
+LLAMA_SPEC_N_MAX=7
 
 # --- SearXNG web search --- see docs/gateway.md#web-search
 # Auto-filled by update.sh — leave blank.

CONTEXT.md

@@ -17,7 +17,7 @@ _Avoid_: using it as a synonym for Server
 ### Serving models
 
 **Backend**:
-A model server process that serves one model to the gateway (`llama-server`, `qwen-classifier`).
+A model server process that serves one model to the gateway (`llama-server`).
 _Avoid_: provider, model server
 
 **Slot**:
@@ -28,9 +28,9 @@ _Avoid_: parallel request, lane
 The large model that does the actual work for workloads.
 _Avoid_: chat model
 
-**Classifier model**:
+**Draft model**:
-The small model that judges a coding CLI's tool calls before they run.
+The small speculative-decoding drafter that runs beside the main model inside `llama-server` (DFlash2).
-_Avoid_: fast model (`fastModel` only when quoting qwen-code's config key)
+_Avoid_: classifier model (the separate classifier backend was removed)
 
 ### Reaching the models
 

README.md

@@ -2,7 +2,7 @@
 
 > **Source of truth:** [`docker-compose.yml`](docker-compose.yml) (what runs) and [`scripts/update.sh`](scripts/update.sh) (how it's deployed). The docs explain *why*; if they disagree with the source, the source wins.
 
-A self-hosted inference stack on one AMD GPU server. Two llama.cpp **backends**, the main model and the classifier model, sit behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches them through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
+A self-hosted inference stack on one AMD GPU server. A llama.cpp **backend** serving the main model sits behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches it through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
 
 ## Quickstart
 

docker-compose.yml

@@ -28,8 +28,9 @@ services:
       --cache-type-k q8_0
       --cache-type-v q8_0
       --cache-reuse 256
-      --spec-type draft-mtp 
+      --spec-type draft-dflash
-      --spec-draft-n-max 2
+      -md /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
+      --spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7}
       --jinja
     # see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port)
     expose:
@@ -37,43 +38,6 @@ services:
     restart: unless-stopped
     networks: [ai-stack]
 
-  # see docs/architecture.md#qwen-classifier
-  qwen-classifier:
-    image: ghcr.io/ggml-org/llama.cpp:server-rocm
-    container_name: qwen-classifier
-    devices:
-      - /dev/kfd
-      - /dev/dri
-    group_add:
-      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
-      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
-    security_opt:
-      - seccomp=unconfined
-    ipc: host
-    # see docs/architecture.md#gpu-access
-    environment:
-      - GPU_MAX_HW_QUEUES=1
-    volumes:
-      - models:/models
-    command: >
-      -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
-      --host 0.0.0.0
-      --port 8080
-      --n-gpu-layers 28
-      --ctx-size 50000
-      --parallel 1
-      --batch-size 512
-      --ubatch-size 128
-      --flash-attn on
-      --reasoning off
-      --cache-type-k q4_0
-      --cache-type-v q4_0
-      --jinja
-    expose:
-      - "8080"
-    restart: unless-stopped
-    networks: [ai-stack]
-
   # see docs/operations.md#model-downloads (also covers downloader-classifier)
   downloader:
     image: curlimages/curl:latest
@@ -89,7 +53,7 @@ services:
         curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
         https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
 
-  downloader-classifier:
+  downloader-draft:
     image: curlimages/curl:latest
     profiles: ["tools"]
     user: root
@@ -98,10 +62,10 @@ services:
     entrypoint: ["sh", "-c"]
     command:
       - >
-        test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
+        test -f /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} &&
         echo "already downloaded, skipping" ||
-        curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+        curl -L --fail --create-dirs -o /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
-        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+        https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf}
 
   # see docs/architecture.md#omniroute; runtime config lives in its dashboard, see docs/gateway.md
   omniroute:

docs/architecture.md

@@ -4,14 +4,14 @@
 
 Terms as defined in [`CONTEXT.md`](../CONTEXT.md).
 
-Every workload calls the **gateway** (`omniroute`) with its own API key. The gateway forwards each call over a **provider connection** to one of two **backends** on the one GPU: `llama-server` (main model) or `qwen-classifier` (classifier model). `qdrant` sits beside them on the same Docker network. It isn't a model, so it isn't behind the gateway.
+Every workload calls the **gateway** (`omniroute`) with its own API key. The gateway forwards each call over a **provider connection** to the one **backend** on the GPU: `llama-server` (main model). `qdrant` sits beside them on the same Docker network. It isn't a model, so it isn't behind the gateway.
 
 ## GPU access
 
-Both backends share the one GPU, and both carry the same GPU settings in compose:
+`llama-server` carries these GPU settings in compose:
 
 - **`group_add` uses numeric GIDs, not group names.** Docker resolves a *named* `group_add` entry against the container's own `/etc/group`, not the host's. That fails unpredictably when the image doesn't define the group, and it gets worse when several GPU services start at once. `scripts/update.sh` resolves the host's real GIDs into `HOST_VIDEO_GID` / `HOST_RENDER_GID` (see [operations → Host-resolved values](operations.md#host-resolved-values)).
-- **`GPU_MAX_HW_QUEUES`** works around [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706): the card sits pinned at full load and boost clock whenever two HIP contexts touch it. With two backends running, there are always two.
+- **`GPU_MAX_HW_QUEUES`** works around [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706): the card sits pinned at full load and boost clock whenever two HIP contexts touch it. Keep it set on every GPU service.
 
 Background: [`research/rocm-gpu-pin-and-render-group.md`](research/rocm-gpu-pin-and-render-group.md). Its service names are older than the current ones, but the findings still apply.
 
@@ -27,20 +27,10 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it,
 - **`--cache-reuse`** reuses cached KV for any matching prompt chunk instead of re-prefilling from scratch. That fixes the root cause behind the gateway's non-ping SSE timeout ([`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md)). It pairs with OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot so that there is a matching prefix to reuse.
 - **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
 - **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).
+- **Speculative decoding (DFlash2):** `--spec-type draft-dflash` with a separate drafter GGUF (`LLAMA_DRAFT_MODEL_FILE`), fetched by `downloader-draft`. It replaced MTP (draft heads inside the target GGUF). `LLAMA_SPEC_N_MAX` is the draft length. Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low.
+- **No classifier backend:** the separate `qwen-classifier` was removed to free VRAM for the drafter. Anything that needs a small "fast" model must use the main model.
 - **No host port**: see [network access → Backends](network-access.md#backends).
 
-## qwen-classifier
-
-The classifier model's backend. It serves qwen-code's Auto Mode classifier (`fastModel`), which judges every tool call before it runs.
-
-- **Why a separate backend:** when the classifier was an alias of the main model, every classification queued behind whatever heavy generation already held the main backend's slots, sometimes for minutes.
-- **Why GPU, not CPU:** a CPU-only instance was tried first. Real classification calls carry a conversation transcript, and they ran past the gateway's timeout and retry-looped. It now runs on the GPU with a partial layer count (`--n-gpu-layers`), which leaves VRAM headroom for the main model.
-- **Context size:** qwen-code hard-caps the classifier transcript in its own source (`MAX_TRANSCRIPT_MESSAGES`, `MAX_HISTORICAL_ACTION_CHARS` in `packages/core/src/permissions/classifier-transcript.ts`). `--ctx-size` gives margin over that worst case, and nothing more. The backend has a single slot, so its whole context is per-slot.
-- **VRAM:** measured usage came out well above the weights+KV estimate. The cause was a missing `--flash-attn`: without it, the unfused attention buffers grow roughly quadratically with context. Batch size was the wrong lever. Check with `rocm-smi` after any change to the context size or the model.
-- **`--reasoning off`** is a no-cost safety net against [llama.cpp#20809](https://github.com/ggml-org/llama.cpp/issues/20809). Some builds misdetect Qwen3-Instruct-2507 as a thinking model and route tool calls into `reasoning_content`. Nobody has confirmed whether the current image still needs it.
-- **Model choice:** the smallest Qwen3 with long native context and no RoPE scaling. See [`research/fast-model-choice.md`](research/fast-model-choice.md) and [`research/classifier-context-window-scaling.md`](research/classifier-context-window-scaling.md).
-- Its provider connection needs specific gateway settings: see [gateway → Provider connections](gateway.md#provider-connections).
-
 ## omniroute
 
 The gateway. Its runtime configuration (provider connections, API keys) lives in its own database and is managed through the dashboard, not in this repo: see [`gateway.md`](gateway.md).

docs/coding-cli-setup/index.md

@@ -24,6 +24,6 @@ Terms as defined in [`CONTEXT.md`](../../CONTEXT.md). Every coding CLI is a work
 | [Claude Code](claude-code.md) | Anthropic Messages | `ANTHROPIC_BASE_URL` env var |
 | [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `config.toml` provider block |
 | [OpenCode](opencode.md) | OpenAI Chat Completions | `opencode.json` provider block |
-| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model + classifier model) | `~/.qwen/settings.json` `modelProviders.openai` |
+| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model) | `~/.qwen/settings.json` `modelProviders.openai` |
 
 If requests fail with a semaphore timeout under concurrent load, see [gateway → Timeouts](../gateway.md#timeouts).

docs/coding-cli-setup/qwen-code.md

@@ -4,11 +4,11 @@
 
 > **Source of truth:** [Qwen Code's repo](https://github.com/QwenLM/qwen-code) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win.
 
-Qwen Code speaks plain **OpenAI Chat Completions**. Unlike the other CLIs, it needs *two* models: the main model, and the classifier model as its `fastModel` for Auto Mode's action classifier. Each has its own provider connection in the gateway, and both are reached through the same gateway URL.
+Qwen Code speaks plain **OpenAI Chat Completions**. It uses the main model only. There is no separate classifier backend any more ([architecture → llama-server](../architecture.md#llama-server)).
 
-## Why a second model exists
+## Auto Mode's classifier (`fastModel`)
 
-Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call: it decides whether to auto-approve or block a shell command or tool call before it runs. It runs on its own backend, `qwen-classifier`. Why that backend exists and how it's sized: [architecture → qwen-classifier](../architecture.md#qwen-classifier).
+Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call. It calls `fastModel`. With no classifier backend, either point `fastModel` at the main model (it then queues behind the main backend's slots, the reason the separate backend once existed), or leave `fastModel` unset and don't use Auto Mode.
 
 ## `~/.qwen/settings.json`
 
@@ -22,16 +22,6 @@ Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gat
         "envKey": "OMNIROUTE_API_KEY",
         "baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1",
         "generationConfig": { "contextWindowSize": <LLAMA_CTX_SIZE / LLAMA_PARALLEL> }
-      },
-      {
-        "id": "<classifier-provider-id-in-omniroute>",
-        "name": "qwen3-4b-classifier",
-        "envKey": "OMNIROUTE_API_KEY",
-        "baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1",
-        "generationConfig": {
-          "contextWindowSize": <qwen-classifier --ctx-size>,
-          "extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
-        }
       }
     ]
   },
@@ -40,14 +30,12 @@ Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gat
     "name": "<main-model-provider-id-in-omniroute>",
     "baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1"
   },
-  "fastModel": "<classifier-provider-id-in-omniroute>"
+  "fastModel": "<main-model-provider-id-in-omniroute>"
 }
 ```
 
-- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=<qwen-code-cli API key>` before launching. Both providers can share one API key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
+- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=<qwen-code-cli API key>` before launching.
 - **The main model's `contextWindowSize` is the per-slot context**, not `LLAMA_CTX_SIZE` itself ([overview](index.md)).
-- **The classifier's `contextWindowSize` is its whole `--ctx-size`.** It has a single slot, so there's no per-slot division ([architecture → qwen-classifier](../architecture.md#qwen-classifier)).
-- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see [`research/fast-model-choice.md`](../research/fast-model-choice.md) §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
 - Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
 
 ## Web search via OmniRoute
@@ -107,7 +95,7 @@ It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above —
 
 ## Auto Mode tuning
 
-Auto Mode's action classifier calls the classifier model above. Even on its own GPU-resident backend, give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call:
+Auto Mode's action classifier calls `fastModel`, now the main model, so it can queue behind a heavy generation. Give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call:
 
 ```json
 {
@@ -122,6 +110,6 @@ Auto Mode's action classifier calls the classifier model above. Even on its own
 
 `hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each.
 
-Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see [gateway → Dashboard](../gateway.md#dashboard)). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.
+The main model's provider connection already carries the long per-connection timeout (`providerSpecificData.timeoutMs`, see [gateway → Timeouts](../gateway.md#timeouts)), which now covers classifier calls too.
 
 Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.

docs/gateway.md

@@ -13,7 +13,7 @@ Browse to `http://<server>:<OMNIROUTE_DASHBOARD_PORT>` from the LAN. The dashboa
 Each backend is registered once in the dashboard (Providers → add a **llama.cpp** provider) or via `POST /api/providers`. The base URL is `http://<service>:<container port>` on the stack network, using the service name and port from compose. None of this is checked into the repo. These docs assume the main model is registered as `qwen3.8-27b-local`.
 
 - **`OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` / `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`** have to be on, because backend URLs are private addresses. They relax OmniRoute's SSRF guard, which normally blocks registering a provider at a private or local address. Anyone with dashboard access can therefore point the gateway at any host on the stack network or the LAN.
-- **Classifier connection:** turn OmniRoute's memory-tool injection off for it. Injected tools break the classifier's output ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)). Also give it a generous per-connection timeout (see [Timeouts](#timeouts)).
+- **Stale classifier connection:** the `qwen-classifier` backend no longer exists. Delete its provider connection in the dashboard, or repoint it at the main model. If repointed, turn OmniRoute's memory-tool injection off for it ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)).
 - Background on registering local backends: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md).
 
 ## API keys

docs/network-access.md

@@ -22,7 +22,7 @@ Terms as defined in [`CONTEXT.md`](../CONTEXT.md). The stack doesn't run its own
 
 ## Backends
 
-`llama-server` and `qwen-classifier` publish no host port (`expose` only). They have no authentication, so they're reachable only on the stack's Docker network, through the gateway. Making them directly reachable would put an unauthenticated inference endpoint on the network. That's a deliberate security change, not a config tweak.
+`llama-server` publishes no host port (`expose` only). It has no authentication, so it's reachable only on the stack's Docker network, through the gateway. Making it directly reachable would put an unauthenticated inference endpoint on the network. That's a deliberate security change, not a config tweak.
 
 ## qdrant
 

scripts/update.sh

@@ -175,7 +175,7 @@ docker compose build --pull
 
 echo "==> ensuring models are downloaded (skips already-present files)"
 docker compose --profile tools run --rm downloader
-docker compose --profile tools run --rm downloader-classifier
+docker compose --profile tools run --rm downloader-draft
 
 echo "==> bringing up omniroute"
 docker compose up -d --wait omniroute