diff --git a/.env.example b/.env.example index e49125b..05144f3 100644 --- a/.env.example +++ b/.env.example @@ -6,8 +6,8 @@ LLAMA_GPU_LAYERS=999 LLAMA_CTX_SIZE=262144 LLAMA_PARALLEL=2 -# see docs/architecture.md#qwen-classifier -LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf +LLAMA_DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf +LLAMA_SPEC_N_MAX=7 # --- SearXNG web search --- see docs/gateway.md#web-search # Auto-filled by update.sh — leave blank. diff --git a/CONTEXT.md b/CONTEXT.md index f1c93e3..cbab291 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,7 +17,7 @@ _Avoid_: using it as a synonym for Server ### Serving models **Backend**: -A model server process that serves one model to the gateway (`llama-server`, `qwen-classifier`). +A model server process that serves one model to the gateway (`llama-server`). _Avoid_: provider, model server **Slot**: @@ -28,9 +28,9 @@ _Avoid_: parallel request, lane The large model that does the actual work for workloads. _Avoid_: chat model -**Classifier model**: -The small model that judges a coding CLI's tool calls before they run. -_Avoid_: fast model (`fastModel` only when quoting qwen-code's config key) +**Draft model**: +The small speculative-decoding drafter that runs beside the main model inside `llama-server` (DFlash2). +_Avoid_: classifier model (the separate classifier backend was removed) ### Reaching the models diff --git a/README.md b/README.md index 771f76d..1c8ae18 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > **Source of truth:** [`docker-compose.yml`](docker-compose.yml) (what runs) and [`scripts/update.sh`](scripts/update.sh) (how it's deployed). The docs explain *why*; if they disagree with the source, the source wins. -A self-hosted inference stack on one AMD GPU server. Two llama.cpp **backends**, the main model and the classifier model, sit behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches them through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md). +A self-hosted inference stack on one AMD GPU server. A llama.cpp **backend** serving the main model sits behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches it through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md). ## Quickstart diff --git a/docker-compose.yml b/docker-compose.yml index 1baebca..c0da56c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -28,8 +28,9 @@ services: --cache-type-k q8_0 --cache-type-v q8_0 --cache-reuse 256 - --spec-type draft-mtp - --spec-draft-n-max 2 + --spec-type draft-dflash + -md /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} + --spec-draft-n-max ${LLAMA_SPEC_N_MAX:-7} --jinja # see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port) expose: @@ -37,43 +38,6 @@ services: restart: unless-stopped networks: [ai-stack] - # see docs/architecture.md#qwen-classifier - qwen-classifier: - image: ghcr.io/ggml-org/llama.cpp:server-rocm - container_name: qwen-classifier - devices: - - /dev/kfd - - /dev/dri - group_add: - - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}" - - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}" - security_opt: - - seccomp=unconfined - ipc: host - # see docs/architecture.md#gpu-access - environment: - - GPU_MAX_HW_QUEUES=1 - volumes: - - models:/models - command: > - -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} - --host 0.0.0.0 - --port 8080 - --n-gpu-layers 28 - --ctx-size 50000 - --parallel 1 - --batch-size 512 - --ubatch-size 128 - --flash-attn on - --reasoning off - --cache-type-k q4_0 - --cache-type-v q4_0 - --jinja - expose: - - "8080" - restart: unless-stopped - networks: [ai-stack] - # see docs/operations.md#model-downloads (also covers downloader-classifier) downloader: image: curlimages/curl:latest @@ -89,7 +53,7 @@ services: curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} - downloader-classifier: + downloader-draft: image: curlimages/curl:latest profiles: ["tools"] user: root @@ -98,10 +62,10 @@ services: entrypoint: ["sh", "-c"] command: - > - test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} && + test -f /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} && echo "already downloaded, skipping" || - curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} - https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} + curl -L --fail --create-dirs -o /models/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} + https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/resolve/main/${LLAMA_DRAFT_MODEL_FILE:-Qwen3.8-27B-DFlash2-Q4_K_M.gguf} # see docs/architecture.md#omniroute; runtime config lives in its dashboard, see docs/gateway.md omniroute: diff --git a/docs/architecture.md b/docs/architecture.md index 706f2f0..86cee0e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -4,14 +4,14 @@ Terms as defined in [`CONTEXT.md`](../CONTEXT.md). -Every workload calls the **gateway** (`omniroute`) with its own API key. The gateway forwards each call over a **provider connection** to one of two **backends** on the one GPU: `llama-server` (main model) or `qwen-classifier` (classifier model). `qdrant` sits beside them on the same Docker network. It isn't a model, so it isn't behind the gateway. +Every workload calls the **gateway** (`omniroute`) with its own API key. The gateway forwards each call over a **provider connection** to the one **backend** on the GPU: `llama-server` (main model). `qdrant` sits beside them on the same Docker network. It isn't a model, so it isn't behind the gateway. ## GPU access -Both backends share the one GPU, and both carry the same GPU settings in compose: +`llama-server` carries these GPU settings in compose: - **`group_add` uses numeric GIDs, not group names.** Docker resolves a *named* `group_add` entry against the container's own `/etc/group`, not the host's. That fails unpredictably when the image doesn't define the group, and it gets worse when several GPU services start at once. `scripts/update.sh` resolves the host's real GIDs into `HOST_VIDEO_GID` / `HOST_RENDER_GID` (see [operations → Host-resolved values](operations.md#host-resolved-values)). -- **`GPU_MAX_HW_QUEUES`** works around [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706): the card sits pinned at full load and boost clock whenever two HIP contexts touch it. With two backends running, there are always two. +- **`GPU_MAX_HW_QUEUES`** works around [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706): the card sits pinned at full load and boost clock whenever two HIP contexts touch it. Keep it set on every GPU service. Background: [`research/rocm-gpu-pin-and-render-group.md`](research/rocm-gpu-pin-and-render-group.md). Its service names are older than the current ones, but the findings still apply. @@ -27,20 +27,10 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it, - **`--cache-reuse`** reuses cached KV for any matching prompt chunk instead of re-prefilling from scratch. That fixes the root cause behind the gateway's non-ping SSE timeout ([`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md)). It pairs with OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot so that there is a matching prefix to reuse. - **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright. - **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md). +- **Speculative decoding (DFlash2):** `--spec-type draft-dflash` with a separate drafter GGUF (`LLAMA_DRAFT_MODEL_FILE`), fetched by `downloader-draft`. It replaced MTP (draft heads inside the target GGUF). `LLAMA_SPEC_N_MAX` is the draft length. Upstream: [llama.cpp#27342](https://github.com/ggml-org/llama.cpp/pull/27342). Draft acceptance can collapse at high slot counts ([llama.cpp#27117](https://github.com/ggml-org/llama.cpp/issues/27117)), another reason `LLAMA_PARALLEL` stays low. +- **No classifier backend:** the separate `qwen-classifier` was removed to free VRAM for the drafter. Anything that needs a small "fast" model must use the main model. - **No host port**: see [network access → Backends](network-access.md#backends). -## qwen-classifier - -The classifier model's backend. It serves qwen-code's Auto Mode classifier (`fastModel`), which judges every tool call before it runs. - -- **Why a separate backend:** when the classifier was an alias of the main model, every classification queued behind whatever heavy generation already held the main backend's slots, sometimes for minutes. -- **Why GPU, not CPU:** a CPU-only instance was tried first. Real classification calls carry a conversation transcript, and they ran past the gateway's timeout and retry-looped. It now runs on the GPU with a partial layer count (`--n-gpu-layers`), which leaves VRAM headroom for the main model. -- **Context size:** qwen-code hard-caps the classifier transcript in its own source (`MAX_TRANSCRIPT_MESSAGES`, `MAX_HISTORICAL_ACTION_CHARS` in `packages/core/src/permissions/classifier-transcript.ts`). `--ctx-size` gives margin over that worst case, and nothing more. The backend has a single slot, so its whole context is per-slot. -- **VRAM:** measured usage came out well above the weights+KV estimate. The cause was a missing `--flash-attn`: without it, the unfused attention buffers grow roughly quadratically with context. Batch size was the wrong lever. Check with `rocm-smi` after any change to the context size or the model. -- **`--reasoning off`** is a no-cost safety net against [llama.cpp#20809](https://github.com/ggml-org/llama.cpp/issues/20809). Some builds misdetect Qwen3-Instruct-2507 as a thinking model and route tool calls into `reasoning_content`. Nobody has confirmed whether the current image still needs it. -- **Model choice:** the smallest Qwen3 with long native context and no RoPE scaling. See [`research/fast-model-choice.md`](research/fast-model-choice.md) and [`research/classifier-context-window-scaling.md`](research/classifier-context-window-scaling.md). -- Its provider connection needs specific gateway settings: see [gateway → Provider connections](gateway.md#provider-connections). - ## omniroute The gateway. Its runtime configuration (provider connections, API keys) lives in its own database and is managed through the dashboard, not in this repo: see [`gateway.md`](gateway.md). diff --git a/docs/coding-cli-setup/index.md b/docs/coding-cli-setup/index.md index 8f07e0d..bf46a80 100644 --- a/docs/coding-cli-setup/index.md +++ b/docs/coding-cli-setup/index.md @@ -24,6 +24,6 @@ Terms as defined in [`CONTEXT.md`](../../CONTEXT.md). Every coding CLI is a work | [Claude Code](claude-code.md) | Anthropic Messages | `ANTHROPIC_BASE_URL` env var | | [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `config.toml` provider block | | [OpenCode](opencode.md) | OpenAI Chat Completions | `opencode.json` provider block | -| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model + classifier model) | `~/.qwen/settings.json` `modelProviders.openai` | +| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model) | `~/.qwen/settings.json` `modelProviders.openai` | If requests fail with a semaphore timeout under concurrent load, see [gateway → Timeouts](../gateway.md#timeouts). diff --git a/docs/coding-cli-setup/qwen-code.md b/docs/coding-cli-setup/qwen-code.md index ee388dc..986ba6b 100644 --- a/docs/coding-cli-setup/qwen-code.md +++ b/docs/coding-cli-setup/qwen-code.md @@ -4,11 +4,11 @@ > **Source of truth:** [Qwen Code's repo](https://github.com/QwenLM/qwen-code) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win. -Qwen Code speaks plain **OpenAI Chat Completions**. Unlike the other CLIs, it needs *two* models: the main model, and the classifier model as its `fastModel` for Auto Mode's action classifier. Each has its own provider connection in the gateway, and both are reached through the same gateway URL. +Qwen Code speaks plain **OpenAI Chat Completions**. It uses the main model only. There is no separate classifier backend any more ([architecture → llama-server](../architecture.md#llama-server)). -## Why a second model exists +## Auto Mode's classifier (`fastModel`) -Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call: it decides whether to auto-approve or block a shell command or tool call before it runs. It runs on its own backend, `qwen-classifier`. Why that backend exists and how it's sized: [architecture → qwen-classifier](../architecture.md#qwen-classifier). +Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call. It calls `fastModel`. With no classifier backend, either point `fastModel` at the main model (it then queues behind the main backend's slots, the reason the separate backend once existed), or leave `fastModel` unset and don't use Auto Mode. ## `~/.qwen/settings.json` @@ -22,16 +22,6 @@ Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gat "envKey": "OMNIROUTE_API_KEY", "baseUrl": "http://:/v1", "generationConfig": { "contextWindowSize": } - }, - { - "id": "", - "name": "qwen3-4b-classifier", - "envKey": "OMNIROUTE_API_KEY", - "baseUrl": "http://:/v1", - "generationConfig": { - "contextWindowSize": , - "extra_body": { "chat_template_kwargs": { "enable_thinking": false } } - } } ] }, @@ -40,14 +30,12 @@ Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gat "name": "", "baseUrl": "http://:/v1" }, - "fastModel": "" + "fastModel": "" } ``` -- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=` before launching. Both providers can share one API key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls. +- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=` before launching. - **The main model's `contextWindowSize` is the per-slot context**, not `LLAMA_CTX_SIZE` itself ([overview](index.md)). -- **The classifier's `contextWindowSize` is its whole `--ctx-size`.** It has a single slot, so there's no per-slot division ([architecture → qwen-classifier](../architecture.md#qwen-classifier)). -- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see [`research/fast-model-choice.md`](../research/fast-model-choice.md) §3) — this is belt-and-suspenders for any future fast-model swap that isn't. - Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required. ## Web search via OmniRoute @@ -107,7 +95,7 @@ It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — ## Auto Mode tuning -Auto Mode's action classifier calls the classifier model above. Even on its own GPU-resident backend, give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call: +Auto Mode's action classifier calls `fastModel`, now the main model, so it can queue behind a heavy generation. Give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call: ```json { @@ -122,6 +110,6 @@ Auto Mode's action classifier calls the classifier model above. Even on its own `hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each. -Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see [gateway → Dashboard](../gateway.md#dashboard)). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can. +The main model's provider connection already carries the long per-connection timeout (`providerSpecificData.timeoutMs`, see [gateway → Timeouts](../gateway.md#timeouts)), which now covers classifier calls too. Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines. diff --git a/docs/gateway.md b/docs/gateway.md index 9b9638c..87a7e25 100644 --- a/docs/gateway.md +++ b/docs/gateway.md @@ -13,7 +13,7 @@ Browse to `http://:` from the LAN. The dashboa Each backend is registered once in the dashboard (Providers → add a **llama.cpp** provider) or via `POST /api/providers`. The base URL is `http://:` on the stack network, using the service name and port from compose. None of this is checked into the repo. These docs assume the main model is registered as `qwen3.8-27b-local`. - **`OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` / `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`** have to be on, because backend URLs are private addresses. They relax OmniRoute's SSRF guard, which normally blocks registering a provider at a private or local address. Anyone with dashboard access can therefore point the gateway at any host on the stack network or the LAN. -- **Classifier connection:** turn OmniRoute's memory-tool injection off for it. Injected tools break the classifier's output ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)). Also give it a generous per-connection timeout (see [Timeouts](#timeouts)). +- **Stale classifier connection:** the `qwen-classifier` backend no longer exists. Delete its provider connection in the dashboard, or repoint it at the main model. If repointed, turn OmniRoute's memory-tool injection off for it ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)). - Background on registering local backends: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md). ## API keys diff --git a/docs/network-access.md b/docs/network-access.md index 3a84a1f..2b0b3c5 100644 --- a/docs/network-access.md +++ b/docs/network-access.md @@ -22,7 +22,7 @@ Terms as defined in [`CONTEXT.md`](../CONTEXT.md). The stack doesn't run its own ## Backends -`llama-server` and `qwen-classifier` publish no host port (`expose` only). They have no authentication, so they're reachable only on the stack's Docker network, through the gateway. Making them directly reachable would put an unauthenticated inference endpoint on the network. That's a deliberate security change, not a config tweak. +`llama-server` publishes no host port (`expose` only). It has no authentication, so it's reachable only on the stack's Docker network, through the gateway. Making it directly reachable would put an unauthenticated inference endpoint on the network. That's a deliberate security change, not a config tweak. ## qdrant diff --git a/scripts/update.sh b/scripts/update.sh index 7d0f8c1..5fdd374 100755 --- a/scripts/update.sh +++ b/scripts/update.sh @@ -175,7 +175,7 @@ docker compose build --pull echo "==> ensuring models are downloaded (skips already-present files)" docker compose --profile tools run --rm downloader -docker compose --profile tools run --rm downloader-classifier +docker compose --profile tools run --rm downloader-draft echo "==> bringing up omniroute" docker compose up -d --wait omniroute