LLM-Server/docs/coding-cli-setup/qwen-code.md

Qwen Code CLI

← back to overview

Qwen Code speaks plain OpenAI Chat Completions, and — unlike the other CLIs — needs two models: the main chat model, and a fastModel for Auto Mode's action classifier. Both are registered as separate providers in OmniRoute but reachable through the same gateway URL.

Why a second model exists

Auto Mode's action classifier (permissions.autoMode) is qwen-code's per-tool-call safety gate — it decides whether to auto-approve or block a shell command / tool call before it runs. It was originally aliased onto the main 27B model's own OmniRoute connection. That broke two ways in practice (see docs/research/fast-model-choice.md for the model research, and the issue-tracker history for the full incident):

• Queued behind heavy work. Every classification call competed for the main model's 2 GPU slots with whatever real generation was already running, so a classifier check could sit blocked for minutes.
• CPU-only was tried first and was too slow. Isolating the classifier onto its own CPU-only llama.cpp instance avoided the GPU queue entirely, but real classification calls (which can carry a non-trivial conversation transcript, not just the bare tool call) blew past OmniRoute's request timeout and retry-looped.

The fix: a dedicated, GPU-resident qwen-classifier service (docker-compose.yml) running a small model (Qwen3-4B-Instruct-2507) on its own partial GPU offload — enough layers on the R9700 to be fast, sized to leave real VRAM headroom next to the 27B model rather than trusting a naive weights+KV estimate (see that service's comment block in docker-compose.yml for the actual measured numbers and the two wrong turns — batch-size tuning, then flash-attn — before partial offload turned out to be the real lever).

~/.qwen/settings.json

{
  "modelProviders": {
    "openai": [
      {
        "id": "<main-model-provider-id-in-omniroute>",
        "name": "qwen3.8-27b-local",
        "envKey": "OMNIROUTE_API_KEY",
        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
        "generationConfig": { "contextWindowSize": 131072 }
      },
      {
        "id": "<classifier-provider-id-in-omniroute>",
        "name": "qwen3-4b-classifier",
        "envKey": "OMNIROUTE_API_KEY",
        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
        "generationConfig": {
          "contextWindowSize": 65536,
          "extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
        }
      }
    ]
  },
  "security": { "auth": { "selectedType": "openai" } },
  "model": {
    "name": "<main-model-provider-id-in-omniroute>",
    "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
  },
  "fastModel": "<classifier-provider-id-in-omniroute>"
}

• envKey names the environment variable Qwen Code reads the virtual key from — set OMNIROUTE_API_KEY=<qwen-code-cli virtual key> before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
• contextWindowSize for the main model is per-slot, not LLAMA_CTX_SIZE itself — llama.cpp divides --ctx-size across LLAMA_PARALLEL concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's limit.context). Compute it from .env: LLAMA_CTX_SIZE / LLAMA_PARALLEL = 262144 / 2 = 131072.
• The classifier's contextWindowSize (65536) is not per-slot math — qwen-classifier runs --parallel 1, so its whole --ctx-size belongs to the one slot. 65536 isn't a guess either: qwen-code's own source hard-caps the classifier transcript (MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000/message in packages/core/src/permissions/classifier-transcript.ts) — worst case is ~40-50K tokens, so 65536 gives real margin without wasting VRAM the way the original 131072 (copied from the main model's entry, not an actual qwen-code requirement) would have.
• enable_thinking: false on the classifier matters for parseability, though Qwen3-4B-Instruct-2507 is already architecturally non-thinking (see fast-model-choice.md §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
• Qwen Code also recognizes advisorModel, visionModel, compactionModel, imageModel for other model roles — none are wired up in this stack; only fastModel is required.

Web search via OmniRoute

Qwen Code's own built-in web search (tools.webSearch.enabled) has nothing to search with here — leave it false. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's /v1/search REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. ~/.qwen/mcp-servers/omniroute-search/index.mjs (needs @modelcontextprotocol/sdk and zod: npm init -y && npm i @modelcontextprotocol/sdk zod in that directory):

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home";
const API_KEY = process.env.OMNIROUTE_API_KEY;

if (!API_KEY) {
  console.error("OMNIROUTE_API_KEY is not set in the environment.");
  process.exit(1);
}

const server = new McpServer({ name: "omniroute-search", version: "1.0.0" });

server.registerTool(
  "search",
  {
    description: "Web/news search via OmniRoute's /v1/search endpoint.",
    inputSchema: { query: z.string().describe("Search query") },
  },
  async ({ query }) => {
    const res = await fetch(`${BASE_URL}/v1/search`, {
      method: "POST",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` },
      body: JSON.stringify({ query }),
    });
    const text = await res.text();
    if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true };
    return { content: [{ type: "text", text }] };
  }
);

await server.connect(new StdioServerTransport());

Register it in ~/.qwen/settings.json:

{
  "mcpServers": {
    "omniroute-search": { "command": "node", "args": ["<path-to>/index.mjs"] }
  },
  "tools": { "webSearch": { "enabled": false } }
}

It reuses the same OMNIROUTE_API_KEY env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.

Non-interactive mode (qwen -p ...) needs this tool explicitly allow-listed. MCP tools require interactive confirmation by default; --approval-mode auto alone doesn't bypass that for a non-interactive run — pass --allowed-tools mcp__omniroute-search__search (or -y for full YOLO) alongside -p, or the search call never reaches the classifier at all and silently no-ops. Confirmed live: without the allow-list, only the tool calls the CLI's non-interactive gate lets through end up as classifier requests.

Auto Mode tuning

Auto Mode's action classifier calls the fast model above. Even on the dedicated GPU-resident instance, give it real timeout headroom rather than trusting OmniRoute's default — and since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:

{
  "permissions": {
    "autoMode": {
      "classifier": { "timeouts": { "stage1Ms": 600000 } },
      "hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] }
    }
  }
}

hints.allow entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each.

Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (providerSpecificData.timeoutMs, dashboard or PATCH /api/providers/{id} — not a .env value, see docs/network-access.md for reaching the dashboard API). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.

Everything else in ~/.qwen/settings.json (hooks, security.auth's underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.