haylanandClaude Bot
17747694bd
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2.3 KiB
Pointing a coding CLI at this stack
Source of truth: each CLI's own config docs (linked per page), plus
docker-compose.yml→omniroutefor the gateway port. This page explains how to wire them up; ports and model settings live in those sources. If they disagree, the sources win.
Terms as defined in CONTEXT.md. Every coding CLI is a workload: it reaches the models through the gateway, never a backend directly (network access). The gateway speaks two wire formats for the same models:
- OpenAI-compatible:
http://<server>:<OMNIROUTE_API_PORT>/v1 - Anthropic Messages:
http://<server>:<OMNIROUTE_API_PORT>(OmniRoute's/v1/messages)
<server> is the server's LAN address or proxy-ai.home, or localhost on the server itself. The main model's name is whatever it was registered as in the gateway. These docs use qwen3.8-27b-local (gateway → Provider connections).
Each CLI needs its own API key, labelled e.g. claude-code-cli, kimi-cli, opencode-cli (gateway → API keys).
Context window: wherever a CLI asks for the main model's context size, use the per-slot context, LLAMA_CTX_SIZE / LLAMA_PARALLEL from the server's .env, not the raw context size (architecture → llama-server).
Tool-calling risk. The main model's tool calling has open llama.cpp upstream parser bugs (
research/qwen3.8-27b-tool-calling.md). Every CLI below inherits that risk, whatever its wire format. Test it on your own workflow before relying on it for unattended multi-step agent work.
Per-CLI setup
| CLI | Wire format | Config |
|---|---|---|
| Claude Code | Anthropic Messages | ANTHROPIC_BASE_URL env var |
| Kimi CLI | OpenAI Chat Completions | config.toml provider block |
| OpenCode | OpenAI Chat Completions | opencode.json provider block |
| Qwen Code | OpenAI Chat Completions (main model) | ~/.qwen/settings.json modelProviders.openai |
If requests fail with a semaphore timeout under concurrent load, see gateway → Timeouts.