haylanandClaude-Bot
930e407053
- docker-compose.yml: add --reasoning off to qwen-classifier per fast-model-choice.md's own recommendation (ggml-org/llama.cpp#20809 safety net) — missed in the original rollout, caught while writing this up. - docs/coding-cli-setup/qwen-code.md: rewritten to match what's actually deployed (qwen-classifier, partial GPU offload, 65536 ctx, Q4_K_XL) — previously described an unimplemented llama-server-fast/8192-ctx plan. Documents the non-interactive MCP tool allow-list gap found live-testing. - docs/coding-cli-setup/opencode.md: fix stale 65536 example that didn't match its own documented LLAMA_CTX_SIZE/LLAMA_PARALLEL formula (131072). - docs/research/fast-model-choice.md: implementation note recording where the actual rollout diverged from this doc's original recommendations (service name, quant, context size, CPU-first-then-GPU path). - docs/research/omniroute-account-semaphore-timeout.md: new — the hardcoded 30s per-connection semaphore timeout found during the pr-agent investigation, root-caused against OmniRoute's own source, and the maxConcurrent:null + providerSpecificData.timeoutMs fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3.1 KiB
Pointing a coding-agent CLI at this stack
This stack routes through the AI gateway (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see docker-compose.yml). The gateway exposes:
- OpenAI-compatible:
http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1 - Anthropic Messages API (OmniRoute's own
/v1/messagesendpoint, translating to the OpenAI-compatible backend):http://<ai-box>:${OMNIROUTE_PORT:-4000}
Both serve the same underlying model — Qwen3.8-27B-UD-Q4_K_XL.gguf, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume qwen3.8-27b-local for continuity) — behind whichever wire format the client speaks.
<ai-box> is this machine's LAN address, or proxy-ai.home if your local DNS resolves that hostname directly to the box — see docs/network-access.md. If you're running a coding CLI from this machine itself, localhost works too.
Each CLI needs its own virtual key — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, <workload>-<purpose> naming, e.g. claude-code-cli, kimi-cli, opencode-cli). No budget set by default. These are the machine's interactive/high-priority workloads per docs/proxy-request-priority.md.
Read this before relying on it for real work. Qwen3.8-27B's tool-calling has documented, open llama.cpp upstream bugs (parser fails on text before
<tool_call>, tool calls emitted as inert XML inside thinking blocks — seedocs/research/qwen3.8-27b-tool-calling.md). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in issue #5 (and the proxy-specific smoke test in issue #17).
Per-CLI setup
Summary
| CLI | Wire format | Endpoint | Config |
|---|---|---|---|
| Claude Code | Anthropic Messages | http://<ai-box>:${OMNIROUTE_PORT:-4000} |
ANTHROPIC_BASE_URL env var |
| Kimi CLI | OpenAI Chat Completions | http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1 |
config.toml provider block |
| OpenCode | OpenAI Chat Completions | http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1 |
opencode.json provider block |
| Qwen Code | OpenAI Chat Completions (2 models: chat + fastModel) |
http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1 |
~/.qwen/settings.json modelProviders.openai |
Further reading: docs/research/qwen3.8-27b-tool-calling.md, docs/proxy-key-onboarding.md, docs/research/omniroute-account-semaphore-timeout.md (a connection that can only handle a few concurrent requests — like llama-server or qwen-classifier — hits a hardcoded 30s reject once more requests queue up than its maxConcurrent, unless configured around it).