LLM-Server/docs/coding-cli-setup.md

Pointing a coding-agent CLI at this stack

This stack routes through the AI gateway (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see docker-compose.yml). The gateway exposes:

• OpenAI-compatible: http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1
• Anthropic Messages API (OmniRoute's own /v1/messages endpoint, translating to the OpenAI-compatible backend): http://<ai-box>:${OMNIROUTE_PORT:-4000}

Both serve the same underlying model — Qwen3.8-27B-UD-Q4_K_XL.gguf, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes qwen3.8-27b-local for continuity) — behind whichever wire format the client speaks.

<ai-box> is this machine's LAN address, or proxy-ai.home if your local DNS resolves that hostname directly to the box — see docs/network-access.md. If you're running a coding CLI from this machine itself, localhost works too.

Each CLI needs its own virtual key — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, <workload>-<purpose> naming, e.g. claude-code-cli, kimi-cli, opencode-cli). No budget set by default. These are the machine's interactive/high-priority workloads per docs/proxy-request-priority.md.

Read this before relying on it for real work. Qwen3.8-27B's tool-calling has documented, open llama.cpp upstream bugs (parser fails on text before <tool_call>, tool calls emitted as inert XML inside thinking blocks — see docs/research/qwen3.8-27b-tool-calling.md). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in issue #5 (and the proxy-specific smoke test in issue #17).

Claude Code CLI

Claude Code speaks the Anthropic Messages API — point it at the gateway's unified endpoint, not llama.cpp directly:

export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
claude

Requires llama.cpp's --jinja flag (already set in docker-compose.yml) — without it, tool-use requests fail outright.

Kimi CLI

Kimi CLI speaks plain OpenAI Chat Completions. Configure a provider block in its config file (config.toml):

[providers.openai]
type = "openai"
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
api_key = "<kimi-cli virtual key>"

If Kimi CLI's response parsing gets confused by Qwen's <think>...</think> reasoning tags, check its reasoning_key setting — it's configurable for non-standard local server responses.

OpenCode CLI

Confirmed project: anomalyco/opencode (renamed from sst/opencode — don't confuse with the unrelated opencode-ai/opencode Go TUI). Docs: https://opencode.ai/docs/

Install:

curl -fsSL https://opencode.ai/install | bash

Config (opencode.json, project root or ~/.config/opencode/opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "aiproxy": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "AI proxy (local)",
      "options": {
        "baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
        "apiKey": "<opencode-cli virtual key>"
      },
      "models": {
        "qwen3.8-27b-local": {
          "name": "Qwen3.8-27B",
          "limit": { "context": 65536, "output": 8192 }
        }
      }
    }
  }
}

Set limit.context to match whatever LLAMA_CTX_SIZE this stack is actually running with (.env), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.

Select the model with aiproxy/qwen3.8-27b-local.

OpenCode-specific risks (on top of the shared Qwen3.8-27B tool-calling risk above):

• Requires llama.cpp's --jinja flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
• anomalyco/opencode#20669 (closed as "not planned" — a live, unfixed risk): OpenCode's bash tool crashes if the model omits the optional description field on a tool call; some local backends return finish_reason: tool_calls with an empty array, which can hang the agent loop instead of stopping cleanly.
• Thinking-mode handling (options.reasoningEffort) is undocumented for models that emit inline <think> tags rather than a native reasoning API field — expect no effect from that config on this model; untested.

Summary

CLI	Wire format	Endpoint	Config
Claude Code	Anthropic Messages	http://<ai-box>:${OMNIROUTE_PORT:-4000}	ANTHROPIC_BASE_URL env var
Kimi CLI	OpenAI Chat Completions	http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1	config.toml provider block
OpenCode	OpenAI Chat Completions	http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1	opencode.json provider block

Further reading: docs/research/qwen3.8-27b-tool-calling.md, docs/research/opencode-cli-setup.md, docs/proxy-key-onboarding.md.