LLM-Server/docs/coding-cli-setup.md

# Pointing a coding-agent CLI at this stack

This stack routes through the [AI proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (LiteLLM) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The proxy exposes:

- **OpenAI-compatible**: `http://<ai-box>:4000/v1` (or `${LITELLM_PORT}` if you changed it in `.env`)
- **Anthropic Messages API** (LiteLLM's own unified `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:4000`

Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the proxy as `qwen3.8-27b-local` — behind whichever wire format the client speaks.

`<ai-box>` is this machine's LAN address, or `proxy.ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.

**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (LiteLLM's Admin UI, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.

> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).

## Claude Code CLI

Claude Code speaks the **Anthropic Messages API** — point it at the proxy's unified endpoint, not llama.cpp directly:

```bash
export ANTHROPIC_BASE_URL=http://<ai-box>:4000
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
claude
```

Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.

## Kimi CLI

Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):

```toml
[providers.openai]
type = "openai"
base_url = "http://<ai-box>:4000/v1"
api_key = "<kimi-cli virtual key>"
```

If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.

## OpenCode CLI

Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/

**Install**:
```bash
curl -fsSL https://opencode.ai/install | bash
```

**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "aiproxy": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "AI proxy (local)",
      "options": {
        "baseURL": "http://<ai-box>:4000/v1",
        "apiKey": "<opencode-cli virtual key>"
      },
      "models": {
        "qwen3.8-27b-local": {
          "name": "Qwen3.8-27B",
          "limit": { "context": 65536, "output": 8192 }
        }
      }
    }
  }
}
```

Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.

Select the model with `aiproxy/qwen3.8-27b-local`.

**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above):
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.

## Summary

| CLI | Wire format | Endpoint | Config |
|---|---|---|---|
| Claude Code | Anthropic Messages | `http://<ai-box>:4000` | `ANTHROPIC_BASE_URL` env var |
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `config.toml` provider block |
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `opencode.json` provider block |

Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.