From 7d8c9324b260b3a3fecd48dad4eb1ad93e232215 Mon Sep 17 00:00:00 2001 From: Haylan Date: Mon, 24 Aug 2026 18:19:56 +0200 Subject: [PATCH] Write local-usage docs for Claude Code CLI, Kimi CLI, and OpenCode CLI Resolves wayfinder ticket #6. Covers the two server endpoints (Anthropic shim vs OpenAI-compatible), exact config for all three CLIs, and surfaces the shared Qwen3.8-27B tool-calling risk plus OpenCode's own documented local-backend brittleness up front rather than burying it. Co-Authored-By: Claude Sonnet 5 --- README.md | 2 +- docs/coding-cli-setup.md | 87 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 docs/coding-cli-setup.md diff --git a/README.md b/README.md index e5294f2..edb19ef 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,6 @@ docker compose up -d - llama.cpp OpenAI-compatible API: http://localhost:8080/v1 - llama.cpp Anthropic Messages API (for Claude Code CLI): http://localhost:8080/v1/messages -Pointing Claude Code CLI or Kimi CLI at the local endpoint is documented separately — see [issue #6](https://git.arthurerlich.de/haylan/LLM-Server/issues/6) once resolved. +Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md). **Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`. diff --git a/docs/coding-cli-setup.md b/docs/coding-cli-setup.md new file mode 100644 index 0000000..93e1798 --- /dev/null +++ b/docs/coding-cli-setup.md @@ -0,0 +1,87 @@ +# Pointing a coding-agent CLI at this stack + +This stack's llama.cpp server exposes two endpoints once `docker compose up` is running (see `docker-compose.yml`): + +- **OpenAI-compatible**: `http://localhost:8080/v1` (or `${LLAMA_PORT}` if you changed it in `.env`) +- **Anthropic Messages API shim**: `http://localhost:8080` (adds `/v1/messages`) + +Both serve the same model — `Qwen3.8-27B-UD-Q4_K_XL.gguf` — behind whichever wire format the client speaks. + +> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before ``, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5). + +## Claude Code CLI + +Claude Code speaks the **Anthropic Messages API** — point it at the shim, not the OpenAI-compatible endpoint: + +```bash +export ANTHROPIC_BASE_URL=http://localhost:8080 +export ANTHROPIC_API_KEY=local # value is unchecked by llama.cpp, but the client requires it set +claude +``` + +Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright. + +## Kimi CLI + +Kimi CLI speaks plain **OpenAI Chat Completions** — no shim needed. Configure a provider block in its config file (`config.toml`): + +```toml +[providers.openai] +type = "openai" +base_url = "http://localhost:8080/v1" +api_key = "local" +``` + +If Kimi CLI's response parsing gets confused by Qwen's `...` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses. + +## OpenCode CLI + +Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/ + +**Install**: +```bash +curl -fsSL https://opencode.ai/install | bash +``` + +**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`): + +```json +{ + "$schema": "https://opencode.ai/config.json", + "provider": { + "llamacpp": { + "npm": "@ai-sdk/openai-compatible", + "name": "llama.cpp (local)", + "options": { + "baseURL": "http://localhost:8080/v1", + "apiKey": "sk-local-not-checked" + }, + "models": { + "qwen3.8-27b": { + "name": "Qwen3.8-27B", + "limit": { "context": 65536, "output": 8192 } + } + } + } + } +} +``` + +Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server. + +Select the model with `llamacpp/qwen3.8-27b`. + +**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above): +- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500. +- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly. +- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `` tags rather than a native reasoning API field — expect no effect from that config on this model; untested. + +## Summary + +| CLI | Wire format | Endpoint | Config | +|---|---|---|---| +| Claude Code | Anthropic Messages | `http://localhost:8080` | `ANTHROPIC_BASE_URL` env var | +| Kimi CLI | OpenAI Chat Completions | `http://localhost:8080/v1` | `config.toml` provider block | +| OpenCode | OpenAI Chat Completions | `http://localhost:8080/v1` | `opencode.json` provider block | + +Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`.