LLM-Server/docs/coding-cli-setup.md

Pointing a coding-agent CLI at this stack

This stack's llama.cpp server exposes two endpoints once docker compose up is running (see docker-compose.yml):

• OpenAI-compatible: http://localhost:8080/v1 (or ${LLAMA_PORT} if you changed it in .env)
• Anthropic Messages API shim: http://localhost:8080 (adds /v1/messages)

Both serve the same model — Qwen3.8-27B-UD-Q4_K_XL.gguf — behind whichever wire format the client speaks.

Read this before relying on it for real work. Qwen3.8-27B's tool-calling has documented, open llama.cpp upstream bugs (parser fails on text before <tool_call>, tool calls emitted as inert XML inside thinking blocks — see docs/research/qwen3.8-27b-tool-calling.md). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in issue #5.

Claude Code CLI

Claude Code speaks the Anthropic Messages API — point it at the shim, not the OpenAI-compatible endpoint:

export ANTHROPIC_BASE_URL=http://localhost:8080
export ANTHROPIC_API_KEY=local   # value is unchecked by llama.cpp, but the client requires it set
claude

Requires llama.cpp's --jinja flag (already set in docker-compose.yml) — without it, tool-use requests fail outright.

Kimi CLI

Kimi CLI speaks plain OpenAI Chat Completions — no shim needed. Configure a provider block in its config file (config.toml):

[providers.openai]
type = "openai"
base_url = "http://localhost:8080/v1"
api_key = "local"

If Kimi CLI's response parsing gets confused by Qwen's <think>...</think> reasoning tags, check its reasoning_key setting — it's configurable for non-standard local server responses.

OpenCode CLI

Confirmed project: anomalyco/opencode (renamed from sst/opencode — don't confuse with the unrelated opencode-ai/opencode Go TUI). Docs: https://opencode.ai/docs/

Install:

curl -fsSL https://opencode.ai/install | bash

Config (opencode.json, project root or ~/.config/opencode/opencode.json):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llamacpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local)",
      "options": {
        "baseURL": "http://localhost:8080/v1",
        "apiKey": "sk-local-not-checked"
      },
      "models": {
        "qwen3.8-27b": {
          "name": "Qwen3.8-27B",
          "limit": { "context": 65536, "output": 8192 }
        }
      }
    }
  }
}

Set limit.context to match whatever LLAMA_CTX_SIZE this stack is actually running with (.env), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.

Select the model with llamacpp/qwen3.8-27b.

OpenCode-specific risks (on top of the shared Qwen3.8-27B tool-calling risk above):

• Requires llama.cpp's --jinja flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
• anomalyco/opencode#20669 (closed as "not planned" — a live, unfixed risk): OpenCode's bash tool crashes if the model omits the optional description field on a tool call; some local backends return finish_reason: tool_calls with an empty array, which can hang the agent loop instead of stopping cleanly.
• Thinking-mode handling (options.reasoningEffort) is undocumented for models that emit inline <think> tags rather than a native reasoning API field — expect no effect from that config on this model; untested.

Summary

CLI	Wire format	Endpoint	Config
Claude Code	Anthropic Messages	http://localhost:8080	ANTHROPIC_BASE_URL env var
Kimi CLI	OpenAI Chat Completions	http://localhost:8080/v1	config.toml provider block
OpenCode	OpenAI Chat Completions	http://localhost:8080/v1	opencode.json provider block

Further reading: docs/research/qwen3.8-27b-tool-calling.md, docs/research/opencode-cli-setup.md.