diff --git a/README.md b/README.md index 9942f3c..2ce6de1 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/ llama.cpp's own API is internal-only — everything routes through the AI gateway below. -Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md). +Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md). **Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`. @@ -26,15 +26,10 @@ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fr - Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`. - Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). -Coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting). +Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting). **Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further. ### Web search The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35). - -## What's not here - -- **Open WebUI** — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle. -- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own built-in memory feature (opt-in via the dashboard, Settings → Memory): vector store is its bundled `sqlite-vec`, embeddings are a local ONNX model (Transformers.js, ~400MB, downloaded into the `omniroute-data` volume on first use) — no external services, no static config here. diff --git a/docs/coding-cli-setup.md b/docs/coding-cli-setup.md deleted file mode 100644 index 8e9f34f..0000000 --- a/docs/coding-cli-setup.md +++ /dev/null @@ -1,91 +0,0 @@ -# Pointing a coding-agent CLI at this stack - -This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes: - -- **OpenAI-compatible**: `http://:${OMNIROUTE_PORT:-4000}/v1` -- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://:${OMNIROUTE_PORT:-4000}` - -Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks. - -`` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too. - -**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `-` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`. - -> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before ``, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)). - -## Claude Code CLI - -Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly: - -```bash -export ANTHROPIC_BASE_URL=http://:${OMNIROUTE_PORT:-4000} -export ANTHROPIC_API_KEY= -claude -``` - -Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright. - -## Kimi CLI - -Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`): - -```toml -[providers.openai] -type = "openai" -base_url = "http://:${OMNIROUTE_PORT:-4000}/v1" -api_key = "" -``` - -If Kimi CLI's response parsing gets confused by Qwen's `...` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses. - -## OpenCode CLI - -Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/ - -**Install**: -```bash -curl -fsSL https://opencode.ai/install | bash -``` - -**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`): - -```json -{ - "$schema": "https://opencode.ai/config.json", - "provider": { - "aiproxy": { - "npm": "@ai-sdk/openai-compatible", - "name": "AI proxy (local)", - "options": { - "baseURL": "http://:${OMNIROUTE_PORT:-4000}/v1", - "apiKey": "" - }, - "models": { - "qwen3.8-27b-local": { - "name": "Qwen3.8-27B", - "limit": { "context": 65536, "output": 8192 } - } - } - } - } -} -``` - -Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server. - -Select the model with `aiproxy/qwen3.8-27b-local`. - -**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above): -- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500. -- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly. -- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `` tags rather than a native reasoning API field — expect no effect from that config on this model; untested. - -## Summary - -| CLI | Wire format | Endpoint | Config | -|---|---|---|---| -| Claude Code | Anthropic Messages | `http://:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var | -| Kimi CLI | OpenAI Chat Completions | `http://:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block | -| OpenCode | OpenAI Chat Completions | `http://:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block | - -Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`. diff --git a/docs/coding-cli-setup/claude-code.md b/docs/coding-cli-setup/claude-code.md new file mode 100644 index 0000000..9951795 --- /dev/null +++ b/docs/coding-cli-setup/claude-code.md @@ -0,0 +1,13 @@ +# Claude Code CLI + +[← back to overview](index.md) + +Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly: + +```bash +export ANTHROPIC_BASE_URL=http://:${OMNIROUTE_PORT:-4000} +export ANTHROPIC_API_KEY= +claude +``` + +Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright. diff --git a/docs/coding-cli-setup/index.md b/docs/coding-cli-setup/index.md new file mode 100644 index 0000000..bf7c368 --- /dev/null +++ b/docs/coding-cli-setup/index.md @@ -0,0 +1,34 @@ +# Pointing a coding-agent CLI at this stack + +[← back to README](../../README.md) + +This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes: + +- **OpenAI-compatible**: `http://:${OMNIROUTE_PORT:-4000}/v1` +- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://:${OMNIROUTE_PORT:-4000}` + +Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks. + +`` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too. + +**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `-` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`. + +> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before ``, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)). + +## Per-CLI setup + +- [Claude Code CLI](claude-code.md) +- [Kimi CLI](kimi-cli.md) +- [OpenCode CLI](opencode.md) +- [Qwen Code CLI](qwen-code.md) + +## Summary + +| CLI | Wire format | Endpoint | Config | +|---|---|---|---| +| [Claude Code](claude-code.md) | Anthropic Messages | `http://:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var | +| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block | +| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block | +| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` | + +Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`. diff --git a/docs/coding-cli-setup/kimi-cli.md b/docs/coding-cli-setup/kimi-cli.md new file mode 100644 index 0000000..7c55338 --- /dev/null +++ b/docs/coding-cli-setup/kimi-cli.md @@ -0,0 +1,14 @@ +# Kimi CLI + +[← back to overview](index.md) + +Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`): + +```toml +[providers.openai] +type = "openai" +base_url = "http://:${OMNIROUTE_PORT:-4000}/v1" +api_key = "" +``` + +If Kimi CLI's response parsing gets confused by Qwen's `...` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses. diff --git a/docs/coding-cli-setup/opencode.md b/docs/coding-cli-setup/opencode.md new file mode 100644 index 0000000..1131280 --- /dev/null +++ b/docs/coding-cli-setup/opencode.md @@ -0,0 +1,45 @@ +# OpenCode CLI + +[← back to overview](index.md) + +Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/ + +**Install**: +```bash +curl -fsSL https://opencode.ai/install | bash +``` + +**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`): + +```json +{ + "$schema": "https://opencode.ai/config.json", + "provider": { + "aiproxy": { + "npm": "@ai-sdk/openai-compatible", + "name": "AI proxy (local)", + "options": { + "baseURL": "http://:${OMNIROUTE_PORT:-4000}/v1", + "apiKey": "" + }, + "models": { + "qwen3.8-27b-local": { + "name": "Qwen3.8-27B", + "limit": { "context": 65536, "output": 8192 } + } + } + } + } +} +``` + +Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server. + +Select the model with `aiproxy/qwen3.8-27b-local`. + +**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)): +- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500. +- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly. +- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `` tags rather than a native reasoning API field — expect no effect from that config on this model; untested. + +Further reading: `docs/research/opencode-cli-setup.md`. diff --git a/docs/coding-cli-setup/qwen-code.md b/docs/coding-cli-setup/qwen-code.md new file mode 100644 index 0000000..4040817 --- /dev/null +++ b/docs/coding-cli-setup/qwen-code.md @@ -0,0 +1,116 @@ +# Qwen Code CLI + +[← back to overview](index.md) + +Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier (a separate, always-resident, always-fast instance so classification doesn't queue behind chat prefill; see `docker-compose.yml`'s `llama-server-fast` service and `docs/research/fast-model-choice.md`). Both are registered as separate providers in OmniRoute but reachable through the same gateway URL. Config lives in `~/.qwen/settings.json`: + +```json +{ + "modelProviders": { + "openai": [ + { + "id": "", + "name": "qwen3.8-27b-local", + "envKey": "OMNIROUTE_API_KEY", + "baseUrl": "http://:${OMNIROUTE_PORT:-4000}/v1", + "generationConfig": { "contextWindowSize": 131072 } + }, + { + "id": "", + "name": "qwen3.8-27b-classifier", + "envKey": "OMNIROUTE_API_KEY", + "baseUrl": "http://:${OMNIROUTE_PORT:-4000}/v1", + "generationConfig": { + "contextWindowSize": 8192, + "extra_body": { "chat_template_kwargs": { "enable_thinking": false } } + } + } + ] + }, + "security": { "auth": { "selectedType": "openai" } }, + "model": { + "name": "", + "baseUrl": "http://:${OMNIROUTE_PORT:-4000}/v1" + }, + "fastModel": "" +} +``` + +- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls. +- **`contextWindowSize` is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it per model from `.env`: + - Main model: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**. + - Fast model: `LLAMA_FAST_CTX_SIZE / LLAMA_FAST_PARALLEL` = `8192 / 1` = **8192**. Undersizing this one specifically breaks Auto Mode ("Classifier stage 1 unavailable") once `hints.allow`/`softDeny`/`hardDeny` entries and recent-action history push a classifier call past it — see the `LLAMA_FAST_CTX_SIZE` comment in `.env.example` before raising it instead of `LLAMA_FAST_PARALLEL`. +- `enable_thinking: false` on the fast model matters: the fast model file (`Qwen3-4B-Instruct-2507`) is already non-thinking, but this also suppresses `` output on any fast-model swap that isn't, keeping classifier responses parseable. +- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required. + +## Web search via OmniRoute + +Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory): + +```js +import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; +import { z } from "zod"; + +const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home"; +const API_KEY = process.env.OMNIROUTE_API_KEY; + +if (!API_KEY) { + console.error("OMNIROUTE_API_KEY is not set in the environment."); + process.exit(1); +} + +const server = new McpServer({ name: "omniroute-search", version: "1.0.0" }); + +server.registerTool( + "search", + { + description: "Web/news search via OmniRoute's /v1/search endpoint.", + inputSchema: { query: z.string().describe("Search query") }, + }, + async ({ query }) => { + const res = await fetch(`${BASE_URL}/v1/search`, { + method: "POST", + headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` }, + body: JSON.stringify({ query }), + }); + const text = await res.text(); + if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true }; + return { content: [{ type: "text", text }] }; + } +); + +await server.connect(new StdioServerTransport()); +``` + +Register it in `~/.qwen/settings.json`: + +```json +{ + "mcpServers": { + "omniroute-search": { "command": "node", "args": ["/index.mjs"] } + }, + "tools": { "webSearch": { "enabled": false } } +} +``` + +It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions. + +## Auto Mode tuning + +Auto Mode's action classifier calls the fast model above — its own request can queue behind other stack traffic before the fast llama-server instance is warm, so the default classifier timeout is worth raising. And since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call: + +```json +{ + "permissions": { + "autoMode": { + "classifier": { "timeouts": { "stage1Ms": 600000 } }, + "hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] } + } + } +} +``` + +`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each (see the `LLAMA_FAST_CTX_SIZE` note above for why that ceiling matters). + +Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines. diff --git a/docs/network-access.md b/docs/network-access.md index 5bcc51f..fc80ea3 100644 --- a/docs/network-access.md +++ b/docs/network-access.md @@ -4,7 +4,7 @@ This stack has no chat UI — every client is a coding CLI reaching the AI gatew ## llama.cpp's raw API stays LAN-only — deliberately -The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) don't reach it directly at all now; they go through the gateway below, same as everything else. +The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else. If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.