Author SHA1 Message Date
haylanandClaude-Bot 6f4e736da8 docs(coding-cli-setup): split per-CLI docs into their own files
Move docs/coding-cli-setup.md to docs/coding-cli-setup/ with one file
per CLI (claude-code, kimi-cli, opencode, qwen-code) plus a shared
index.md for the gateway intro, tool-calling risk note, and summary
table.

Also fixes the qwen-code doc: context sizes are per-slot
(LLAMA_CTX_SIZE / LLAMA_PARALLEL), not raw LLAMA_CTX_SIZE (same fix
applied to OpenCode's limit.context); documents the fastModel
classifier provider and its own context math; adds the omniroute-search
MCP server (SearXNG web search) and Auto Mode permissions tuning that
were missing from the original qwen-code section.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-08 12:30:08 +02:00
haylanandClaude-Bot 1fcf30e9a1 fix(update.sh): stop config sync loop from dying silently on a missing key
current=$(grep ...) ran before checking whether the key existed in .env
at all. Under set -euo pipefail, a key missing from .env (e.g. a new
tunable this branch just added to .env.example) made that grep exit 1,
pipefail propagated it, and set -e killed the script instantly with no
output - looked exactly like a gum hang but never reached ensure_gum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
2026-09-07 20:09:31 +02:00
haylanandClaude-Bot e151aa6ffe feat(update.sh): vendor gum binary for the R9700's offline install
The server has no outbound internet access, so the curl download in
ensure_gum always failed silently and fell back to plain prompts.
Vendor the x86_64 release tarball under scripts/vendor/ and check it
before attempting a download - download stays as a fallback for other
archs or a version bump without a re-vendor.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
2026-09-07 20:04:56 +02:00
haylanandClaude-Bot feb7469f0b fix(update.sh): report why gum auto-install failed instead of failing silently
curl -fsSL | tar swallowed curl errors entirely, so a network failure
fetching gum looked identical to a successful skip - just silently
dropped into the plain-prompt fallback with no explanation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
2026-09-07 20:01:20 +02:00
haylan 7a654ead91 chore: ignore .qwen/temp 2026-09-07 19:54:38 +02:00
haylanandClaude-Bot 2bfe6dbd29 docs: point knowledge.proxy-ai.home at Neo4j's browser
Documents the NPM route for the RAG knowledge graph alongside the
existing proxy-ai.home/search.home entries. Neo4j (not Qdrant) gets
the hostname — it's the human-facing graph browser; Qdrant's
dashboard stays on its raw port for now.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
2026-09-07 19:40:51 +02:00
haylanandClaude-Bot 20f2ec3ab2 feat: add qdrant and neo4j for RAG vector/graph storage
Adds the two databases as infra services (own volume, ai-stack
network, host-published UI ports like comfyui) for an upcoming RAG
pipeline. Extraction/chunking/orchestration code is out of scope for
this repo — it's app logic that calls into these DBs and llama-server,
not compose infra.

Neo4j password follows the omniroute secret pattern: blank in
.env.example, auto-generated by scripts/update.sh.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
2026-09-07 19:20:14 +02:00
haylan 5b7548dc7c Merge pull request 'Fix llama-server-fast context-size exhaustion breaking Auto Mode' (#49) from fix-fastmodel-context-size into main
Reviewed-on: #49
2026-09-06 20:21:53 +00:00
haylan ea7b05fb99 feat: interactive per-key conflict resolution in update.sh's config sync
Replaces the previous commit's blind force-overwrite with a real
choice, per user feedback: force-overwriting server config without
asking was the wrong default.

- A tracked config value (real default in .env.example) that already
  matches .env is left alone silently — no prompt, no noise.
- A value that DIFFERS is a conflict, shown on one screen (all
  conflicts together, not one prompt per key) via gum
  (charmbracelet/gum) — single static binary fetched as a release
  tarball into .cache/gum/ (gitignored), no build step, no package
  manager dependency. Falls back to a plain read-based prompt if gum
  can't be fetched (offline, unsupported arch).
- Non-interactive (no TTY — cron, CI, piped): any conflict is a hard
  error (exit 1, lists every conflicting key) unless --force is
  passed, which accepts every new value automatically — matches how
  this PR's own fix needs to land unattended.
- Secrets and host-resolved values are completely unaffected either
  way — untouched by this loop, same as before.

Verified in an isolated sandbox against the exact scenario from this
PR (stale LLAMA_FAST_PARALLEL=2 vs git's 1):
- no TTY, no --force: exits 1, prints the diff, doesn't touch .env
- no TTY, --force: LLAMA_FAST_PARALLEL corrected 2 -> 1, an
  OMNIROUTE secret confirmed untouched (not regenerated)
docker compose config -q still passes.

Not verified: the interactive gum path itself (needs a real TTY,
couldn't allocate a pty in this sandbox) — worth confirming for real
on the server, including that gum's release asset naming actually
matches what ensure_gum() expects.

Refs #5
2026-09-06 22:20:27 +02:00
haylan 9def240a8e feat: update.sh force-syncs tracked config from .env.example
Follows directly from the previous commit's caveat: this PR's own fix
(LLAMA_FAST_PARALLEL=2 -> 1) wouldn't have taken effect on the server
without a manual .env edit, because set_if_blank never touches an
already-set value — by design, for secrets, but the same logic was
silently protecting stale copies of ordinary tunable config too.

Every KEY=VALUE line in .env.example with a real (non-blank) default
is now force-synced into .env on every run. Secrets and host-resolved
values are unaffected — .env.example already leaves those blank on
purpose, so the sync loop naturally skips them and they keep going
through set_if_blank as before.

Trade-off, called out in both the script's header and the sync loop's
own comment: there's no such thing as a persistent server-only
override for these keys anymore — a hand-edited value not reflected
in git gets reverted on the next run. That's the intended behavior.

Verified against a simulated stale .env matching the real scenario
from this PR: LLAMA_FAST_PARALLEL correctly overwritten 2 -> 1, an
OMNIROUTE secret left untouched. bash -n and docker compose config -q
both pass.

Refs #5
2026-09-06 22:10:51 +02:00
haylanandClaude-Bot 52a92f6508 fix: llama-server-fast context-size exhaustion breaking Auto Mode classifier
Real failure: "Auto Mode couldn't classify this action (Classifier
stage 1 unavailable)". Reproduced directly against the server:

  {"error":{"message":"[400]: request (6186 tokens) exceeds the
  available context size (4096 tokens)"...

LLAMA_FAST_CTX_SIZE=8192 is the TOTAL across every LLAMA_FAST_PARALLEL
slot, not per-request — the main model's own .env.example comment
already calls this out, missed it when llama-server-fast was set up
(#44). With PARALLEL=2 that's 4096/slot, too small for a real
classifier call (hints + environment + recent tool-call history).

Fixed by dropping to a single slot (LLAMA_FAST_PARALLEL=1) rather than
raising ctx-size — this service doesn't need concurrent classifier
calls the way the main model needs concurrent chat sessions, so this
costs no extra VRAM. The full 8192 now goes to the one slot.

docker compose config -q validated.

Refs #5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
2026-09-06 22:06:59 +02:00
13 changed files with 413 additions and 104 deletions
+20 -4
View File
@@ -116,11 +116,22 @@ LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model # Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
# is dense too. # is dense too.
LLAMA_FAST_GPU_LAYERS=999 LLAMA_FAST_GPU_LAYERS=999
# Classifier transcripts are truncated/bounded by qwen-code itself (see its # --ctx-size is the TOTAL across every LLAMA_FAST_PARALLEL slot, not per
# own Auto Mode docs) — no need for anywhere near the 27B's huge context. # request — same halving already called out for the main model above.
# 8192 keeps this instance's KV cache negligible. # Was PARALLEL=2, silently halving this to 4096/slot — too small: a real
# classifier call (hints + environment + recent tool-call history) hit
# "exceeds the available context size (4096 tokens)" in practice, which
# qwen-code surfaces as "Auto Mode couldn't classify this action
# (Classifier stage 1 unavailable)" — see issue #5. Fixed by dropping to
# a single slot instead of raising ctx-size (no extra VRAM, and this
# service doesn't need concurrent classifier calls the way the main
# model needs concurrent chat sessions) — the full 8192 now goes to the
# one slot. If hints.allow/softDeny/hardDeny ever approach their
# 50-entries-each ceiling, raise LLAMA_FAST_CTX_SIZE instead — qwen-code
# caps those at 200 chars x 150 entries plus 40,000 chars of
# historical-action context, which can exceed 8192 tokens worst-case.
LLAMA_FAST_CTX_SIZE=8192 LLAMA_FAST_CTX_SIZE=8192
LLAMA_FAST_PARALLEL=2 LLAMA_FAST_PARALLEL=1
# --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/ # --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
# image-generation-model-choice.md and issue #42) --- # image-generation-model-choice.md and issue #42) ---
@@ -130,3 +141,8 @@ LLAMA_FAST_PARALLEL=2
COMFYUI_DIFFUSION_MODEL_FILE=qwen_image_fp8_e4m3fn.safetensors COMFYUI_DIFFUSION_MODEL_FILE=qwen_image_fp8_e4m3fn.safetensors
COMFYUI_TEXT_ENCODER_FILE=qwen_2.5_vl_7b_fp8_scaled.safetensors COMFYUI_TEXT_ENCODER_FILE=qwen_2.5_vl_7b_fp8_scaled.safetensors
COMFYUI_VAE_FILE=qwen_image_vae.safetensors COMFYUI_VAE_FILE=qwen_image_vae.safetensors
# --- RAG databases (qdrant + neo4j, see wayfinder notes) ---
# No auth on qdrant (its default) — same trust boundary as llama-server:
# ai-stack is not exposed off-box. Random, filled in automatically:
NEO4J_PASSWORD=
+2
View File
@@ -4,3 +4,5 @@
# to be committed to this repo. # to be committed to this repo.
data/ data/
.leankg/ .leankg/
.cache/
.qwen/temp
+2 -7
View File
@@ -14,7 +14,7 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/
llama.cpp's own API is internal-only — everything routes through the AI gateway below. llama.cpp's own API is internal-only — everything routes through the AI gateway below.
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md). Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`. **Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
@@ -26,15 +26,10 @@ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fr
- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`. - Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). - Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
Coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting). Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further. **Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
### Web search ### Web search
The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35). The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
## What's not here
- **Open WebUI** — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle.
- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own built-in memory feature (opt-in via the dashboard, Settings → Memory): vector store is its bundled `sqlite-vec`, embeddings are a local ONNX model (Transformers.js, ~400MB, downloaded into the `omniroute-data` volume on first use) — no external services, no static config here.
+31 -1
View File
@@ -70,7 +70,7 @@ services:
--port 8080 --port 8080
--n-gpu-layers ${LLAMA_FAST_GPU_LAYERS:-999} --n-gpu-layers ${LLAMA_FAST_GPU_LAYERS:-999}
--ctx-size ${LLAMA_FAST_CTX_SIZE:-8192} --ctx-size ${LLAMA_FAST_CTX_SIZE:-8192}
--parallel ${LLAMA_FAST_PARALLEL:-2} --parallel ${LLAMA_FAST_PARALLEL:-1}
--flash-attn on --flash-attn on
--cache-type-k q8_0 --cache-type-k q8_0
--cache-type-v q8_0 --cache-type-v q8_0
@@ -303,6 +303,34 @@ services:
depends_on: depends_on:
- llama-server - llama-server
# RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
# directly like comfyui above, not gatewayed through omniroute (it isn't an
# LLM provider).
qdrant:
image: qdrant/qdrant:latest
container_name: qdrant
volumes:
- qdrant-data:/qdrant/storage
ports:
- "6333:6333"
restart: unless-stopped
networks: [ai-stack]
# RAG graph store, native vector index too (can absorb qdrant's job later
# if the two-DB split proves unnecessary — see wayfinder notes).
neo4j:
image: neo4j:5-community
container_name: neo4j
environment:
- NEO4J_AUTH=neo4j/${NEO4J_PASSWORD:?run scripts/update.sh first to resolve this}
volumes:
- neo4j-data:/data
ports:
- "7474:7474" # browser UI
- "7687:7687" # bolt
restart: unless-stopped
networks: [ai-stack]
networks: networks:
ai-stack: ai-stack:
@@ -310,3 +338,5 @@ volumes:
models: models:
omniroute-data: omniroute-data:
comfyui-data: comfyui-data:
qdrant-data:
neo4j-data:
-91
View File
@@ -1,91 +0,0 @@
# Pointing a coding-agent CLI at this stack
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
## Claude Code CLI
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
```bash
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
claude
```
Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
## Kimi CLI
Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
```toml
[providers.openai]
type = "openai"
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
api_key = "<kimi-cli virtual key>"
```
If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
## OpenCode CLI
Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
**Install**:
```bash
curl -fsSL https://opencode.ai/install | bash
```
**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"aiproxy": {
"npm": "@ai-sdk/openai-compatible",
"name": "AI proxy (local)",
"options": {
"baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"apiKey": "<opencode-cli virtual key>"
},
"models": {
"qwen3.8-27b-local": {
"name": "Qwen3.8-27B",
"limit": { "context": 65536, "output": 8192 }
}
}
}
}
}
```
Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.
Select the model with `aiproxy/qwen3.8-27b-local`.
**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above):
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
## Summary
| CLI | Wire format | Endpoint | Config |
|---|---|---|---|
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
+13
View File
@@ -0,0 +1,13 @@
# Claude Code CLI
[← back to overview](index.md)
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
```bash
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
claude
```
Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
+34
View File
@@ -0,0 +1,34 @@
# Pointing a coding-agent CLI at this stack
[← back to README](../../README.md)
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
## Per-CLI setup
- [Claude Code CLI](claude-code.md)
- [Kimi CLI](kimi-cli.md)
- [OpenCode CLI](opencode.md)
- [Qwen Code CLI](qwen-code.md)
## Summary
| CLI | Wire format | Endpoint | Config |
|---|---|---|---|
| [Claude Code](claude-code.md) | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` |
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`.
+14
View File
@@ -0,0 +1,14 @@
# Kimi CLI
[← back to overview](index.md)
Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
```toml
[providers.openai]
type = "openai"
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
api_key = "<kimi-cli virtual key>"
```
If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
+45
View File
@@ -0,0 +1,45 @@
# OpenCode CLI
[← back to overview](index.md)
Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
**Install**:
```bash
curl -fsSL https://opencode.ai/install | bash
```
**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
```json
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"aiproxy": {
"npm": "@ai-sdk/openai-compatible",
"name": "AI proxy (local)",
"options": {
"baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"apiKey": "<opencode-cli virtual key>"
},
"models": {
"qwen3.8-27b-local": {
"name": "Qwen3.8-27B",
"limit": { "context": 65536, "output": 8192 }
}
}
}
}
}
```
Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server.
Select the model with `aiproxy/qwen3.8-27b-local`.
**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)):
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
Further reading: `docs/research/opencode-cli-setup.md`.
+116
View File
@@ -0,0 +1,116 @@
# Qwen Code CLI
[← back to overview](index.md)
Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier (a separate, always-resident, always-fast instance so classification doesn't queue behind chat prefill; see `docker-compose.yml`'s `llama-server-fast` service and `docs/research/fast-model-choice.md`). Both are registered as separate providers in OmniRoute but reachable through the same gateway URL. Config lives in `~/.qwen/settings.json`:
```json
{
"modelProviders": {
"openai": [
{
"id": "<main-model-provider-id-in-omniroute>",
"name": "qwen3.8-27b-local",
"envKey": "OMNIROUTE_API_KEY",
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"generationConfig": { "contextWindowSize": 131072 }
},
{
"id": "<fast-model-provider-id-in-omniroute>",
"name": "qwen3.8-27b-classifier",
"envKey": "OMNIROUTE_API_KEY",
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"generationConfig": {
"contextWindowSize": 8192,
"extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
}
}
]
},
"security": { "auth": { "selectedType": "openai" } },
"model": {
"name": "<main-model-provider-id-in-omniroute>",
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
},
"fastModel": "<fast-model-provider-id-in-omniroute>"
}
```
- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=<qwen-code-cli virtual key>` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
- **`contextWindowSize` is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it per model from `.env`:
- Main model: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**.
- Fast model: `LLAMA_FAST_CTX_SIZE / LLAMA_FAST_PARALLEL` = `8192 / 1` = **8192**. Undersizing this one specifically breaks Auto Mode ("Classifier stage 1 unavailable") once `hints.allow`/`softDeny`/`hardDeny` entries and recent-action history push a classifier call past it — see the `LLAMA_FAST_CTX_SIZE` comment in `.env.example` before raising it instead of `LLAMA_FAST_PARALLEL`.
- `enable_thinking: false` on the fast model matters: the fast model file (`Qwen3-4B-Instruct-2507`) is already non-thinking, but this also suppresses `<think>` output on any fast-model swap that isn't, keeping classifier responses parseable.
- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
## Web search via OmniRoute
Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
```js
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home";
const API_KEY = process.env.OMNIROUTE_API_KEY;
if (!API_KEY) {
console.error("OMNIROUTE_API_KEY is not set in the environment.");
process.exit(1);
}
const server = new McpServer({ name: "omniroute-search", version: "1.0.0" });
server.registerTool(
"search",
{
description: "Web/news search via OmniRoute's /v1/search endpoint.",
inputSchema: { query: z.string().describe("Search query") },
},
async ({ query }) => {
const res = await fetch(`${BASE_URL}/v1/search`, {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` },
body: JSON.stringify({ query }),
});
const text = await res.text();
if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true };
return { content: [{ type: "text", text }] };
}
);
await server.connect(new StdioServerTransport());
```
Register it in `~/.qwen/settings.json`:
```json
{
"mcpServers": {
"omniroute-search": { "command": "node", "args": ["<path-to>/index.mjs"] }
},
"tools": { "webSearch": { "enabled": false } }
}
```
It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.
## Auto Mode tuning
Auto Mode's action classifier calls the fast model above — its own request can queue behind other stack traffic before the fast llama-server instance is warm, so the default classifier timeout is worth raising. And since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:
```json
{
"permissions": {
"autoMode": {
"classifier": { "timeouts": { "stage1Ms": 600000 } },
"hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] }
}
}
}
```
`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each (see the `LLAMA_FAST_CTX_SIZE` note above for why that ceiling matters).
Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.
+7 -1
View File
@@ -4,7 +4,7 @@ This stack has no chat UI — every client is a coding CLI reaching the AI gatew
## llama.cpp's raw API stays LAN-only — deliberately ## llama.cpp's raw API stays LAN-only — deliberately
The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) don't reach it directly at all now; they go through the gateway below, same as everything else. The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else.
If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture. If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.
@@ -16,3 +16,9 @@ As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migr
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward. - The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname. **Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname.
## RAG knowledge graph (Neo4j) — `knowledge.proxy-ai.home`
Set up as an NPM Proxy Host pointing at this machine's LAN IP on Neo4j's Browser port (`7474`, see `docker-compose.yml`'s `neo4j` service, [PR #50](https://git.arthurerlich.de/haylan/LLM-Server/pulls/50)). Internal-only, same as `proxy-ai.home` — no DMZ/external route, this is admin/dev tooling, not a client-facing endpoint. Bolt (`7687`, the actual query protocol) isn't proxied through NPM at all — clients on the LAN reach it directly at `<this-machine>:7687`.
Qdrant's dashboard (`6333`) stays on its raw LAN IP/port for now — no hostname assigned yet.
+129
View File
@@ -4,6 +4,18 @@
# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what # exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
# changed — safe to run any time, including with nothing to do. # changed — safe to run any time, including with nothing to do.
# #
# Tunable config values (LLAMA_*, ports, timeouts — anything with a real
# default in .env.example) are synced from .env.example every run. A value
# already matching is left alone silently. A value that DIFFERS from the
# server's current .env is a conflict: interactively, you're shown every
# conflict on one screen (via gum) and choose which to accept — unpicked
# keys keep the server's current value. Non-interactively (no TTY — cron,
# CI, piped), any conflict is a hard error unless --force is passed, which
# accepts every new value automatically. Secrets and host-resolved values
# (blank in .env.example — OMNIROUTE_*_SECRET/_KEY/_SALT/_PASSWORD,
# SEARXNG_LAN_IP, COMFYUI_PUID/PGID, HOST_VIDEO_GID/RENDER_GID) are never
# touched by this — they keep going through set_if_blank as before.
#
# omniroute's own routing/provider config (llama-server, search) lives in # omniroute's own routing/provider config (llama-server, search) lives in
# its dashboard, not a checked-in file like the old litellm-config.yaml — # its dashboard, not a checked-in file like the old litellm-config.yaml —
# see issue #31 and docs/proxy-key-onboarding.md. # see issue #31 and docs/proxy-key-onboarding.md.
@@ -14,6 +26,14 @@
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
FORCE=false
for arg in "$@"; do
case "$arg" in
--force) FORCE=true ;;
*) echo "Usage: $0 [--force]" >&2; exit 1 ;;
esac
done
# Must run before anything else touches a file this script itself reads # Must run before anything else touches a file this script itself reads
# (docker-compose.yml, .env.example, this script's own remaining lines) — # (docker-compose.yml, .env.example, this script's own remaining lines) —
# a self-updating script isn't guaranteed atomic against its own file # a self-updating script isn't guaranteed atomic against its own file
@@ -27,6 +47,114 @@ git pull --ff-only
[ -f .env ] || cp .env.example .env [ -f .env ] || cp .env.example .env
echo "==> syncing tracked config values from .env.example"
# ponytail: gum (charmbracelet/gum) is a single static binary. Vendored as
# a release tarball under scripts/vendor/ (checked into git) for the R9700
# box, which has no outbound internet access — the download fallback below
# is only for other archs / when the vendored copy is missing or stale.
# Cached under .cache/gum/ (gitignored) so repeat runs don't re-extract.
GUM_VERSION="0.14.5"
GUM_DIR="$(pwd)/.cache/gum"
GUM_BIN="$GUM_DIR/gum"
ensure_gum() {
command -v gum >/dev/null 2>&1 && { echo "gum"; return; }
[ -x "$GUM_BIN" ] && { echo "$GUM_BIN"; return; }
mkdir -p "$GUM_DIR"
local arch tmpdir vendored
case "$(uname -m)" in
x86_64) arch="x86_64" ;;
aarch64|arm64) arch="arm64" ;;
*) echo "no gum build for $(uname -m), falling back to plain prompts" >&2; echo ""; return ;;
esac
tmpdir="$(mktemp -d)"
vendored="$(pwd)/scripts/vendor/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
if [ -f "$vendored" ]; then
tar -xz -C "$tmpdir" -f "$vendored"
else
local url="https://github.com/charmbracelet/gum/releases/download/v${GUM_VERSION}/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
if ! curl -fsSL "$url" | tar -xz -C "$tmpdir" 2>/dev/null; then
echo "no vendored gum for $arch and couldn't download from $url (no internet egress? falling back to plain prompts)" >&2
fi
fi
if [ -n "$(find "$tmpdir" -name gum -type f 2>/dev/null)" ]; then
find "$tmpdir" -name gum -type f -exec cp {} "$GUM_BIN" \;
chmod +x "$GUM_BIN" 2>/dev/null || true
fi
rm -rf "$tmpdir"
[ -x "$GUM_BIN" ] && echo "$GUM_BIN" || echo ""
}
# Collect every key where .env.example has a real (non-blank) default:
# missing from .env -> just add it (no conflict, nothing to decide);
# present and identical -> leave alone silently; present and different ->
# a conflict to resolve below.
conflict_keys=()
conflict_old=()
conflict_new=()
while IFS='=' read -r key value; do
[ -n "$value" ] || continue
if ! grep -qE "^${key}=" .env; then
echo "${key}=${value}" >> .env
continue
fi
current="$(grep -E "^${key}=" .env | head -1 | cut -d= -f2-)"
if [ "$current" != "$value" ]; then
conflict_keys+=("$key")
conflict_old+=("$current")
conflict_new+=("$value")
fi
done < <(grep -E '^[A-Za-z_][A-Za-z0-9_]*=.+' .env.example)
if [ "${#conflict_keys[@]}" -gt 0 ]; then
if [ "$FORCE" = true ]; then
for i in "${!conflict_keys[@]}"; do
key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
sed -i "s|^${key}=.*|${key}=${new}|" .env
echo "${key}: ${conflict_old[$i]} -> ${new} (--force)"
done
elif [ ! -t 0 ] || [ ! -t 1 ]; then
echo "ERROR: ${#conflict_keys[@]} config value(s) in .env differ from .env.example, and this isn't an interactive terminal:" >&2
for i in "${!conflict_keys[@]}"; do
echo " ${conflict_keys[$i]}: ${conflict_old[$i]} (current) vs ${conflict_new[$i]} (.env.example)" >&2
done
echo "Re-run interactively to choose per-key, or pass --force to accept every new value." >&2
exit 1
else
gum_bin="$(ensure_gum)"
labels=()
for i in "${!conflict_keys[@]}"; do
labels+=("${conflict_keys[$i]}: ${conflict_old[$i]} -> ${conflict_new[$i]}")
done
if [ -n "$gum_bin" ]; then
selected="$(printf '%s\n' "${labels[@]}" | "$gum_bin" choose --no-limit --selected "$(printf '%s\n' "${labels[@]}" | paste -sd,)" --header "Config differs from .env.example — selected keys take the new value, unselected keep the server's current value:")"
else
# ponytail: plain-bash fallback if gum couldn't be fetched (offline,
# unsupported arch) — same one-screen-of-conflicts idea, cruder UI.
echo "Config differs from .env.example. Enter space-separated numbers to KEEP the server's current value (all others take the new value), or press enter to take every new value:"
for i in "${!conflict_keys[@]}"; do
echo " $((i+1))) ${labels[$i]}"
done
read -r -p "> " keep_nums
selected=""
for i in "${!conflict_keys[@]}"; do
case " $keep_nums " in
*" $((i+1)) "*) ;;
*) selected="${selected}${labels[$i]}"$'\n' ;;
esac
done
fi
for i in "${!conflict_keys[@]}"; do
key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
if printf '%s\n' "$selected" | grep -qxF "${labels[$i]}"; then
sed -i "s|^${key}=.*|${key}=${new}|" .env
echo "${key}: ${conflict_old[$i]} -> ${new}"
else
echo "${key}: kept ${conflict_old[$i]} (server value)"
fi
done
fi
fi
# Handles all three cases: the KEY=value line is missing entirely (.env # Handles all three cases: the KEY=value line is missing entirely (.env
# predates that var being added to .env.example — sed can't fix what isn't # predates that var being added to .env.example — sed can't fix what isn't
# there, so this appends it), present but blank, or already set. # there, so this appends it), present but blank, or already set.
@@ -55,6 +183,7 @@ set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)" set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)" set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)" set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
set_if_blank NEO4J_PASSWORD "$(openssl rand -hex 16)"
echo "==> resolving SEARXNG_LAN_IP" echo "==> resolving SEARXNG_LAN_IP"
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just # search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
Binary file not shown.