Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6f4e736da8 | ||
|
|
1fcf30e9a1 | ||
|
|
e151aa6ffe | ||
|
|
feb7469f0b | ||
|
|
7a654ead91 | ||
|
|
2bfe6dbd29 | ||
|
|
20f2ec3ab2 | ||
|
|
5b7548dc7c | ||
|
|
ea7b05fb99 | ||
|
|
9def240a8e | ||
|
|
52a92f6508 | ||
|
|
63938e95c9 | ||
|
|
7b6d3f5802 |
+20
-4
@@ -116,11 +116,22 @@ LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
|
||||
# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
|
||||
# is dense too.
|
||||
LLAMA_FAST_GPU_LAYERS=999
|
||||
# Classifier transcripts are truncated/bounded by qwen-code itself (see its
|
||||
# own Auto Mode docs) — no need for anywhere near the 27B's huge context.
|
||||
# 8192 keeps this instance's KV cache negligible.
|
||||
# --ctx-size is the TOTAL across every LLAMA_FAST_PARALLEL slot, not per
|
||||
# request — same halving already called out for the main model above.
|
||||
# Was PARALLEL=2, silently halving this to 4096/slot — too small: a real
|
||||
# classifier call (hints + environment + recent tool-call history) hit
|
||||
# "exceeds the available context size (4096 tokens)" in practice, which
|
||||
# qwen-code surfaces as "Auto Mode couldn't classify this action
|
||||
# (Classifier stage 1 unavailable)" — see issue #5. Fixed by dropping to
|
||||
# a single slot instead of raising ctx-size (no extra VRAM, and this
|
||||
# service doesn't need concurrent classifier calls the way the main
|
||||
# model needs concurrent chat sessions) — the full 8192 now goes to the
|
||||
# one slot. If hints.allow/softDeny/hardDeny ever approach their
|
||||
# 50-entries-each ceiling, raise LLAMA_FAST_CTX_SIZE instead — qwen-code
|
||||
# caps those at 200 chars x 150 entries plus 40,000 chars of
|
||||
# historical-action context, which can exceed 8192 tokens worst-case.
|
||||
LLAMA_FAST_CTX_SIZE=8192
|
||||
LLAMA_FAST_PARALLEL=2
|
||||
LLAMA_FAST_PARALLEL=1
|
||||
|
||||
# --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
|
||||
# image-generation-model-choice.md and issue #42) ---
|
||||
@@ -130,3 +141,8 @@ LLAMA_FAST_PARALLEL=2
|
||||
COMFYUI_DIFFUSION_MODEL_FILE=qwen_image_fp8_e4m3fn.safetensors
|
||||
COMFYUI_TEXT_ENCODER_FILE=qwen_2.5_vl_7b_fp8_scaled.safetensors
|
||||
COMFYUI_VAE_FILE=qwen_image_vae.safetensors
|
||||
|
||||
# --- RAG databases (qdrant + neo4j, see wayfinder notes) ---
|
||||
# No auth on qdrant (its default) — same trust boundary as llama-server:
|
||||
# ai-stack is not exposed off-box. Random, filled in automatically:
|
||||
NEO4J_PASSWORD=
|
||||
|
||||
@@ -4,3 +4,5 @@
|
||||
# to be committed to this repo.
|
||||
data/
|
||||
.leankg/
|
||||
.cache/
|
||||
.qwen/temp
|
||||
@@ -14,7 +14,7 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/
|
||||
|
||||
llama.cpp's own API is internal-only — everything routes through the AI gateway below.
|
||||
|
||||
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
|
||||
Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).
|
||||
|
||||
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
|
||||
|
||||
@@ -26,15 +26,10 @@ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fr
|
||||
- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
|
||||
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
|
||||
|
||||
Coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
|
||||
Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
|
||||
|
||||
**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
|
||||
|
||||
### Web search
|
||||
|
||||
The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
|
||||
|
||||
## What's not here
|
||||
|
||||
- **Open WebUI** — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle.
|
||||
- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own built-in memory feature (opt-in via the dashboard, Settings → Memory): vector store is its bundled `sqlite-vec`, embeddings are a local ONNX model (Transformers.js, ~400MB, downloaded into the `omniroute-data` volume on first use) — no external services, no static config here.
|
||||
|
||||
+31
-1
@@ -70,7 +70,7 @@ services:
|
||||
--port 8080
|
||||
--n-gpu-layers ${LLAMA_FAST_GPU_LAYERS:-999}
|
||||
--ctx-size ${LLAMA_FAST_CTX_SIZE:-8192}
|
||||
--parallel ${LLAMA_FAST_PARALLEL:-2}
|
||||
--parallel ${LLAMA_FAST_PARALLEL:-1}
|
||||
--flash-attn on
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
@@ -303,6 +303,34 @@ services:
|
||||
depends_on:
|
||||
- llama-server
|
||||
|
||||
# RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
|
||||
# directly like comfyui above, not gatewayed through omniroute (it isn't an
|
||||
# LLM provider).
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
container_name: qdrant
|
||||
volumes:
|
||||
- qdrant-data:/qdrant/storage
|
||||
ports:
|
||||
- "6333:6333"
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
# RAG graph store, native vector index too (can absorb qdrant's job later
|
||||
# if the two-DB split proves unnecessary — see wayfinder notes).
|
||||
neo4j:
|
||||
image: neo4j:5-community
|
||||
container_name: neo4j
|
||||
environment:
|
||||
- NEO4J_AUTH=neo4j/${NEO4J_PASSWORD:?run scripts/update.sh first to resolve this}
|
||||
volumes:
|
||||
- neo4j-data:/data
|
||||
ports:
|
||||
- "7474:7474" # browser UI
|
||||
- "7687:7687" # bolt
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
networks:
|
||||
ai-stack:
|
||||
|
||||
@@ -310,3 +338,5 @@ volumes:
|
||||
models:
|
||||
omniroute-data:
|
||||
comfyui-data:
|
||||
qdrant-data:
|
||||
neo4j-data:
|
||||
|
||||
@@ -1,91 +0,0 @@
|
||||
# Pointing a coding-agent CLI at this stack
|
||||
|
||||
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
|
||||
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
|
||||
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
|
||||
|
||||
`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
|
||||
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
|
||||
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
|
||||
|
||||
## Claude Code CLI
|
||||
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
|
||||
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
|
||||
claude
|
||||
```
|
||||
|
||||
Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
|
||||
|
||||
## Kimi CLI
|
||||
|
||||
Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
|
||||
|
||||
```toml
|
||||
[providers.openai]
|
||||
type = "openai"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
|
||||
api_key = "<kimi-cli virtual key>"
|
||||
```
|
||||
|
||||
If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
|
||||
|
||||
## OpenCode CLI
|
||||
|
||||
Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
|
||||
|
||||
**Install**:
|
||||
```bash
|
||||
curl -fsSL https://opencode.ai/install | bash
|
||||
```
|
||||
|
||||
**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"aiproxy": {
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "AI proxy (local)",
|
||||
"options": {
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
|
||||
"apiKey": "<opencode-cli virtual key>"
|
||||
},
|
||||
"models": {
|
||||
"qwen3.8-27b-local": {
|
||||
"name": "Qwen3.8-27B",
|
||||
"limit": { "context": 65536, "output": 8192 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.
|
||||
|
||||
Select the model with `aiproxy/qwen3.8-27b-local`.
|
||||
|
||||
**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above):
|
||||
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
|
||||
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
|
||||
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
|
||||
|
||||
## Summary
|
||||
|
||||
| CLI | Wire format | Endpoint | Config |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
|
||||
|
||||
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Claude Code CLI
|
||||
|
||||
[← back to overview](index.md)
|
||||
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
|
||||
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
|
||||
claude
|
||||
```
|
||||
|
||||
Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Pointing a coding-agent CLI at this stack
|
||||
|
||||
[← back to README](../../README.md)
|
||||
|
||||
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
|
||||
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
|
||||
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
|
||||
|
||||
`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
|
||||
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
|
||||
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
|
||||
|
||||
## Per-CLI setup
|
||||
|
||||
- [Claude Code CLI](claude-code.md)
|
||||
- [Kimi CLI](kimi-cli.md)
|
||||
- [OpenCode CLI](opencode.md)
|
||||
- [Qwen Code CLI](qwen-code.md)
|
||||
|
||||
## Summary
|
||||
|
||||
| CLI | Wire format | Endpoint | Config |
|
||||
|---|---|---|---|
|
||||
| [Claude Code](claude-code.md) | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
|
||||
| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
|
||||
| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` |
|
||||
|
||||
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Kimi CLI
|
||||
|
||||
[← back to overview](index.md)
|
||||
|
||||
Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
|
||||
|
||||
```toml
|
||||
[providers.openai]
|
||||
type = "openai"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
|
||||
api_key = "<kimi-cli virtual key>"
|
||||
```
|
||||
|
||||
If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
|
||||
@@ -0,0 +1,45 @@
|
||||
# OpenCode CLI
|
||||
|
||||
[← back to overview](index.md)
|
||||
|
||||
Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
|
||||
|
||||
**Install**:
|
||||
```bash
|
||||
curl -fsSL https://opencode.ai/install | bash
|
||||
```
|
||||
|
||||
**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://opencode.ai/config.json",
|
||||
"provider": {
|
||||
"aiproxy": {
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "AI proxy (local)",
|
||||
"options": {
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
|
||||
"apiKey": "<opencode-cli virtual key>"
|
||||
},
|
||||
"models": {
|
||||
"qwen3.8-27b-local": {
|
||||
"name": "Qwen3.8-27B",
|
||||
"limit": { "context": 65536, "output": 8192 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server.
|
||||
|
||||
Select the model with `aiproxy/qwen3.8-27b-local`.
|
||||
|
||||
**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)):
|
||||
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
|
||||
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
|
||||
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
|
||||
|
||||
Further reading: `docs/research/opencode-cli-setup.md`.
|
||||
@@ -0,0 +1,116 @@
|
||||
# Qwen Code CLI
|
||||
|
||||
[← back to overview](index.md)
|
||||
|
||||
Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier (a separate, always-resident, always-fast instance so classification doesn't queue behind chat prefill; see `docker-compose.yml`'s `llama-server-fast` service and `docs/research/fast-model-choice.md`). Both are registered as separate providers in OmniRoute but reachable through the same gateway URL. Config lives in `~/.qwen/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"modelProviders": {
|
||||
"openai": [
|
||||
{
|
||||
"id": "<main-model-provider-id-in-omniroute>",
|
||||
"name": "qwen3.8-27b-local",
|
||||
"envKey": "OMNIROUTE_API_KEY",
|
||||
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
|
||||
"generationConfig": { "contextWindowSize": 131072 }
|
||||
},
|
||||
{
|
||||
"id": "<fast-model-provider-id-in-omniroute>",
|
||||
"name": "qwen3.8-27b-classifier",
|
||||
"envKey": "OMNIROUTE_API_KEY",
|
||||
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
|
||||
"generationConfig": {
|
||||
"contextWindowSize": 8192,
|
||||
"extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"security": { "auth": { "selectedType": "openai" } },
|
||||
"model": {
|
||||
"name": "<main-model-provider-id-in-omniroute>",
|
||||
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
|
||||
},
|
||||
"fastModel": "<fast-model-provider-id-in-omniroute>"
|
||||
}
|
||||
```
|
||||
|
||||
- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=<qwen-code-cli virtual key>` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
|
||||
- **`contextWindowSize` is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it per model from `.env`:
|
||||
- Main model: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**.
|
||||
- Fast model: `LLAMA_FAST_CTX_SIZE / LLAMA_FAST_PARALLEL` = `8192 / 1` = **8192**. Undersizing this one specifically breaks Auto Mode ("Classifier stage 1 unavailable") once `hints.allow`/`softDeny`/`hardDeny` entries and recent-action history push a classifier call past it — see the `LLAMA_FAST_CTX_SIZE` comment in `.env.example` before raising it instead of `LLAMA_FAST_PARALLEL`.
|
||||
- `enable_thinking: false` on the fast model matters: the fast model file (`Qwen3-4B-Instruct-2507`) is already non-thinking, but this also suppresses `<think>` output on any fast-model swap that isn't, keeping classifier responses parseable.
|
||||
- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
|
||||
|
||||
## Web search via OmniRoute
|
||||
|
||||
Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
|
||||
|
||||
```js
|
||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||
import { z } from "zod";
|
||||
|
||||
const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home";
|
||||
const API_KEY = process.env.OMNIROUTE_API_KEY;
|
||||
|
||||
if (!API_KEY) {
|
||||
console.error("OMNIROUTE_API_KEY is not set in the environment.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const server = new McpServer({ name: "omniroute-search", version: "1.0.0" });
|
||||
|
||||
server.registerTool(
|
||||
"search",
|
||||
{
|
||||
description: "Web/news search via OmniRoute's /v1/search endpoint.",
|
||||
inputSchema: { query: z.string().describe("Search query") },
|
||||
},
|
||||
async ({ query }) => {
|
||||
const res = await fetch(`${BASE_URL}/v1/search`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` },
|
||||
body: JSON.stringify({ query }),
|
||||
});
|
||||
const text = await res.text();
|
||||
if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true };
|
||||
return { content: [{ type: "text", text }] };
|
||||
}
|
||||
);
|
||||
|
||||
await server.connect(new StdioServerTransport());
|
||||
```
|
||||
|
||||
Register it in `~/.qwen/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"omniroute-search": { "command": "node", "args": ["<path-to>/index.mjs"] }
|
||||
},
|
||||
"tools": { "webSearch": { "enabled": false } }
|
||||
}
|
||||
```
|
||||
|
||||
It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.
|
||||
|
||||
## Auto Mode tuning
|
||||
|
||||
Auto Mode's action classifier calls the fast model above — its own request can queue behind other stack traffic before the fast llama-server instance is warm, so the default classifier timeout is worth raising. And since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:
|
||||
|
||||
```json
|
||||
{
|
||||
"permissions": {
|
||||
"autoMode": {
|
||||
"classifier": { "timeouts": { "stage1Ms": 600000 } },
|
||||
"hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each (see the `LLAMA_FAST_CTX_SIZE` note above for why that ceiling matters).
|
||||
|
||||
Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.
|
||||
@@ -4,7 +4,7 @@ This stack has no chat UI — every client is a coding CLI reaching the AI gatew
|
||||
|
||||
## llama.cpp's raw API stays LAN-only — deliberately
|
||||
|
||||
The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) don't reach it directly at all now; they go through the gateway below, same as everything else.
|
||||
The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else.
|
||||
|
||||
If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.
|
||||
|
||||
@@ -16,3 +16,9 @@ As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migr
|
||||
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
|
||||
|
||||
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname.
|
||||
|
||||
## RAG knowledge graph (Neo4j) — `knowledge.proxy-ai.home`
|
||||
|
||||
Set up as an NPM Proxy Host pointing at this machine's LAN IP on Neo4j's Browser port (`7474`, see `docker-compose.yml`'s `neo4j` service, [PR #50](https://git.arthurerlich.de/haylan/LLM-Server/pulls/50)). Internal-only, same as `proxy-ai.home` — no DMZ/external route, this is admin/dev tooling, not a client-facing endpoint. Bolt (`7687`, the actual query protocol) isn't proxied through NPM at all — clients on the LAN reach it directly at `<this-machine>:7687`.
|
||||
|
||||
Qdrant's dashboard (`6333`) stays on its raw LAN IP/port for now — no hostname assigned yet.
|
||||
|
||||
+140
-3
@@ -4,6 +4,18 @@
|
||||
# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
|
||||
# changed — safe to run any time, including with nothing to do.
|
||||
#
|
||||
# Tunable config values (LLAMA_*, ports, timeouts — anything with a real
|
||||
# default in .env.example) are synced from .env.example every run. A value
|
||||
# already matching is left alone silently. A value that DIFFERS from the
|
||||
# server's current .env is a conflict: interactively, you're shown every
|
||||
# conflict on one screen (via gum) and choose which to accept — unpicked
|
||||
# keys keep the server's current value. Non-interactively (no TTY — cron,
|
||||
# CI, piped), any conflict is a hard error unless --force is passed, which
|
||||
# accepts every new value automatically. Secrets and host-resolved values
|
||||
# (blank in .env.example — OMNIROUTE_*_SECRET/_KEY/_SALT/_PASSWORD,
|
||||
# SEARXNG_LAN_IP, COMFYUI_PUID/PGID, HOST_VIDEO_GID/RENDER_GID) are never
|
||||
# touched by this — they keep going through set_if_blank as before.
|
||||
#
|
||||
# omniroute's own routing/provider config (llama-server, search) lives in
|
||||
# its dashboard, not a checked-in file like the old litellm-config.yaml —
|
||||
# see issue #31 and docs/proxy-key-onboarding.md.
|
||||
@@ -14,8 +26,135 @@
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
FORCE=false
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--force) FORCE=true ;;
|
||||
*) echo "Usage: $0 [--force]" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Must run before anything else touches a file this script itself reads
|
||||
# (docker-compose.yml, .env.example, this script's own remaining lines) —
|
||||
# a self-updating script isn't guaranteed atomic against its own file
|
||||
# changing mid-run, so pulling later can execute a mix of old and new
|
||||
# script/compose content in one pass. Bit us for real: old GID-resolution
|
||||
# code ran, then this pulled in new var names docker-compose.yml now
|
||||
# requires, and nothing re-ran the (now-current) resolution step for
|
||||
# them — see issue #5's thread.
|
||||
echo "==> git pull"
|
||||
git pull --ff-only
|
||||
|
||||
[ -f .env ] || cp .env.example .env
|
||||
|
||||
echo "==> syncing tracked config values from .env.example"
|
||||
# ponytail: gum (charmbracelet/gum) is a single static binary. Vendored as
|
||||
# a release tarball under scripts/vendor/ (checked into git) for the R9700
|
||||
# box, which has no outbound internet access — the download fallback below
|
||||
# is only for other archs / when the vendored copy is missing or stale.
|
||||
# Cached under .cache/gum/ (gitignored) so repeat runs don't re-extract.
|
||||
GUM_VERSION="0.14.5"
|
||||
GUM_DIR="$(pwd)/.cache/gum"
|
||||
GUM_BIN="$GUM_DIR/gum"
|
||||
ensure_gum() {
|
||||
command -v gum >/dev/null 2>&1 && { echo "gum"; return; }
|
||||
[ -x "$GUM_BIN" ] && { echo "$GUM_BIN"; return; }
|
||||
mkdir -p "$GUM_DIR"
|
||||
local arch tmpdir vendored
|
||||
case "$(uname -m)" in
|
||||
x86_64) arch="x86_64" ;;
|
||||
aarch64|arm64) arch="arm64" ;;
|
||||
*) echo "no gum build for $(uname -m), falling back to plain prompts" >&2; echo ""; return ;;
|
||||
esac
|
||||
tmpdir="$(mktemp -d)"
|
||||
vendored="$(pwd)/scripts/vendor/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
|
||||
if [ -f "$vendored" ]; then
|
||||
tar -xz -C "$tmpdir" -f "$vendored"
|
||||
else
|
||||
local url="https://github.com/charmbracelet/gum/releases/download/v${GUM_VERSION}/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
|
||||
if ! curl -fsSL "$url" | tar -xz -C "$tmpdir" 2>/dev/null; then
|
||||
echo "no vendored gum for $arch and couldn't download from $url (no internet egress? falling back to plain prompts)" >&2
|
||||
fi
|
||||
fi
|
||||
if [ -n "$(find "$tmpdir" -name gum -type f 2>/dev/null)" ]; then
|
||||
find "$tmpdir" -name gum -type f -exec cp {} "$GUM_BIN" \;
|
||||
chmod +x "$GUM_BIN" 2>/dev/null || true
|
||||
fi
|
||||
rm -rf "$tmpdir"
|
||||
[ -x "$GUM_BIN" ] && echo "$GUM_BIN" || echo ""
|
||||
}
|
||||
|
||||
# Collect every key where .env.example has a real (non-blank) default:
|
||||
# missing from .env -> just add it (no conflict, nothing to decide);
|
||||
# present and identical -> leave alone silently; present and different ->
|
||||
# a conflict to resolve below.
|
||||
conflict_keys=()
|
||||
conflict_old=()
|
||||
conflict_new=()
|
||||
while IFS='=' read -r key value; do
|
||||
[ -n "$value" ] || continue
|
||||
if ! grep -qE "^${key}=" .env; then
|
||||
echo "${key}=${value}" >> .env
|
||||
continue
|
||||
fi
|
||||
current="$(grep -E "^${key}=" .env | head -1 | cut -d= -f2-)"
|
||||
if [ "$current" != "$value" ]; then
|
||||
conflict_keys+=("$key")
|
||||
conflict_old+=("$current")
|
||||
conflict_new+=("$value")
|
||||
fi
|
||||
done < <(grep -E '^[A-Za-z_][A-Za-z0-9_]*=.+' .env.example)
|
||||
|
||||
if [ "${#conflict_keys[@]}" -gt 0 ]; then
|
||||
if [ "$FORCE" = true ]; then
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
|
||||
sed -i "s|^${key}=.*|${key}=${new}|" .env
|
||||
echo "${key}: ${conflict_old[$i]} -> ${new} (--force)"
|
||||
done
|
||||
elif [ ! -t 0 ] || [ ! -t 1 ]; then
|
||||
echo "ERROR: ${#conflict_keys[@]} config value(s) in .env differ from .env.example, and this isn't an interactive terminal:" >&2
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
echo " ${conflict_keys[$i]}: ${conflict_old[$i]} (current) vs ${conflict_new[$i]} (.env.example)" >&2
|
||||
done
|
||||
echo "Re-run interactively to choose per-key, or pass --force to accept every new value." >&2
|
||||
exit 1
|
||||
else
|
||||
gum_bin="$(ensure_gum)"
|
||||
labels=()
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
labels+=("${conflict_keys[$i]}: ${conflict_old[$i]} -> ${conflict_new[$i]}")
|
||||
done
|
||||
if [ -n "$gum_bin" ]; then
|
||||
selected="$(printf '%s\n' "${labels[@]}" | "$gum_bin" choose --no-limit --selected "$(printf '%s\n' "${labels[@]}" | paste -sd,)" --header "Config differs from .env.example — selected keys take the new value, unselected keep the server's current value:")"
|
||||
else
|
||||
# ponytail: plain-bash fallback if gum couldn't be fetched (offline,
|
||||
# unsupported arch) — same one-screen-of-conflicts idea, cruder UI.
|
||||
echo "Config differs from .env.example. Enter space-separated numbers to KEEP the server's current value (all others take the new value), or press enter to take every new value:"
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
echo " $((i+1))) ${labels[$i]}"
|
||||
done
|
||||
read -r -p "> " keep_nums
|
||||
selected=""
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
case " $keep_nums " in
|
||||
*" $((i+1)) "*) ;;
|
||||
*) selected="${selected}${labels[$i]}"$'\n' ;;
|
||||
esac
|
||||
done
|
||||
fi
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
|
||||
if printf '%s\n' "$selected" | grep -qxF "${labels[$i]}"; then
|
||||
sed -i "s|^${key}=.*|${key}=${new}|" .env
|
||||
echo "${key}: ${conflict_old[$i]} -> ${new}"
|
||||
else
|
||||
echo "${key}: kept ${conflict_old[$i]} (server value)"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
fi
|
||||
|
||||
# Handles all three cases: the KEY=value line is missing entirely (.env
|
||||
# predates that var being added to .env.example — sed can't fix what isn't
|
||||
# there, so this appends it), present but blank, or already set.
|
||||
@@ -44,6 +183,7 @@ set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
|
||||
set_if_blank NEO4J_PASSWORD "$(openssl rand -hex 16)"
|
||||
|
||||
echo "==> resolving SEARXNG_LAN_IP"
|
||||
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
|
||||
@@ -81,9 +221,6 @@ else
|
||||
echo "HOST_RENDER_GID: no 'render' group on this host, set it manually if still blank."
|
||||
fi
|
||||
|
||||
echo "==> git pull"
|
||||
git pull --ff-only
|
||||
|
||||
echo "==> validating compose config"
|
||||
docker compose config -q
|
||||
|
||||
|
||||
BIN
Binary file not shown.
Reference in New Issue
Block a user