diff --git a/.env.example b/.env.example index 179558c..4899635 100644 --- a/.env.example +++ b/.env.example @@ -39,9 +39,13 @@ LAZYTAINER_INACTIVE_TIMEOUT=900 SEARXNG_LAN_IP= # --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) --- -# API_PORT is the only port published to the host/internet (reverse-proxied -# by NPM) — the dashboard (DASHBOARD_PORT) is never published, see -# docker-compose.yml's omniroute service comment. +# OMNIROUTE_PORT is the host-published port (reverse-proxied by NPM) — kept +# at 4000, same as the old LiteLLM setup, so existing NPM/firewall config +# doesn't need to change. It's mapped via plain Docker port publishing onto +# API_PORT, omniroute's own container-internal port (left at its default, +# not reconfigured to match). The dashboard (DASHBOARD_PORT) is never +# published at all — see docker-compose.yml's omniroute service comment. +OMNIROUTE_PORT=4000 OMNIROUTE_API_PORT=20129 OMNIROUTE_DASHBOARD_PORT=20128 # Random values, filled in automatically by ./scripts/update.sh — leave diff --git a/README.md b/README.md index 50d2337..f9d136b 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload API keys and usage tracking. As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) this is [OmniRoute](https://github.com/diegosouzapw/OmniRoute), replacing the original LiteLLM setup. `./scripts/update.sh` handles most of OmniRoute's secrets (see `.env.example`); per-workload API keys still need minting by hand in the dashboard — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). -- Gateway API: `http://:${OMNIROUTE_API_PORT:-20129}/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md). +- Gateway API: `http://:${OMNIROUTE_PORT:-4000}/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md). - Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`. - Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). diff --git a/docker-compose.yml b/docker-compose.yml index 4281c3c..b353b74 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -72,9 +72,10 @@ services: env_file: .env environment: # Split-port mode: dashboard and API are fully separate ports (unlike - # LiteLLM's single :4000 for both /v1 and /ui) — only API_PORT is - # published below, so the dashboard has no network route in from - # outside this container at all. No NPM path-deny rule needed. + # LiteLLM's single :4000 for both /v1 and /ui) — only API_PORT (via + # the OMNIROUTE_PORT:4000 host mapping below) is published, so the + # dashboard has no network route in from outside this container at + # all. No NPM path-deny rule needed. - API_HOST=0.0.0.0 - API_PORT=${OMNIROUTE_API_PORT:-20129} - DASHBOARD_PORT=${OMNIROUTE_DASHBOARD_PORT:-20128} @@ -87,8 +88,13 @@ services: # network can't resolve search.home on its own. extra_hosts: - "search.home:${SEARXNG_LAN_IP}" + # OMNIROUTE_PORT (host) is decoupled from API_PORT (container-internal, + # left at omniroute's own default) via plain Docker port mapping — the + # gateway needs to stay reachable at the same :4000 this stack has + # always published (NPM/firewall already point there), without having + # to also change omniroute's own internal port config to match. ports: - - "${OMNIROUTE_API_PORT:-20129}:${OMNIROUTE_API_PORT:-20129}" + - "${OMNIROUTE_PORT:-4000}:${OMNIROUTE_API_PORT:-20129}" restart: unless-stopped networks: [ai-stack] # ponytail: TCP-connect check, not an HTTP /healthz GET — the image has diff --git a/docs/coding-cli-setup.md b/docs/coding-cli-setup.md index 78413c8..5d5b8e4 100644 --- a/docs/coding-cli-setup.md +++ b/docs/coding-cli-setup.md @@ -2,8 +2,8 @@ This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes: -- **OpenAI-compatible**: `http://:${OMNIROUTE_API_PORT:-20129}/v1` -- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://:${OMNIROUTE_API_PORT:-20129}` +- **OpenAI-compatible**: `http://:${OMNIROUTE_PORT:-4000}/v1` +- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://:${OMNIROUTE_PORT:-4000}` Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks. @@ -18,7 +18,7 @@ Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, register Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly: ```bash -export ANTHROPIC_BASE_URL=http://:${OMNIROUTE_API_PORT:-20129} +export ANTHROPIC_BASE_URL=http://:${OMNIROUTE_PORT:-4000} export ANTHROPIC_API_KEY= claude ``` @@ -32,7 +32,7 @@ Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in ```toml [providers.openai] type = "openai" -base_url = "http://:${OMNIROUTE_API_PORT:-20129}/v1" +base_url = "http://:${OMNIROUTE_PORT:-4000}/v1" api_key = "" ``` @@ -57,7 +57,7 @@ curl -fsSL https://opencode.ai/install | bash "npm": "@ai-sdk/openai-compatible", "name": "AI proxy (local)", "options": { - "baseURL": "http://:${OMNIROUTE_API_PORT:-20129}/v1", + "baseURL": "http://:${OMNIROUTE_PORT:-4000}/v1", "apiKey": "" }, "models": { @@ -84,8 +84,8 @@ Select the model with `aiproxy/qwen3.8-27b-local`. | CLI | Wire format | Endpoint | Config | |---|---|---|---| -| Claude Code | Anthropic Messages | `http://:${OMNIROUTE_API_PORT:-20129}` | `ANTHROPIC_BASE_URL` env var | -| Kimi CLI | OpenAI Chat Completions | `http://:${OMNIROUTE_API_PORT:-20129}/v1` | `config.toml` provider block | -| OpenCode | OpenAI Chat Completions | `http://:${OMNIROUTE_API_PORT:-20129}/v1` | `opencode.json` provider block | +| Claude Code | Anthropic Messages | `http://:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var | +| Kimi CLI | OpenAI Chat Completions | `http://:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block | +| OpenCode | OpenAI Chat Completions | `http://:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block | Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`. diff --git a/docs/network-access.md b/docs/network-access.md index 13e9d82..4f9e4f3 100644 --- a/docs/network-access.md +++ b/docs/network-access.md @@ -12,7 +12,7 @@ If you later want external CLI access too, that's a deliberate scope change — As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute: -- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `ai.home` internal-only, `ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual). +- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_PORT:-4000}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `ai.home` internal-only, `ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual). - The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward. **Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname. diff --git a/docs/proxy-key-onboarding.md b/docs/proxy-key-onboarding.md index f46da75..d3abfb3 100644 --- a/docs/proxy-key-onboarding.md +++ b/docs/proxy-key-onboarding.md @@ -6,7 +6,7 @@ No workload in this stack itself needs a key right now — every client is exter ## Create the key -1. Log into the omniroute dashboard (`http://:${OMNIROUTE_DASHBOARD_PORT:-20128}` — LAN/host-only, never published publicly, see `docker-compose.yml`'s `omniroute` service). +1. Log into the omniroute dashboard. `DASHBOARD_PORT` (20128) is never published to the host (see `docker-compose.yml`'s `omniroute` service) — from the R9700 box itself, find the container's own address (`docker inspect -f '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`) and browse to `http://:20128` (the host can reach a container's bridge-network IP directly, published port or not). From elsewhere, SSH port-forward instead: `ssh -L 20128::20128 `, then browse `http://localhost:20128`. 2. "Keys" → "Create API key". 3. Label it `-` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `claude-code-cli`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync. 4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.