fix(omniroute): keep the gateway published on host port 4000
Not omniroute's own internal port (API_PORT stays at its default 20129, unreconfigured) - just the Docker port mapping, so existing NPM/firewall config pointed at :4000 keeps working without changes on that end. New OMNIROUTE_PORT env var is the host side of "OMNIROUTE_PORT:API_PORT" in docker-compose.yml's ports: entry. Also corrected docs/proxy-key-onboarding.md's dashboard-access instructions - DASHBOARD_PORT was never published to the host in the first place, so "http://<host>:20128" was never actually reachable as written; documented reaching it via the container's own bridge-network IP or an SSH port-forward instead. llama-server remains unexposed (no ports: entry, only expose:) - unaffected by this change, confirming it stays that way. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
@@ -2,8 +2,8 @@
|
||||
|
||||
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
|
||||
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}`
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
|
||||
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
|
||||
|
||||
@@ -18,7 +18,7 @@ Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, register
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_API_PORT:-20129}
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
|
||||
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
|
||||
claude
|
||||
```
|
||||
@@ -32,7 +32,7 @@ Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in
|
||||
```toml
|
||||
[providers.openai]
|
||||
type = "openai"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
|
||||
api_key = "<kimi-cli virtual key>"
|
||||
```
|
||||
|
||||
@@ -57,7 +57,7 @@ curl -fsSL https://opencode.ai/install | bash
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "AI proxy (local)",
|
||||
"options": {
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1",
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
|
||||
"apiKey": "<opencode-cli virtual key>"
|
||||
},
|
||||
"models": {
|
||||
@@ -84,8 +84,8 @@ Select the model with `aiproxy/qwen3.8-27b-local`.
|
||||
|
||||
| CLI | Wire format | Endpoint | Config |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `opencode.json` provider block |
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
|
||||
|
||||
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
|
||||
|
||||
@@ -12,7 +12,7 @@ If you later want external CLI access too, that's a deliberate scope change —
|
||||
|
||||
As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute:
|
||||
|
||||
- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `ai.home` internal-only, `ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual).
|
||||
- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_PORT:-4000}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `ai.home` internal-only, `ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual).
|
||||
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
|
||||
|
||||
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname.
|
||||
|
||||
@@ -6,7 +6,7 @@ No workload in this stack itself needs a key right now — every client is exter
|
||||
|
||||
## Create the key
|
||||
|
||||
1. Log into the omniroute dashboard (`http://<host>:${OMNIROUTE_DASHBOARD_PORT:-20128}` — LAN/host-only, never published publicly, see `docker-compose.yml`'s `omniroute` service).
|
||||
1. Log into the omniroute dashboard. `DASHBOARD_PORT` (20128) is never published to the host (see `docker-compose.yml`'s `omniroute` service) — from the R9700 box itself, find the container's own address (`docker inspect -f '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`) and browse to `http://<that-ip>:20128` (the host can reach a container's bridge-network IP directly, published port or not). From elsewhere, SSH port-forward instead: `ssh -L 20128:<container-ip>:20128 <host>`, then browse `http://localhost:20128`.
|
||||
2. "Keys" → "Create API key".
|
||||
3. Label it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `claude-code-cli`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync.
|
||||
4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.
|
||||
|
||||
Reference in New Issue
Block a user