haylanandClaude Bot
a0e5e9704b
- CONTEXT.md glossary - README reduced to intro, quickstart and doc index - new architecture.md, gateway.md (from proxy-key-onboarding.md), operations.md; network-access.md rewritten - coding-cli-setup: stale OMNIROUTE_PORT:-4000 endpoint, terms and duplicated per-slot/classifier explanations fixed - research/README.md index; proxy-request-priority.md moved there as superseded Every doc names keys instead of restating values and links its source. Refs #55 (#56, #57, #59, #60) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
63 lines
5.6 KiB
Markdown
63 lines
5.6 KiB
Markdown
# Gateway (OmniRoute)
|
|
|
|
> **Source of truth:** [`docker-compose.yml`](../docker-compose.yml) → `omniroute` service, and [`.env.example`](../.env.example) → OmniRoute section. Provider connections and API keys live in OmniRoute's own database, managed through its dashboard, not in this repo. This page explains *why* and *how*; values live in those sources. If they disagree, the source wins.
|
|
|
|
Terms as defined in [`CONTEXT.md`](../CONTEXT.md). How the gateway fits the rest of the stack: [`architecture.md`](architecture.md#omniroute).
|
|
|
|
## Dashboard
|
|
|
|
Browse to `http://<server>:<OMNIROUTE_DASHBOARD_PORT>` from the LAN. The dashboard isn't routed to the internet: see [network access → Gateway](network-access.md#gateway). For the first login, use `OMNIROUTE_INITIAL_PASSWORD` from the server's `.env`, then change the password in the dashboard.
|
|
|
|
## Provider connections
|
|
|
|
Each backend is registered once in the dashboard (Providers → add a **llama.cpp** provider) or via `POST /api/providers`. The base URL is `http://<service>:<container port>` on the stack network, using the service name and port from compose. None of this is checked into the repo. These docs assume the main model is registered as `qwen3.8-27b-local`.
|
|
|
|
- **`OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` / `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`** have to be on, because backend URLs are private addresses. They relax OmniRoute's SSRF guard, which normally blocks registering a provider at a private or local address. Anyone with dashboard access can therefore point the gateway at any host on the stack network or the LAN.
|
|
- **Classifier connection:** turn OmniRoute's memory-tool injection off for it. Injected tools break the classifier's output ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)). Also give it a generous per-connection timeout (see [Timeouts](#timeouts)).
|
|
- Background on registering local backends: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md).
|
|
|
|
## API keys
|
|
|
|
Every workload gets its own API key, so each has its own visible usage.
|
|
|
|
**Create**
|
|
|
|
1. [Open the dashboard](#dashboard) → "Keys" → "Create API key".
|
|
2. Label it `<workload>-<purpose>`, e.g. `paperless-ocr`, `claude-code-cli`. The dashboard lists keys by label, so the label list is the ledger. There's no separate tracking doc.
|
|
3. Copy the key value. OmniRoute shows it only once.
|
|
|
|
Keys can only be minted by hand for now. `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key, and that flow hasn't been verified against a live instance.
|
|
|
|
**Hand over:** put the key in that workload's own config or `.env`, never in this repo. A key used for [web search](#web-search) needs search permission, not just chat completions.
|
|
|
|
**Revoke** ("Keys" → delete) when the workload is retired or the key may have leaked, then remove it from the workload's config. There's no scheduled rotation.
|
|
|
|
## Timeouts
|
|
|
|
A slow request can be cut off at several layers. Each setting is named below, and its value lives where the list says:
|
|
|
|
- **`STREAM_IDLE_TIMEOUT_MS`** (`.env.example`): the longest allowed gap *between* SSE chunks once streaming has started.
|
|
- **`REQUEST_TIMEOUT_MS`** (`.env.example`): the wait budget for the *first* non-ping SSE event. Retries and cooldowns eat into it, so the effective deadline is whatever budget remains, not a flat timer. See [`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md).
|
|
- **Per-connection `providerSpecificData.timeoutMs`** (dashboard only, or `PATCH /api/providers/{id}`): bounds a request that is waiting behind the backend's busy slots.
|
|
- **Per-connection concurrency semaphore**: a hardcoded reject once more requests queue than the connection's `maxConcurrent`. It is not configurable, only avoidable: [`research/omniroute-account-semaphore-timeout.md`](research/omniroute-account-semaphore-timeout.md).
|
|
- **`OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`** (not set in this repo): [`research/omniroute-direct-response-timeout-outage-2026-09-15.md`](research/omniroute-direct-response-timeout-outage-2026-09-15.md).
|
|
|
|
These are raised above OmniRoute's defaults because contended prefill on the main backend can take a long time before the first token ([architecture → llama-server](architecture.md#llama-server)).
|
|
|
|
## Web search
|
|
|
|
SearXNG runs elsewhere on the LAN as `search.home`. It's registered as a search provider connection and exposed through the gateway's `/v1/search` endpoint. The container can't resolve `search.home` (it's a LAN DNS name), so compose adds an `extra_hosts` entry from `SEARXNG_LAN_IP`, which `update.sh` resolves on the server ([operations → Host-resolved values](operations.md#host-resolved-values)).
|
|
|
|
- Wiring it into Qwen Code: [`coding-cli-setup/qwen-code.md`](coding-cli-setup/qwen-code.md#web-search-via-omniroute).
|
|
- Research: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md) (current setup), [`research/dashscope-websearch-selfhost-options.md`](research/dashscope-websearch-selfhost-options.md) (why DashScope isn't emulated).
|
|
|
|
## Upstream caution
|
|
|
|
OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md` and `PUBLIC_CREDS.md` in [its repo](https://github.com/diegosouzapw/OmniRoute)) describe shipped features for:
|
|
|
|
- evading AI-provider client detection
|
|
- intercepting HTTPS system-wide via a locally installed root CA
|
|
- hiding credentials from secret scanners
|
|
|
|
This stack's configuration uses none of them. They are a real characteristic of the upstream project, though, so weigh them before extending this integration.
|