LLM-Server/docs/gateway.md

Gateway (OmniRoute)

Source of truth: docker-compose.yml → omniroute service, and .env.example → OmniRoute section. Provider connections and API keys live in OmniRoute's own database, managed through its dashboard, not in this repo. This page explains why and how; values live in those sources. If they disagree, the source wins.

Terms as defined in CONTEXT.md. How the gateway fits the rest of the stack: architecture.md.

Dashboard

Browse to http://<server>:<OMNIROUTE_DASHBOARD_PORT> from the LAN. The dashboard isn't routed to the internet: see network access → Gateway. For the first login, use OMNIROUTE_INITIAL_PASSWORD from the server's .env, then change the password in the dashboard.

Provider connections

Each backend is registered once in the dashboard (Providers → add a llama.cpp provider) or via POST /api/providers. The base URL is http://<service>:<container port> on the stack network, using the service name and port from compose. None of this is checked into the repo. These docs assume the main model is registered as qwen3.8-27b-local.

• OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS / OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS have to be on, because backend URLs are private addresses. They relax OmniRoute's SSRF guard, which normally blocks registering a provider at a private or local address. Anyone with dashboard access can therefore point the gateway at any host on the stack network or the LAN.
• Classifier connection: turn OmniRoute's memory-tool injection off for it. Injected tools break the classifier's output (research/omniroute-memory-tool-injection-breaks-classifier.md). Also give it a generous per-connection timeout (see Timeouts).
• Background on registering local backends: research/omniroute-qwen-websearch.md.

API keys

Every workload gets its own API key, so each has its own visible usage.

Create

1. Open the dashboard → "Keys" → "Create API key".
2. Label it <workload>-<purpose>, e.g. paperless-ocr, claude-code-cli. The dashboard lists keys by label, so the label list is the ledger. There's no separate tracking doc.
3. Copy the key value. OmniRoute shows it only once.

Keys can only be minted by hand for now. POST /api/keys needs a dashboard login session (ManagementSessionAuth), not a static bearer key, and that flow hasn't been verified against a live instance.

Hand over: put the key in that workload's own config or .env, never in this repo. A key used for web search needs search permission, not just chat completions.

Revoke ("Keys" → delete) when the workload is retired or the key may have leaked, then remove it from the workload's config. There's no scheduled rotation.

Timeouts

A slow request can be cut off at several layers. Each setting is named below, and its value lives where the list says:

• STREAM_IDLE_TIMEOUT_MS (.env.example): the longest allowed gap between SSE chunks once streaming has started.
• REQUEST_TIMEOUT_MS (.env.example): the wait budget for the first non-ping SSE event. Retries and cooldowns eat into it, so the effective deadline is whatever budget remains, not a flat timer. See research/omniroute-non-ping-sse-stream-timeout.md.
• Per-connection providerSpecificData.timeoutMs (dashboard only, or PATCH /api/providers/{id}): bounds a request that is waiting behind the backend's busy slots.
• Per-connection concurrency semaphore: a hardcoded reject once more requests queue than the connection's maxConcurrent. It is not configurable, only avoidable: research/omniroute-account-semaphore-timeout.md.
• OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS (not set in this repo): research/omniroute-direct-response-timeout-outage-2026-09-15.md.

These are raised above OmniRoute's defaults because contended prefill on the main backend can take a long time before the first token (architecture → llama-server).

Web search

SearXNG runs elsewhere on the LAN as search.home. It's registered as a search provider connection and exposed through the gateway's /v1/search endpoint. The container can't resolve search.home (it's a LAN DNS name), so compose adds an extra_hosts entry from SEARXNG_LAN_IP, which update.sh resolves on the server (operations → Host-resolved values).

• Wiring it into Qwen Code: coding-cli-setup/qwen-code.md.
• Research: research/omniroute-qwen-websearch.md (current setup), research/dashscope-websearch-selfhost-options.md (why DashScope isn't emulated).

Upstream caution

OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md and PUBLIC_CREDS.md in its repo) describe shipped features for:

• evading AI-provider client detection
• intercepting HTTPS system-wide via a locally installed root CA
• hiding credentials from secret scanners

This stack's configuration uses none of them. They are a real characteristic of the upstream project, though, so weigh them before extending this integration.