OmniRoute: per-workload virtual keys, admin UI, and deployment shape (Docker/compose) #36

New Issue

Closed

opened 2026-09-03 17:18:10 +00:00 by haylan · 3 comments

haylan commented 2026-09-03 17:18:10 +00:00

Owner

Copy Link

Copy Source

Part of #31

Question

Current setup: LiteLLM issues a separate virtual key per workload (Open WebUI, coding CLIs, litellm-pgvector's own callback) minted via its /key/generate API, automated in scripts/update.sh; an admin UI (/ui, LAN-only) manages keys/budgets — see docs/proxy-key-onboarding.md, docs/network-access.md.

For OmniRoute, determine:

• Does it support per-workload API-key scoping the same way (README mentions "read/write/admin tiers" and "virtual key pools") — is there a mintable-via-API equivalent to LiteLLM's /key/generate that scripts/update.sh could call the same way?
• Does it ship an admin UI, and can it be kept LAN-only the way LiteLLM's /ui is (see docs/network-access.md's NPM-path-denial setup)?
• What's the actual deployment shape for this Docker-compose-based stack — is there an official Docker image (README mentions multi-arch AMD64+ARM64 images), what config file format does it use in place of litellm-config.yaml, and what env vars/secrets does scripts/update.sh would need to generate or mint (replacing LITELLM_MASTER_KEY/LITELLM_SALT_KEY/etc.)?

Answer is what the eventual docker-compose.yml/config rewrite is built from.

Part of #31 ## Question Current setup: LiteLLM issues a separate virtual key per workload (Open WebUI, coding CLIs, litellm-pgvector's own callback) minted via its `/key/generate` API, automated in `scripts/update.sh`; an admin UI (`/ui`, LAN-only) manages keys/budgets — see `docs/proxy-key-onboarding.md`, `docs/network-access.md`. For OmniRoute, determine: - Does it support per-workload API-key scoping the same way (README mentions "read/write/admin tiers" and "virtual key pools") — is there a mintable-via-API equivalent to LiteLLM's `/key/generate` that `scripts/update.sh` could call the same way? - Does it ship an admin UI, and can it be kept LAN-only the way LiteLLM's `/ui` is (see `docs/network-access.md`'s NPM-path-denial setup)? - What's the actual deployment shape for this Docker-compose-based stack — is there an official Docker image (README mentions multi-arch AMD64+ARM64 images), what config file format does it use in place of `litellm-config.yaml`, and what env vars/secrets does `scripts/update.sh` would need to generate or mint (replacing `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY`/etc.)? Answer is what the eventual docker-compose.yml/config rewrite is built from.

haylan added the wayfinder:research label 2026-09-03 17:18:11 +00:00

haylan self-assigned this 2026-09-03 17:20:41 +00:00

haylan commented 2026-09-03 17:21:12 +00:00

Author

Owner

Copy Link

Copy Source

Findings (from github.com/diegosouzapw/OmniRoute, checked out at release/v3.8.51, its default branch)

1. Per-workload key scoping / mintable-via-API equivalent to /key/generate: yes, close equivalent exists.

• REST API: POST /api/keys — "Create API key" (response includes the full key value), plus GET /api/keys, GET/PATCH/DELETE /api/keys/{id} (per docs/openapi.yaml). This is a workable drop-in for what scripts/update.sh does today with LiteLLM's /key/generate.
• CLI equivalent also exists: omniroute tokens create --name ci --scope read (README, "Remote Mode" section).
• Scope granularity is murkier than LiteLLM's budget/alias model. The README's "read/write/admin tiers" claim isn't backed up in the security docs I could find: docs/security/ROUTE_GUARD_TIERS.md only documents route-level tiers (LOCAL_ONLY, ALWAYS_PROTECTED, MANAGEMENT) plus two named scopes (manage, mcp:connect) — not a documented read/write/admin key-scope contract. docs/security/CLI_TOKEN.md only says "Remote contexts use scoped access tokens instead" with no worked example. So the mechanism (POST /api/keys) is there; the exact scope/permission model scripts/update.sh would need to target isn't concretely documented — expect to reverse-engineer it from the OpenAPI schema at implementation time.

2. Admin UI: yes, and it can plausibly be kept LAN-only — differently from LiteLLM.

• Dashboard serves on http://localhost:20128 by default, same port as the API (single-port mode).
• Unlike LiteLLM (which needs NPM path-denial because /ui and the API share one port with no split), OmniRoute has an explicit split-port mode: env vars API_PORT, DASHBOARD_PORT, API_HOST — "serve Dashboard and API on separate ports for network isolation" (.env.example). That's actually a cleaner LAN-only story: bind/expose only API_PORT externally and never publish DASHBOARD_PORT, no NPM path-deny rule needed. Management endpoints also support requireLogin gating per the OpenAPI overview.

3. Deployment shape:

• Official Docker image: diegosouzapw/omniroute, multi-arch AMD64+ARM64 as the README claims. Only a bare docker run example is documented anywhere I found (docs/getting-started/QUICK-START.md, README) — no docker-compose.yml example ships in the repo or docs, so the compose stanza will need to be written from scratch (image + port(s) + volume + env file), not adapted from an upstream example.
• Config file format: there is no litellm-config.yaml equivalent. Provider/routing config is done through the web dashboard (Providers → Add Provider) at runtime, persisted into its own SQLite DB (/app/data), not a checked-in static config file. All install-time config is via .env.
• Env vars/secrets replacing LITELLM_MASTER_KEY/LITELLM_SALT_KEY (from .env.example):
  • INITIAL_PASSWORD — initial dashboard admin password, bootstrap-only, closest analog to a master credential.
  • JWT_SECRET — signs dashboard session cookies.
  • API_KEY_SECRET — encrypts API keys at rest in SQLite.
  • STORAGE_ENCRYPTION_KEY — encrypts the SQLite DB at rest.
  • MACHINE_ID_SALT and OMNIROUTE_CLI_SALT — salts for machine-id fingerprinting / CLI auth token derivation (closest analog to LITELLM_SALT_KEY).
  • So scripts/update.sh would need to generate/persist 5 secrets on first run instead of LiteLLM's 2.

Caveat worth flagging back on #31

This repo does not read as a normal small-team OSS gateway: default branch release/v3.8.51, claimed 60k+ stars / 550+ contributors for a repo created 2026-02-13, a 511MB repo, and a docs tree with entries like STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md, DEVIN_CLAUDE_BRIDGE.md. Combined with #31's already-recorded caution (README name-dropping this environment's own tooling in agent-credential-control marketing copy), this reinforces treating it as unverified/unusual rather than a mature drop-in LiteLLM replacement — worth a trust/provenance check before wiring in real config, independent of the technical shape answered above.

Sources checked directly: README.md, docs/openapi.yaml, docs/getting-started/QUICK-START.md, docs/security/CLI_TOKEN.md, docs/security/ROUTE_GUARD_TIERS.md, .env.example — all at raw.githubusercontent.com/diegosouzapw/OmniRoute/release/v3.8.51/…

## Findings (from github.com/diegosouzapw/OmniRoute, checked out at `release/v3.8.51`, its default branch) **1. Per-workload key scoping / mintable-via-API equivalent to `/key/generate`: yes, close equivalent exists.** - REST API: `POST /api/keys` — "Create API key" (response includes the full key value), plus `GET /api/keys`, `GET/PATCH/DELETE /api/keys/{id}` (per `docs/openapi.yaml`). This is a workable drop-in for what `scripts/update.sh` does today with LiteLLM's `/key/generate`. - CLI equivalent also exists: `omniroute tokens create --name ci --scope read` (README, "Remote Mode" section). - Scope granularity is murkier than LiteLLM's budget/alias model. The README's "read/write/admin tiers" claim isn't backed up in the security docs I could find: `docs/security/ROUTE_GUARD_TIERS.md` only documents *route*-level tiers (`LOCAL_ONLY`, `ALWAYS_PROTECTED`, `MANAGEMENT`) plus two named scopes (`manage`, `mcp:connect`) — not a documented read/write/admin key-scope contract. `docs/security/CLI_TOKEN.md` only says "Remote contexts use scoped access tokens instead" with no worked example. So the *mechanism* (`POST /api/keys`) is there; the exact scope/permission model scripts/update.sh would need to target isn't concretely documented — expect to reverse-engineer it from the OpenAPI schema at implementation time. **2. Admin UI: yes, and it can plausibly be kept LAN-only — differently from LiteLLM.** - Dashboard serves on `http://localhost:20128` by default, same port as the API (single-port mode). - Unlike LiteLLM (which needs NPM path-denial because `/ui` and the API share one port with no split), OmniRoute has an explicit **split-port mode**: env vars `API_PORT`, `DASHBOARD_PORT`, `API_HOST` — "serve Dashboard and API on separate ports for network isolation" (`.env.example`). That's actually a cleaner LAN-only story: bind/expose only `API_PORT` externally and never publish `DASHBOARD_PORT`, no NPM path-deny rule needed. Management endpoints also support `requireLogin` gating per the OpenAPI overview. **3. Deployment shape:** - Official Docker image: `diegosouzapw/omniroute`, multi-arch AMD64+ARM64 as the README claims. Only a bare `docker run` example is documented anywhere I found (`docs/getting-started/QUICK-START.md`, README) — **no `docker-compose.yml` example ships in the repo or docs**, so the compose stanza will need to be written from scratch (image + port(s) + volume + env file), not adapted from an upstream example. - Config file format: there is no `litellm-config.yaml` equivalent. Provider/routing config is done through the web dashboard (Providers → Add Provider) at runtime, persisted into its own SQLite DB (`/app/data`), not a checked-in static config file. All install-time config is via `.env`. - Env vars/secrets replacing `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` (from `.env.example`): - `INITIAL_PASSWORD` — initial dashboard admin password, bootstrap-only, closest analog to a master credential. - `JWT_SECRET` — signs dashboard session cookies. - `API_KEY_SECRET` — encrypts API keys at rest in SQLite. - `STORAGE_ENCRYPTION_KEY` — encrypts the SQLite DB at rest. - `MACHINE_ID_SALT` and `OMNIROUTE_CLI_SALT` — salts for machine-id fingerprinting / CLI auth token derivation (closest analog to `LITELLM_SALT_KEY`). - So `scripts/update.sh` would need to generate/persist 5 secrets on first run instead of LiteLLM's 2. ## Caveat worth flagging back on #31 This repo does not read as a normal small-team OSS gateway: default branch `release/v3.8.51`, claimed 60k+ stars / 550+ contributors for a repo created 2026-02-13, a 511MB repo, and a docs tree with entries like `STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md`, `DEVIN_CLAUDE_BRIDGE.md`. Combined with #31's already-recorded caution (README name-dropping this environment's own tooling in agent-credential-control marketing copy), this reinforces treating it as unverified/unusual rather than a mature drop-in LiteLLM replacement — worth a trust/provenance check before wiring in real config, independent of the technical shape answered above. Sources checked directly: README.md, docs/openapi.yaml, docs/getting-started/QUICK-START.md, docs/security/CLI_TOKEN.md, docs/security/ROUTE_GUARD_TIERS.md, .env.example — all at raw.githubusercontent.com/diegosouzapw/OmniRoute/release/v3.8.51/…

haylan closed this issue 2026-09-03 17:21:22 +00:00

haylan referenced this issue 2026-09-03 17:21:52 +00:00

Migrate the AI gateway from LiteLLM to OmniRoute #31

haylan commented 2026-09-03 17:28:21 +00:00

Author

Owner

Copy Link

Copy Source

Correction: the earlier resolution comment's claim that this repo's docs include STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, and PUBLIC_CREDS.md does not hold up. Verified directly:

• A full, non-truncated repo tree listing (GET /repos/diegosouzapw/OmniRoute/git/trees/main?recursive=1, 1,077 entries) contains no file matching STEALTH, MITM, TPROXY, DECRYPT, or PUBLIC_CREDS anywhere.
• raw.githubusercontent.com/.../STEALTH_GUIDE.md and .../MITM-TPROXY-DECRYPT.md both 404.
• Control test: raw.githubusercontent.com/.../docs/security/SOCKET_DEV_FINDINGS.md (a real file this and ticket #35 both cite) fetches fine via the identical method — so the 404s are real, not a fetch-method problem.

This was very likely WebFetch summarizer hallucination — the same failure mode ticket #32's resolution explicitly caught and cross-checked around ("14 local providers" that also wasn't real). The genuinely-verified findings from #33 (Socket.dev-flagged, maintainer-attested, real credential-handling CVEs patched in 3.8.5→3.8.6; root-CA/MITM-install and keychain-import as documented opt-in capabilities) still stand on their own and don't need embellishment.

**Correction**: the earlier resolution comment's claim that this repo's docs include `STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, and `PUBLIC_CREDS.md` does not hold up. Verified directly: - A full, non-truncated repo tree listing (`GET /repos/diegosouzapw/OmniRoute/git/trees/main?recursive=1`, 1,077 entries) contains no file matching STEALTH, MITM, TPROXY, DECRYPT, or PUBLIC_CREDS anywhere. - `raw.githubusercontent.com/.../STEALTH_GUIDE.md` and `.../MITM-TPROXY-DECRYPT.md` both 404. - Control test: `raw.githubusercontent.com/.../docs/security/SOCKET_DEV_FINDINGS.md` (a real file this and ticket #35 both cite) fetches fine via the identical method — so the 404s are real, not a fetch-method problem. This was very likely WebFetch summarizer hallucination — the same failure mode ticket #32's resolution explicitly caught and cross-checked around ("14 local providers" that also wasn't real). The genuinely-verified findings from #33 (Socket.dev-flagged, maintainer-attested, real credential-handling CVEs patched in 3.8.5→3.8.6; root-CA/MITM-install and keychain-import as documented opt-in capabilities) still stand on their own and don't need embellishment.

haylan commented 2026-09-03 17:33:00 +00:00

Author

Owner

Copy Link

Copy Source

Second correction — the previous correction was wrong. The files exist. I'd checked the `main` branch; the original research (and this repo's actual release line) is on `release/v3.8.51`. Confirmed via `GET /repos/diegosouzapw/OmniRoute/contents/docs/security?ref=release/v3.8.51`: `STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, and `PUBLIC_CREDS.md` are all present (alongside `AGENTROUTER_WAF.md`, `SUPPLY_CHAIN.md`, `BAN_DETECTION.md`), and `docs/DEVIN_CLAUDE_BRIDGE.md` also exists. Read all three directly:

• STEALTH_GUIDE.md: client-fingerprint spoofing to impersonate official AI CLI tools (Claude Code named specifically, plus Codex/Cursor/Kimi/GitHub) against provider-side detection — fake user-agents/headers, reimplemented integrity-check hashing, zero-width-joiner obfuscation of client identity — plus a MITM proxy against Google's Antigravity backend explicitly to route around free-tier quota enforcement. Its own text: the credit-override env var "is the single most common ToS-violation report we see."
• MITM-TPROXY-DECRYPT.md: root-privileged kernel TPROXY interception decrypting all outbound HTTPS traffic on the host (not scoped to AI providers), via a dynamically-generated 10-year root CA installed into the system trust store.
• PUBLIC_CREDS.md: XOR-masks embedded OAuth/API credentials specifically to defeat GitHub Secret Scanning and Semgrep pattern matching.

This is a materially different finding than "has some disclosed, patched vulnerabilities" — these are documented, shipped features whose purpose is defeating detection (provider-side client detection, OS trust-store/traffic interception, and static secret-scanners), not incidental risk. Recommending against integrating this project into the stack at all, independent of the earlier technical-fit findings (#32/#35/this ticket) which remain accurate on their own terms.

**Second correction — the previous correction was wrong.** The files exist. I'd checked the \`main\` branch; the original research (and this repo's actual release line) is on \`release/v3.8.51\`. Confirmed via \`GET /repos/diegosouzapw/OmniRoute/contents/docs/security?ref=release/v3.8.51\`: \`STEALTH_GUIDE.md\`, \`MITM-TPROXY-DECRYPT.md\`, and \`PUBLIC_CREDS.md\` are all present (alongside \`AGENTROUTER_WAF.md\`, \`SUPPLY_CHAIN.md\`, \`BAN_DETECTION.md\`), and \`docs/DEVIN_CLAUDE_BRIDGE.md\` also exists. Read all three directly: - **STEALTH_GUIDE.md**: client-fingerprint spoofing to impersonate official AI CLI tools (Claude Code named specifically, plus Codex/Cursor/Kimi/GitHub) against provider-side detection — fake user-agents/headers, reimplemented integrity-check hashing, zero-width-joiner obfuscation of client identity — plus a MITM proxy against Google's Antigravity backend explicitly to route around free-tier quota enforcement. Its own text: the credit-override env var "is the single most common ToS-violation report we see." - **MITM-TPROXY-DECRYPT.md**: root-privileged kernel TPROXY interception decrypting *all* outbound HTTPS traffic on the host (not scoped to AI providers), via a dynamically-generated 10-year root CA installed into the system trust store. - **PUBLIC_CREDS.md**: XOR-masks embedded OAuth/API credentials specifically to defeat GitHub Secret Scanning and Semgrep pattern matching. This is a materially different finding than "has some disclosed, patched vulnerabilities" — these are documented, shipped features whose purpose is defeating detection (provider-side client detection, OS trust-store/traffic interception, and static secret-scanners), not incidental risk. Recommending against integrating this project into the stack at all, independent of the earlier technical-fit findings (#32/#35/this ticket) which remain accurate on their own terms.

haylan referenced this issue 2026-09-03 17:38:36 +00:00

OmniRoute: docker-compose service design and secrets/config plan #37

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

tool-cassefier-model

feat-rag-databases

No results found.

Labels

Clear labels

wayfinder:grilling

Wayfinder grilling ticket

wayfinder:map

Wayfinder map issue

wayfinder:prototype

Wayfinder prototype ticket

wayfinder:research

Wayfinder research ticket

wayfinder:task

Wayfinder task ticket

No labels wayfinder:research

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

haylan (Arthur Erlich)

No Assignees haylan

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: haylan/LLM-Server#36