Migrate the AI gateway from LiteLLM to OmniRoute #31

New Issue

2026-09-03T17:17:29Z

haylan commented

2026-09-03 17:17:29 +00:00

Destination

Replace LiteLLM — and everything built on top of it (litellm-pgvector knowledgebase, SearXNG search wiring, per-workload virtual keys) — with OmniRoute as this stack's sole AI gateway. End state: docker-compose.yml, the gateway's config file, scripts/update.sh, and docs all point at OmniRoute instead of LiteLLM; no cloud provider API keys are configured (this stack only ever routes to local llama.cpp chat/embedding backends).

Notes

Domain docs: CONTEXT.md (if present) + docs/adr/, per docs/agents/domain.md.
Deploy convention (CLAUDE.md): any change to docker-compose.yml, a gateway config file, .env.example, or scripts/* must be committed/pushed, then applied via ./scripts/update.sh on the R9700 server — this map does not itself deploy.
Standing caution — upgraded to a recommendation against integrating this project. docs/security/ on OmniRoute's release/v3.8.51 branch (its actual release line — verified directly, contents API, not summarized) contains, and this session read verbatim: STEALTH_GUIDE.md (client-fingerprint spoofing to impersonate official AI CLI tools including Claude Code against provider-side detection, plus a MITM proxy against Google's own backend explicitly to route around free-tier quota enforcement — the doc's own words: this "is the single most common ToS-violation report we see"), MITM-TPROXY-DECRYPT.md (root-privileged kernel TPROXY interception decrypting all outbound HTTPS traffic on the host via a dynamically-installed root CA, not scoped to AI providers), and PUBLIC_CREDS.md (XOR-masks embedded credentials specifically to defeat GitHub Secret Scanning/Semgrep). These are documented, shipped features, not incidental bugs — their shared purpose is defeating detection. Combined with #33's real, patched-but-disclosed credential-handling CVEs, this project should not be integrated into this stack. (Session history note: this finding was first reported, then wrongly retracted as a hallucination after checking the wrong branch — main instead of release/v3.8.51 — then re-confirmed with direct verbatim reads. See #36's comments for the full back-and-forth if it matters later.)
Current baseline this replaces: AI gateway/proxy (#9), LiteLLM gateway search/knowledgebase/memory (#21).

Decisions so far

OmniRoute: can it route to arbitrary OpenAI-compatible local endpoints (llama-server), not just Ollama? — not Ollama-only: a named llama-cpp provider preset (configurable base URL, regression-tested) handles chat, and a generic openai-compatible-embeddings provider_node (tested working over LAN IPs) handles embeddings — register two connections mirroring LiteLLM's model_list + api_base.
OmniRoute due-diligence: maintainer, repo history, npm package trust — real, popular, actively-maintained project (60.8k★, 550+ contributors, MIT); had two real credential-handling vulnerabilities (Socket.dev-flagged, patched 3.8.5→3.8.6). Superseded by the Notes finding above: the project also ships detection-evasion features by design.
OmniRoute: memory/knowledgebase parity with litellm-pgvector — no parity (no fact-file ingestion, no file_search equivalent, no local-embedding-server hookup, decay is really just a 30-day retention cutoff not type-based, only external backend is Qdrant not pgvector) — keep vendor/litellm-pgvector.
OmniRoute: web-search tool equivalent to the SearXNG standalone endpoint — parity confirmed: native SearXNG provider behind a standalone /v1/search REST endpoint, independent of model tool-calling. Networking isn't drop-in: no documented LAN mDNS/extra_hosts handling, and OmniRoute's own SSRF/egress flags need setting to reach a private SearXNG instance.
OmniRoute: per-workload virtual keys, admin UI, and deployment shape (Docker/compose) — POST /api/keys is a workable /key/generate equivalent; dashboard supports split-port mode for a LAN-only admin UI; no official docker-compose example, config lives in a dashboard-managed SQLite DB not a static YAML file; 5 secrets replace LiteLLM's 2. Also the ticket that surfaced the STEALTH_GUIDE.md/MITM-TPROXY-DECRYPT.md/PUBLIC_CREDS.md finding — see Notes above.

Not yet specified

If this map is abandoned per the Notes recommendation: nothing further to specify — the map would close without a docker-compose/config redesign ticket ever being opened.
If overridden and pursued anyway: cutover/rollback plan, priority-scheduling equivalent, per-workload key minting automation against the underdocumented POST /api/keys scope model, Open WebUI integration changes, documentation updates, and the concrete docker-compose.yml/config design itself all remain unspecified.

Out of scope

OmniRoute's cloud-provider routing, free-tier provider pools, and token-compression marketplace — this stack has no cloud API keys and none are being added.

## Destination Replace LiteLLM — and everything built on top of it (litellm-pgvector knowledgebase, SearXNG search wiring, per-workload virtual keys) — with [OmniRoute](https://github.com/diegosouzapw/OmniRoute) as this stack's sole AI gateway. End state: `docker-compose.yml`, the gateway's config file, `scripts/update.sh`, and docs all point at OmniRoute instead of LiteLLM; no cloud provider API keys are configured (this stack only ever routes to local llama.cpp chat/embedding backends). ## Notes - Domain docs: `CONTEXT.md` (if present) + `docs/adr/`, per `docs/agents/domain.md`. - Deploy convention (`CLAUDE.md`): any change to `docker-compose.yml`, a gateway config file, `.env.example`, or `scripts/*` must be committed/pushed, then applied via `./scripts/update.sh` on the R9700 server — this map does not itself deploy. - **Standing caution — upgraded to a recommendation against integrating this project.** `docs/security/` on OmniRoute's `release/v3.8.51` branch (its actual release line — verified directly, contents API, not summarized) contains, and this session read verbatim: `STEALTH_GUIDE.md` (client-fingerprint spoofing to impersonate official AI CLI tools including Claude Code against provider-side detection, plus a MITM proxy against Google's own backend explicitly to route around free-tier quota enforcement — the doc's own words: this "is the single most common ToS-violation report we see"), `MITM-TPROXY-DECRYPT.md` (root-privileged kernel TPROXY interception decrypting *all* outbound HTTPS traffic on the host via a dynamically-installed root CA, not scoped to AI providers), and `PUBLIC_CREDS.md` (XOR-masks embedded credentials specifically to defeat GitHub Secret Scanning/Semgrep). These are documented, shipped features, not incidental bugs — their shared purpose is defeating detection. Combined with #33's real, patched-but-disclosed credential-handling CVEs, this project should not be integrated into this stack. (Session history note: this finding was first reported, then wrongly retracted as a hallucination after checking the wrong branch — `main` instead of `release/v3.8.51` — then re-confirmed with direct verbatim reads. See #36's comments for the full back-and-forth if it matters later.) - Current baseline this replaces: AI gateway/proxy (#9), LiteLLM gateway search/knowledgebase/memory (#21). ## Decisions so far - [OmniRoute: can it route to arbitrary OpenAI-compatible local endpoints (llama-server), not just Ollama?](https://git.arthurerlich.de/haylan/LLM-Server/issues/32) — not Ollama-only: a named `llama-cpp` provider preset (configurable base URL, regression-tested) handles chat, and a generic `openai-compatible-embeddings` provider_node (tested working over LAN IPs) handles embeddings — register two connections mirroring LiteLLM's `model_list` + `api_base`. - [OmniRoute due-diligence: maintainer, repo history, npm package trust](https://git.arthurerlich.de/haylan/LLM-Server/issues/33) — real, popular, actively-maintained project (60.8k★, 550+ contributors, MIT); had two real credential-handling vulnerabilities (Socket.dev-flagged, patched 3.8.5→3.8.6). Superseded by the Notes finding above: the project also ships detection-evasion features by design. - [OmniRoute: memory/knowledgebase parity with litellm-pgvector](https://git.arthurerlich.de/haylan/LLM-Server/issues/34) — no parity (no fact-file ingestion, no file_search equivalent, no local-embedding-server hookup, decay is really just a 30-day retention cutoff not type-based, only external backend is Qdrant not pgvector) — keep `vendor/litellm-pgvector`. - [OmniRoute: web-search tool equivalent to the SearXNG standalone endpoint](https://git.arthurerlich.de/haylan/LLM-Server/issues/35) — parity confirmed: native SearXNG provider behind a standalone `/v1/search` REST endpoint, independent of model tool-calling. Networking isn't drop-in: no documented LAN mDNS/`extra_hosts` handling, and OmniRoute's own SSRF/egress flags need setting to reach a private SearXNG instance. - [OmniRoute: per-workload virtual keys, admin UI, and deployment shape (Docker/compose)](https://git.arthurerlich.de/haylan/LLM-Server/issues/36) — `POST /api/keys` is a workable `/key/generate` equivalent; dashboard supports split-port mode for a LAN-only admin UI; no official docker-compose example, config lives in a dashboard-managed SQLite DB not a static YAML file; 5 secrets replace LiteLLM's 2. Also the ticket that surfaced the STEALTH_GUIDE.md/MITM-TPROXY-DECRYPT.md/PUBLIC_CREDS.md finding — see Notes above. ## Not yet specified - If this map is abandoned per the Notes recommendation: nothing further to specify — the map would close without a docker-compose/config redesign ticket ever being opened. - If overridden and pursued anyway: cutover/rollback plan, priority-scheduling equivalent, per-workload key minting automation against the underdocumented `POST /api/keys` scope model, Open WebUI integration changes, documentation updates, and the concrete docker-compose.yml/config design itself all remain unspecified. ## Out of scope - OmniRoute's cloud-provider routing, free-tier provider pools, and token-compression marketplace — this stack has no cloud API keys and none are being added.

haylan added the wayfinder:map label 2026-09-03 17:17:29 +00:00

haylan referenced this issue

2026-09-03 17:17:39 +00:00

OmniRoute: can it route to arbitrary OpenAI-compatible local endpoints (llama-server), not just Ollama? #32

haylan referenced this issue

2026-09-03 17:17:47 +00:00

OmniRoute due-diligence: maintainer, repo history, npm package trust #33

haylan referenced this issue

2026-09-03 17:17:58 +00:00

OmniRoute: memory/knowledgebase parity with litellm-pgvector #34

haylan referenced this issue

2026-09-03 17:18:05 +00:00