feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature

LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37):
replace the litellm/litellm-db services with omniroute, split-port mode
(API_PORT published/reverse-proxied, DASHBOARD_PORT never published -
tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place
of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as
omniroute providers post-boot (no static config.yaml equivalent). No
scripted per-workload key minting yet - omniroute's POST /api/keys needs a
dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is
a manual step for now (docs/proxy-key-onboarding.md).

Caveat carried into the map and README: OmniRoute's own docs
(docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md
on its release/v3.8.51 branch) describe shipped features for AI-provider
client-detection evasion, system-wide HTTPS interception via a locally
installed root CA, and hiding credentials from secret scanners. Proceeding
anyway was an explicit, informed user decision.

Also drops the gateway-level memory/knowledgebase feature entirely (user:
"I don't need it") - litellm-pgvector, pgvector-db, embedding-server,
scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory-
knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated
and untouched. litellm-config.yaml deleted (was kept as a rollback
reference, but there's no rollback path to a feature being deliberately
removed).

Not yet verified against real hardware - see issue #31's open tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
2026-09-03 19:49:35 +02:00
co-authored by Claude-Bot
parent 4fe910a5f3
commit 472e3a4738
23 changed files with 133 additions and 1893 deletions
+5 -7
View File
@@ -15,13 +15,11 @@ The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and
If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture (the raw API would need its own auth in front of it).
## The AI proxy (LiteLLM) — `proxy.ai.home` / `proxy.ai.haylan.ch`
## The AI gateway (OmniRoute) — `proxy.ai.home` / `proxy.ai.haylan.ch`
Once the gateway from [issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) is deployed, it gets its own hostnames, same NPM pattern as Open WebUI above:
As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute — same NPM pattern as Open WebUI above, but a cleaner split than LiteLLM's ever was:
- **`proxy.ai.home`** — internal only, fronts the full LiteLLM port (API + Admin UI).
- **`proxy.ai.haylan.ch`** — external, via the DMZ/NPM. Fronts only the inference API paths.
- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port.
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
**Every proxy call already requires a valid virtual key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.
**LiteLLM's Admin UI (`/ui`) stays LAN-only**, same reasoning as llama.cpp's raw API: it manages every workload's keys and budgets, so it doesn't belong on the public internet. LiteLLM serves `/ui` on the same port as its API by default, so `proxy.ai.haylan.ch`'s NPM Proxy Host needs an explicit rule denying the `/ui` path (a "Deny" custom location, same UI as the "Advanced" tab used for other NPM hosts) — `proxy.ai.home` has no such restriction and reaches both the API and the Admin UI.
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.