haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature

Browse Source 
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37):
replace the litellm/litellm-db services with omniroute, split-port mode
(API_PORT published/reverse-proxied, DASHBOARD_PORT never published -
tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place
of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as
omniroute providers post-boot (no static config.yaml equivalent). No
scripted per-workload key minting yet - omniroute's POST /api/keys needs a
dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is
a manual step for now (docs/proxy-key-onboarding.md).

Caveat carried into the map and README: OmniRoute's own docs
(docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md
on its release/v3.8.51 branch) describe shipped features for AI-provider
client-detection evasion, system-wide HTTPS interception via a locally
installed root CA, and hiding credentials from secret scanners. Proceeding
anyway was an explicit, informed user decision.

Also drops the gateway-level memory/knowledgebase feature entirely (user:
"I don't need it") - litellm-pgvector, pgvector-db, embedding-server,
scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory-
knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated
and untouched. litellm-config.yaml deleted (was kept as a rollback
reference, but there's no rollback path to a feature being deliberately
removed).

Not yet verified against real hardware - see issue #31's open tickets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
haylan
2026-09-03 19:49:35 +02:00
co-authored by Claude-Bot
parent 4fe910a5f3
commit 472e3a4738
23 changed files with 133 additions and 1893 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/proxy-request-priority.md
+3 -1
View File
@@ -1,6 +1,8 @@
# Request priority on the AI proxy
One local model instance (llama.cpp on the single R9700) serves every workload through the LiteLLM proxy ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
**Stale as of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (LiteLLM → OmniRoute migration)** — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
## Tiers