haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(omniroute): add dedicated embedding-server for memory feature

Browse Source 
llama.cpp loads one model per process and the running Qwen3.8-27B chat
model isn't embedding-trained, so this is a second, CPU-only
llama-server instance (bge-small-en-v1.5, 384-dim) rather than adding
--embeddings to the chat one — see docs/research/litellm-knowledgebase.md
#3.

Downloader extended to fetch both GGUFs into the shared models volume.
No host port published — OmniRoute reaches it via the ai-stack network
DNS name (embedding-server:8081).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qx22CV9EUS3hGATucQav13
This commit is contained in:
haylan
2026-09-03 21:43:20 +02:00
co-authored by Claude-Bot
parent 885de477ba
commit 9e1362c22c
3 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -37,4 +37,4 @@ The gateway also fronts SearXNG-backed web search — see `docs/research/litellm
## What's not here
- **Open WebUI** — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle.
- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own memory feature, backed by `qdrant` (bare service — wired up as a memory provider by hand in the dashboard, no static config here).
- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own memory feature, backed by `qdrant` and `embedding-server` (bge-small-en-v1.5, CPU-only) — both bare services, wired up as a memory provider by hand in the dashboard, no static routing config here.