haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(omniroute): add dedicated embedding-server for memory feature

Browse Source 
llama.cpp loads one model per process and the running Qwen3.8-27B chat
model isn't embedding-trained, so this is a second, CPU-only
llama-server instance (bge-small-en-v1.5, 384-dim) rather than adding
--embeddings to the chat one — see docs/research/litellm-knowledgebase.md
#3.

Downloader extended to fetch both GGUFs into the shared models volume.
No host port published — OmniRoute reaches it via the ai-stack network
DNS name (embedding-server:8081).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qx22CV9EUS3hGATucQav13
This commit is contained in:
haylan
2026-09-03 21:43:20 +02:00
co-authored by Claude-Bot
parent 885de477ba
commit 9e1362c22c
3 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+29 -2
View File
@@ -39,7 +39,7 @@ services:
# `docker compose --profile tools run --rm downloader`. Folded into
# scripts/update.sh, which runs this every time; the `test -f` guard is
# what makes that safe to re-run without re-downloading. Keeps the model
# file inside the named `models` volume instead of a host bind-mount.
# files inside the named `models` volume instead of a host bind-mount.
downloader:
image: curlimages/curl:latest
profiles: ["tools"]
@@ -54,7 +54,34 @@ services:
test -f /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} &&
echo "already downloaded, skipping" ||
curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf};
test -f /models/${EMBEDDING_MODEL_FILE:-bge-small-en-v1.5-q8_0.gguf} &&
echo "already downloaded, skipping" ||
curl -L --fail --create-dirs -o /models/${EMBEDDING_MODEL_FILE:-bge-small-en-v1.5-q8_0.gguf}
https://huggingface.co/ggml-org/bge-small-en-v1.5-Q8_0-GGUF/resolve/main/${EMBEDDING_MODEL_FILE:-bge-small-en-v1.5-q8_0.gguf}
# Dedicated embedding model for OmniRoute's memory feature. llama.cpp loads
# one model per process and the already-running Qwen3.8-27B chat model
# isn't embedding-trained, so this is a second, separate llama-server
# instance rather than adding --embeddings to the chat one — see
# docs/research/litellm-knowledgebase.md #3. bge-small-en-v1.5 outputs
# 384-dim vectors — use that as the Qdrant collection size.
# ponytail: CPU image, no GPU devices — a 33M-param embedding model is
# fast enough on CPU and this avoids VRAM contention with llama-server's
# 27B chat model on the same GPU.
embedding-server:
image: ghcr.io/ggml-org/llama.cpp:server
container_name: embedding-server
volumes:
- models:/models
command: >
-m /models/${EMBEDDING_MODEL_FILE:-bge-small-en-v1.5-q8_0.gguf}
--host 0.0.0.0
--port 8081
--embeddings
--pooling mean
restart: unless-stopped
networks: [ai-stack]
# Vector store for OmniRoute's memory feature — wired up in the OmniRoute
# dashboard as a memory service, not via static config here. Not published