feat(omniroute): add dedicated embedding-server for memory feature
llama.cpp loads one model per process and the running Qwen3.8-27B chat model isn't embedding-trained, so this is a second, CPU-only llama-server instance (bge-small-en-v1.5, 384-dim) rather than adding --embeddings to the chat one — see docs/research/litellm-knowledgebase.md #3. Downloader extended to fetch both GGUFs into the shared models volume. No host port published — OmniRoute reaches it via the ai-stack network DNS name (embedding-server:8081). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qx22CV9EUS3hGATucQav13
This commit is contained in:
@@ -28,6 +28,12 @@ LLAMA_GPU_LAYERS=999
|
||||
# of dropping context: --cache-type-k/v q8_0 roughly halves it.
|
||||
LLAMA_CTX_SIZE=131072
|
||||
|
||||
# --- embedding-server (OmniRoute memory) ---
|
||||
# bge-small-en-v1.5, CPU-only, 384-dim output — see docker-compose.yml's
|
||||
# embedding-server service comment for why this is a second model/process
|
||||
# rather than reusing llama-server.
|
||||
EMBEDDING_MODEL_FILE=bge-small-en-v1.5-q8_0.gguf
|
||||
|
||||
# --- Lazytainer ---
|
||||
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
||||
LAZYTAINER_INACTIVE_TIMEOUT=900
|
||||
|
||||
Reference in New Issue
Block a user