LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37):
replace the litellm/litellm-db services with omniroute, split-port mode
(API_PORT published/reverse-proxied, DASHBOARD_PORT never published -
tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place
of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as
omniroute providers post-boot (no static config.yaml equivalent). No
scripted per-workload key minting yet - omniroute's POST /api/keys needs a
dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is
a manual step for now (docs/proxy-key-onboarding.md).
Caveat carried into the map and README: OmniRoute's own docs
(docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md
on its release/v3.8.51 branch) describe shipped features for AI-provider
client-detection evasion, system-wide HTTPS interception via a locally
installed root CA, and hiding credentials from secret scanners. Proceeding
anyway was an explicit, informed user decision.
Also drops the gateway-level memory/knowledgebase feature entirely (user:
"I don't need it") - litellm-pgvector, pgvector-db, embedding-server,
scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory-
knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated
and untouched. litellm-config.yaml deleted (was kept as a rollback
reference, but there's no rollback path to a feature being deliberately
removed).
Not yet verified against real hardware - see issue #31's open tickets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K
fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700,
~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table.
Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense
model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/
--cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that
there's no explicit SSD offload tier to enable — llama.cpp's default mmap
already falls back to disk implicitly if GPU+RAM run out.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
litellm.aembedding can't infer a provider from a bare model name plus a
custom api_base — litellm-pgvector's embedding_service.py was hitting
'litellm.BadRequestError: LLM Provider NOT provided' on every query-time
embedding (i.e. every search). Same openai/ prefix already used for
qwen3.8-27b-local and local-embedding in litellm-config.yaml.
Today's embedding-server crash-loop (missing nomic-embed-text GGUF) was a
manual step nobody ran. update.sh now runs both downloader profiles
itself, every time, before bringing services up -- no separate command to
remember.
- docker-compose.yml: downloader/downloader-embedding commands gain a
`test -f ... && skip || curl ...` guard, so re-running update.sh never
re-downloads an existing model file.
- scripts/update.sh: runs both profiles after image pull/build, before
service recreation.
- scripts/download-model.sh removed -- folded in, redundant standalone
script.
- README.md / docs/memory-knowledgebase.md updated accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
lazytainer panics on every start with 'Could not determine container ID
of lazytainer': its self-detection (vmorganp/Lazytainer,
configureFromLabels()) matches os.Hostname() against the Docker
container-ID list, but network_mode: host makes the container inherit
the host machine's hostname instead of its own container ID, so the
match never succeeds. Upstream's own example compose file doesn't use
host networking (default bridge + published ports) — nothing about
lazytainer requires it.
It also couldn't have seen the traffic it's meant to watch: llama-server
has no published host port (issue #15) and only exists on the ai-stack
bridge network, which host networking has no visibility into. Joining
ai-stack instead fixes both problems at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
This reverts commit abeadc49c8.
Restores vendor/litellm-pgvector/ and the vector_store_registry wiring
(in-band file_search tool-call support) at the user's request, after
re-confirming against docs.litellm.ai/docs/completion/knowledgebase and
litellm-pgvector's own README that pg_vector is still not an in-process
vector_store_registry backend -- it requires this same standalone
connector service either way, so there is no simpler 'native' path that
was missed. Trading back in: 793 lines of vendored code, the untested
Prisma migration, and the git-context build risk noted in VENDORED.md
(all flagged as unverified against real hardware in issue #24), in
exchange for the file_search in-band tool call memory-retrieval did not
support.
Conflicts resolved on top of later commits (Redis, update.sh key-minting
fold-in):
- .env.example / docs/memory-knowledgebase.md: kept the auto-mint-via-
update.sh language, renamed MEMORY_RETRIEVAL_* back to
LITELLM_PGVECTOR_*.
- scripts/generate-secrets.sh: left deleted -- its job was folded into
update.sh in 24d749b, unrelated to this revert.
- scripts/update.sh: renamed the MEMORY_RETRIEVAL_* secret/mint calls to
LITELLM_PGVECTOR_* to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
New redis service (redis:7-alpine, password-protected, no persistence
volume — everything it holds is cache/coordination state). litellm gets
REDIS_HOST/REDIS_PORT/REDIS_PASSWORD, which is all LiteLLM needs to use it
for router state, rate limits/budgets, and cache invalidation — no
litellm-config.yaml block required (docs.litellm.ai/docs/proxy/caching).
REDIS_PASSWORD added to .env.example and update.sh's auto-generated
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per docs/research/langchain-pgvector-vs-litellm-pgvector.md (issue #25):
the vendored litellm-pgvector connector (793 lines, Prisma migrations, a
fragile git-context build) is replaced by a ~90-line FastAPI service
(services/memory-retrieval/) wrapping langchain_postgres.PGVector directly
against pgvector-db. Same gateway boundary — it still calls litellm for
embeddings, nothing talks to Postgres or the model directly except this
service.
- New services/memory-retrieval/ (main.py, Dockerfile, requirements.txt):
POST /ingest, POST /query, GET /health.
- docker-compose.yml: litellm-pgvector service replaced by memory-retrieval;
pgvector-db and embedding-server untouched.
- litellm-config.yaml: vector_store_registry block removed (no
langchain_postgres provider exists to register against; callers query
memory-retrieval directly instead of an in-band file_search tool call —
that mechanism was never confirmed working per issue #24 anyway).
- scripts/ingest-memory.sh rewritten for the new /ingest endpoint (same
per-line chunking, no dedup).
- .env vars renamed: LITELLM_PGVECTOR_API_KEY/LITELLM_PGVECTOR_EMBEDDING_KEY
-> MEMORY_RETRIEVAL_API_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY.
- vendor/litellm-pgvector/ removed entirely.
- docs/memory-knowledgebase.md updated for the new setup/query flow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The server's Docker/BuildKit couldn't do a git-context build of a public
github.com repo (fails with "could not read Username ... terminal prompts
disabled" — an auth-shaped error for what should be an anonymous clone).
Rather than debug that, vendor litellm-pgvector's small source tree
directly (vendor/litellm-pgvector/, see VENDORED.md for provenance/update
steps) and build from the local path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- search_tools block in litellm-config.yaml (SearXNG as a first-class
search_provider, standalone /v1/search endpoint, not a model tool) plus
extra_hosts on the litellm service so it can resolve search.home.
- New embedding-server (nomic-embed-text-v1.5 on a second llama.cpp
instance), pgvector-db, and litellm-pgvector services — LiteLLM's native
knowledgebase feature has no Qdrant backend, so this is the only
self-hosted path (docs/research/litellm-knowledgebase.md).
- vector_store_registry + local-embedding model entry in
litellm-config.yaml, wiring it together.
- scripts/ingest-memory.sh to load data/memory.md and
data/claude-legacy-memory.md into the knowledgebase.
- docs/memory-knowledgebase.md documenting the whole setup; data/
gitignored (personal memory content, not meant to be committed).
- New .env vars (SEARXNG_LAN_IP, PGVECTOR_DB_PASSWORD,
LITELLM_PGVECTOR_API_KEY, LITELLM_PGVECTOR_EMBEDDING_KEY,
EMBEDDING_MODEL_FILE) and generate-secrets.sh support for the
auto-generatable ones.
Resolves#22 and #23 (wayfinder map #21). Not yet verified on real
hardware — see #24.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The curlimages/curl image drops to non-root curl_user (uid 100) by
default, but the models named volume is created root-owned by Docker,
so writes into it failed with 'Permission denied'.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Open WebUI now points at litellm instead of llama-server directly, using a
provisioned virtual key. llama-server's host port is dropped (internal-only
on the ai-stack network) since the proxy is the only intended entry point
now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with
per-CLI virtual keys instead of the old shared dummy key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with
custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a
priority-scheduling stub (per #16, needs real-hardware smoke test), and
required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real
hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp
directly, migration is #15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #4. Wires up the locked decisions from the map:
- llama.cpp (ghcr.io/ggml-org/llama.cpp:server-rocm, gfx1201) serving
Qwen3.8-27B-UD-Q4_K_XL.gguf, port published for direct Claude Code CLI /
Kimi CLI access alongside Open WebUI.
- Open WebUI with WEBUI_AUTH on, RAG+Memory wired to a standalone Qdrant
service.
- Lazytainer labels on llama-server for a 15 min idle-stop.
- Named Docker volumes only (models, qdrant-data, openwebui-data) — no host
bind-mounts.
- One-off 'downloader' compose profile instead of a host-side script with
its own dependencies, wrapped by scripts/download-model.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude-Bot