LiteLLM gateway: search, knowledgebase, and gateway-level memory #21

New Issue

2026-09-02T19:18:43Z

haylan commented

2026-09-02 19:18:43 +00:00

Destination

The LiteLLM gateway (already fronting llama.cpp per AI gateway/proxy) gains three capabilities, wired at the gateway layer so every client behind it (Open WebUI, coding CLIs, future workloads) benefits uniformly rather than each client configuring its own: (a) web search via the local SearXNG instance, (b) a vector store / knowledgebase for RAG, and (c) persistent memory — the facts in data/memory.md and data/claude-legacy-memory.md — ingested into that knowledgebase and retrievable by any model call through LiteLLM. Ends in real config/compose changes committed to the repo, not just decisions.

Notes

Prior context: builds on AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack (LiteLLM already deployed, litellm-config.yaml, docker-compose.yml). Qdrant is already running as a standalone service (qdrant, ai-stack network) — currently only consumed by Open WebUI's own RAG/Memory, per Local AI inference stack.

User-provided starting points:

SearXNG instance already running on the LAN: http://search.home/
LiteLLM search docs: https://docs.litellm.ai/docs/search
LiteLLM knowledgebase/vector-store docs: https://docs.litellm.ai/docs/completion/knowledgebase
Memory source files: data/memory.md, data/claude-legacy-memory.md (Claude-memory-style dated fact lists)

Standing decision from charting: memory must be served from LiteLLM itself, not from Open WebUI's separate Memory/RAG feature — Open WebUI is "just a playground to chat" in the user's words, so the gateway is the right layer, keeping memory available to every client (coding CLIs included), not just Open WebUI.

Out of scope for this map: the Open WebUI bug where responses only render after a manual page reload — that's a live bug, not a planning decision, and is being debugged separately outside this map.

Skills to consult per ticket: /research for LiteLLM's actual search/knowledgebase mechanics (don't assume the docs' shape without checking); /grilling + /domain-modeling if a new open decision surfaces (e.g. embedding model choice, memory-ingestion pipeline design).

Tickets

How does LiteLLM's SearXNG web-search integration work, and what does wiring it in require? (#22)
How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? (#23)
LangChain+pgvector direct RAG vs. the litellm-pgvector connector — which fits this stack? (#25)
Verify search + knowledgebase wiring on the real R9700 box (#24)

Decisions so far

Search settled: How does LiteLLM's SearXNG web-search integration work, and what does wiring it in require? (#22) — SearXNG is a first-class search_provider, wired as a standalone /v1/search/searxng-search REST endpoint (not a model tool, not auto-injected), sidestepping Qwen3.8-27B's known-flaky tool-calling entirely. Needed extra_hosts on the litellm service to resolve search.home. Implemented in litellm-config.yaml / docker-compose.yml on main; docs in docs/research/litellm-searxng-search.md and docs/memory-knowledgebase.md.
Knowledgebase settled (superseded once, see below): How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? (#23) — LiteLLM's native vector-store feature has no Qdrant backend; the existing Qdrant service can't be reused. First attempt: a litellm-pgvector companion service vendored from its upstream repo, wired via vector_store_registry. Superseded by #25 below.
Knowledgebase connector switched: LangChain+pgvector direct RAG vs. the litellm-pgvector connector — which fits this stack? (#25) — the vendored litellm-pgvector connector (793 lines, Prisma migrations, a fragile git-context build) is replaced by services/memory-retrieval/, a ~90-line FastAPI service wrapping langchain_postgres.PGVector directly against pgvector-db. Same gateway boundary (still calls litellm for embeddings). pgvector-db and embedding-server (nomic-embed-text-v1.5 on a second llama.cpp instance) survive unchanged from #23's original resolution. vector_store_registry removed from litellm-config.yaml — no in-band file_search tool call; callers query memory-retrieval's /query endpoint directly instead. scripts/ingest-memory.sh rewritten for the new /ingest endpoint. Implemented on main; docs in docs/research/langchain-pgvector-vs-litellm-pgvector.md and docs/memory-knowledgebase.md. Not yet verified on real hardware — see Verify search + knowledgebase wiring on the real R9700 box (#24).

Not yet specified

Whether/how a memory-write path (the model or a client saving new facts back into the knowledgebase, not just reading it) is ever wanted — everything built so far is read-only retrieval plus a manual ingestion script; no such write-back mechanism exists or was requested.

Out of scope

The Open WebUI "responses only show after reload" bug — being debugged separately, not a wayfinder decision.

## Destination The LiteLLM gateway (already fronting llama.cpp per [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)) gains three capabilities, wired at the **gateway layer** so every client behind it (Open WebUI, coding CLIs, future workloads) benefits uniformly rather than each client configuring its own: (a) **web search** via the local SearXNG instance, (b) a **vector store / knowledgebase** for RAG, and (c) **persistent memory** — the facts in `data/memory.md` and `data/claude-legacy-memory.md` — ingested into that knowledgebase and retrievable by any model call through LiteLLM. Ends in real config/compose changes committed to the repo, not just decisions. ## Notes **Prior context**: builds on [AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (LiteLLM already deployed, `litellm-config.yaml`, `docker-compose.yml`). Qdrant is already running as a standalone service (`qdrant`, ai-stack network) — currently only consumed by Open WebUI's own RAG/Memory, per [Local AI inference stack](https://git.arthurerlich.de/haylan/LLM-Server/issues/1). **User-provided starting points**: - SearXNG instance already running on the LAN: `http://search.home/` - LiteLLM search docs: https://docs.litellm.ai/docs/search - LiteLLM knowledgebase/vector-store docs: https://docs.litellm.ai/docs/completion/knowledgebase - Memory source files: `data/memory.md`, `data/claude-legacy-memory.md` (Claude-memory-style dated fact lists) **Standing decision from charting**: memory must be served **from LiteLLM itself**, not from Open WebUI's separate Memory/RAG feature — Open WebUI is "just a playground to chat" in the user's words, so the gateway is the right layer, keeping memory available to every client (coding CLIs included), not just Open WebUI. **Out of scope for this map**: the Open WebUI bug where responses only render after a manual page reload — that's a live bug, not a planning decision, and is being debugged separately outside this map. **Skills to consult per ticket**: `/research` for LiteLLM's actual search/knowledgebase mechanics (don't assume the docs' shape without checking); `/grilling` + `/domain-modeling` if a new open decision surfaces (e.g. embedding model choice, memory-ingestion pipeline design). ## Tickets - [x] How does LiteLLM's SearXNG web-search integration work, and what does wiring it in require? (#22) - [x] How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? (#23) - [x] LangChain+pgvector direct RAG vs. the litellm-pgvector connector — which fits this stack? (#25) - [ ] Verify search + knowledgebase wiring on the real R9700 box (#24) ## Decisions so far - **Search settled**: [How does LiteLLM's SearXNG web-search integration work, and what does wiring it in require? (#22)](https://git.arthurerlich.de/haylan/LLM-Server/issues/22) — SearXNG is a first-class `search_provider`, wired as a standalone `/v1/search/searxng-search` REST endpoint (not a model tool, not auto-injected), sidestepping Qwen3.8-27B's known-flaky tool-calling entirely. Needed `extra_hosts` on the `litellm` service to resolve `search.home`. Implemented in `litellm-config.yaml` / `docker-compose.yml` on main; docs in `docs/research/litellm-searxng-search.md` and `docs/memory-knowledgebase.md`. - **Knowledgebase settled (superseded once, see below)**: [How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? (#23)](https://git.arthurerlich.de/haylan/LLM-Server/issues/23) — LiteLLM's native vector-store feature has **no Qdrant backend**; the existing Qdrant service can't be reused. First attempt: a `litellm-pgvector` companion service vendored from its upstream repo, wired via `vector_store_registry`. Superseded by #25 below. - **Knowledgebase connector switched**: [LangChain+pgvector direct RAG vs. the litellm-pgvector connector — which fits this stack? (#25)](https://git.arthurerlich.de/haylan/LLM-Server/issues/25) — the vendored `litellm-pgvector` connector (793 lines, Prisma migrations, a fragile git-context build) is replaced by `services/memory-retrieval/`, a ~90-line FastAPI service wrapping `langchain_postgres.PGVector` directly against `pgvector-db`. Same gateway boundary (still calls `litellm` for embeddings). `pgvector-db` and `embedding-server` (nomic-embed-text-v1.5 on a second llama.cpp instance) survive unchanged from #23's original resolution. `vector_store_registry` removed from `litellm-config.yaml` — no in-band `file_search` tool call; callers query `memory-retrieval`'s `/query` endpoint directly instead. `scripts/ingest-memory.sh` rewritten for the new `/ingest` endpoint. Implemented on main; docs in `docs/research/langchain-pgvector-vs-litellm-pgvector.md` and `docs/memory-knowledgebase.md`. **Not yet verified on real hardware** — see [Verify search + knowledgebase wiring on the real R9700 box (#24)](https://git.arthurerlich.de/haylan/LLM-Server/issues/24). ## Not yet specified - Whether/how a memory-*write* path (the model or a client saving new facts back into the knowledgebase, not just reading it) is ever wanted — everything built so far is read-only retrieval plus a manual ingestion script; no such write-back mechanism exists or was requested. ## Out of scope - The Open WebUI "responses only show after reload" bug — being debugged separately, not a wayfinder decision.

haylan added the wayfinder:map label 2026-09-02 19:18:43 +00:00

haylan referenced this issue

2026-09-02 19:19:18 +00:00

How does LiteLLM's SearXNG web-search integration work, and what does wiring it in require? #22

haylan referenced this issue

2026-09-02 19:19:19 +00:00

How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? #23

haylan referenced this issue

2026-09-02 19:27:36 +00:00

How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? #23

haylan referenced this issue

2026-09-02 19:37:44 +00:00