LLM-Server/docs/memory-knowledgebase.md

Knowledgebase, memory, and web search

Three gateway-level capabilities added on top of the AI gateway/proxy, so every client behind LiteLLM gets them — not just Open WebUI. See issue #21 for the rationale.

Not yet verified on real hardware — see issue #24.

Web search (SearXNG)

litellm-config.yaml's search_tools block wires the LAN's SearXNG instance in as a standalone REST endpoint, not a model-callable tool — call it directly:

curl http://<proxy>:4000/v1/search/searxng-search \
  -H "Authorization: Bearer <a virtual key>" \
  -H "Content-Type: application/json" \
  -d '{"query": "...", "max_results": 5}'

Because this doesn't ask the model to emit a tool call, it sidesteps Qwen3.8-27B's known-flaky tool-calling (docs/research/qwen3.8-27b-tool-calling.md) entirely. Open WebUI's own web-search setting can point at this endpoint the same way.

Requires SEARXNG_LAN_IP set in .env (SearXNG's stable LAN IP — use a static DHCP reservation) so the litellm container can resolve search.home via extra_hosts. Full research: docs/research/litellm-searxng-search.md.

Knowledgebase (vector store / RAG)

LiteLLM's native knowledgebase feature has no Qdrant backend — the qdrant service in this stack only serves Open WebUI's own separate RAG/Memory feature and is unrelated to this — and no langchain_postgres backend either. Instead, a small in-repo service wraps LangChain's PGVector directly against its own Postgres+pgvector database (pgvector-db). This replaced an earlier attempt to vendor the third-party litellm-pgvector connector — see docs/research/langchain-pgvector-vs-litellm-pgvector.md for why. One consequence: the OpenAI-style file_search tool call on a /chat/completions request doesn't work here (no vector_store_registry entry) — query memory-retrieval directly instead (see below).

New pieces:

• embedding-server — a second llama.cpp instance (small footprint, nomic-embed-text-v1.5) serving /v1/embeddings. The chat model isn't embedding-trained and llama.cpp serves one model per process, so this can't just be a flag on llama-server.
• pgvector-db — Postgres with the pgvector extension, separate from litellm-db.
• memory-retrieval (services/memory-retrieval/) — a ~90-line FastAPI app wrapping langchain_postgres.PGVector, exposing POST /ingest and POST /query. Calls back into litellm for embeddings, same gateway boundary as everything else here.
• litellm-config.yaml's local-embedding model entry, which memory-retrieval calls through.

First-time setup

docker compose --profile tools run --rm downloader-embedding   # fetch the embedding model
docker compose up -d embedding-server pgvector-db memory-retrieval

Create a memory-retrieval virtual key in LiteLLM's Admin UI (per docs/proxy-key-onboarding.md) and set it as MEMORY_RETRIEVAL_EMBEDDING_KEY in .env — it calls back into litellm for embeddings, same as any other workload.

Loading memory into it

data/memory.md and data/claude-legacy-memory.md — Claude-memory-style fact files — get loaded via:

./scripts/ingest-memory.sh

One chunk per fact/paragraph line, tagged with source/section metadata. Re-run after editing either file (see the script's header comment for the no-dedup caveat).

Querying it

Directly against memory-retrieval — no in-band file_search tool call (see above):

curl -X POST http://memory-retrieval:8000/query \
  -H "Authorization: Bearer <MEMORY_RETRIEVAL_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"query": "...", "k": 5}'

A caller wanting RAG-augmented chat does the "search then send" pattern: query /query, splice the results into the prompt, then call litellm as normal.