haylanandClaude-Bot
abeadc49c8
Per docs/research/langchain-pgvector-vs-litellm-pgvector.md (issue #25): the vendored litellm-pgvector connector (793 lines, Prisma migrations, a fragile git-context build) is replaced by a ~90-line FastAPI service (services/memory-retrieval/) wrapping langchain_postgres.PGVector directly against pgvector-db. Same gateway boundary — it still calls litellm for embeddings, nothing talks to Postgres or the model directly except this service. - New services/memory-retrieval/ (main.py, Dockerfile, requirements.txt): POST /ingest, POST /query, GET /health. - docker-compose.yml: litellm-pgvector service replaced by memory-retrieval; pgvector-db and embedding-server untouched. - litellm-config.yaml: vector_store_registry block removed (no langchain_postgres provider exists to register against; callers query memory-retrieval directly instead of an in-band file_search tool call — that mechanism was never confirmed working per issue #24 anyway). - scripts/ingest-memory.sh rewritten for the new /ingest endpoint (same per-line chunking, no dedup). - .env vars renamed: LITELLM_PGVECTOR_API_KEY/LITELLM_PGVECTOR_EMBEDDING_KEY -> MEMORY_RETRIEVAL_API_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY. - vendor/litellm-pgvector/ removed entirely. - docs/memory-knowledgebase.md updated for the new setup/query flow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
64 lines
3.9 KiB
Markdown
64 lines
3.9 KiB
Markdown
# Knowledgebase, memory, and web search
|
|
|
|
Three gateway-level capabilities added on top of the [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9), so every client behind LiteLLM gets them — not just Open WebUI. See [issue #21](https://git.arthurerlich.de/haylan/LLM-Server/issues/21) for the rationale.
|
|
|
|
**Not yet verified on real hardware** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24).
|
|
|
|
## Web search (SearXNG)
|
|
|
|
`litellm-config.yaml`'s `search_tools` block wires the LAN's SearXNG instance in as a **standalone REST endpoint**, not a model-callable tool — call it directly:
|
|
|
|
```bash
|
|
curl http://<proxy>:4000/v1/search/searxng-search \
|
|
-H "Authorization: Bearer <a virtual key>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"query": "...", "max_results": 5}'
|
|
```
|
|
|
|
Because this doesn't ask the model to emit a tool call, it sidesteps Qwen3.8-27B's known-flaky tool-calling (`docs/research/qwen3.8-27b-tool-calling.md`) entirely. Open WebUI's own web-search setting can point at this endpoint the same way.
|
|
|
|
Requires `SEARXNG_LAN_IP` set in `.env` (SearXNG's stable LAN IP — use a static DHCP reservation) so the `litellm` container can resolve `search.home` via `extra_hosts`. Full research: `docs/research/litellm-searxng-search.md`.
|
|
|
|
## Knowledgebase (vector store / RAG)
|
|
|
|
LiteLLM's native knowledgebase feature has **no Qdrant backend** — the `qdrant` service in this stack only serves Open WebUI's own separate RAG/Memory feature and is unrelated to this — and no `langchain_postgres` backend either. Instead, a small in-repo service wraps LangChain's `PGVector` directly against its own Postgres+pgvector database (`pgvector-db`). This replaced an earlier attempt to vendor the third-party `litellm-pgvector` connector — see `docs/research/langchain-pgvector-vs-litellm-pgvector.md` for why. One consequence: the OpenAI-style `file_search` tool call on a `/chat/completions` request doesn't work here (no `vector_store_registry` entry) — query `memory-retrieval` directly instead (see below).
|
|
|
|
New pieces:
|
|
|
|
- **`embedding-server`** — a second llama.cpp instance (small footprint, `nomic-embed-text-v1.5`) serving `/v1/embeddings`. The chat model isn't embedding-trained and llama.cpp serves one model per process, so this can't just be a flag on `llama-server`.
|
|
- **`pgvector-db`** — Postgres with the pgvector extension, separate from `litellm-db`.
|
|
- **`memory-retrieval`** (`services/memory-retrieval/`) — a ~90-line FastAPI app wrapping `langchain_postgres.PGVector`, exposing `POST /ingest` and `POST /query`. Calls back into `litellm` for embeddings, same gateway boundary as everything else here.
|
|
- `litellm-config.yaml`'s `local-embedding` model entry, which `memory-retrieval` calls through.
|
|
|
|
### First-time setup
|
|
|
|
```bash
|
|
docker compose --profile tools run --rm downloader-embedding # fetch the embedding model
|
|
docker compose up -d embedding-server pgvector-db memory-retrieval
|
|
```
|
|
|
|
Create a `memory-retrieval` virtual key in LiteLLM's Admin UI (per `docs/proxy-key-onboarding.md`) and set it as `MEMORY_RETRIEVAL_EMBEDDING_KEY` in `.env` — it calls back into `litellm` for embeddings, same as any other workload.
|
|
|
|
### Loading memory into it
|
|
|
|
`data/memory.md` and `data/claude-legacy-memory.md` — Claude-memory-style fact files — get loaded via:
|
|
|
|
```bash
|
|
./scripts/ingest-memory.sh
|
|
```
|
|
|
|
One chunk per fact/paragraph line, tagged with `source`/`section` metadata. Re-run after editing either file (see the script's header comment for the no-dedup caveat).
|
|
|
|
### Querying it
|
|
|
|
Directly against `memory-retrieval` — no in-band `file_search` tool call (see above):
|
|
|
|
```bash
|
|
curl -X POST http://memory-retrieval:8000/query \
|
|
-H "Authorization: Bearer <MEMORY_RETRIEVAL_API_KEY>" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"query": "...", "k": 5}'
|
|
```
|
|
|
|
A caller wanting RAG-augmented chat does the "search then send" pattern: query `/query`, splice the results into the prompt, then call `litellm` as normal.
|