Revert "feat(knowledgebase): replace litellm-pgvector connector with memory-retrieval"
This reverts commitabeadc49c8. Restores vendor/litellm-pgvector/ and the vector_store_registry wiring (in-band file_search tool-call support) at the user's request, after re-confirming against docs.litellm.ai/docs/completion/knowledgebase and litellm-pgvector's own README that pg_vector is still not an in-process vector_store_registry backend -- it requires this same standalone connector service either way, so there is no simpler 'native' path that was missed. Trading back in: 793 lines of vendored code, the untested Prisma migration, and the git-context build risk noted in VENDORED.md (all flagged as unverified against real hardware in issue #24), in exchange for the file_search in-band tool call memory-retrieval did not support. Conflicts resolved on top of later commits (Redis, update.sh key-minting fold-in): - .env.example / docs/memory-knowledgebase.md: kept the auto-mint-via- update.sh language, renamed MEMORY_RETRIEVAL_* back to LITELLM_PGVECTOR_*. - scripts/generate-secrets.sh: left deleted -- its job was folded into update.sh in24d749b, unrelated to this revert. - scripts/update.sh: renamed the MEMORY_RETRIEVAL_* secret/mint calls to LITELLM_PGVECTOR_* to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
Three gateway-level capabilities added on top of the [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9), so every client behind LiteLLM gets them — not just Open WebUI. See [issue #21](https://git.arthurerlich.de/haylan/LLM-Server/issues/21) for the rationale.
|
||||
|
||||
**Not yet verified on real hardware** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24).
|
||||
**Not yet verified on real hardware** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24). In particular: `litellm-pgvector`'s Prisma migrations on first boot, and the exact `vector_store_registry` field names for the `pg_vector` provider.
|
||||
|
||||
## Web search (SearXNG)
|
||||
|
||||
@@ -21,23 +21,23 @@ Requires `SEARXNG_LAN_IP` set in `.env` so the `litellm` container can resolve `
|
||||
|
||||
## Knowledgebase (vector store / RAG)
|
||||
|
||||
LiteLLM's native knowledgebase feature has **no Qdrant backend** — the `qdrant` service in this stack only serves Open WebUI's own separate RAG/Memory feature and is unrelated to this — and no `langchain_postgres` backend either. Instead, a small in-repo service wraps LangChain's `PGVector` directly against its own Postgres+pgvector database (`pgvector-db`). This replaced an earlier attempt to vendor the third-party `litellm-pgvector` connector — see `docs/research/langchain-pgvector-vs-litellm-pgvector.md` for why. One consequence: the OpenAI-style `file_search` tool call on a `/chat/completions` request doesn't work here (no `vector_store_registry` entry) — query `memory-retrieval` directly instead (see below).
|
||||
LiteLLM's native knowledgebase feature has **no Qdrant backend** — the `qdrant` service in this stack only serves Open WebUI's own separate RAG/Memory feature and is unrelated to this. The only self-hosted path is [litellm-pgvector](https://github.com/BerriAI/litellm-pgvector), a companion service backed by its own Postgres+pgvector database (`pgvector-db`), which this stack now runs alongside `litellm`. Full research: `docs/research/litellm-knowledgebase.md`.
|
||||
|
||||
New pieces:
|
||||
|
||||
- **`embedding-server`** — a second llama.cpp instance (small footprint, `nomic-embed-text-v1.5`) serving `/v1/embeddings`. The chat model isn't embedding-trained and llama.cpp serves one model per process, so this can't just be a flag on `llama-server`.
|
||||
- **`pgvector-db`** — Postgres with the pgvector extension, separate from `litellm-db`.
|
||||
- **`memory-retrieval`** (`services/memory-retrieval/`) — a ~90-line FastAPI app wrapping `langchain_postgres.PGVector`, exposing `POST /ingest` and `POST /query`. Calls back into `litellm` for embeddings, same gateway boundary as everything else here.
|
||||
- `litellm-config.yaml`'s `local-embedding` model entry, which `memory-retrieval` calls through.
|
||||
- **`litellm-pgvector`** — the connector service; no published image exists, so it's built from a vendored copy of the upstream repo at `vendor/litellm-pgvector/` (see that dir's `VENDORED.md`) — a remote git build context failed on the server's Docker/BuildKit setup.
|
||||
- `litellm-config.yaml`'s `local-embedding` model entry and `vector_store_registry` block, tying it together.
|
||||
|
||||
### First-time setup
|
||||
|
||||
```bash
|
||||
docker compose --profile tools run --rm downloader-embedding # fetch the embedding model
|
||||
docker compose up -d embedding-server pgvector-db memory-retrieval
|
||||
docker compose up -d embedding-server pgvector-db litellm-pgvector
|
||||
```
|
||||
|
||||
`./scripts/update.sh` mints `MEMORY_RETRIEVAL_EMBEDDING_KEY` automatically (a `memory-retrieval` virtual key via LiteLLM's own API) if it's blank — it calls back into `litellm` for embeddings, same as any other workload. See `docs/proxy-key-onboarding.md` if a mint fails and it needs doing by hand.
|
||||
`./scripts/update.sh` mints `LITELLM_PGVECTOR_EMBEDDING_KEY` automatically (a `litellm-pgvector` virtual key via LiteLLM's own API) if it's blank — it calls back into `litellm` for embeddings, same as any other workload. See `docs/proxy-key-onboarding.md` if a mint fails and it needs doing by hand.
|
||||
|
||||
### Loading memory into it
|
||||
|
||||
@@ -51,13 +51,14 @@ One chunk per fact/paragraph line, tagged with `source`/`section` metadata. Re-r
|
||||
|
||||
### Querying it
|
||||
|
||||
Directly against `memory-retrieval` — no in-band `file_search` tool call (see above):
|
||||
Via the OpenAI Assistants-style `file_search` tool on a chat completion:
|
||||
|
||||
```bash
|
||||
curl -X POST http://memory-retrieval:8000/query \
|
||||
-H "Authorization: Bearer <MEMORY_RETRIEVAL_API_KEY>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "...", "k": 5}'
|
||||
```json
|
||||
{
|
||||
"model": "qwen3.8-27b-local",
|
||||
"messages": [...],
|
||||
"tools": [{"type": "file_search", "vector_store_ids": ["memory-and-notes"]}]
|
||||
}
|
||||
```
|
||||
|
||||
A caller wanting RAG-augmented chat does the "search then send" pattern: query `/query`, splice the results into the prompt, then call `litellm` as normal.
|
||||
or directly: `POST /v1/vector_stores/memory-and-notes/search` with `{"query": "..."}`.
|
||||
|
||||
Reference in New Issue
Block a user