306 lines
16 KiB
Markdown
306 lines
16 KiB
Markdown
# Research: LiteLLM vector store / knowledgebase feature (for Gitea issue #23)
|
||
|
||
**Question:** Can LiteLLM's built-in "knowledgebase" / vector store feature be
|
||
used to load two Markdown fact files (`data/memory.md`,
|
||
`data/claude-legacy-memory.md`) into a queryable knowledge base, and if so,
|
||
what does that require from this stack (config shape, backend, embedding
|
||
model, ingestion mechanism)?
|
||
|
||
**Bottom line: no, not against this stack's existing Qdrant instance, and not
|
||
without adding a dedicated embedding model.** LiteLLM's vector store feature
|
||
is a *routing/registry* layer over a small set of natively-integrated
|
||
backends (Bedrock Knowledge Bases, OpenAI Vector Stores, Azure
|
||
Vector/AI-Search, Vertex AI RAG/Search, Gemini File Search, RAGFlow) plus one
|
||
self-hosted option — Postgres+pgvector, via a **separate companion service**
|
||
(`BerriAI/litellm-pgvector`), not Qdrant. There is no Qdrant provider at all.
|
||
Ingesting the two fact files would also require running a dedicated
|
||
embedding-capable model, which this stack doesn't currently have (the one
|
||
llama.cpp instance serves a chat model, not started with `--embeddings`).
|
||
|
||
## 1. Config shape in `config.yaml`
|
||
|
||
Top-level block is **`vector_store_registry`** (this is the current key name;
|
||
an earlier PR that introduced the feature used `vector_stores` — see
|
||
Provenance note below), a list of entries:
|
||
|
||
```yaml
|
||
vector_store_registry:
|
||
- vector_store_name: "my-knowledgebase" # optional friendly name
|
||
litellm_params:
|
||
vector_store_id: "T37J8R4WTM" # required, provider's own ID
|
||
custom_llm_provider: "bedrock" # required — selects backend
|
||
vector_store_description: "..." # optional
|
||
vector_store_metadata: {} # optional
|
||
litellm_credential_name: "..." # optional, reuse a named credential
|
||
embedding_model: "..." # backend-dependent — see §3
|
||
```
|
||
|
||
Source: [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase)
|
||
(fetched directly). Earlier shape (same idea, older field names) documented
|
||
in [BerriAI/litellm PR #10448](https://github.com/BerriAI/litellm/pull/10448)
|
||
("[Feat] Vector Stores/KnowledgeBases - Allow defining Vector Store Configs"):
|
||
|
||
```yaml
|
||
vector_stores:
|
||
- vector_store_name: "bedrock-litellm-website-knowledgebase"
|
||
litellm_params:
|
||
custom_llm_provider: "bedrock"
|
||
id: "T37J8R4WTM"
|
||
```
|
||
|
||
This is a **registry of pointers to knowledge bases that already exist on
|
||
the backend provider** — it is not itself a vector database. `model_list`
|
||
entries are not where a store is attached; `vector_store_registry` is its
|
||
own top-level sibling of `model_list`.
|
||
|
||
### Referencing a vector store from a request
|
||
|
||
Not via a `model_list` entry — via the **`tools`** array of a
|
||
`/chat/completions` (or `/v1/responses`) request, OpenAI Assistants-style
|
||
`file_search` tool:
|
||
|
||
```json
|
||
{
|
||
"model": "gpt-4o",
|
||
"messages": [...],
|
||
"tools": [
|
||
{
|
||
"type": "file_search",
|
||
"vector_store_ids": ["T37J8R4WTM"]
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
LiteLLM intercepts the tool call, looks up the referenced ID in
|
||
`vector_store_registry`, calls that backend's native search, and injects the
|
||
retrieved chunks into the prompt before calling the target model. Source:
|
||
[docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
|
||
|
||
There are also direct, OpenAI-compatible proxy endpoints for managing/using a
|
||
store outside of a chat completion — see §4.
|
||
|
||
## 2. Backend providers — Qdrant is NOT one of them
|
||
|
||
Per the official docs page and the PR that introduced the feature, the
|
||
natively-supported `custom_llm_provider` values are:
|
||
|
||
| Provider | `custom_llm_provider` value | Notes |
|
||
|---|---|---|
|
||
| AWS Bedrock Knowledge Bases | `bedrock` | Original/reference implementation ([PR #10448](https://github.com/BerriAI/litellm/pull/10448)) |
|
||
| OpenAI Vector Stores | `openai` | Wraps OpenAI's own vector store API |
|
||
| Azure Vector Stores | `azure` | Assistants-API-only per docs |
|
||
| Azure AI Search | `azure_ai_search` (vector search capability) | |
|
||
| Vertex AI RAG Engine | `vertex_ai` | Added in [PR #15781](https://github.com/BerriAI/litellm/pull/15781) |
|
||
| Vertex AI Search API | `vertex_ai/search_api` | |
|
||
| Gemini File Search | — | |
|
||
| RAGFlow Datasets | — | Dataset mgmt only; search unsupported per docs |
|
||
| Postgres + pgvector | `pg_vector` (via a **separate** connector service) | See below — not built into the litellm proxy image |
|
||
|
||
Source: [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase),
|
||
cross-checked against [PR #12595](https://github.com/BerriAI/litellm/pull/12595)
|
||
("[Feat] Vector Stores - Add Vertex RAG Engine API as a provider") and
|
||
[PR #15781](https://github.com/BerriAI/litellm/pull/15781) ("(feat) Vector
|
||
Stores: support Vertex AI Search API").
|
||
|
||
**Qdrant is not listed anywhere** in the knowledgebase docs, the vector
|
||
store provider PRs, or the pgvector connector's own README. LiteLLM does use
|
||
Qdrant in one unrelated feature — **semantic response caching**
|
||
(`docs.litellm.ai/docs/caching/all_caches`, `qdrant_api_base` /
|
||
`qdrant_api_key` / `qdrant_collection_name` config) — but that is a cache for
|
||
LLM *responses*, not the vector-store/knowledgebase (RAG) feature, and shares
|
||
no config or code path with `vector_store_registry`. It would not let
|
||
LiteLLM search Qdrant-held documents as a knowledge base.
|
||
|
||
### The pgvector option is a separate microservice, not a built-in backend
|
||
|
||
[`BerriAI/litellm-pgvector`](https://github.com/BerriAI/litellm-pgvector) is
|
||
its own repo/container: a FastAPI app that exposes OpenAI-compatible vector
|
||
store endpoints backed by Postgres with the `pgvector` extension, calling
|
||
back out to a LiteLLM proxy's `/embeddings` endpoint to generate embeddings.
|
||
It is registered into `vector_store_registry` like any other backend (with
|
||
`custom_llm_provider: pg_vector` pointed at this companion service's URL),
|
||
but it is **not Qdrant** and **not part of the main `litellm` proxy image**
|
||
already running in this stack — it would mean deploying and operating a
|
||
fourth service (on top of `litellm`, `litellm-db`, and `llama-server`), with
|
||
its own Postgres database using the pgvector extension (the existing
|
||
`litellm-db` Postgres image, `postgres:16-alpine`, does not have pgvector
|
||
installed).
|
||
|
||
### Verdict on the existing Qdrant instance
|
||
|
||
**LiteLLM's knowledgebase/vector_store feature cannot point at this repo's
|
||
existing standalone `qdrant` service.** There is no Qdrant provider type for
|
||
`vector_store_registry`. To use LiteLLM's native feature at all, this stack
|
||
would need to either integrate with a cloud-native backend (Bedrock/Vertex/
|
||
Azure/OpenAI — none of which apply, this stack is local-only) or stand up
|
||
the separate `litellm-pgvector` + pgvector-enabled Postgres stack — an
|
||
entirely different vector store technology from the Qdrant already running
|
||
for Open WebUI. The existing Qdrant collection Open WebUI uses for its own
|
||
RAG/Memory feature is unrelated to and unreachable from LiteLLM's
|
||
knowledgebase feature.
|
||
|
||
## 3. Embedding model requirement
|
||
|
||
LiteLLM's vector store feature **does call an embedding endpoint itself**
|
||
when a backend needs one (pgvector explicitly; the managed cloud backends
|
||
handle embedding server-side). The field is `embedding_model` inside a
|
||
`vector_store_registry` entry's `litellm_params`, confirmed in
|
||
[BerriAI/litellm issue #23980](https://github.com/BerriAI/litellm/issues/23980)
|
||
("[Bug]: Vector store creation fails when using model mapping public model
|
||
name for embedding_model"), which shows:
|
||
|
||
```json
|
||
"litellm_params": {
|
||
"vector_bucket_name": "my-embeddings",
|
||
"index_name": "test-index",
|
||
"aws_region_name": "us-east-1",
|
||
"embedding_model": "test-vector-store/bedrock/amazon.nova-2-multimodal-embeddings-v1:0"
|
||
}
|
||
```
|
||
|
||
For the pgvector connector specifically, the embedding model is configured
|
||
via its own env vars (not `vector_store_registry`, since it's a separate
|
||
service): `EMBEDDING__MODEL`, `EMBEDDING__BASE_URL` (pointed at a LiteLLM
|
||
proxy), `EMBEDDING__API_KEY`, `EMBEDDING__DIMENSIONS` — e.g.
|
||
`EMBEDDING__MODEL=text-embedding-ada-002`,
|
||
`EMBEDDING__BASE_URL=http://litellm:4000`. Source:
|
||
[BerriAI/litellm-pgvector README](https://github.com/BerriAI/litellm-pgvector/blob/main/README.md).
|
||
This confirms: **yes, the embedding model must be reachable as a model
|
||
LiteLLM's proxy can call** — i.e. it needs its own `model_list` entry with
|
||
`mode: embedding` (LiteLLM's standard way of declaring an embedding-capable
|
||
model — see [docs.litellm.ai/docs/embedding/supported_embedding](https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/embedding/supported_embedding.md)),
|
||
so the pgvector service (or LiteLLM itself for backends that embed
|
||
internally) can call `POST /embeddings` against it through the proxy.
|
||
|
||
### Does llama.cpp (this repo's model server) support `/embeddings`?
|
||
|
||
Yes, but not enabled as currently configured, and not well-suited to the
|
||
model already loaded. llama.cpp's server (`tools/server`, the same
|
||
`ghcr.io/ggml-org/llama.cpp:server-rocm` image this repo uses per
|
||
`docker-compose.yml`) exposes an OpenAI-compatible `POST /v1/embeddings`
|
||
route (and a native `/embedding` route), **but only when started with the
|
||
`--embeddings` flag** — source:
|
||
[ggml-org/llama.cpp tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
|
||
This repo's `llama-server` command in `docker-compose.yml` (lines 16–22) does
|
||
not pass `--embeddings`, so the currently-running instance does not serve
|
||
embeddings at all.
|
||
|
||
Even if the flag were added, llama.cpp loads **one model per server
|
||
process** — the flag would make the already-loaded Qwen3.8-27B **chat**
|
||
model (per `litellm-config.yaml`, `qwen3.8-27b-local` /
|
||
`Qwen3.8-27B-UD-Q4_K_XL.gguf`) emit pooled hidden-state vectors as
|
||
"embeddings," but a generalist instruction-tuned chat model is not what
|
||
that's trained for — embedding quality from a non-embedding-trained model is
|
||
materially worse than a purpose-trained embedding model (e.g. BGE, Nomic
|
||
Embed, mxbai-embed, gte). **A separate, dedicated embedding model/server
|
||
would be needed** — either a second `llama-server` container loaded with a
|
||
small GGUF embedding model (`--embeddings` flag on), or a different
|
||
embedding-serving stack — and it would need its own `model_list` entry in
|
||
`litellm-config.yaml` with `mode: embedding` for LiteLLM/litellm-pgvector to
|
||
call it.
|
||
|
||
## 4. How documents actually get ingested
|
||
|
||
Two ingestion paths exist depending on backend, both are HTTP APIs — there
|
||
is no admin-UI "add document" flow beyond store *creation*, and no CLI:
|
||
|
||
**a) OpenAI-compatible vector-store file API** (used for the `openai`
|
||
backend, and shown as the general pattern in the docs):
|
||
|
||
- `POST /v1/vector_stores` — create a store. Body:
|
||
```json
|
||
{
|
||
"name": "My Document Store",
|
||
"file_ids": ["file-abc123"],
|
||
"chunking_strategy": {
|
||
"type": "static",
|
||
"static": {"max_chunk_size_tokens": 800, "chunk_overlap_tokens": 400}
|
||
},
|
||
"metadata": {"key": "value"}
|
||
}
|
||
```
|
||
Requires files already uploaded through a separate Files API to obtain
|
||
`file_id`s first — the docs page does not show a files-upload endpoint
|
||
under the vector-store docs directly (this is OpenAI's own two-step
|
||
upload-then-attach flow, proxied through). Source:
|
||
[docs.litellm.ai/docs/vector_stores/create](https://docs.litellm.ai/docs/vector_stores/create).
|
||
- `POST /v1/vector_stores/{vector_store_id}/search` — query it:
|
||
```json
|
||
{
|
||
"query": "What is the capital of France?",
|
||
"filters": {"file_ids": ["file-abc123"]},
|
||
"max_num_results": 5,
|
||
"ranking_options": {"score_threshold": 0.7},
|
||
"rewrite_query": true
|
||
}
|
||
```
|
||
Source: [docs.litellm.ai/docs/vector_stores/search](https://docs.litellm.ai/docs/vector_stores/search).
|
||
- `GET /vector_store/list` — list registered stores. Source:
|
||
[docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
|
||
- LiteLLM Admin UI: **Experimental → Vector Stores → Create Vector Store**
|
||
exists for registering/creating stores, and the **Logs** page shows
|
||
vector-store search queries/scores after use — but this is store
|
||
management and observability, not a bulk document-ingestion UI. Source:
|
||
[docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
|
||
|
||
**b) `litellm-pgvector` connector's own embeddings API** (only path relevant
|
||
if this repo went the self-hosted pgvector route, since Qdrant isn't
|
||
supported at all): direct chunk-level ingestion, no file upload step —
|
||
|
||
- `POST /v1/vector_stores/{id}/embeddings` — single chunk:
|
||
```json
|
||
{"content": "...", "embedding": [/* optional, else computed server-side */], "metadata": {}}
|
||
```
|
||
- `POST /v1/vector_stores/{id}/embeddings/batch` — array of the same shape,
|
||
for bulk loading.
|
||
|
||
Source: [BerriAI/litellm-pgvector README](https://github.com/BerriAI/litellm-pgvector/blob/main/README.md).
|
||
|
||
For a follow-up ticket that wants to programmatically load
|
||
`data/memory.md` / `data/claude-legacy-memory.md` (dated fact lists) into a
|
||
knowledge base, the realistic path through LiteLLM's native feature would be
|
||
the pgvector connector's batch-embeddings endpoint (chunk the Markdown into
|
||
facts/sections client-side, POST each as a batch) — **but that requires
|
||
first standing up**: (1) a pgvector-enabled Postgres, (2) the
|
||
`litellm-pgvector` service, and (3) a dedicated embedding model server
|
||
registered in `litellm-config.yaml`. None of that reuses the Qdrant instance
|
||
already running in this stack.
|
||
|
||
## 5. Repo context read for this research
|
||
|
||
- `litellm-config.yaml` — current config has one `model_list` entry
|
||
(`qwen3.8-27b-local`, chat-only, via llama.cpp), `router_settings`
|
||
(priority scheduling), `general_settings.master_key`. No
|
||
`vector_store_registry` block exists yet.
|
||
- `docker-compose.yml` — confirms `qdrant` service (image `qdrant/qdrant`,
|
||
network `ai-stack`, no host port, only `open-webui` currently consumes it
|
||
via `VECTOR_DB=qdrant` / `QDRANT_URI=http://qdrant:6333`); confirms
|
||
`llama-server` command has no `--embeddings` flag and loads a single GGUF
|
||
(`Qwen3.8-27B-UD-Q4_K_XL.gguf`); confirms `litellm-db` is plain
|
||
`postgres:16-alpine` (no pgvector extension installed).
|
||
- `CLAUDE.md` — points to `docs/agents/issue-tracker.md` (Gitea issues via
|
||
`tea`) and `docs/agents/domain.md` (domain docs convention).
|
||
- `docs/agents/domain.md` — says to check `CONTEXT.md` and `docs/adr/` at
|
||
repo root before exploring, but "if any of these files don't exist,
|
||
proceed silently." Neither `CONTEXT.md` nor `docs/adr/` exist in this repo
|
||
yet, so there's no glossary/ADR conflict to flag.
|
||
- `docs/research/` — repeated existing precedent (e.g.
|
||
`docs/research/proxy-shadow-pricing.md`, `docs/research/voidllm-evaluation.md`,
|
||
`docs/research/qwen3.8-27b-tool-calling.md`) confirms this is the
|
||
established location and Markdown format for this kind of investigation;
|
||
this file follows that convention.
|
||
|
||
## Provenance note on the top-level config key name
|
||
|
||
The docs page fetched live (`docs.litellm.ai/docs/completion/knowledgebase`)
|
||
shows `vector_store_registry` as the current top-level key. The original
|
||
feature PR ([#10448](https://github.com/BerriAI/litellm/pull/10448)) used
|
||
`vector_stores` in its example YAML. If implementing against a specific
|
||
pinned LiteLLM version, verify the exact key against that version's docs/
|
||
source rather than assuming either name — this repo's `docker-compose.yml`
|
||
pins `ghcr.io/berriai/litellm:main-stable`, a rolling tag, so the schema in
|
||
whatever image is actually pulled should be spot-checked (e.g. `GET /openapi.json`
|
||
against the running proxy, or grepping the image's installed
|
||
`litellm/types/router.py` / proxy schema) before writing config against it.
|