LLM-Server/docs/research/litellm-knowledgebase.md

Research: LiteLLM vector store / knowledgebase feature (for Gitea issue #23)

Question: Can LiteLLM's built-in "knowledgebase" / vector store feature be used to load two Markdown fact files (data/memory.md, data/claude-legacy-memory.md) into a queryable knowledge base, and if so, what does that require from this stack (config shape, backend, embedding model, ingestion mechanism)?

Bottom line: no, not against this stack's existing Qdrant instance, and not without adding a dedicated embedding model. LiteLLM's vector store feature is a routing/registry layer over a small set of natively-integrated backends (Bedrock Knowledge Bases, OpenAI Vector Stores, Azure Vector/AI-Search, Vertex AI RAG/Search, Gemini File Search, RAGFlow) plus one self-hosted option — Postgres+pgvector, via a separate companion service (BerriAI/litellm-pgvector), not Qdrant. There is no Qdrant provider at all. Ingesting the two fact files would also require running a dedicated embedding-capable model, which this stack doesn't currently have (the one llama.cpp instance serves a chat model, not started with --embeddings).

1. Config shape in config.yaml

Top-level block is vector_store_registry (this is the current key name; an earlier PR that introduced the feature used vector_stores — see Provenance note below), a list of entries:

vector_store_registry:
  - vector_store_name: "my-knowledgebase"      # optional friendly name
    litellm_params:
      vector_store_id: "T37J8R4WTM"             # required, provider's own ID
      custom_llm_provider: "bedrock"            # required — selects backend
      vector_store_description: "..."           # optional
      vector_store_metadata: {}                 # optional
      litellm_credential_name: "..."            # optional, reuse a named credential
      embedding_model: "..."                    # backend-dependent — see §3

Source: docs.litellm.ai/docs/completion/knowledgebase (fetched directly). Earlier shape (same idea, older field names) documented in BerriAI/litellm PR #10448 ("[Feat] Vector Stores/KnowledgeBases - Allow defining Vector Store Configs"):

vector_stores:
  - vector_store_name: "bedrock-litellm-website-knowledgebase"
    litellm_params:
      custom_llm_provider: "bedrock"
      id: "T37J8R4WTM"

This is a registry of pointers to knowledge bases that already exist on the backend provider — it is not itself a vector database. model_list entries are not where a store is attached; vector_store_registry is its own top-level sibling of model_list.

Referencing a vector store from a request

Not via a model_list entry — via the tools array of a /chat/completions (or /v1/responses) request, OpenAI Assistants-style file_search tool:

{
  "model": "gpt-4o",
  "messages": [...],
  "tools": [
    {
      "type": "file_search",
      "vector_store_ids": ["T37J8R4WTM"]
    }
  ]
}

LiteLLM intercepts the tool call, looks up the referenced ID in vector_store_registry, calls that backend's native search, and injects the retrieved chunks into the prompt before calling the target model. Source: docs.litellm.ai/docs/completion/knowledgebase.

There are also direct, OpenAI-compatible proxy endpoints for managing/using a store outside of a chat completion — see §4.

2. Backend providers — Qdrant is NOT one of them

Per the official docs page and the PR that introduced the feature, the natively-supported custom_llm_provider values are:

Provider	custom_llm_provider value	Notes
AWS Bedrock Knowledge Bases	bedrock	Original/reference implementation (PR #10448)
OpenAI Vector Stores	openai	Wraps OpenAI's own vector store API
Azure Vector Stores	azure	Assistants-API-only per docs
Azure AI Search	azure_ai_search (vector search capability)
Vertex AI RAG Engine	vertex_ai	Added in PR #15781
Vertex AI Search API	vertex_ai/search_api
Gemini File Search	—
RAGFlow Datasets	—	Dataset mgmt only; search unsupported per docs
Postgres + pgvector	pg_vector (via a separate connector service)	See below — not built into the litellm proxy image

Source: docs.litellm.ai/docs/completion/knowledgebase, cross-checked against PR #12595 ("[Feat] Vector Stores - Add Vertex RAG Engine API as a provider") and PR #15781 ("(feat) Vector Stores: support Vertex AI Search API").

Qdrant is not listed anywhere in the knowledgebase docs, the vector store provider PRs, or the pgvector connector's own README. LiteLLM does use Qdrant in one unrelated feature — semantic response caching (docs.litellm.ai/docs/caching/all_caches, qdrant_api_base / qdrant_api_key / qdrant_collection_name config) — but that is a cache for LLM responses, not the vector-store/knowledgebase (RAG) feature, and shares no config or code path with vector_store_registry. It would not let LiteLLM search Qdrant-held documents as a knowledge base.

The pgvector option is a separate microservice, not a built-in backend

BerriAI/litellm-pgvector is its own repo/container: a FastAPI app that exposes OpenAI-compatible vector store endpoints backed by Postgres with the pgvector extension, calling back out to a LiteLLM proxy's /embeddings endpoint to generate embeddings. It is registered into vector_store_registry like any other backend (with custom_llm_provider: pg_vector pointed at this companion service's URL), but it is not Qdrant and not part of the main litellm proxy image already running in this stack — it would mean deploying and operating a fourth service (on top of litellm, litellm-db, and llama-server), with its own Postgres database using the pgvector extension (the existing litellm-db Postgres image, postgres:16-alpine, does not have pgvector installed).

Verdict on the existing Qdrant instance

LiteLLM's knowledgebase/vector_store feature cannot point at this repo's existing standalone qdrant service. There is no Qdrant provider type for vector_store_registry. To use LiteLLM's native feature at all, this stack would need to either integrate with a cloud-native backend (Bedrock/Vertex/ Azure/OpenAI — none of which apply, this stack is local-only) or stand up the separate litellm-pgvector + pgvector-enabled Postgres stack — an entirely different vector store technology from the Qdrant already running for Open WebUI. The existing Qdrant collection Open WebUI uses for its own RAG/Memory feature is unrelated to and unreachable from LiteLLM's knowledgebase feature.

3. Embedding model requirement

LiteLLM's vector store feature does call an embedding endpoint itself when a backend needs one (pgvector explicitly; the managed cloud backends handle embedding server-side). The field is embedding_model inside a vector_store_registry entry's litellm_params, confirmed in BerriAI/litellm issue #23980 ("[Bug]: Vector store creation fails when using model mapping public model name for embedding_model"), which shows:

"litellm_params": {
  "vector_bucket_name": "my-embeddings",
  "index_name": "test-index",
  "aws_region_name": "us-east-1",
  "embedding_model": "test-vector-store/bedrock/amazon.nova-2-multimodal-embeddings-v1:0"
}

For the pgvector connector specifically, the embedding model is configured via its own env vars (not vector_store_registry, since it's a separate service): EMBEDDING__MODEL, EMBEDDING__BASE_URL (pointed at a LiteLLM proxy), EMBEDDING__API_KEY, EMBEDDING__DIMENSIONS — e.g. EMBEDDING__MODEL=text-embedding-ada-002, EMBEDDING__BASE_URL=http://litellm:4000. Source: BerriAI/litellm-pgvector README. This confirms: yes, the embedding model must be reachable as a model LiteLLM's proxy can call — i.e. it needs its own model_list entry with mode: embedding (LiteLLM's standard way of declaring an embedding-capable model — see docs.litellm.ai/docs/embedding/supported_embedding), so the pgvector service (or LiteLLM itself for backends that embed internally) can call POST /embeddings against it through the proxy.

Does llama.cpp (this repo's model server) support /embeddings?

Yes, but not enabled as currently configured, and not well-suited to the model already loaded. llama.cpp's server (tools/server, the same ghcr.io/ggml-org/llama.cpp:server-rocm image this repo uses per docker-compose.yml) exposes an OpenAI-compatible POST /v1/embeddings route (and a native /embedding route), but only when started with the --embeddings flag — source: ggml-org/llama.cpp tools/server/README.md. This repo's llama-server command in docker-compose.yml (lines 16–22) does not pass --embeddings, so the currently-running instance does not serve embeddings at all.

Even if the flag were added, llama.cpp loads one model per server process — the flag would make the already-loaded Qwen3.8-27B chat model (per litellm-config.yaml, qwen3.8-27b-local / Qwen3.8-27B-UD-Q4_K_XL.gguf) emit pooled hidden-state vectors as "embeddings," but a generalist instruction-tuned chat model is not what that's trained for — embedding quality from a non-embedding-trained model is materially worse than a purpose-trained embedding model (e.g. BGE, Nomic Embed, mxbai-embed, gte). A separate, dedicated embedding model/server would be needed — either a second llama-server container loaded with a small GGUF embedding model (--embeddings flag on), or a different embedding-serving stack — and it would need its own model_list entry in litellm-config.yaml with mode: embedding for LiteLLM/litellm-pgvector to call it.

4. How documents actually get ingested

Two ingestion paths exist depending on backend, both are HTTP APIs — there is no admin-UI "add document" flow beyond store creation, and no CLI:

a) OpenAI-compatible vector-store file API (used for the openai backend, and shown as the general pattern in the docs):

• POST /v1/vector_stores — create a store. Body:

  {
    "name": "My Document Store",
    "file_ids": ["file-abc123"],
    "chunking_strategy": {
      "type": "static",
      "static": {"max_chunk_size_tokens": 800, "chunk_overlap_tokens": 400}
    },
    "metadata": {"key": "value"}
  }
  

  Requires files already uploaded through a separate Files API to obtain file_ids first — the docs page does not show a files-upload endpoint under the vector-store docs directly (this is OpenAI's own two-step upload-then-attach flow, proxied through). Source: docs.litellm.ai/docs/vector_stores/create.
• POST /v1/vector_stores/{vector_store_id}/search — query it:

  {
    "query": "What is the capital of France?",
    "filters": {"file_ids": ["file-abc123"]},
    "max_num_results": 5,
    "ranking_options": {"score_threshold": 0.7},
    "rewrite_query": true
  }
  

  Source: docs.litellm.ai/docs/vector_stores/search.
• GET /vector_store/list — list registered stores. Source: docs.litellm.ai/docs/completion/knowledgebase.
• LiteLLM Admin UI: Experimental → Vector Stores → Create Vector Store exists for registering/creating stores, and the Logs page shows vector-store search queries/scores after use — but this is store management and observability, not a bulk document-ingestion UI. Source: docs.litellm.ai/docs/completion/knowledgebase.

b) litellm-pgvector connector's own embeddings API (only path relevant if this repo went the self-hosted pgvector route, since Qdrant isn't supported at all): direct chunk-level ingestion, no file upload step —

• POST /v1/vector_stores/{id}/embeddings — single chunk:

  {"content": "...", "embedding": [/* optional, else computed server-side */], "metadata": {}}
  

• POST /v1/vector_stores/{id}/embeddings/batch — array of the same shape, for bulk loading.

Source: BerriAI/litellm-pgvector README.

For a follow-up ticket that wants to programmatically load data/memory.md / data/claude-legacy-memory.md (dated fact lists) into a knowledge base, the realistic path through LiteLLM's native feature would be the pgvector connector's batch-embeddings endpoint (chunk the Markdown into facts/sections client-side, POST each as a batch) — but that requires first standing up: (1) a pgvector-enabled Postgres, (2) the litellm-pgvector service, and (3) a dedicated embedding model server registered in litellm-config.yaml. None of that reuses the Qdrant instance already running in this stack.

5. Repo context read for this research

• litellm-config.yaml — current config has one model_list entry (qwen3.8-27b-local, chat-only, via llama.cpp), router_settings (priority scheduling), general_settings.master_key. No vector_store_registry block exists yet.
• docker-compose.yml — confirms qdrant service (image qdrant/qdrant, network ai-stack, no host port, only open-webui currently consumes it via VECTOR_DB=qdrant / QDRANT_URI=http://qdrant:6333); confirms llama-server command has no --embeddings flag and loads a single GGUF (Qwen3.8-27B-UD-Q4_K_XL.gguf); confirms litellm-db is plain postgres:16-alpine (no pgvector extension installed).
• CLAUDE.md — points to docs/agents/issue-tracker.md (Gitea issues via tea) and docs/agents/domain.md (domain docs convention).
• docs/agents/domain.md — says to check CONTEXT.md and docs/adr/ at repo root before exploring, but "if any of these files don't exist, proceed silently." Neither CONTEXT.md nor docs/adr/ exist in this repo yet, so there's no glossary/ADR conflict to flag.
• docs/research/ — repeated existing precedent (e.g. docs/research/proxy-shadow-pricing.md, docs/research/voidllm-evaluation.md, docs/research/qwen3.8-27b-tool-calling.md) confirms this is the established location and Markdown format for this kind of investigation; this file follows that convention.

Provenance note on the top-level config key name

The docs page fetched live (docs.litellm.ai/docs/completion/knowledgebase) shows vector_store_registry as the current top-level key. The original feature PR (#10448) used vector_stores in its example YAML. If implementing against a specific pinned LiteLLM version, verify the exact key against that version's docs/ source rather than assuming either name — this repo's docker-compose.yml pins ghcr.io/berriai/litellm:main-stable, a rolling tag, so the schema in whatever image is actually pulled should be spot-checked (e.g. GET /openapi.json against the running proxy, or grepping the image's installed litellm/types/router.py / proxy schema) before writing config against it.