haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

How does LiteLLM's knowledgebase/vector-store feature work, and what does it need? #23

New Issue
Closed
opened 2026-09-02 19:18:55 +00:00 by haylan · 1 comment
haylan commented 2026-09-02 19:18:55 +00:00
Owner
Copy Link
Copy Source

Part of #21

Question

How does LiteLLM's vector store / knowledgebase feature (https://docs.litellm.ai/docs/completion/knowledgebase) actually work — config shape in litellm-config.yaml, how it integrates with an existing standalone Qdrant instance (already running as the qdrant service on the ai-stack network, currently only consumed by Open WebUI), what embedding model/endpoint it requires, and how documents/facts get ingested into it (API call, CLI, file watch, etc).

Part of #21 ## Question How does LiteLLM's vector store / knowledgebase feature (https://docs.litellm.ai/docs/completion/knowledgebase) actually work — config shape in `litellm-config.yaml`, how it integrates with an existing standalone Qdrant instance (already running as the `qdrant` service on the `ai-stack` network, currently only consumed by Open WebUI), what embedding model/endpoint it requires, and how documents/facts get ingested into it (API call, CLI, file watch, etc).
haylan added the wayfinder:research label 2026-09-02 19:18:55 +00:00
haylan commented 2026-09-02 19:27:36 +00:00
Author
Owner
Copy Link
Copy Source

Findings (full detail in docs/research/litellm-knowledgebase.md on branch research/litellm-knowledgebase):

  1. Config shape: top-level vector_store_registry list (not under model_list), each entry with litellm_params.vector_store_id, custom_llm_provider, plus optional description/metadata/credential/embedding_model. Referenced from a request via tools: [{"type": "file_search", "vector_store_ids": [...]}].

  2. Qdrant reuse: not possible. Supported custom_llm_provider backends are bedrock, openai, azure, azure_ai_search, vertex_ai/vertex_ai/search_api, Gemini File Search, RAGFlow, and Postgres+pgvector (via a separate companion service, BerriAI/litellm-pgvector — not the main litellm image). Qdrant isn't one of them; the only Qdrant integration LiteLLM has is unrelated (semantic response caching). The existing qdrant service in this repo can't back this feature.

  3. Embedding model: required, and needs new infra. LiteLLM (or the pgvector connector) calls an embedding endpoint itself, needing its own model_list entry (mode: embedding). llama.cpp supports OpenAI-compatible /v1/embeddings, but only with --embeddings on the command line (not set today), and it's single-model-per-process — currently loaded with the Qwen3 chat model. A dedicated embedding model/server is needed, not a repurposed chat model.

  4. Ingestion: HTTP APIs only, no CLI/bulk-upload UI. Realistic self-hosted path (pgvector, since Qdrant isn't supported): POST /v1/vector_stores/{id}/embeddings / /embeddings/batch on the separate litellm-pgvector service, taking raw chunk content + metadata directly.

Net implication for #21's destination: reusing the existing Qdrant service isn't an option for LiteLLM's native knowledgebase feature. Getting this working requires standing up pgvector-Postgres + the litellm-pgvector companion service + a dedicated embedding model/server — none of which exist in this stack today. That's a real infra decision, not a config tweak — worth a grilling session before ticketing the build-out (Qdrant-reuse-only vs. add pgvector, and whether a second GPU-resident embedding model is acceptable next to the chat model).

**Findings** (full detail in `docs/research/litellm-knowledgebase.md` on branch `research/litellm-knowledgebase`): 1. **Config shape**: top-level `vector_store_registry` list (not under `model_list`), each entry with `litellm_params.vector_store_id`, `custom_llm_provider`, plus optional description/metadata/credential/`embedding_model`. Referenced from a request via `tools: [{"type": "file_search", "vector_store_ids": [...]}]`. 2. **Qdrant reuse: not possible.** Supported `custom_llm_provider` backends are `bedrock`, `openai`, `azure`, `azure_ai_search`, `vertex_ai`/`vertex_ai/search_api`, Gemini File Search, RAGFlow, and Postgres+pgvector (via a separate companion service, `BerriAI/litellm-pgvector` — not the main litellm image). Qdrant isn't one of them; the only Qdrant integration LiteLLM has is unrelated (semantic response caching). The existing `qdrant` service in this repo can't back this feature. 3. **Embedding model: required, and needs new infra.** LiteLLM (or the pgvector connector) calls an embedding endpoint itself, needing its own `model_list` entry (`mode: embedding`). llama.cpp supports OpenAI-compatible `/v1/embeddings`, but only with `--embeddings` on the command line (not set today), and it's single-model-per-process — currently loaded with the Qwen3 chat model. A dedicated embedding model/server is needed, not a repurposed chat model. 4. **Ingestion**: HTTP APIs only, no CLI/bulk-upload UI. Realistic self-hosted path (pgvector, since Qdrant isn't supported): `POST /v1/vector_stores/{id}/embeddings` / `/embeddings/batch` on the separate `litellm-pgvector` service, taking raw chunk content + metadata directly. **Net implication for #21's destination**: reusing the existing Qdrant service isn't an option for LiteLLM's native knowledgebase feature. Getting this working requires standing up pgvector-Postgres + the `litellm-pgvector` companion service + a dedicated embedding model/server — none of which exist in this stack today. That's a real infra decision, not a config tweak — worth a grilling session before ticketing the build-out (Qdrant-reuse-only vs. add pgvector, and whether a second GPU-resident embedding model is acceptable next to the chat model).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: haylan/LLM-Server#23