haylan
124053cf89
The vector_store_registry block in litellm-config.yaml only seeds the store into litellm's in-memory registry at boot — the Admin UI's Vector Stores page (/ui/vector-stores) reads litellm's own DB (LiteLLM_ManagedVectorStoresTable) instead, via /vector_store/list. A config-only entry there gets silently deleted from memory the first time anyone loads that page, since /vector_store/list treats the DB as the source of truth and removes anything not also present there. update.sh now registers it in the DB too, via POST /vector_store/new (idempotent, same pattern as the existing virtual-key minting) — with the literal resolved key, not the os.environ/... form used in litellm-config.yaml, since the management API doesn't do config.yaml-style substitution on request bodies. Found in the process: /vector_store/update in this litellm version can't touch litellm_params at all (VectorStoreUpdateRequest has no such field, so PGVECTOR_API_KEY sent through it is silently dropped) — worth knowing if this key ever needs rotating; documented in update.sh's comment. Verified live: registered via the API, confirmed the row appears in /vector_store/list (what the Admin UI page reads) with the real key, and re-verified search still works end-to-end afterward.
98 lines
5.0 KiB
YAML
98 lines
5.0 KiB
YAML
model_list:
|
|
- model_name: qwen3.8-27b-local
|
|
litellm_params:
|
|
# Static name — llama.cpp serves whatever model it loaded regardless of
|
|
# what's requested here; this string isn't shell-expanded (this file
|
|
# isn't docker-compose.yml, .env vars don't reach it).
|
|
model: openai/qwen3.8-27b-local
|
|
api_base: http://llama-server:8080/v1
|
|
api_key: local
|
|
# Qwen3 is a reasoning model — it spends output tokens on
|
|
# reasoning_content before ever writing content. Callers that don't
|
|
# set their own max_tokens (Open WebUI's default request didn't) hit
|
|
# llama.cpp's low default, so the model runs out mid-thought and
|
|
# content comes back empty. This is a floor, not a cap — any caller
|
|
# that passes its own max_tokens still overrides it.
|
|
# Raised from 4096: confirmed in the wild (llama-server logs) that
|
|
# 4096 wasn't enough — reasoning_content alone ate the whole budget on
|
|
# a real request (n_gen = 4096 exactly, no answer ever written). At
|
|
# ~26.7 t/s and a 65536-token context window, 16384 is a ~10-minute
|
|
# worst case, not the full ~20-minute worst case 32768 would be.
|
|
max_tokens: 16384
|
|
model_info:
|
|
# Shadow cloud-cost estimate — priced against Claude Sonnet 5's published
|
|
# rate, not real spend (this proxy only ever routes to the local model).
|
|
# Source: https://platform.claude.com/docs/en/about-claude/pricing,
|
|
# checked 2026-08-25. Update these two numbers if that page changes.
|
|
input_cost_per_token: 0.000002 # $2 / MTok
|
|
output_cost_per_token: 0.00001 # $10 / MTok
|
|
|
|
- model_name: local-embedding
|
|
litellm_params:
|
|
# Served by the dedicated embedding-server (nomic-embed-text-v1.5), not
|
|
# the chat model — see docker-compose.yml. Called by litellm-pgvector
|
|
# to embed knowledgebase content, and available directly at
|
|
# /v1/embeddings for anything else that wants it.
|
|
model: openai/local-embedding
|
|
api_base: http://embedding-server:8080/v1
|
|
api_key: local
|
|
model_info:
|
|
mode: embedding
|
|
|
|
# SearXNG-backed web search — a standalone REST endpoint (/v1/search/searxng-search),
|
|
# NOT a model-callable tool and not auto-injected into chat completions. See
|
|
# docs/research/litellm-searxng-search.md. Requires the litellm container to
|
|
# resolve search.home — see the `extra_hosts` entry in docker-compose.yml.
|
|
search_tools:
|
|
- search_tool_name: searxng-search
|
|
litellm_params:
|
|
search_provider: searxng
|
|
api_base: http://search.home/
|
|
|
|
# Knowledgebase / RAG, backed by the litellm-pgvector companion service (NOT
|
|
# Qdrant — LiteLLM's native vector-store feature has no Qdrant provider, see
|
|
# docs/research/litellm-knowledgebase.md). vector_store_id is this proxy's
|
|
# own identifier for the store, not assigned by a backend.
|
|
# Smoke-tested end-to-end against a running deploy (issue #24): search via
|
|
# both /v1/vector_stores/{id}/search directly and the file_search tool on a
|
|
# chat completion. Needed several fixes beyond this block to work — a
|
|
# missing api_key here, litellm-pgvector's Prisma schema never having been
|
|
# pushed, a 1536- vs 768-dim mismatch, and its create endpoint ignoring any
|
|
# caller-supplied id — see scripts/update.sh, scripts/ingest-memory.sh, and
|
|
# vendor/litellm-pgvector/'s local patches (models.py, main.py,
|
|
# prisma/schema.prisma).
|
|
#
|
|
# This block only seeds the store into litellm's in-memory registry at
|
|
# boot — it does NOT make it appear on the Admin UI's Vector Stores page
|
|
# (/ui/vector-stores). That page reads litellm's own DB
|
|
# (LiteLLM_ManagedVectorStoresTable), a separate registration scripts/
|
|
# update.sh also does via POST /vector_store/new. Keep both in sync by
|
|
# hand if you change api_base/api_key here — see update.sh's comment on
|
|
# why there's no automatic sync from this block to the DB row.
|
|
vector_store_registry:
|
|
- vector_store_name: memory-and-notes
|
|
litellm_params:
|
|
vector_store_id: "memory-and-notes"
|
|
custom_llm_provider: pg_vector
|
|
api_base: http://litellm-pgvector:8000
|
|
# Required by litellm's pg_vector provider (see
|
|
# PGVectorStoreConfig.validate_environment in litellm's source) — it's
|
|
# the Bearer token litellm-pgvector's own API checks against its
|
|
# SERVER_API_KEY. Was missing entirely, which is why every vector
|
|
# store call was failing with "Incorrect API key provided: None"
|
|
# before litellm-pgvector was ever reached. See issue #24.
|
|
api_key: os.environ/LITELLM_PGVECTOR_API_KEY
|
|
embedding_model: local-embedding
|
|
|
|
router_settings:
|
|
# ponytail: LiteLLM's request-prioritization scheduler is beta (see
|
|
# docs/proxy-request-priority.md) — exact settings key/shape must be
|
|
# confirmed against LiteLLM's current docs and smoke-tested against
|
|
# llama.cpp before workloads depend on it. Redis is available (see the
|
|
# litellm service's REDIS_* env vars in docker-compose.yml) if the
|
|
# scheduler needs shared state for it.
|
|
enable_priority_scheduling: true
|
|
|
|
general_settings:
|
|
master_key: os.environ/LITELLM_MASTER_KEY
|