haylanandClaude-Bot
b4dc83949e
New redis service (redis:7-alpine, password-protected, no persistence volume — everything it holds is cache/coordination state). litellm gets REDIS_HOST/REDIS_PORT/REDIS_PASSWORD, which is all LiteLLM needs to use it for router state, rate limits/budgets, and cache invalidation — no litellm-config.yaml block required (docs.litellm.ai/docs/proxy/caching). REDIS_PASSWORD added to .env.example and update.sh's auto-generated secrets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
66 lines
3.1 KiB
YAML
66 lines
3.1 KiB
YAML
model_list:
|
|
- model_name: qwen3.8-27b-local
|
|
litellm_params:
|
|
# Static name — llama.cpp serves whatever model it loaded regardless of
|
|
# what's requested here; this string isn't shell-expanded (this file
|
|
# isn't docker-compose.yml, .env vars don't reach it).
|
|
model: openai/qwen3.8-27b-local
|
|
api_base: http://llama-server:8080/v1
|
|
api_key: local
|
|
# Qwen3 is a reasoning model — it spends output tokens on
|
|
# reasoning_content before ever writing content. Callers that don't
|
|
# set their own max_tokens (Open WebUI's default request didn't) hit
|
|
# llama.cpp's low default, so the model runs out mid-thought and
|
|
# content comes back empty. This is a floor, not a cap — any caller
|
|
# that passes its own max_tokens still overrides it.
|
|
max_tokens: 4096
|
|
model_info:
|
|
# Shadow cloud-cost estimate — priced against Claude Sonnet 5's published
|
|
# rate, not real spend (this proxy only ever routes to the local model).
|
|
# Source: https://platform.claude.com/docs/en/about-claude/pricing,
|
|
# checked 2026-08-25. Update these two numbers if that page changes.
|
|
input_cost_per_token: 0.000002 # $2 / MTok
|
|
output_cost_per_token: 0.00001 # $10 / MTok
|
|
|
|
- model_name: local-embedding
|
|
litellm_params:
|
|
# Served by the dedicated embedding-server (nomic-embed-text-v1.5), not
|
|
# the chat model — see docker-compose.yml. Called by memory-retrieval
|
|
# to embed knowledgebase content, and available directly at
|
|
# /v1/embeddings for anything else that wants it.
|
|
model: openai/local-embedding
|
|
api_base: http://embedding-server:8080/v1
|
|
api_key: local
|
|
model_info:
|
|
mode: embedding
|
|
|
|
# SearXNG-backed web search — a standalone REST endpoint (/v1/search/searxng-search),
|
|
# NOT a model-callable tool and not auto-injected into chat completions. See
|
|
# docs/research/litellm-searxng-search.md. Requires the litellm container to
|
|
# resolve search.home — see the `extra_hosts` entry in docker-compose.yml.
|
|
search_tools:
|
|
- search_tool_name: searxng-search
|
|
litellm_params:
|
|
search_provider: searxng
|
|
api_base: http://search.home/
|
|
|
|
# Knowledgebase / RAG lives outside LiteLLM's own registry now — see the
|
|
# memory-retrieval service (docker-compose.yml) and
|
|
# docs/research/langchain-pgvector-vs-litellm-pgvector.md. LiteLLM's native
|
|
# vector_store_registry has no Qdrant provider and no langchain_postgres
|
|
# provider either, so registering a store here isn't an option; callers
|
|
# query memory-retrieval's /query endpoint directly instead of an in-band
|
|
# file_search tool call.
|
|
|
|
router_settings:
|
|
# ponytail: LiteLLM's request-prioritization scheduler is beta (see
|
|
# docs/proxy-request-priority.md) — exact settings key/shape must be
|
|
# confirmed against LiteLLM's current docs and smoke-tested against
|
|
# llama.cpp before workloads depend on it. Redis is available (see the
|
|
# litellm service's REDIS_* env vars in docker-compose.yml) if the
|
|
# scheduler needs shared state for it.
|
|
enable_priority_scheduling: true
|
|
|
|
general_settings:
|
|
master_key: os.environ/LITELLM_MASTER_KEY
|