haylanandClaude-Bot
5cb34b19f3
Open WebUI now points at litellm instead of llama-server directly, using a provisioned virtual key. llama-server's host port is dropped (internal-only on the ai-stack network) since the proxy is the only intended entry point now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with per-CLI virtual keys instead of the old shared dummy key. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
28 lines
1.3 KiB
YAML
28 lines
1.3 KiB
YAML
model_list:
|
|
- model_name: qwen3.8-27b-local
|
|
litellm_params:
|
|
# Static name — llama.cpp serves whatever model it loaded regardless of
|
|
# what's requested here; this string isn't shell-expanded (this file
|
|
# isn't docker-compose.yml, .env vars don't reach it).
|
|
model: openai/qwen3.8-27b-local
|
|
api_base: http://llama-server:8080/v1
|
|
api_key: local
|
|
model_info:
|
|
# Shadow cloud-cost estimate — priced against Claude Sonnet 5's published
|
|
# rate, not real spend (this proxy only ever routes to the local model).
|
|
# Source: https://platform.claude.com/docs/en/about-claude/pricing,
|
|
# checked 2026-08-25. Update these two numbers if that page changes.
|
|
input_cost_per_token: 0.000002 # $2 / MTok
|
|
output_cost_per_token: 0.00001 # $10 / MTok
|
|
|
|
router_settings:
|
|
# ponytail: LiteLLM's request-prioritization scheduler is beta (see
|
|
# docs/proxy-request-priority.md) — exact settings key/shape must be
|
|
# confirmed against LiteLLM's current docs and smoke-tested against
|
|
# llama.cpp before workloads depend on it. Single-instance deployment,
|
|
# no Redis configured — add one only if the scheduler turns out to need it.
|
|
enable_priority_scheduling: true
|
|
|
|
general_settings:
|
|
master_key: os.environ/LITELLM_MASTER_KEY
|