Author LiteLLM AI proxy service (resolves #14)
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a priority-scheduling stub (per #16, needs real-hardware smoke test), and required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp directly, migration is #15. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
model_list:
|
||||
- model_name: qwen3.8-27b-local
|
||||
litellm_params:
|
||||
model: openai/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
api_base: http://llama-server:8080/v1
|
||||
api_key: local
|
||||
model_info:
|
||||
# Shadow cloud-cost estimate — priced against Claude Sonnet 5's published
|
||||
# rate, not real spend (this proxy only ever routes to the local model).
|
||||
# Source: https://platform.claude.com/docs/en/about-claude/pricing,
|
||||
# checked 2026-08-25. Update these two numbers if that page changes.
|
||||
input_cost_per_token: 0.000002 # $2 / MTok
|
||||
output_cost_per_token: 0.00001 # $10 / MTok
|
||||
|
||||
router_settings:
|
||||
# ponytail: LiteLLM's request-prioritization scheduler is beta (see
|
||||
# docs/proxy-request-priority.md) — exact settings key/shape must be
|
||||
# confirmed against LiteLLM's current docs and smoke-tested against
|
||||
# llama.cpp before workloads depend on it. Single-instance deployment,
|
||||
# no Redis configured — add one only if the scheduler turns out to need it.
|
||||
enable_priority_scheduling: true
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
Reference in New Issue
Block a user