haylanandClaude-Bot
0aefb36a48
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a priority-scheduling stub (per #16, needs real-hardware smoke test), and required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp directly, migration is #15. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
31 lines
1.2 KiB
Bash
31 lines
1.2 KiB
Bash
# Copy to .env and adjust. All values below are defaults baked into
|
|
# docker-compose.yml — only uncomment/change what you actually want to override.
|
|
|
|
# --- llama.cpp / model ---
|
|
LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
|
|
LLAMA_GPU_LAYERS=999
|
|
# 65536 (64K) fits comfortably in 32GB VRAM alongside the model weights.
|
|
# Raise toward 131072 if you need more context; see docs/research/qwen3.8-27b-quant.md
|
|
# for the VRAM math at larger context sizes.
|
|
LLAMA_CTX_SIZE=65536
|
|
LLAMA_PORT=8080
|
|
|
|
# --- Open WebUI ---
|
|
WEBUI_PORT=3000
|
|
# Dummy key — llama.cpp's OpenAI-compatible endpoint doesn't check it, but
|
|
# Open WebUI requires the field to be non-empty.
|
|
OPENAI_API_KEY=local
|
|
|
|
# --- Lazytainer ---
|
|
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
|
LAZYTAINER_INACTIVE_TIMEOUT=900
|
|
|
|
# --- LiteLLM proxy (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
|
|
LITELLM_PORT=4000
|
|
# Required — generate real random values before first run, e.g. `openssl rand -hex 32`.
|
|
# LITELLM_SALT_KEY encrypts stored data; do not change it after the first run
|
|
# (existing encrypted data becomes unreadable if you do).
|
|
LITELLM_MASTER_KEY=
|
|
LITELLM_SALT_KEY=
|
|
LITELLM_DB_PASSWORD=litellm
|