haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Author LiteLLM AI proxy service (resolves #14)

Browse Source 
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with
custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a
priority-scheduling stub (per #16, needs real-hardware smoke test), and
required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real
hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp
directly, migration is #15.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-08-25 07:11:15 +02:00
co-authored by Claude-Bot
parent 0342090e1c
commit 0aefb36a48
4 changed files with 77 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+11
View File
@@ -19,3 +19,14 @@ docker compose up -d
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
## AI proxy (LiteLLM)
An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). Before `docker compose up -d`, set `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` in `.env` (see `.env.example`).
- Proxy API: `http://<this-machine>:4000/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
- Admin UI (`/ui`, key/budget management): LAN-only — see `docs/network-access.md`.
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
- Request priority across workloads: [`docs/proxy-request-priority.md`](docs/proxy-request-priority.md).
**Not yet done**: Open WebUI and the coding CLIs still talk to llama.cpp directly, not through this proxy — that migration is [issue #15](https://git.arthurerlich.de/haylan/LLM-Server/issues/15). **Not yet verified**: this config hasn't been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).