LLM-Server

Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.

See the wayfinder map (issue #1) for the full architecture rationale and open questions.

Quickstart

cp .env.example .env
# set LITELLM_MASTER_KEY / LITELLM_SALT_KEY (openssl rand -hex 32), see .env.example
./scripts/download-model.sh
docker compose up -d litellm litellm-db llama-server qdrant   # bring the proxy up first

Log into LiteLLM's Admin UI (http://<this-machine>:4000/ui), create an openwebui virtual key (see docs/proxy-key-onboarding.md), set OPENWEBUI_LITELLM_KEY in .env to it, then:

docker compose up -d

• Open WebUI: http://<this-machine>:3000 locally, or ai.home / ai.haylan.ch once routed through Nginx Proxy Manager — see docs/network-access.md. First signup becomes the admin account (WEBUI_AUTH is on).
• llama.cpp's own API is internal-only now — everything routes through the AI proxy below.

Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see docs/coding-cli-setup.md.

Known risk: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see docs/research/qwen3.8-27b-tool-calling.md.

AI proxy (LiteLLM)

An AI gateway/proxy fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). Before docker compose up -d, set LITELLM_MASTER_KEY and LITELLM_SALT_KEY in .env (see .env.example).

• Proxy API: http://<this-machine>:4000/v1 locally, or proxy.ai.home / proxy.ai.haylan.ch once routed through NPM — see docs/network-access.md.
• Admin UI (/ui, key/budget management): LAN-only — see docs/network-access.md.
• Issuing a key for a new workload: docs/proxy-key-onboarding.md.
• Request priority across workloads: docs/proxy-request-priority.md.

Open WebUI and the coding CLIs (see docs/coding-cli-setup.md) route through the proxy now — llama-server has no published host port anymore. Not yet verified: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see docs/proxy-request-priority.md) — see issue #17.