# LLM-Server

Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.

See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.

## Quickstart

```bash
cp .env.example .env      # adjust if needed
./scripts/download-model.sh
docker compose up -d
```

- Open WebUI: `http://<this-machine>:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on).
- llama.cpp OpenAI-compatible API: `http://<this-machine>:8080/v1` — **LAN-only, not proxied**, see `docs/network-access.md`.
- llama.cpp Anthropic Messages API (for Claude Code CLI): `http://<this-machine>:8080/v1/messages` — same LAN-only scope.

Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).

**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.

## AI proxy (LiteLLM)

An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). Before `docker compose up -d`, set `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` in `.env` (see `.env.example`).

- Proxy API: `http://<this-machine>:4000/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
- Admin UI (`/ui`, key/budget management): LAN-only — see `docs/network-access.md`.
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
- Request priority across workloads: [`docs/proxy-request-priority.md`](docs/proxy-request-priority.md).

**Not yet done**: Open WebUI and the coding CLIs still talk to llama.cpp directly, not through this proxy — that migration is [issue #15](https://git.arthurerlich.de/haylan/LLM-Server/issues/15). **Not yet verified**: this config hasn't been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).