Author docker-compose stack: llama.cpp (ROCm) + Open WebUI + Qdrant + Lazytainer

Resolves wayfinder ticket #4. Wires up the locked decisions from the map:
- llama.cpp (ghcr.io/ggml-org/llama.cpp:server-rocm, gfx1201) serving
  Qwen3.8-27B-UD-Q4_K_XL.gguf, port published for direct Claude Code CLI /
  Kimi CLI access alongside Open WebUI.
- Open WebUI with WEBUI_AUTH on, RAG+Memory wired to a standalone Qdrant
  service.
- Lazytainer labels on llama-server for a 15 min idle-stop.
- Named Docker volumes only (models, qdrant-data, openwebui-data) — no host
  bind-mounts.
- One-off 'downloader' compose profile instead of a host-side script with
  its own dependencies, wrapped by scripts/download-model.sh.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-24 13:01:00 +02:00
co-authored by Claude-Bot
parent 19d62aefd3
commit f0e473048d
4 changed files with 147 additions and 0 deletions
+21
View File
@@ -0,0 +1,21 @@
# LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
## Quickstart
```bash
cp .env.example .env # adjust if needed
./scripts/download-model.sh
docker compose up -d
```
- Open WebUI: http://localhost:3000 (first signup becomes the admin account — `WEBUI_AUTH` is on)
- llama.cpp OpenAI-compatible API: http://localhost:8080/v1
- llama.cpp Anthropic Messages API (for Claude Code CLI): http://localhost:8080/v1/messages
Pointing Claude Code CLI or Kimi CLI at the local endpoint is documented separately — see [issue #6](https://git.arthurerlich.de/haylan/LLM-Server/issues/6) once resolved.
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.