haylanandClaude-Bot
c01ef8965d
Resolves wayfinder ticket #8. The stack is reached over LAN/WAN hostnames, not localhost: - Open WebUI: ai.home (LAN) and ai.haylan.ch (external, via DMZ) routed through the existing Nginx Proxy Manager instance to the published WEBUI_PORT — no new reverse-proxy service needed in this repo. - llama.cpp's raw API stays LAN-only and unregistered in NPM (no auth of its own); coding CLIs reach it via the box's LAN IP or ai.home directly. New docs/network-access.md documents the NPM proxy-host setup and the LAN-only decision. coding-cli-setup.md and README updated to stop referencing localhost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
22 lines
1.3 KiB
Markdown
22 lines
1.3 KiB
Markdown
# LLM-Server
|
|
|
|
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
|
|
|
|
See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
cp .env.example .env # adjust if needed
|
|
./scripts/download-model.sh
|
|
docker compose up -d
|
|
```
|
|
|
|
- Open WebUI: `http://<this-machine>:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on).
|
|
- llama.cpp OpenAI-compatible API: `http://<this-machine>:8080/v1` — **LAN-only, not proxied**, see `docs/network-access.md`.
|
|
- llama.cpp Anthropic Messages API (for Claude Code CLI): `http://<this-machine>:8080/v1/messages` — same LAN-only scope.
|
|
|
|
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
|
|
|
|
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
|