haylanandClaude-Bot
7d8c9324b2
Resolves wayfinder ticket #6. Covers the two server endpoints (Anthropic shim vs OpenAI-compatible), exact config for all three CLIs, and surfaces the shared Qwen3.8-27B tool-calling risk plus OpenCode's own documented local-backend brittleness up front rather than burying it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
22 lines
1.1 KiB
Markdown
22 lines
1.1 KiB
Markdown
# LLM-Server
|
|
|
|
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
|
|
|
|
See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
cp .env.example .env # adjust if needed
|
|
./scripts/download-model.sh
|
|
docker compose up -d
|
|
```
|
|
|
|
- Open WebUI: http://localhost:3000 (first signup becomes the admin account — `WEBUI_AUTH` is on)
|
|
- llama.cpp OpenAI-compatible API: http://localhost:8080/v1
|
|
- llama.cpp Anthropic Messages API (for Claude Code CLI): http://localhost:8080/v1/messages
|
|
|
|
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
|
|
|
|
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
|