haylanandClaude-Bot
b0e37b2f4c
One script to bring the running stack in sync after any change: git pull, validate compose config, rebuild the litellm-pgvector local build, re-pull images, recreate what changed. Documented in README's quickstart, and CLAUDE.md now tells agents to run it after touching compose/config/scripts rather than just describing a change as done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
45 lines
3.4 KiB
Markdown
45 lines
3.4 KiB
Markdown
# LLM-Server
|
|
|
|
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
|
|
|
|
See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
# set LITELLM_MASTER_KEY / LITELLM_SALT_KEY (openssl rand -hex 32), see .env.example
|
|
./scripts/download-model.sh
|
|
docker compose up -d litellm litellm-db llama-server qdrant # bring the proxy up first
|
|
```
|
|
|
|
Log into LiteLLM's Admin UI (`http://<this-machine>:4000/ui`), create an `openwebui` virtual key (see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md)), set `OPENWEBUI_LITELLM_KEY` in `.env` to it, then:
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
- Open WebUI: `http://<this-machine>:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on).
|
|
- llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
|
|
|
|
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
|
|
|
|
**Applying updates**: after pulling changes (or editing `docker-compose.yml`/`litellm-config.yaml`/`.env` yourself), run `./scripts/update.sh` — pulls, validates the compose config, rebuilds/re-pulls images, and recreates whatever changed. Safe to run any time.
|
|
|
|
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
|
|
|
|
## AI proxy (LiteLLM)
|
|
|
|
An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). Before `docker compose up -d`, set `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` in `.env` (see `.env.example`).
|
|
|
|
- Proxy API: `http://<this-machine>:4000/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
|
|
- Admin UI (`/ui`, key/budget management): LAN-only — see `docs/network-access.md`.
|
|
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
|
|
- Request priority across workloads: [`docs/proxy-request-priority.md`](docs/proxy-request-priority.md).
|
|
|
|
Open WebUI and the coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the proxy now — llama-server has no published host port anymore. **Not yet verified**: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).
|
|
|
|
### Web search, knowledgebase, and memory
|
|
|
|
The gateway also fronts SearXNG-backed web search and a pgvector-backed knowledgebase (loaded with `data/memory.md` / `data/claude-legacy-memory.md`), wired at the LiteLLM layer so every client gets them, not just Open WebUI — see [`docs/memory-knowledgebase.md`](docs/memory-knowledgebase.md). **Not yet verified on real hardware** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24).
|