haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Migrate Open WebUI and coding CLIs to the AI proxy (resolves #15)

Browse Source 
Open WebUI now points at litellm instead of llama-server directly, using a
provisioned virtual key. llama-server's host port is dropped (internal-only
on the ai-stack network) since the proxy is the only intended entry point
now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with
per-CLI virtual keys instead of the old shared dummy key.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-08-25 07:18:03 +02:00
co-authored by Claude-Bot
parent 0aefb36a48
commit 5cb34b19f3
5 changed files with 52 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+10 -4
View File
@@ -7,14 +7,20 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/
## Quickstart
```bash
cp .env.example .env # adjust if needed
cp .env.example .env
# set LITELLM_MASTER_KEY / LITELLM_SALT_KEY (openssl rand -hex 32), see .env.example
./scripts/download-model.sh
docker compose up -d litellm litellm-db llama-server qdrant # bring the proxy up first
```
Log into LiteLLM's Admin UI (`http://<this-machine>:4000/ui`), create an `openwebui` virtual key (see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md)), set `OPENWEBUI_LITELLM_KEY` in `.env` to it, then:
```bash
docker compose up -d
```
- Open WebUI: `http://<this-machine>:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on).
- llama.cpp OpenAI-compatible API: `http://<this-machine>:8080/v1` — **LAN-only, not proxied**, see `docs/network-access.md`.
- llama.cpp Anthropic Messages API (for Claude Code CLI): `http://<this-machine>:8080/v1/messages` — same LAN-only scope.
- llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
@@ -29,4 +35,4 @@ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fr
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
- Request priority across workloads: [`docs/proxy-request-priority.md`](docs/proxy-request-priority.md).
**Not yet done**: Open WebUI and the coding CLIs still talk to llama.cpp directly, not through this proxy — that migration is [issue #15](https://git.arthurerlich.de/haylan/LLM-Server/issues/15). **Not yet verified**: this config hasn't been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).
Open WebUI and the coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the proxy now — llama-server has no published host port anymore. **Not yet verified**: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).