From 3bbda098b34a7c8a68ecf4d16a3e1f37630e4e01 Mon Sep 17 00:00:00 2001 From: ArthurErlich Date: Thu, 3 Sep 2026 19:55:31 +0200 Subject: [PATCH] feat(stack): remove Open WebUI and Qdrant No longer needed - every client is a coding CLI behind the OmniRoute gateway, not a chat UI. Drops the open-webui and qdrant services, WEBUI_PORT/OPENWEBUI_OMNIROUTE_KEY env vars, and the openwebui-data/ qdrant-data volumes. Qdrant only ever served Open WebUI's own built-in memory/RAG (unrelated to the gateway-level knowledgebase removed in 472e3a4), so it goes too rather than sit unused. Docs updated: README, docs/network-access.md (ai.home/ai.haylan.ch section was entirely about Open WebUI, rewritten around the gateway), docs/proxy-key-onboarding.md, docs/proxy-request-priority.md. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197 --- .env.example | 14 ++++------- README.md | 18 +++++++------ docker-compose.yml | 46 ++-------------------------------- docs/network-access.md | 21 ++++++---------- docs/proxy-key-onboarding.md | 4 +-- docs/proxy-request-priority.md | 2 +- scripts/update.sh | 18 ++++++------- 7 files changed, 35 insertions(+), 88 deletions(-) diff --git a/.env.example b/.env.example index a944641..179558c 100644 --- a/.env.example +++ b/.env.example @@ -28,10 +28,6 @@ LLAMA_GPU_LAYERS=999 # of dropping context: --cache-type-k/v q8_0 roughly halves it. LLAMA_CTX_SIZE=131072 -# --- Open WebUI --- -WEBUI_PORT=8008 -# No scripted mint yet — set OPENWEBUI_OMNIROUTE_KEY below by hand instead. - # --- Lazytainer --- # Seconds of inactivity before llama-server is stopped. 900 = 15 min. LAZYTAINER_INACTIVE_TIMEOUT=900 @@ -63,8 +59,8 @@ OMNIROUTE_STORAGE_ENCRYPTION_KEY= # Per-deployment salts — random is fine, just needs to be stable: OMNIROUTE_MACHINE_ID_SALT= OMNIROUTE_CLI_SALT= -# Per-workload virtual keys — omniroute has no scripted /key/generate -# equivalent yet (its key-creation endpoint needs a dashboard login session, -# not a static bearer key — see issue #37), so mint these by hand in the -# dashboard for now. See docs/proxy-key-onboarding.md. -OPENWEBUI_OMNIROUTE_KEY= +# Per-workload virtual keys (one per client that calls the gateway) have no +# scripted /key/generate equivalent yet — omniroute's key-creation endpoint +# needs a dashboard login session, not a static bearer key (see issue #37). +# Mint them by hand in the dashboard, add a KEY=value line here per workload +# as you onboard one. See docs/proxy-key-onboarding.md. diff --git a/README.md b/README.md index 1c3ccb0..50d2337 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # LLM-Server -Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle. +Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle. See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions. @@ -10,10 +10,9 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/ ./scripts/update.sh ``` -`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Per-workload API keys (e.g. `OPENWEBUI_OMNIROUTE_KEY`) have no scripted mint yet — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed. +`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed. -- Open WebUI: `http://:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on). -- llama.cpp's own API is internal-only now — everything routes through the AI proxy below. +llama.cpp's own API is internal-only — everything routes through the AI gateway below. Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md). @@ -27,12 +26,17 @@ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fr - Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`. - Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). -Open WebUI and the coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting, Redis integration). +Coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting). **Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further. ### Web search -The gateway also fronts SearXNG-backed web search, wired at the gateway layer so every client gets it, not just Open WebUI — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35). +The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35). -The gateway-level knowledgebase/memory feature (litellm-pgvector, pgvector-db, a dedicated embedding model) that used to sit alongside this was removed — this stack doesn't use it. Open WebUI's own built-in memory/RAG (backed by `qdrant`) is unrelated and unaffected. +## What's not here + +Two features from earlier iterations of this stack were deliberately removed, not just left unfinished: + +- **Open WebUI + Qdrant** — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle. +- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). diff --git a/docker-compose.yml b/docker-compose.yml index a3427c8..acb0871 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -20,8 +20,8 @@ services: --n-gpu-layers ${LLAMA_GPU_LAYERS:-999} --ctx-size ${LLAMA_CTX_SIZE:-131072} --jinja - # No published host port: llama-server is reached only via the litellm - # proxy on the ai-stack docker network now — see issue #15. Its + # No published host port: llama-server is reached only via the omniroute + # gateway on the ai-stack docker network now — see issue #15. Its # unauthenticated API no longer needs to be LAN-reachable directly. expose: - "8080" @@ -56,46 +56,6 @@ services: curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} - qdrant: - image: qdrant/qdrant:latest - container_name: qdrant - volumes: - - qdrant-data:/qdrant/storage - restart: unless-stopped - networks: [ai-stack] - healthcheck: - test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/localhost/6333'"] - interval: 10s - timeout: 5s - retries: 5 - - open-webui: - image: ghcr.io/open-webui/open-webui:main - container_name: open-webui - depends_on: - qdrant: - condition: service_healthy - omniroute: - condition: service_healthy - volumes: - - openwebui-data:/app/backend/data - env_file: .env - environment: - - WEBUI_AUTH=True - # Routed through the omniroute gateway, not llama-server directly — see - # issue #15 (original rationale) and #31 (litellm -> omniroute - # migration). OPENAI_API_KEY must be a per-workload key created for - # Open WebUI in the omniroute dashboard (Keys -> Create, label - # "openwebui") — no scripted mint yet, see docs/proxy-key-onboarding.md. - - OPENAI_API_BASE_URL=http://omniroute:${OMNIROUTE_API_PORT:-20129}/v1 - - OPENAI_API_KEY=${OPENWEBUI_OMNIROUTE_KEY} - - VECTOR_DB=qdrant - - QDRANT_URI=http://qdrant:6333 - ports: - - "${WEBUI_PORT:-8008}:8080" - restart: unless-stopped - networks: [ai-stack] - # Replaces litellm — see issue #31 (wayfinder map) for the full migration # rationale/findings. No static config.yaml equivalent: provider routing # (llama-server, searxng-search) is registered once through the dashboard @@ -165,6 +125,4 @@ networks: volumes: models: - qdrant-data: - openwebui-data: omniroute-data: diff --git a/docs/network-access.md b/docs/network-access.md index 44f31ea..13e9d82 100644 --- a/docs/network-access.md +++ b/docs/network-access.md @@ -1,25 +1,18 @@ -# Network access: ai.home / ai.haylan.ch +# Network access: proxy.ai.home / proxy.ai.haylan.ch -Open WebUI is meant to be reachable as **`ai.home`** on the LAN and **`ai.haylan.ch`** from outside. This stack doesn't run its own reverse proxy — it publishes Open WebUI's port to the host (`${WEBUI_PORT:-3000}`, see `docker-compose.yml`) and relies on the **existing Nginx Proxy Manager (NPM)** instance already fronting other self-hosted services on this network. - -## What to set up in NPM - -Two Proxy Hosts, both pointing at this machine's LAN IP on port `${WEBUI_PORT:-3000}` (Open WebUI): - -- **`ai.home`** — internal only, no external DNS/TLS needed unless you want it. -- **`ai.haylan.ch`** — external, reachable via the DMZ already forwarding it to NPM; let NPM issue/manage the TLS cert as it does for other services. +This stack has no chat UI — every client is a coding CLI reaching the AI gateway (OmniRoute). It doesn't run its own reverse proxy — it publishes the gateway's API port to the host and relies on the **existing Nginx Proxy Manager (NPM)** instance already fronting other self-hosted services on this network. ## llama.cpp's raw API stays LAN-only — deliberately -The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable via `ai.haylan.ch`. It has no authentication of its own (unlike Open WebUI, which has login enabled) — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) reach it directly over the LAN, using this machine's LAN IP or `ai.home` if your local DNS resolves that hostname straight to the box (bypassing NPM, which only fronts ports 80/443). +The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) don't reach it directly at all now; they go through the gateway below, same as everything else. -If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture (the raw API would need its own auth in front of it). +If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture. ## The AI gateway (OmniRoute) — `proxy.ai.home` / `proxy.ai.haylan.ch` -As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute — same NPM pattern as Open WebUI above, but a cleaner split than LiteLLM's ever was: +As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute: -- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port. +- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `ai.home` internal-only, `ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual). - The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward. -**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname. +**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname. diff --git a/docs/proxy-key-onboarding.md b/docs/proxy-key-onboarding.md index 7f7c54e..f46da75 100644 --- a/docs/proxy-key-onboarding.md +++ b/docs/proxy-key-onboarding.md @@ -2,13 +2,13 @@ How to issue a new per-workload API key against the OmniRoute gateway (see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) — the LiteLLM → OmniRoute migration; original gateway rationale in [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend. -`OPENWEBUI_OMNIROUTE_KEY` — the one key this stack's own services need — has no scripted mint yet: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key — including that one — by hand for now, via the dashboard steps below. +No workload in this stack itself needs a key right now — every client is external (a coding CLI, or another self-hosted service). There's no scripted mint yet either way: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key by hand for now, via the dashboard steps below. ## Create the key 1. Log into the omniroute dashboard (`http://:${OMNIROUTE_DASHBOARD_PORT:-20128}` — LAN/host-only, never published publicly, see `docker-compose.yml`'s `omniroute` service). 2. "Keys" → "Create API key". -3. Label it `-` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `openwebui`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync. +3. Label it `-` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `claude-code-cli`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync. 4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs. Once `POST /api/keys`'s session-auth flow is worked out (issue #37), the equivalent `curl` here can replace this manual step, the way `update.sh` used to automate LiteLLM's `/key/generate`. diff --git a/docs/proxy-request-priority.md b/docs/proxy-request-priority.md index 4d7cf22..6b12188 100644 --- a/docs/proxy-request-priority.md +++ b/docs/proxy-request-priority.md @@ -8,7 +8,7 @@ One local model instance (llama.cpp on the single R9700) serves every workload t Two tiers, assigned per workload's virtual key (per `docs/proxy-key-onboarding.md`): -- **High priority** (interactive — someone's waiting): Open WebUI chat, coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review. +- **High priority** (interactive — someone's waiting): coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review. - **Low priority** (batch — nobody's watching a spinner): Paperless OCR/tagging, Nextcloud Memories face-recognition, AI watermark removal. ## Mechanism diff --git a/scripts/update.sh b/scripts/update.sh index 8c6e012..a3b03a3 100755 --- a/scripts/update.sh +++ b/scripts/update.sh @@ -73,17 +73,13 @@ docker compose --profile tools run --rm downloader echo "==> bringing up omniroute" docker compose up -d --wait omniroute -# ponytail: no scripted key-minting yet, unlike the old LiteLLM -# /key/generate flow — omniroute's POST /api/keys needs a dashboard login -# session (ManagementSessionAuth), not a static bearer key, and that flow -# hasn't been verified against a live instance (see issue #37). Until then, -# check for the workload keys and just remind rather than fail. -set -a && . ./.env && set +a -if grep -qE "^OPENWEBUI_OMNIROUTE_KEY=.*[^[:space:]]" .env; then - echo "OPENWEBUI_OMNIROUTE_KEY: already set, skipping." -else - echo "OPENWEBUI_OMNIROUTE_KEY: blank — mint it by hand in the omniroute dashboard (http://localhost:\${OMNIROUTE_DASHBOARD_PORT:-20128}) and set it in .env. See docs/proxy-key-onboarding.md." -fi +# ponytail: no scripted key-minting yet, unlike the old LiteLLM /key/generate +# flow — omniroute's POST /api/keys needs a dashboard login session +# (ManagementSessionAuth), not a static bearer key, and that flow hasn't +# been verified against a live instance (see issue #37). No in-stack +# workload needs a key right now (nothing left calls the gateway besides +# coding CLIs, which mint their own by hand per docs/proxy-key-onboarding.md) +# — revisit this script once that flow is automatable. echo "==> recreating changed services" docker compose up -d --remove-orphans