haylanandClaude-Bot
472e3a4738
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
4.1 KiB
LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
See the wayfinder map (issue #1) for the full architecture rationale and open questions.
Quickstart
./scripts/update.sh
update.sh creates .env from .env.example if missing, fills in every random secret it can generate itself (via openssl, SEARXNG_LAN_IP resolved from search.home on this host), downloads the model GGUF into the models volume if it's not there yet, then pulls/builds/brings up the whole stack. Per-workload API keys (e.g. OPENWEBUI_OMNIROUTE_KEY) have no scripted mint yet — see docs/proxy-key-onboarding.md. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
- Open WebUI:
http://<this-machine>:3000locally, orai.home/ai.haylan.chonce routed through Nginx Proxy Manager — seedocs/network-access.md. First signup becomes the admin account (WEBUI_AUTHis on). - llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see docs/coding-cli-setup.md.
Known risk: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see docs/research/qwen3.8-27b-tool-calling.md.
AI gateway (OmniRoute)
An AI gateway/proxy fronts llama.cpp: per-workload API keys and usage tracking. As of issue #31 this is OmniRoute, replacing the original LiteLLM setup. ./scripts/update.sh handles most of OmniRoute's secrets (see .env.example); per-workload API keys still need minting by hand in the dashboard — see docs/proxy-key-onboarding.md.
- Gateway API:
http://<this-machine>:${OMNIROUTE_API_PORT:-20129}/v1locally, orproxy.ai.home/proxy.ai.haylan.chonce routed through NPM — seedocs/network-access.md. - Dashboard (key/provider management): LAN/host-only, never published to the internet — see
docs/network-access.md. - Issuing a key for a new workload:
docs/proxy-key-onboarding.md.
Open WebUI and the coding CLIs (see docs/coding-cli-setup.md) route through the gateway — llama-server has no published host port. Not yet verified: none of this has been smoke-tested on real hardware yet — see issue #31's tickets for the open items (provider registration, per-workload key minting, Redis integration).
Note on this choice: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
Web search
The gateway also fronts SearXNG-backed web search, wired at the gateway layer so every client gets it, not just Open WebUI — see docs/research/litellm-searxng-search.md for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
The gateway-level knowledgebase/memory feature (litellm-pgvector, pgvector-db, a dedicated embedding model) that used to sit alongside this was removed — this stack doesn't use it. Open WebUI's own built-in memory/RAG (backed by qdrant) is unrelated and unaffected.