T

haylanandClaude-Bot c38375c0f4 fix(omniroute): add required WS bridge secret, memory ceiling, shutdown grace period

Cross-checked the deployment against OmniRoute's own docs
(docs/reference/ENVIRONMENT.md, docs/guides/DOCKER_GUIDE.md) and found
three gaps from the original migration:

- OMNIROUTE_WS_BRIDGE_SECRET was entirely missing - ENVIRONMENT.md marks
  it REQUIRED (production), for the internal Codex Responses WebSocket
  bridge. docker compose config validated fine without it (compose
  doesn't know omniroute's own required-var list), so this went
  unnoticed until checking the docs directly.
- No mem_limit/OMNIROUTE_MEMORY_MB - the Docker guide is explicit that
  the 1024MB default heap is dashboard-only sized; coding-agent workloads
  (every client this stack has) need OMNIROUTE_MEMORY_MB=8192 and a
  10+ GiB container ceiling. Set both.
- No stop_grace_period - the guide's --stop-timeout 40 equivalent, so
  SQLite WAL changes checkpoint back into the main DB file on shutdown
  instead of getting killed mid-write.

Redis checked and confirmed correctly absent - OmniRoute uses SQLite
only, no Redis anywhere in its docs.

Still open: whether API_PORT actually isolates /dashboard and /api/*
from the published port, or bridges everything through (see issue #31)
- OmniRoute's own ARCHITECTURE.md doesn't document split-port mode as a
real security boundary, and the live "[API Bridge] ... -> dashboard"
log line is ambiguous. Waiting on a live curl test against
proxy-ai.home before treating that as resolved.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197

2026-09-03 20:37:49 +02:00

docs

docs: fix gateway hostname to proxy-ai.home/proxy-ai.haylan.ch

2026-09-03 20:28:21 +02:00

scripts

fix(omniroute): add required WS bridge secret, memory ceiling, shutdown grace period

2026-09-03 20:37:49 +02:00

.env.example

fix(omniroute): add required WS bridge secret, memory ceiling, shutdown grace period

2026-09-03 20:37:49 +02:00

.gitignore

Revert "feat(knowledgebase): replace litellm-pgvector connector with memory-retrieval"

2026-09-02 22:45:08 +02:00

CLAUDE.md

feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature

2026-09-03 19:49:35 +02:00

docker-compose.yml

fix(omniroute): add required WS bridge secret, memory ceiling, shutdown grace period

2026-09-03 20:37:49 +02:00

README.md

docs: fix gateway hostname to proxy-ai.home/proxy-ai.haylan.ch

2026-09-03 20:28:21 +02:00

README.md

LLM-Server

Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle.

See the wayfinder map (issue #1) for the full architecture rationale and open questions.

Quickstart

./scripts/update.sh

update.sh creates .env from .env.example if missing, fills in every random secret it can generate itself (via openssl, SEARXNG_LAN_IP resolved from search.home on this host), downloads the model GGUF into the models volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.

llama.cpp's own API is internal-only — everything routes through the AI gateway below.

Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see docs/coding-cli-setup.md.

Known risk: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see docs/research/qwen3.8-27b-tool-calling.md.

AI gateway (OmniRoute)

An AI gateway/proxy fronts llama.cpp: per-workload API keys and usage tracking. As of issue #31 this is OmniRoute, replacing the original LiteLLM setup. ./scripts/update.sh handles most of OmniRoute's secrets (see .env.example); per-workload API keys still need minting by hand in the dashboard — see docs/proxy-key-onboarding.md.

Gateway API: http://<this-machine>:${OMNIROUTE_PORT:-4000}/v1 locally, or proxy-ai.home / proxy-ai.haylan.ch once routed through NPM — see docs/network-access.md.
Dashboard (key/provider management): LAN/host-only, never published to the internet — see docs/network-access.md.
Issuing a key for a new workload: docs/proxy-key-onboarding.md.

Coding CLIs (see docs/coding-cli-setup.md) route through the gateway — llama-server has no published host port. Not yet verified: none of this has been smoke-tested on real hardware yet — see issue #31's tickets for the open items (provider registration, per-workload key minting).

Note on this choice: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.

Web search

The gateway also fronts SearXNG-backed web search — see docs/research/litellm-searxng-search.md for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).

What's not here

Two features from earlier iterations of this stack were deliberately removed, not just left unfinished:

Open WebUI + Qdrant — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle.
Gateway-level knowledgebase/memory (litellm-pgvector, pgvector-db, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34).