haylan
ea7b05fb99
Replaces the previous commit's blind force-overwrite with a real choice, per user feedback: force-overwriting server config without asking was the wrong default. - A tracked config value (real default in .env.example) that already matches .env is left alone silently — no prompt, no noise. - A value that DIFFERS is a conflict, shown on one screen (all conflicts together, not one prompt per key) via gum (charmbracelet/gum) — single static binary fetched as a release tarball into .cache/gum/ (gitignored), no build step, no package manager dependency. Falls back to a plain read-based prompt if gum can't be fetched (offline, unsupported arch). - Non-interactive (no TTY — cron, CI, piped): any conflict is a hard error (exit 1, lists every conflicting key) unless --force is passed, which accepts every new value automatically — matches how this PR's own fix needs to land unattended. - Secrets and host-resolved values are completely unaffected either way — untouched by this loop, same as before. Verified in an isolated sandbox against the exact scenario from this PR (stale LLAMA_FAST_PARALLEL=2 vs git's 1): - no TTY, no --force: exits 1, prints the diff, doesn't touch .env - no TTY, --force: LLAMA_FAST_PARALLEL corrected 2 -> 1, an OMNIROUTE secret confirmed untouched (not regenerated) docker compose config -q still passes. Not verified: the interactive gum path itself (needs a real TTY, couldn't allocate a pty in this sandbox) — worth confirming for real on the server, including that gum's release asset naming actually matches what ensure_gum() expects. Refs #5
LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle.
See the wayfinder map (issue #1) for the full architecture rationale and open questions.
Quickstart
./scripts/update.sh
update.sh creates .env from .env.example if missing, fills in every random secret it can generate itself (via openssl, SEARXNG_LAN_IP resolved from search.home on this host), downloads the model GGUF into the models volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
llama.cpp's own API is internal-only — everything routes through the AI gateway below.
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see docs/coding-cli-setup.md.
Known risk: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see docs/research/qwen3.8-27b-tool-calling.md.
AI gateway (OmniRoute)
An AI gateway/proxy fronts llama.cpp: per-workload API keys and usage tracking. As of issue #31 this is OmniRoute, replacing the original LiteLLM setup. ./scripts/update.sh handles most of OmniRoute's secrets (see .env.example); per-workload API keys still need minting by hand in the dashboard — see docs/proxy-key-onboarding.md.
- Gateway API:
http://<this-machine>:${OMNIROUTE_PORT:-4000}/v1locally, orproxy-ai.home/proxy-ai.haylan.chonce routed through NPM — seedocs/network-access.md. - Dashboard (key/provider management): LAN/host-only, never published to the internet — see
docs/network-access.md. - Issuing a key for a new workload:
docs/proxy-key-onboarding.md.
Coding CLIs (see docs/coding-cli-setup.md) route through the gateway — llama-server has no published host port. Not yet verified: none of this has been smoke-tested on real hardware yet — see issue #31's tickets for the open items (provider registration, per-workload key minting).
Note on this choice: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
Web search
The gateway also fronts SearXNG-backed web search — see docs/research/litellm-searxng-search.md for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
What's not here
- Open WebUI — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle.
- Gateway-level knowledgebase/memory (
litellm-pgvector,pgvector-db, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own built-in memory feature (opt-in via the dashboard, Settings → Memory): vector store is its bundledsqlite-vec, embeddings are a local ONNX model (Transformers.js, ~400MB, downloaded into theomniroute-datavolume on first use) — no external services, no static config here.