haylanandClaude-Bot
472e3a4738
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
71 lines
3.6 KiB
Bash
71 lines
3.6 KiB
Bash
# Copy to .env and adjust — or just run ./scripts/update.sh, which creates
|
|
# .env from this file and fills in every secret/key below it can generate
|
|
# itself (see each var's comment). All values below are defaults baked into
|
|
# docker-compose.yml — only uncomment/change what you actually want to
|
|
# override.
|
|
|
|
# --- llama.cpp / model ---
|
|
LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
|
|
# 999 = every layer on GPU (this model is dense, not MoE, and already fits
|
|
# fully in 32GB VRAM — see docs/research/qwen3.8-27b-quant.md). Lower this
|
|
# to leave that many fewer layers on GPU and push the rest to CPU/system RAM
|
|
# if something else is contending for VRAM — llama.cpp has no separate
|
|
# "RAM offload" flag, --n-gpu-layers *is* the RAM-offload knob for a dense
|
|
# model. Don't reach for --n-cpu-moe/--cpu-moe/--override-tensor "exps" —
|
|
# those target Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B), not this
|
|
# one, and are no-ops here.
|
|
# There's no separate "then SSD" tier to enable either: llama.cpp mmaps the
|
|
# model file by default (no --no-mmap here), so if GPU+RAM ever can't hold
|
|
# the working set, the OS pages the rest in from disk automatically — an
|
|
# implicit, slow last resort, not a config knob. An explicit tiered SSD
|
|
# offload has been an open llama.cpp feature request since 2025 (still
|
|
# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
|
|
LLAMA_GPU_LAYERS=999
|
|
# 131072 (128K): ~25.6GB (17.6GB weights + ~8GB KV cache) on the 32GB
|
|
# R9700, ~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full
|
|
# table (64K only used ~19.6GB/~12GB headroom, but real usage was burning
|
|
# through 64K fast). If headroom gets tight, quantize the KV cache instead
|
|
# of dropping context: --cache-type-k/v q8_0 roughly halves it.
|
|
LLAMA_CTX_SIZE=131072
|
|
|
|
# --- Open WebUI ---
|
|
WEBUI_PORT=8008
|
|
# No scripted mint yet — set OPENWEBUI_OMNIROUTE_KEY below by hand instead.
|
|
|
|
# --- Lazytainer ---
|
|
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
|
LAZYTAINER_INACTIVE_TIMEOUT=900
|
|
|
|
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
|
|
# Resolved automatically by ./scripts/update.sh from search.home on this
|
|
# host — leave blank. Only set by hand if that resolution fails (e.g.
|
|
# search.home isn't a static DHCP reservation and its IP drifted).
|
|
SEARXNG_LAN_IP=
|
|
|
|
# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
|
|
# API_PORT is the only port published to the host/internet (reverse-proxied
|
|
# by NPM) — the dashboard (DASHBOARD_PORT) is never published, see
|
|
# docker-compose.yml's omniroute service comment.
|
|
OMNIROUTE_API_PORT=20129
|
|
OMNIROUTE_DASHBOARD_PORT=20128
|
|
# Random values, filled in automatically by ./scripts/update.sh — leave
|
|
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
|
|
# change it there afterwards — this is only the first-boot value):
|
|
OMNIROUTE_INITIAL_PASSWORD=
|
|
# Signs dashboard session cookies:
|
|
OMNIROUTE_JWT_SECRET=
|
|
# Encrypts API key values at rest in omniroute's SQLite DB:
|
|
OMNIROUTE_API_KEY_SECRET=
|
|
# Encrypts the whole SQLite DB at rest. Do not change after first run —
|
|
# existing encrypted data becomes unreadable if you do (same caveat as
|
|
# LiteLLM's old LITELLM_SALT_KEY):
|
|
OMNIROUTE_STORAGE_ENCRYPTION_KEY=
|
|
# Per-deployment salts — random is fine, just needs to be stable:
|
|
OMNIROUTE_MACHINE_ID_SALT=
|
|
OMNIROUTE_CLI_SALT=
|
|
# Per-workload virtual keys — omniroute has no scripted /key/generate
|
|
# equivalent yet (its key-creation endpoint needs a dashboard login session,
|
|
# not a static bearer key — see issue #37), so mint these by hand in the
|
|
# dashboard for now. See docs/proxy-key-onboarding.md.
|
|
OPENWEBUI_OMNIROUTE_KEY=
|