feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
+31
-89
@@ -1,9 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
# The one command to run after any change to this repo (compose file,
|
||||
# litellm-config.yaml, .env, or a git pull) to bring the running stack in
|
||||
# sync. Ensures secrets/keys exist, pulls, validates, rebuilds/re-pulls
|
||||
# images, and recreates only what changed — safe to run any time, including
|
||||
# with nothing to do.
|
||||
# The one command to run after any change to this repo (compose file, .env,
|
||||
# or a git pull) to bring the running stack in sync. Ensures secrets/keys
|
||||
# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
|
||||
# changed — safe to run any time, including with nothing to do.
|
||||
#
|
||||
# omniroute's own routing/provider config (llama-server, search) lives in
|
||||
# its dashboard, not a checked-in file like the old litellm-config.yaml —
|
||||
# see issue #31 and docs/proxy-key-onboarding.md.
|
||||
#
|
||||
# ponytail: no rollback/backup logic — this is a single-user homelab box,
|
||||
# not a fleet. If a bad config lands, `git revert` + re-run is the recovery
|
||||
@@ -31,19 +34,20 @@ set_if_blank() {
|
||||
|
||||
echo "==> filling in missing secrets"
|
||||
# Random values — safe to re-run, never overwrites what's already set.
|
||||
# LITELLM_SALT_KEY especially: never change it after first run, existing
|
||||
# encrypted data becomes unreadable if you do.
|
||||
set_if_blank LITELLM_MASTER_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank LITELLM_SALT_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank LITELLM_DB_PASSWORD "$(openssl rand -hex 32)"
|
||||
set_if_blank REDIS_PASSWORD "$(openssl rand -hex 32)"
|
||||
set_if_blank UI_PASSWORD "$(openssl rand -hex 16)"
|
||||
set_if_blank PGVECTOR_DB_PASSWORD "$(openssl rand -hex 32)"
|
||||
set_if_blank LITELLM_PGVECTOR_API_KEY "$(openssl rand -hex 32)"
|
||||
# OMNIROUTE_STORAGE_ENCRYPTION_KEY especially: never change it after first
|
||||
# run, existing encrypted data becomes unreadable if you do (same caveat as
|
||||
# LiteLLM's old LITELLM_SALT_KEY).
|
||||
set_if_blank OMNIROUTE_INITIAL_PASSWORD "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_JWT_SECRET "$(openssl rand -base64 48)"
|
||||
set_if_blank OMNIROUTE_API_KEY_SECRET "$(openssl rand -hex 32)"
|
||||
set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
|
||||
|
||||
echo "==> resolving SEARXNG_LAN_IP"
|
||||
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
|
||||
# not from inside the litellm container (see docs/research/litellm-searxng-search.md).
|
||||
# not from inside the omniroute container (see docs/research/litellm-searxng-search.md,
|
||||
# still the relevant background even though omniroute replaced litellm — see issue #31).
|
||||
searxng_ip="$(getent hosts search.home 2>/dev/null | awk '{print $1}' | head -1)"
|
||||
if [ -n "$searxng_ip" ]; then
|
||||
set_if_blank SEARXNG_LAN_IP "$searxng_ip"
|
||||
@@ -65,86 +69,24 @@ docker compose build --pull
|
||||
|
||||
echo "==> ensuring models are downloaded (skips already-present files)"
|
||||
docker compose --profile tools run --rm downloader
|
||||
docker compose --profile tools run --rm downloader-embedding
|
||||
|
||||
echo "==> bringing up litellm (needed to mint virtual keys below)"
|
||||
docker compose up -d --wait litellm-db litellm
|
||||
echo "==> bringing up omniroute"
|
||||
docker compose up -d --wait omniroute
|
||||
|
||||
# OPENWEBUI_LITELLM_KEY / LITELLM_PGVECTOR_EMBEDDING_KEY are per-workload
|
||||
# virtual keys, not random secrets — minted via LiteLLM's own API
|
||||
# (docs/proxy-key-onboarding.md documents the manual Admin UI route; this is
|
||||
# the same thing over the REST endpoint LITELLM_MASTER_KEY already
|
||||
# authenticates against).
|
||||
set -a && . ./.env && set +a
|
||||
mint_key_if_blank() {
|
||||
local key="$1" alias="$2"
|
||||
if grep -qE "^${key}=.*[^[:space:]]" .env; then
|
||||
echo "${key}: already set, skipping."
|
||||
return
|
||||
fi
|
||||
local minted
|
||||
minted=$(curl -sf -X POST "http://localhost:${LITELLM_PORT:-4000}/key/generate" \
|
||||
-H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"key_alias\": \"${alias}\"}" | jq -r '.key')
|
||||
if [ -n "$minted" ] && [ "$minted" != "null" ]; then
|
||||
# Same missing-line-vs-blank-line handling as set_if_blank above.
|
||||
if grep -qE "^${key}=" .env; then
|
||||
sed -i "s|^${key}=.*|${key}=${minted}|" .env
|
||||
else
|
||||
echo "${key}=${minted}" >> .env
|
||||
fi
|
||||
echo "${key}: minted."
|
||||
else
|
||||
echo "${key}: mint failed, create it by hand per docs/proxy-key-onboarding.md."
|
||||
fi
|
||||
}
|
||||
mint_key_if_blank OPENWEBUI_LITELLM_KEY openwebui
|
||||
mint_key_if_blank LITELLM_PGVECTOR_EMBEDDING_KEY litellm-pgvector
|
||||
# ponytail: no scripted key-minting yet, unlike the old LiteLLM
|
||||
# /key/generate flow — omniroute's POST /api/keys needs a dashboard login
|
||||
# session (ManagementSessionAuth), not a static bearer key, and that flow
|
||||
# hasn't been verified against a live instance (see issue #37). Until then,
|
||||
# check for the workload keys and just remind rather than fail.
|
||||
set -a && . ./.env && set +a
|
||||
if grep -qE "^OPENWEBUI_OMNIROUTE_KEY=.*[^[:space:]]" .env; then
|
||||
echo "OPENWEBUI_OMNIROUTE_KEY: already set, skipping."
|
||||
else
|
||||
echo "OPENWEBUI_OMNIROUTE_KEY: blank — mint it by hand in the omniroute dashboard (http://localhost:\${OMNIROUTE_DASHBOARD_PORT:-20128}) and set it in .env. See docs/proxy-key-onboarding.md."
|
||||
fi
|
||||
|
||||
echo "==> recreating changed services"
|
||||
docker compose up -d --remove-orphans
|
||||
|
||||
# litellm-pgvector's Dockerfile only runs `prisma generate` (codegen) at
|
||||
# build time — nothing ever applied the schema to pgvector-db itself, so the
|
||||
# vector_stores/embeddings tables plain didn't exist until this was added
|
||||
# (see issue #24). --accept-data-loss is the same "no rollback/backup logic,
|
||||
# git revert is the recovery path" tradeoff as the rest of this script — a
|
||||
# schema-incompatible change here would need a manual look regardless.
|
||||
echo "==> syncing litellm-pgvector's database schema"
|
||||
docker compose up -d --wait pgvector-db litellm-pgvector
|
||||
docker compose exec -T litellm-pgvector prisma db push --accept-data-loss
|
||||
|
||||
# Registers memory-and-notes in litellm's own DB (LiteLLM_ManagedVectorStoresTable),
|
||||
# not just litellm-config.yaml's vector_store_registry block. Both matter for
|
||||
# different reasons: config.yaml seeds it into memory at boot (works even
|
||||
# before this script has ever run against a fresh DB); the DB row is what
|
||||
# /vector_store/list — and so the Admin UI's Vector Stores page — actually
|
||||
# shows, since that endpoint only auto-syncs a config-only entry into the DB
|
||||
# view once a DB row with the same id exists (see issue #24 follow-up).
|
||||
#
|
||||
# Must pass the real key, not the os.environ/... form used in
|
||||
# litellm-config.yaml — this hits the live management API, not the
|
||||
# config.yaml loader, so there's no env-substitution pass over the request
|
||||
# body. Ignores failure if the row already exists (no update-in-place: see
|
||||
# below).
|
||||
#
|
||||
# No update-if-changed path — the DB row is otherwise never touched once
|
||||
# created (/vector_store/update in this litellm version can't set
|
||||
# litellm_params at all — VectorStoreUpdateRequest has no such field, so an
|
||||
# update silently no-ops on it). If LITELLM_PGVECTOR_API_KEY ever rotates,
|
||||
# fix this row by hand: /vector_store/delete then re-run this script.
|
||||
echo "==> registering memory-and-notes vector store with litellm (for the Admin UI)"
|
||||
curl -sf -X POST "http://localhost:${LITELLM_PORT:-4000}/vector_store/new" \
|
||||
-H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$(jq -n --arg key "${LITELLM_PGVECTOR_API_KEY}" '{
|
||||
vector_store_id: "memory-and-notes",
|
||||
custom_llm_provider: "pg_vector",
|
||||
vector_store_name: "memory-and-notes",
|
||||
litellm_params: {api_base: "http://litellm-pgvector:8000", api_key: $key}
|
||||
}')" > /dev/null 2>&1 || echo "memory-and-notes: already registered (or registration failed — check by hand if this is a fresh deploy)."
|
||||
|
||||
echo "==> status"
|
||||
docker compose ps
|
||||
|
||||
Reference in New Issue
Block a user