Follows directly from the previous commit's caveat: this PR's own fix
(LLAMA_FAST_PARALLEL=2 -> 1) wouldn't have taken effect on the server
without a manual .env edit, because set_if_blank never touches an
already-set value — by design, for secrets, but the same logic was
silently protecting stale copies of ordinary tunable config too.
Every KEY=VALUE line in .env.example with a real (non-blank) default
is now force-synced into .env on every run. Secrets and host-resolved
values are unaffected — .env.example already leaves those blank on
purpose, so the sync loop naturally skips them and they keep going
through set_if_blank as before.
Trade-off, called out in both the script's header and the sync loop's
own comment: there's no such thing as a persistent server-only
override for these keys anymore — a hand-edited value not reflected
in git gets reverted on the next run. That's the intended behavior.
Verified against a simulated stale .env matching the real scenario
from this PR: LLAMA_FAST_PARALLEL correctly overwritten 2 -> 1, an
OMNIROUTE secret left untouched. bash -n and docker compose config -q
both pass.
Refs #5
Real failure on the server after #47 merged: the old GID-resolution
code ran (found the old COMFYUI_VIDEO_GID/COMFYUI_RENDER_GID vars
"already set"), then git pull swapped every file on disk out from
under the still-running script — including docker-compose.yml, now
requiring HOST_VIDEO_GID/HOST_RENDER_GID — but the resolution step
that would populate those had already run under the old code and
never re-ran. compose validation then failed on the new required
vars that were never set.
A self-updating script isn't atomic against its own file changing
mid-run. Move git pull to the very first thing the script does, so
every run is consistently either fully old or fully new code, never
a mix.
bash -n and docker compose config -q both validated.
Two real-hardware findings from issue #5, both researched and fixed
together (docs/research/rocm-gpu-pin-and-render-group.md):
- GPU_MAX_HW_QUEUES=1 on llama-server and llama-server-fast. Confirmed
on real hardware: either container alone is fine (3% GPU, low
power), only two concurrent HIP contexts pin the R9700 at 100%/
boost-clock (ROCm/ROCm#5706, an MES firmware bug). The env var is
validated on the exact image this stack uses, per-process by design
— applying it to both containers is the correct scope.
- group_add switched from plain names (video/render) to resolved
numeric GIDs (HOST_VIDEO_GID/HOST_RENDER_GID) on all three GPU
services. The "unable to find group render: no matching entries in
group file" error confirmed new since the second GPU service was
added is a known Docker bug (docker/cli#4714): group_add by name
resolves against the container's own /etc/group, not the host's,
and multiple GPU services starting concurrently race on that lookup.
Numeric GIDs skip resolution entirely. scripts/update.sh's existing
comfyui-only GID resolution is generalized to resolve these once for
all three services.
docker compose config -q validated (fails fast with a clear error if
the GIDs aren't resolved yet, passes once they are).
Refs #5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Two frontier tickets from the ComfyUI map (#38), both unblocked now
that their blockers (#39 model choice, #40 lazytainer research) are
resolved.
- downloader-comfyui service (docker-compose.yml) + COMFYUI_* vars
(.env.example): fetches Qwen-Image FP8 diffusion/text-encoder/VAE
weights from Comfy-Org/Qwen-Image_ComfyUI, same test -f guard
pattern as the existing downloaders. Closes#42.
- scripts/switch-model.sh: swaps GPU residency between llama-server
and comfyui via direct `docker compose stop`/`up -d`, bypassing
lazytainer per docs/research/lazytainer-omniroute-idle-stop.md
(its packet-threshold detector can't distinguish OmniRoute's health
checks from real traffic, so idle-stop can't be relied on for a
deliberate swap). llama-server-fast stays resident throughout —
not part of this swap. Closes#43.
- scripts/update.sh: runs the new downloader profile.
docker compose config -q validated clean.
Refs #38, #42, #43
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507,
Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for
use as qwen-code CLI's Auto Mode classifier fastModel. Model choice
researched in docs/research/fast-model-choice.md: architecturally
non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added
defensively per a known (closed) llama.cpp misdetection bug.
- docker-compose.yml: llama-server-fast + downloader-fast services,
omniroute depends_on updated
- .env.example: LLAMA_FAST_* vars
- scripts/update.sh: runs the new downloader profile
Refs #44
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Adds the comfyui service (yurisasc/comfyui-rocm7.1, gfx1201-tuned for the
R9700) on the ai-stack network, published on host port 8138 for a planned
external nginx route to comfy.home. Resolves PUID/PGID/VIDEO_GID/RENDER_GID
from the host in scripts/update.sh, same pattern as SEARXNG_LAN_IP.
OmniRoute provider registration (http://comfyui:8188) is still the same
manual dashboard/POST-/api/providers flow already used for llama-server —
not scripted, per docs/proxy-key-onboarding.md.
Part of wayfinder map #38.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zCwaWJQuKgXDUfRPBqDS7
Cross-checked the deployment against OmniRoute's own docs
(docs/reference/ENVIRONMENT.md, docs/guides/DOCKER_GUIDE.md) and found
three gaps from the original migration:
- OMNIROUTE_WS_BRIDGE_SECRET was entirely missing - ENVIRONMENT.md marks
it REQUIRED (production), for the internal Codex Responses WebSocket
bridge. docker compose config validated fine without it (compose
doesn't know omniroute's own required-var list), so this went
unnoticed until checking the docs directly.
- No mem_limit/OMNIROUTE_MEMORY_MB - the Docker guide is explicit that
the 1024MB default heap is dashboard-only sized; coding-agent workloads
(every client this stack has) need OMNIROUTE_MEMORY_MB=8192 and a
10+ GiB container ceiling. Set both.
- No stop_grace_period - the guide's --stop-timeout 40 equivalent, so
SQLite WAL changes checkpoint back into the main DB file on shutdown
instead of getting killed mid-write.
Redis checked and confirmed correctly absent - OmniRoute uses SQLite
only, no Redis anywhere in its docs.
Still open: whether API_PORT actually isolates /dashboard and /api/*
from the published port, or bridges everything through (see issue #31)
- OmniRoute's own ARCHITECTURE.md doesn't document split-port mode as a
real security boundary, and the live "[API Bridge] ... -> dashboard"
log line is ambiguous. Waiting on a live curl test against
proxy-ai.home before treating that as resolved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
No longer needed - every client is a coding CLI behind the OmniRoute
gateway, not a chat UI. Drops the open-webui and qdrant services,
WEBUI_PORT/OPENWEBUI_OMNIROUTE_KEY env vars, and the openwebui-data/
qdrant-data volumes. Qdrant only ever served Open WebUI's own built-in
memory/RAG (unrelated to the gateway-level knowledgebase removed in
472e3a4), so it goes too rather than sit unused.
Docs updated: README, docs/network-access.md (ai.home/ai.haylan.ch
section was entirely about Open WebUI, rewritten around the gateway),
docs/proxy-key-onboarding.md, docs/proxy-request-priority.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37):
replace the litellm/litellm-db services with omniroute, split-port mode
(API_PORT published/reverse-proxied, DASHBOARD_PORT never published -
tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place
of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as
omniroute providers post-boot (no static config.yaml equivalent). No
scripted per-workload key minting yet - omniroute's POST /api/keys needs a
dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is
a manual step for now (docs/proxy-key-onboarding.md).
Caveat carried into the map and README: OmniRoute's own docs
(docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md
on its release/v3.8.51 branch) describe shipped features for AI-provider
client-detection evasion, system-wide HTTPS interception via a locally
installed root CA, and hiding credentials from secret scanners. Proceeding
anyway was an explicit, informed user decision.
Also drops the gateway-level memory/knowledgebase feature entirely (user:
"I don't need it") - litellm-pgvector, pgvector-db, embedding-server,
scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory-
knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated
and untouched. litellm-config.yaml deleted (was kept as a rollback
reference, but there's no rollback path to a feature being deliberately
removed).
Not yet verified against real hardware - see issue #31's open tickets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
The vector_store_registry block in litellm-config.yaml only seeds the
store into litellm's in-memory registry at boot — the Admin UI's Vector
Stores page (/ui/vector-stores) reads litellm's own DB
(LiteLLM_ManagedVectorStoresTable) instead, via /vector_store/list. A
config-only entry there gets silently deleted from memory the first time
anyone loads that page, since /vector_store/list treats the DB as the
source of truth and removes anything not also present there.
update.sh now registers it in the DB too, via POST /vector_store/new
(idempotent, same pattern as the existing virtual-key minting) — with the
literal resolved key, not the os.environ/... form used in
litellm-config.yaml, since the management API doesn't do config.yaml-style
substitution on request bodies.
Found in the process: /vector_store/update in this litellm version can't
touch litellm_params at all (VectorStoreUpdateRequest has no such field,
so PGVECTOR_API_KEY sent through it is silently dropped) — worth knowing
if this key ever needs rotating; documented in update.sh's comment.
Verified live: registered via the API, confirmed the row appears in
/vector_store/list (what the Admin UI page reads) with the real key, and
re-verified search still works end-to-end afterward.
litellm-pgvector's Dockerfile only runs 'prisma generate' (codegen) at
build time — nothing ever ran 'prisma db push' against pgvector-db, so the
vector_stores/embeddings tables plain didn't exist on a fresh deploy. Every
vector store call failed with 'relation "vector_stores" does not exist'.
update.sh now runs the push itself after bringing litellm-pgvector up.
Two bugs, either one fatal:
- litellm-pgvector's embeddings endpoints take a precomputed vector per
chunk (they don't call the embedding model themselves) — the script
posted {content, metadata} with no embedding field, guaranteed 422.
- The batch endpoint expects {"embeddings": [...]}; the script posted a
bare JSON array as the body.
Now embeds each file's chunks via LiteLLM's /v1/embeddings first (using
LITELLM_PGVECTOR_EMBEDDING_KEY, already minted for exactly this) before
batch-inserting. Also made both source files optional — a missing file is
skipped, not a hard failure, since neither exists in this checkout yet.
Today's embedding-server crash-loop (missing nomic-embed-text GGUF) was a
manual step nobody ran. update.sh now runs both downloader profiles
itself, every time, before bringing services up -- no separate command to
remember.
- docker-compose.yml: downloader/downloader-embedding commands gain a
`test -f ... && skip || curl ...` guard, so re-running update.sh never
re-downloads an existing model file.
- scripts/update.sh: runs both profiles after image pull/build, before
service recreation.
- scripts/download-model.sh removed -- folded in, redundant standalone
script.
- README.md / docs/memory-knowledgebase.md updated accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
This reverts commit abeadc49c8.
Restores vendor/litellm-pgvector/ and the vector_store_registry wiring
(in-band file_search tool-call support) at the user's request, after
re-confirming against docs.litellm.ai/docs/completion/knowledgebase and
litellm-pgvector's own README that pg_vector is still not an in-process
vector_store_registry backend -- it requires this same standalone
connector service either way, so there is no simpler 'native' path that
was missed. Trading back in: 793 lines of vendored code, the untested
Prisma migration, and the git-context build risk noted in VENDORED.md
(all flagged as unverified against real hardware in issue #24), in
exchange for the file_search in-band tool call memory-retrieval did not
support.
Conflicts resolved on top of later commits (Redis, update.sh key-minting
fold-in):
- .env.example / docs/memory-knowledgebase.md: kept the auto-mint-via-
update.sh language, renamed MEMORY_RETRIEVAL_* back to
LITELLM_PGVECTOR_*.
- scripts/generate-secrets.sh: left deleted -- its job was folded into
update.sh in 24d749b, unrelated to this revert.
- scripts/update.sh: renamed the MEMORY_RETRIEVAL_* secret/mint calls to
LITELLM_PGVECTOR_* to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
New redis service (redis:7-alpine, password-protected, no persistence
volume — everything it holds is cache/coordination state). litellm gets
REDIS_HOST/REDIS_PORT/REDIS_PASSWORD, which is all LiteLLM needs to use it
for router state, rate limits/budgets, and cache invalidation — no
litellm-config.yaml block required (docs.litellm.ai/docs/proxy/caching).
REDIS_PASSWORD added to .env.example and update.sh's auto-generated
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_if_blank/mint_key_if_blank used sed -i "s|^KEY=.*|...|", which only
replaces an existing KEY= line — a no-op if the line isn't there at all.
Any .env created before a var was added to .env.example (SEARXNG_LAN_IP,
PGVECTOR_DB_PASSWORD, MEMORY_RETRIEVAL_*, etc.) silently never got that var
filled in. Both helpers now check for "line missing" as a third case and
append it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update.sh now creates .env from .env.example if missing, idempotently fills
in every random secret (same logic generate-secrets.sh had, now removed),
resolves SEARXNG_LAN_IP from search.home via the host's own DNS, and mints
OPENWEBUI_LITELLM_KEY / MEMORY_RETRIEVAL_EMBEDDING_KEY through LiteLLM's own
/key/generate API once litellm is up — no more manual Admin UI step for the
stack's own two workload keys. Docs updated to point at update.sh as the
one command; docs/proxy-key-onboarding.md keeps the manual/API steps as the
fallback and for onboarding other workloads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per docs/research/langchain-pgvector-vs-litellm-pgvector.md (issue #25):
the vendored litellm-pgvector connector (793 lines, Prisma migrations, a
fragile git-context build) is replaced by a ~90-line FastAPI service
(services/memory-retrieval/) wrapping langchain_postgres.PGVector directly
against pgvector-db. Same gateway boundary — it still calls litellm for
embeddings, nothing talks to Postgres or the model directly except this
service.
- New services/memory-retrieval/ (main.py, Dockerfile, requirements.txt):
POST /ingest, POST /query, GET /health.
- docker-compose.yml: litellm-pgvector service replaced by memory-retrieval;
pgvector-db and embedding-server untouched.
- litellm-config.yaml: vector_store_registry block removed (no
langchain_postgres provider exists to register against; callers query
memory-retrieval directly instead of an in-band file_search tool call —
that mechanism was never confirmed working per issue #24 anyway).
- scripts/ingest-memory.sh rewritten for the new /ingest endpoint (same
per-line chunking, no dedup).
- .env vars renamed: LITELLM_PGVECTOR_API_KEY/LITELLM_PGVECTOR_EMBEDDING_KEY
-> MEMORY_RETRIEVAL_API_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY.
- vendor/litellm-pgvector/ removed entirely.
- docs/memory-knowledgebase.md updated for the new setup/query flow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One script to bring the running stack in sync after any change: git pull,
validate compose config, rebuild the litellm-pgvector local build, re-pull
images, recreate what changed. Documented in README's quickstart, and
CLAUDE.md now tells agents to run it after touching compose/config/scripts
rather than just describing a change as done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- search_tools block in litellm-config.yaml (SearXNG as a first-class
search_provider, standalone /v1/search endpoint, not a model tool) plus
extra_hosts on the litellm service so it can resolve search.home.
- New embedding-server (nomic-embed-text-v1.5 on a second llama.cpp
instance), pgvector-db, and litellm-pgvector services — LiteLLM's native
knowledgebase feature has no Qdrant backend, so this is the only
self-hosted path (docs/research/litellm-knowledgebase.md).
- vector_store_registry + local-embedding model entry in
litellm-config.yaml, wiring it together.
- scripts/ingest-memory.sh to load data/memory.md and
data/claude-legacy-memory.md into the knowledgebase.
- docs/memory-knowledgebase.md documenting the whole setup; data/
gitignored (personal memory content, not meant to be committed).
- New .env vars (SEARXNG_LAN_IP, PGVECTOR_DB_PASSWORD,
LITELLM_PGVECTOR_API_KEY, LITELLM_PGVECTOR_EMBEDDING_KEY,
EMBEDDING_MODEL_FILE) and generate-secrets.sh support for the
auto-generatable ones.
Resolves#22 and #23 (wayfinder map #21). Not yet verified on real
hardware — see #24.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Without these, LiteLLM's /ui falls back to username 'admin' and the
master key as the password, forcing the master key to be typed into
a browser. generate-secrets.sh now fills UI_PASSWORD like the other
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Resolves wayfinder ticket #4. Wires up the locked decisions from the map:
- llama.cpp (ghcr.io/ggml-org/llama.cpp:server-rocm, gfx1201) serving
Qwen3.8-27B-UD-Q4_K_XL.gguf, port published for direct Claude Code CLI /
Kimi CLI access alongside Open WebUI.
- Open WebUI with WEBUI_AUTH on, RAG+Memory wired to a standalone Qdrant
service.
- Lazytainer labels on llama-server for a 15 min idle-stop.
- Named Docker volumes only (models, qdrant-data, openwebui-data) — no host
bind-mounts.
- One-off 'downloader' compose profile instead of a host-side script with
its own dependencies, wrapped by scripts/download-model.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylan
