Batch/ubatch reduction barely moved measured VRAM (~768MB free, same as
before) — wrong lever. llama-server runs with --flash-attn on; this
service didn't. Without it, the unfused attention compute buffer at
65536 ctx is far larger than flash-attn's fused workspace, which is
what the naive weights+KV estimate missed. Matches llama-server's flag.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Measured live: full-offload weights+KV (~4.9GiB estimate) actually used
~5.85GiB, leaving only ~700MB free on the R9700 — too tight, real OOM
risk for either GPU process. The gap was compute-buffer/graph overhead
the naive estimate didn't account for. Drop --batch-size/--ubatch-size
well below llama-server's defaults (2048/512) to shrink it — a
single-request classifier has no batching throughput to lose.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CPU-only was too slow in practice: real classification calls blew past
OmniRoute's 60s timeout and retry-looped (504/499). Moved to GPU.
Also traced qwen-code's actual classifier transcript cap in its source
(MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000/message) —
worst case is ~40-50K tokens, not the 131072 originally set in
settings.json (copied from the main model's entry, not a real qwen-code
requirement). Dropped ctx-size to 65536 (~1.5x margin) so Q4_K_XL
weights + q4_0/q4_0 KV fit fully on GPU (~4.9GiB) inside the ~6.1GiB
free on the R9700, instead of needing partial CPU/GPU offload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Q4_K_M-class file this originally specced didn't exist yet on
gameserver (classifier crash-looped: "No such file or directory").
A Q8_K_XL GGUF for the same model was already sitting in the models
volume from something earlier — point at that instead of downloading a
new file, and drop the KV cache quant to q4_0/q4_0 to keep total RAM
comfortable now that the weights are the larger Q8 variant.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was
aliased onto llama-server's own 27B connection, so every tool-call safety
check queued behind whatever heavy generation was already running on that
model's 2 GPU slots.
Add qwen-classifier: a separate llama.cpp instance, CPU-only, running
Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context,
qwen-code's requirement, without lossy RoPE scaling). Structurally isolated
from llama-server's queue instead of sharing it. Sized for gameserver's
~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M-
class weights (~2.3GiB) fits comfortably, with better KV quality than the
q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Assess whether the Colibrì disk-streaming MoE inference engine fits the AMD ROCm single-GPU setup or fills a gap beyond the existing llama.cpp/OmniRoute/Qdrant/Neo4j/ComfyUI stack. Conclusion: worth a passing watch, not worth integrating today.
Move docs/coding-cli-setup.md to docs/coding-cli-setup/ with one file
per CLI (claude-code, kimi-cli, opencode, qwen-code) plus a shared
index.md for the gateway intro, tool-calling risk note, and summary
table.
Also fixes the qwen-code doc: context sizes are per-slot
(LLAMA_CTX_SIZE / LLAMA_PARALLEL), not raw LLAMA_CTX_SIZE (same fix
applied to OpenCode's limit.context); documents the fastModel
classifier provider and its own context math; adds the omniroute-search
MCP server (SearXNG web search) and Auto Mode permissions tuning that
were missing from the original qwen-code section.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
current=$(grep ...) ran before checking whether the key existed in .env
at all. Under set -euo pipefail, a key missing from .env (e.g. a new
tunable this branch just added to .env.example) made that grep exit 1,
pipefail propagated it, and set -e killed the script instantly with no
output - looked exactly like a gum hang but never reached ensure_gum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
The server has no outbound internet access, so the curl download in
ensure_gum always failed silently and fell back to plain prompts.
Vendor the x86_64 release tarball under scripts/vendor/ and check it
before attempting a download - download stays as a fallback for other
archs or a version bump without a re-vendor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
curl -fsSL | tar swallowed curl errors entirely, so a network failure
fetching gum looked identical to a successful skip - just silently
dropped into the plain-prompt fallback with no explanation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
Documents the NPM route for the RAG knowledge graph alongside the
existing proxy-ai.home/search.home entries. Neo4j (not Qdrant) gets
the hostname — it's the human-facing graph browser; Qdrant's
dashboard stays on its raw port for now.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
Drops the second always-resident llama.cpp instance, its downloader,
and the omniroute depends_on entry. Also strips the now-dead
LLAMA_FAST_* block from .env.example and the stale VRAM-budget comment
in scripts/switch-model.sh that assumed this service was always up.
Note: this was qwen-code's Auto Mode Stage 1 classifier (fastModel) —
see docs/research/fast-model-choice.md and issue #44. Auto Mode will
lose that classifier until/unless it's reconfigured to route
elsewhere or fall back to prompt-only classification.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
Adds the two databases as infra services (own volume, ai-stack
network, host-published UI ports like comfyui) for an upcoming RAG
pipeline. Extraction/chunking/orchestration code is out of scope for
this repo — it's app logic that calls into these DBs and llama-server,
not compose infra.
Neo4j password follows the omniroute secret pattern: blank in
.env.example, auto-generated by scripts/update.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
Replaces the previous commit's blind force-overwrite with a real
choice, per user feedback: force-overwriting server config without
asking was the wrong default.
- A tracked config value (real default in .env.example) that already
matches .env is left alone silently — no prompt, no noise.
- A value that DIFFERS is a conflict, shown on one screen (all
conflicts together, not one prompt per key) via gum
(charmbracelet/gum) — single static binary fetched as a release
tarball into .cache/gum/ (gitignored), no build step, no package
manager dependency. Falls back to a plain read-based prompt if gum
can't be fetched (offline, unsupported arch).
- Non-interactive (no TTY — cron, CI, piped): any conflict is a hard
error (exit 1, lists every conflicting key) unless --force is
passed, which accepts every new value automatically — matches how
this PR's own fix needs to land unattended.
- Secrets and host-resolved values are completely unaffected either
way — untouched by this loop, same as before.
Verified in an isolated sandbox against the exact scenario from this
PR (stale LLAMA_FAST_PARALLEL=2 vs git's 1):
- no TTY, no --force: exits 1, prints the diff, doesn't touch .env
- no TTY, --force: LLAMA_FAST_PARALLEL corrected 2 -> 1, an
OMNIROUTE secret confirmed untouched (not regenerated)
docker compose config -q still passes.
Not verified: the interactive gum path itself (needs a real TTY,
couldn't allocate a pty in this sandbox) — worth confirming for real
on the server, including that gum's release asset naming actually
matches what ensure_gum() expects.
Refs #5
Follows directly from the previous commit's caveat: this PR's own fix
(LLAMA_FAST_PARALLEL=2 -> 1) wouldn't have taken effect on the server
without a manual .env edit, because set_if_blank never touches an
already-set value — by design, for secrets, but the same logic was
silently protecting stale copies of ordinary tunable config too.
Every KEY=VALUE line in .env.example with a real (non-blank) default
is now force-synced into .env on every run. Secrets and host-resolved
values are unaffected — .env.example already leaves those blank on
purpose, so the sync loop naturally skips them and they keep going
through set_if_blank as before.
Trade-off, called out in both the script's header and the sync loop's
own comment: there's no such thing as a persistent server-only
override for these keys anymore — a hand-edited value not reflected
in git gets reverted on the next run. That's the intended behavior.
Verified against a simulated stale .env matching the real scenario
from this PR: LLAMA_FAST_PARALLEL correctly overwritten 2 -> 1, an
OMNIROUTE secret left untouched. bash -n and docker compose config -q
both pass.
Refs #5
Real failure: "Auto Mode couldn't classify this action (Classifier
stage 1 unavailable)". Reproduced directly against the server:
{"error":{"message":"[400]: request (6186 tokens) exceeds the
available context size (4096 tokens)"...
LLAMA_FAST_CTX_SIZE=8192 is the TOTAL across every LLAMA_FAST_PARALLEL
slot, not per-request — the main model's own .env.example comment
already calls this out, missed it when llama-server-fast was set up
(#44). With PARALLEL=2 that's 4096/slot, too small for a real
classifier call (hints + environment + recent tool-call history).
Fixed by dropping to a single slot (LLAMA_FAST_PARALLEL=1) rather than
raising ctx-size — this service doesn't need concurrent classifier
calls the way the main model needs concurrent chat sessions, so this
costs no extra VRAM. The full 8192 now goes to the one slot.
docker compose config -q validated.
Refs #5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Real failure on the server after #47 merged: the old GID-resolution
code ran (found the old COMFYUI_VIDEO_GID/COMFYUI_RENDER_GID vars
"already set"), then git pull swapped every file on disk out from
under the still-running script — including docker-compose.yml, now
requiring HOST_VIDEO_GID/HOST_RENDER_GID — but the resolution step
that would populate those had already run under the old code and
never re-ran. compose validation then failed on the new required
vars that were never set.
A self-updating script isn't atomic against its own file changing
mid-run. Move git pull to the very first thing the script does, so
every run is consistently either fully old or fully new code, never
a mix.
bash -n and docker compose config -q both validated.
Two real-hardware findings from issue #5, both researched and fixed
together (docs/research/rocm-gpu-pin-and-render-group.md):
- GPU_MAX_HW_QUEUES=1 on llama-server and llama-server-fast. Confirmed
on real hardware: either container alone is fine (3% GPU, low
power), only two concurrent HIP contexts pin the R9700 at 100%/
boost-clock (ROCm/ROCm#5706, an MES firmware bug). The env var is
validated on the exact image this stack uses, per-process by design
— applying it to both containers is the correct scope.
- group_add switched from plain names (video/render) to resolved
numeric GIDs (HOST_VIDEO_GID/HOST_RENDER_GID) on all three GPU
services. The "unable to find group render: no matching entries in
group file" error confirmed new since the second GPU service was
added is a known Docker bug (docker/cli#4714): group_add by name
resolves against the container's own /etc/group, not the host's,
and multiple GPU services starting concurrently race on that lookup.
Numeric GIDs skip resolution entirely. scripts/update.sh's existing
comfyui-only GID resolution is generalized to resolve these once for
all three services.
docker compose config -q validated (fails fast with a clear error if
the GIDs aren't resolved yet, passes once they are).
Refs #5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Two frontier tickets from the ComfyUI map (#38), both unblocked now
that their blockers (#39 model choice, #40 lazytainer research) are
resolved.
- downloader-comfyui service (docker-compose.yml) + COMFYUI_* vars
(.env.example): fetches Qwen-Image FP8 diffusion/text-encoder/VAE
weights from Comfy-Org/Qwen-Image_ComfyUI, same test -f guard
pattern as the existing downloaders. Closes#42.
- scripts/switch-model.sh: swaps GPU residency between llama-server
and comfyui via direct `docker compose stop`/`up -d`, bypassing
lazytainer per docs/research/lazytainer-omniroute-idle-stop.md
(its packet-threshold detector can't distinguish OmniRoute's health
checks from real traffic, so idle-stop can't be relied on for a
deliberate swap). llama-server-fast stays resident throughout —
not part of this swap. Closes#43.
- scripts/update.sh: runs the new downloader profile.
docker compose config -q validated clean.
Refs #38, #42, #43
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507,
Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for
use as qwen-code CLI's Auto Mode classifier fastModel. Model choice
researched in docs/research/fast-model-choice.md: architecturally
non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added
defensively per a known (closed) llama.cpp misdetection bug.
- docker-compose.yml: llama-server-fast + downloader-fast services,
omniroute depends_on updated
- .env.example: LLAMA_FAST_* vars
- scripts/update.sh: runs the new downloader profile
Refs #44
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Adds the comfyui service (yurisasc/comfyui-rocm7.1, gfx1201-tuned for the
R9700) on the ai-stack network, published on host port 8138 for a planned
external nginx route to comfy.home. Resolves PUID/PGID/VIDEO_GID/RENDER_GID
from the host in scripts/update.sh, same pattern as SEARXNG_LAN_IP.
OmniRoute provider registration (http://comfyui:8188) is still the same
manual dashboard/POST-/api/providers flow already used for llama-server —
not scripted, per docs/proxy-key-onboarding.md.
Part of wayfinder map #38.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zCwaWJQuKgXDUfRPBqDS7
Closes issue #39 — with llama-server stopped and the full ~32GB R9700
available, Qwen-Image (Apache-2.0, 20B MMDiT) at FP8 precision (~25GB)
is the recommended upgrade from FLUX.1-schnell: it has the cleanest
license of the candidates and is the only one with a ComfyUI workflow
pre-validated specifically on this GPU architecture (gfx1201/R9700),
per kyuz0/amd-r9700-comfy. HunyuanImage-3.0 is ruled out (CUDA-only,
multi-GPU datacenter VRAM floor); Krea-2 flagged as promising but too
new for R9700-specific field evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zCwaWJQuKgXDUfRPBqDS7
A real session hit 'request (66192 tokens) exceeds the available
context size (65536 tokens)' — --parallel 2 was splitting the
131072 total into 65536/slot, too small for actual usage.
Add --cache-type-k/v q8_0 (roughly halves KV memory) so the model's
true max context (262144, already the documented .env.example goal)
fits in the same ~25.6GB/6GB-headroom footprint the old 131072 fp16
setting used, instead of shrinking per-slot context to fit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cuts prefill time with no accuracy cost, per markaicode.com's
llama.cpp timeout writeup — directly helps the prefill-vs-stream-idle
contention issue fixed in the prior two commits.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contended prefill (see LLAMA_PARALLEL) can outrun the ~95s dashboard
value before first token, causing OmniRoute to cancel still-working
requests. Track it in git via STREAM_IDLE_TIMEOUT_MS instead of a
dashboard-only setting.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
OmniRoute's memory feature is self-contained: its bundled sqlite-vec
vector store plus a local ONNX embedding model (Transformers.js,
~400MB, fetched into the omniroute-data volume on first use) replace
the external qdrant + bge-small-en-v1.5 embedding-server pair, which
was never wired up in the dashboard. Two fewer containers, no
second GGUF download, no EMBEDDING_MODEL_FILE var. Memory stays
opt-in via the dashboard (Settings -> Memory, transformers source);
nothing here changes the gateway's static config.
llama.cpp loads one model per process and the running Qwen3.8-27B chat
model isn't embedding-trained, so this is a second, CPU-only
llama-server instance (bge-small-en-v1.5, 384-dim) rather than adding
--embeddings to the chat one — see docs/research/litellm-knowledgebase.md
#3.
Downloader extended to fetch both GGUFs into the shared models volume.
No host port published — OmniRoute reaches it via the ai-stack network
DNS name (embedding-server:8081).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qx22CV9EUS3hGATucQav13
Bare service, no static config — wired up as a memory provider by
hand in the OmniRoute dashboard. Not published to the host; only
OmniRoute (same ai-stack network) talks to it. Also drops two stale
comments left over from the OMNIROUTE_PORT:4000 removal.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qx22CV9EUS3hGATucQav13
Cross-checked the deployment against OmniRoute's own docs
(docs/reference/ENVIRONMENT.md, docs/guides/DOCKER_GUIDE.md) and found
three gaps from the original migration:
- OMNIROUTE_WS_BRIDGE_SECRET was entirely missing - ENVIRONMENT.md marks
it REQUIRED (production), for the internal Codex Responses WebSocket
bridge. docker compose config validated fine without it (compose
doesn't know omniroute's own required-var list), so this went
unnoticed until checking the docs directly.
- No mem_limit/OMNIROUTE_MEMORY_MB - the Docker guide is explicit that
the 1024MB default heap is dashboard-only sized; coding-agent workloads
(every client this stack has) need OMNIROUTE_MEMORY_MB=8192 and a
10+ GiB container ceiling. Set both.
- No stop_grace_period - the guide's --stop-timeout 40 equivalent, so
SQLite WAL changes checkpoint back into the main DB file on shutdown
instead of getting killed mid-write.
Redis checked and confirmed correctly absent - OmniRoute uses SQLite
only, no Redis anywhere in its docs.
Still open: whether API_PORT actually isolates /dashboard and /api/*
from the published port, or bridges everything through (see issue #31)
- OmniRoute's own ARCHITECTURE.md doesn't document split-port mode as a
real security boundary, and the live "[API Bridge] ... -> dashboard"
log line is ambiguous. Waiting on a live curl test against
proxy-ai.home before treating that as resolved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
Not omniroute's own internal port (API_PORT stays at its default 20129,
unreconfigured) - just the Docker port mapping, so existing NPM/firewall
config pointed at :4000 keeps working without changes on that end. New
OMNIROUTE_PORT env var is the host side of "OMNIROUTE_PORT:API_PORT" in
docker-compose.yml's ports: entry.
Also corrected docs/proxy-key-onboarding.md's dashboard-access
instructions - DASHBOARD_PORT was never published to the host in the
first place, so "http://<host>:20128" was never actually reachable as
written; documented reaching it via the container's own bridge-network IP
or an SSH port-forward instead.
llama-server remains unexposed (no ports: entry, only expose:) -
unaffected by this change, confirming it stays that way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
Confirmed live on the R9700: omniroute starts up fine ("[API Bridge]
Listening on 0.0.0.0:20129") but docker reported it unhealthy forever -
the healthcheck's python3 -c "..." command can never run (which python3
wget curl node found only node in the image), so it failed every single
check regardless of actual app health.
Switched to a node-based TCP-connect check on the same port instead of an
HTTP GET against /healthz - OmniRoute's own Docker guide already treats a
bare TCP probe as an acceptable liveness check, and this sidesteps needing
to confirm /healthz's exact path/response shape on this image.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
No longer needed - every client is a coding CLI behind the OmniRoute
gateway, not a chat UI. Drops the open-webui and qdrant services,
WEBUI_PORT/OPENWEBUI_OMNIROUTE_KEY env vars, and the openwebui-data/
qdrant-data volumes. Qdrant only ever served Open WebUI's own built-in
memory/RAG (unrelated to the gateway-level knowledgebase removed in
472e3a4), so it goes too rather than sit unused.
Docs updated: README, docs/network-access.md (ai.home/ai.haylan.ch
section was entirely about Open WebUI, rewritten around the gateway),
docs/proxy-key-onboarding.md, docs/proxy-request-priority.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37):
replace the litellm/litellm-db services with omniroute, split-port mode
(API_PORT published/reverse-proxied, DASHBOARD_PORT never published -
tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place
of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as
omniroute providers post-boot (no static config.yaml equivalent). No
scripted per-workload key minting yet - omniroute's POST /api/keys needs a
dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is
a manual step for now (docs/proxy-key-onboarding.md).
Caveat carried into the map and README: OmniRoute's own docs
(docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md
on its release/v3.8.51 branch) describe shipped features for AI-provider
client-detection evasion, system-wide HTTPS interception via a locally
installed root CA, and hiding credentials from secret scanners. Proceeding
anyway was an explicit, informed user decision.
Also drops the gateway-level memory/knowledgebase feature entirely (user:
"I don't need it") - litellm-pgvector, pgvector-db, embedding-server,
scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory-
knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated
and untouched. litellm-config.yaml deleted (was kept as a rollback
reference, but there's no rollback path to a feature being deliberately
removed).
Not yet verified against real hardware - see issue #31's open tickets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
tea comments create doesn't exist (add/a is the subcommand); a wayfinder
session hit this live resolving OmniRoute-migration tickets. Also note
that concurrent ticket resolutions racing to edit the same map issue
(a full-body replace, no append) can clobber each other's Decisions-so-far
lines -- observed the same session across 5 parallel research tickets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
The vector_store_registry block in litellm-config.yaml only seeds the
store into litellm's in-memory registry at boot — the Admin UI's Vector
Stores page (/ui/vector-stores) reads litellm's own DB
(LiteLLM_ManagedVectorStoresTable) instead, via /vector_store/list. A
config-only entry there gets silently deleted from memory the first time
anyone loads that page, since /vector_store/list treats the DB as the
source of truth and removes anything not also present there.
update.sh now registers it in the DB too, via POST /vector_store/new
(idempotent, same pattern as the existing virtual-key minting) — with the
literal resolved key, not the os.environ/... form used in
litellm-config.yaml, since the management API doesn't do config.yaml-style
substitution on request bodies.
Found in the process: /vector_store/update in this litellm version can't
touch litellm_params at all (VectorStoreUpdateRequest has no such field,
so PGVECTOR_API_KEY sent through it is silently dropped) — worth knowing
if this key ever needs rotating; documented in update.sh's comment.
Verified live: registered via the API, confirmed the row appears in
/vector_store/list (what the Admin UI page reads) with the real key, and
re-verified search still works end-to-end afterward.
memory-knowledgebase.md no longer says 'not yet verified' — it's been
smoke-tested end-to-end (direct search + the file_search tool on a chat
completion) and the bugs found are fixed in the preceding commits.
VENDORED.md documents the three local patches on top of the upstream
litellm-pgvector commit so a future re-vendor doesn't silently drop them.
litellm-pgvector's Dockerfile only runs 'prisma generate' (codegen) at
build time — nothing ever ran 'prisma db push' against pgvector-db, so the
vector_stores/embeddings tables plain didn't exist on a fresh deploy. Every
vector store call failed with 'relation "vector_stores" does not exist'.
update.sh now runs the push itself after bringing litellm-pgvector up.
Two bugs, either one fatal:
- litellm-pgvector's embeddings endpoints take a precomputed vector per
chunk (they don't call the embedding model themselves) — the script
posted {content, metadata} with no embedding field, guaranteed 422.
- The batch endpoint expects {"embeddings": [...]}; the script posted a
bare JSON array as the body.
Now embeds each file's chunks via LiteLLM's /v1/embeddings first (using
LITELLM_PGVECTOR_EMBEDDING_KEY, already minted for exactly this) before
batch-inserting. Also made both source files optional — a missing file is
skipped, not a hard failure, since neither exists in this checkout yet.
litellm.aembedding can't infer a provider from a bare model name plus a
custom api_base — litellm-pgvector's embedding_service.py was hitting
'litellm.BadRequestError: LLM Provider NOT provided' on every query-time
embedding (i.e. every search). Same openai/ prefix already used for
qwen3.8-27b-local and local-embedding in litellm-config.yaml.
POST /v1/vector_stores always minted a random UUID for the new store's id,
ignoring the request entirely. litellm-config.yaml's vector_store_registry
addresses this store by a fixed id (memory-and-notes), which could never
match a real row as a result — every search/write 404'd.
Added an optional id field to VectorStoreCreateRequest; create_vector_store
uses it when given, falls back to a random UUID otherwise (unchanged
behavior for callers that don't care).
schema.prisma hardcoded vector(1536), OpenAI ada-002's size. This stack's
actual embedding model is nomic-embed-text-v1.5, confirmed 768-dim live
against /v1/embeddings. Left as-is, the first insert after a schema push
would fail on a pgvector dimension mismatch.
Without it, litellm's pg_vector provider sent Authorization: Bearer None,
which round-tripped to the real api.openai.com and came back with
'Incorrect API key provided: None' — masking the actual problem and making
every vector store call fail. Points at litellm-pgvector's own
LITELLM_PGVECTOR_API_KEY (its SERVER_API_KEY), already present in .env.
Smoke-tested end-to-end against the live deploy for issue #24, alongside
the litellm-pgvector fixes in the following commits.
Today's embedding-server crash-loop (missing nomic-embed-text GGUF) was a
manual step nobody ran. update.sh now runs both downloader profiles
itself, every time, before bringing services up -- no separate command to
remember.
- docker-compose.yml: downloader/downloader-embedding commands gain a
`test -f ... && skip || curl ...` guard, so re-running update.sh never
re-downloads an existing model file.
- scripts/update.sh: runs both profiles after image pull/build, before
service recreation.
- scripts/download-model.sh removed -- folded in, redundant standalone
script.
- README.md / docs/memory-knowledgebase.md updated accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
Confirmed in llama-server logs: a real request hit exactly n_gen=4096
(the old floor) and returned no answer -- reasoning_content alone ate
the whole budget before any content was written, exactly the failure
mode this config's own comment predicted. 16384 is the user-chosen
middle ground between 8192 and 32768: ~10 min worst-case at ~26.7 t/s,
well under the 65536-token context window.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
lazytainer panics on every start with 'Could not determine container ID
of lazytainer': its self-detection (vmorganp/Lazytainer,
configureFromLabels()) matches os.Hostname() against the Docker
container-ID list, but network_mode: host makes the container inherit
the host machine's hostname instead of its own container ID, so the
match never succeeds. Upstream's own example compose file doesn't use
host networking (default bridge + published ports) — nothing about
lazytainer requires it.
It also couldn't have seen the traffic it's meant to watch: llama-server
has no published host port (issue #15) and only exists on the ai-stack
bridge network, which host networking has no visibility into. Joining
ai-stack instead fixes both problems at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
README.md and docs/proxy-key-onboarding.md were edited by later commits
(b996b1f, 24d749b) that the litellm-pgvector revert didn't touch, so they
still named the now-removed MEMORY_RETRIEVAL_EMBEDDING_KEY var. Left
docs/research/langchain-pgvector-vs-litellm-pgvector.md as-is — it's a
dated record of the decision being reversed, not live config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
This reverts commit abeadc49c8.
Restores vendor/litellm-pgvector/ and the vector_store_registry wiring
(in-band file_search tool-call support) at the user's request, after
re-confirming against docs.litellm.ai/docs/completion/knowledgebase and
litellm-pgvector's own README that pg_vector is still not an in-process
vector_store_registry backend -- it requires this same standalone
connector service either way, so there is no simpler 'native' path that
was missed. Trading back in: 793 lines of vendored code, the untested
Prisma migration, and the git-context build risk noted in VENDORED.md
(all flagged as unverified against real hardware in issue #24), in
exchange for the file_search in-band tool call memory-retrieval did not
support.
Conflicts resolved on top of later commits (Redis, update.sh key-minting
fold-in):
- .env.example / docs/memory-knowledgebase.md: kept the auto-mint-via-
update.sh language, renamed MEMORY_RETRIEVAL_* back to
LITELLM_PGVECTOR_*.
- scripts/generate-secrets.sh: left deleted -- its job was folded into
update.sh in 24d749b, unrelated to this revert.
- scripts/update.sh: renamed the MEMORY_RETRIEVAL_* secret/mint calls to
LITELLM_PGVECTOR_* to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
New redis service (redis:7-alpine, password-protected, no persistence
volume — everything it holds is cache/coordination state). litellm gets
REDIS_HOST/REDIS_PORT/REDIS_PASSWORD, which is all LiteLLM needs to use it
for router state, rate limits/budgets, and cache invalidation — no
litellm-config.yaml block required (docs.litellm.ai/docs/proxy/caching).
REDIS_PASSWORD added to .env.example and update.sh's auto-generated
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
set_if_blank/mint_key_if_blank used sed -i "s|^KEY=.*|...|", which only
replaces an existing KEY= line — a no-op if the line isn't there at all.
Any .env created before a var was added to .env.example (SEARXNG_LAN_IP,
PGVECTOR_DB_PASSWORD, MEMORY_RETRIEVAL_*, etc.) silently never got that var
filled in. Both helpers now check for "line missing" as a third case and
append it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
update.sh now creates .env from .env.example if missing, idempotently fills
in every random secret (same logic generate-secrets.sh had, now removed),
resolves SEARXNG_LAN_IP from search.home via the host's own DNS, and mints
OPENWEBUI_LITELLM_KEY / MEMORY_RETRIEVAL_EMBEDDING_KEY through LiteLLM's own
/key/generate API once litellm is up — no more manual Admin UI step for the
stack's own two workload keys. Docs updated to point at update.sh as the
one command; docs/proxy-key-onboarding.md keeps the manual/API steps as the
fallback and for onboarding other workloads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per docs/research/langchain-pgvector-vs-litellm-pgvector.md (issue #25):
the vendored litellm-pgvector connector (793 lines, Prisma migrations, a
fragile git-context build) is replaced by a ~90-line FastAPI service
(services/memory-retrieval/) wrapping langchain_postgres.PGVector directly
against pgvector-db. Same gateway boundary — it still calls litellm for
embeddings, nothing talks to Postgres or the model directly except this
service.
- New services/memory-retrieval/ (main.py, Dockerfile, requirements.txt):
POST /ingest, POST /query, GET /health.
- docker-compose.yml: litellm-pgvector service replaced by memory-retrieval;
pgvector-db and embedding-server untouched.
- litellm-config.yaml: vector_store_registry block removed (no
langchain_postgres provider exists to register against; callers query
memory-retrieval directly instead of an in-band file_search tool call —
that mechanism was never confirmed working per issue #24 anyway).
- scripts/ingest-memory.sh rewritten for the new /ingest endpoint (same
per-line chunking, no dedup).
- .env vars renamed: LITELLM_PGVECTOR_API_KEY/LITELLM_PGVECTOR_EMBEDDING_KEY
-> MEMORY_RETRIEVAL_API_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY.
- vendor/litellm-pgvector/ removed entirely.
- docs/memory-knowledgebase.md updated for the new setup/query flow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verdict: switch. See docs/research/langchain-pgvector-vs-litellm-pgvector.md
for full reasoning, LOC comparison, and a concrete replacement service spec.
Research/doc only — no compose/config/vendor changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The server's Docker/BuildKit couldn't do a git-context build of a public
github.com repo (fails with "could not read Username ... terminal prompts
disabled" — an auth-shaped error for what should be an anonymous clone).
Rather than debug that, vendor litellm-pgvector's small source tree
directly (vendor/litellm-pgvector/, see VENDORED.md for provenance/update
steps) and build from the local path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One script to bring the running stack in sync after any change: git pull,
validate compose config, rebuild the litellm-pgvector local build, re-pull
images, recreate what changed. Documented in README's quickstart, and
CLAUDE.md now tells agents to run it after touching compose/config/scripts
rather than just describing a change as done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- search_tools block in litellm-config.yaml (SearXNG as a first-class
search_provider, standalone /v1/search endpoint, not a model tool) plus
extra_hosts on the litellm service so it can resolve search.home.
- New embedding-server (nomic-embed-text-v1.5 on a second llama.cpp
instance), pgvector-db, and litellm-pgvector services — LiteLLM's native
knowledgebase feature has no Qdrant backend, so this is the only
self-hosted path (docs/research/litellm-knowledgebase.md).
- vector_store_registry + local-embedding model entry in
litellm-config.yaml, wiring it together.
- scripts/ingest-memory.sh to load data/memory.md and
data/claude-legacy-memory.md into the knowledgebase.
- docs/memory-knowledgebase.md documenting the whole setup; data/
gitignored (personal memory content, not meant to be committed).
- New .env vars (SEARXNG_LAN_IP, PGVECTOR_DB_PASSWORD,
LITELLM_PGVECTOR_API_KEY, LITELLM_PGVECTOR_EMBEDDING_KEY,
EMBEDDING_MODEL_FILE) and generate-secrets.sh support for the
auto-generatable ones.
Resolves#22 and #23 (wayfinder map #21). Not yet verified on real
hardware — see #24.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers whether SearXNG is natively supported by LiteLLM's /v1/search
feature, whether it's a model-tool or automatic retrieval, what
docker-compose.yml networking change is needed for the litellm container
to reach the LAN's search.home host, and how it interacts with this
project's known-flaky Qwen3.8-27B tool-calling.
Research only — litellm-config.yaml and docker-compose.yml are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude-Bot