Today's embedding-server crash-loop (missing nomic-embed-text GGUF) was a
manual step nobody ran. update.sh now runs both downloader profiles
itself, every time, before bringing services up -- no separate command to
remember.
- docker-compose.yml: downloader/downloader-embedding commands gain a
`test -f ... && skip || curl ...` guard, so re-running update.sh never
re-downloads an existing model file.
- scripts/update.sh: runs both profiles after image pull/build, before
service recreation.
- scripts/download-model.sh removed -- folded in, redundant standalone
script.
- README.md / docs/memory-knowledgebase.md updated accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
README.md and docs/proxy-key-onboarding.md were edited by later commits
(b996b1f, 24d749b) that the litellm-pgvector revert didn't touch, so they
still named the now-removed MEMORY_RETRIEVAL_EMBEDDING_KEY var. Left
docs/research/langchain-pgvector-vs-litellm-pgvector.md as-is — it's a
dated record of the decision being reversed, not live config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
This reverts commit abeadc49c8.
Restores vendor/litellm-pgvector/ and the vector_store_registry wiring
(in-band file_search tool-call support) at the user's request, after
re-confirming against docs.litellm.ai/docs/completion/knowledgebase and
litellm-pgvector's own README that pg_vector is still not an in-process
vector_store_registry backend -- it requires this same standalone
connector service either way, so there is no simpler 'native' path that
was missed. Trading back in: 793 lines of vendored code, the untested
Prisma migration, and the git-context build risk noted in VENDORED.md
(all flagged as unverified against real hardware in issue #24), in
exchange for the file_search in-band tool call memory-retrieval did not
support.
Conflicts resolved on top of later commits (Redis, update.sh key-minting
fold-in):
- .env.example / docs/memory-knowledgebase.md: kept the auto-mint-via-
update.sh language, renamed MEMORY_RETRIEVAL_* back to
LITELLM_PGVECTOR_*.
- scripts/generate-secrets.sh: left deleted -- its job was folded into
update.sh in 24d749b, unrelated to this revert.
- scripts/update.sh: renamed the MEMORY_RETRIEVAL_* secret/mint calls to
LITELLM_PGVECTOR_* to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
update.sh now creates .env from .env.example if missing, idempotently fills
in every random secret (same logic generate-secrets.sh had, now removed),
resolves SEARXNG_LAN_IP from search.home via the host's own DNS, and mints
OPENWEBUI_LITELLM_KEY / MEMORY_RETRIEVAL_EMBEDDING_KEY through LiteLLM's own
/key/generate API once litellm is up — no more manual Admin UI step for the
stack's own two workload keys. Docs updated to point at update.sh as the
one command; docs/proxy-key-onboarding.md keeps the manual/API steps as the
fallback and for onboarding other workloads.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per docs/research/langchain-pgvector-vs-litellm-pgvector.md (issue #25):
the vendored litellm-pgvector connector (793 lines, Prisma migrations, a
fragile git-context build) is replaced by a ~90-line FastAPI service
(services/memory-retrieval/) wrapping langchain_postgres.PGVector directly
against pgvector-db. Same gateway boundary — it still calls litellm for
embeddings, nothing talks to Postgres or the model directly except this
service.
- New services/memory-retrieval/ (main.py, Dockerfile, requirements.txt):
POST /ingest, POST /query, GET /health.
- docker-compose.yml: litellm-pgvector service replaced by memory-retrieval;
pgvector-db and embedding-server untouched.
- litellm-config.yaml: vector_store_registry block removed (no
langchain_postgres provider exists to register against; callers query
memory-retrieval directly instead of an in-band file_search tool call —
that mechanism was never confirmed working per issue #24 anyway).
- scripts/ingest-memory.sh rewritten for the new /ingest endpoint (same
per-line chunking, no dedup).
- .env vars renamed: LITELLM_PGVECTOR_API_KEY/LITELLM_PGVECTOR_EMBEDDING_KEY
-> MEMORY_RETRIEVAL_API_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY.
- vendor/litellm-pgvector/ removed entirely.
- docs/memory-knowledgebase.md updated for the new setup/query flow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verdict: switch. See docs/research/langchain-pgvector-vs-litellm-pgvector.md
for full reasoning, LOC comparison, and a concrete replacement service spec.
Research/doc only — no compose/config/vendor changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The server's Docker/BuildKit couldn't do a git-context build of a public
github.com repo (fails with "could not read Username ... terminal prompts
disabled" — an auth-shaped error for what should be an anonymous clone).
Rather than debug that, vendor litellm-pgvector's small source tree
directly (vendor/litellm-pgvector/, see VENDORED.md for provenance/update
steps) and build from the local path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- search_tools block in litellm-config.yaml (SearXNG as a first-class
search_provider, standalone /v1/search endpoint, not a model tool) plus
extra_hosts on the litellm service so it can resolve search.home.
- New embedding-server (nomic-embed-text-v1.5 on a second llama.cpp
instance), pgvector-db, and litellm-pgvector services — LiteLLM's native
knowledgebase feature has no Qdrant backend, so this is the only
self-hosted path (docs/research/litellm-knowledgebase.md).
- vector_store_registry + local-embedding model entry in
litellm-config.yaml, wiring it together.
- scripts/ingest-memory.sh to load data/memory.md and
data/claude-legacy-memory.md into the knowledgebase.
- docs/memory-knowledgebase.md documenting the whole setup; data/
gitignored (personal memory content, not meant to be committed).
- New .env vars (SEARXNG_LAN_IP, PGVECTOR_DB_PASSWORD,
LITELLM_PGVECTOR_API_KEY, LITELLM_PGVECTOR_EMBEDDING_KEY,
EMBEDDING_MODEL_FILE) and generate-secrets.sh support for the
auto-generatable ones.
Resolves#22 and #23 (wayfinder map #21). Not yet verified on real
hardware — see #24.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers whether SearXNG is natively supported by LiteLLM's /v1/search
feature, whether it's a model-tool or automatic retrieval, what
docker-compose.yml networking change is needed for the litellm container
to reach the LAN's search.home host, and how it interacts with this
project's known-flaky Qwen3.8-27B tool-calling.
Research only — litellm-config.yaml and docker-compose.yml are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers whether the $20/mo Claude Pro subscription can be used as a
LiteLLM backend instead of a pay-per-token Anthropic API key. Finding:
no — Pro/Max OAuth credentials are contractually restricted to Claude
Code / native Anthropic apps (Consumer ToS + Usage Policy, actively
enforced server-side since March 2026). Documents the supported path:
a separate Anthropic Console API key as a model_list entry, same shape
as the existing llama.cpp entry. Same pattern applies to other
providers (ChatGPT Plus, Gemini subscriptions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified against the live GitHub repo/docs (not secondhand): real,
active, functioning AI gateway, ~5 months old, effectively
single-maintainer, BSL 1.1. Fails two of the requirements from
docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages
inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic
can't be routed through it) and no request-priority/queuing feature.
Recommendation: stay on LiteLLM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open WebUI now points at litellm instead of llama-server directly, using a
provisioned virtual key. llama-server's host port is dropped (internal-only
on the ai-stack network) since the proxy is the only intended entry point
now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with
per-CLI virtual keys instead of the old shared dummy key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #8. The stack is reached over LAN/WAN hostnames,
not localhost:
- Open WebUI: ai.home (LAN) and ai.haylan.ch (external, via DMZ) routed
through the existing Nginx Proxy Manager instance to the published
WEBUI_PORT — no new reverse-proxy service needed in this repo.
- llama.cpp's raw API stays LAN-only and unregistered in NPM (no auth of
its own); coding CLIs reach it via the box's LAN IP or ai.home directly.
New docs/network-access.md documents the NPM proxy-host setup and the
LAN-only decision. coding-cli-setup.md and README updated to stop
referencing localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #6. Covers the two server endpoints (Anthropic
shim vs OpenAI-compatible), exact config for all three CLIs, and surfaces
the shared Qwen3.8-27B tool-calling risk plus OpenCode's own documented
local-backend brittleness up front rather than burying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers issue #7. Confirms sst/opencode moved to anomalyco/opencode,
documents install methods, opencode.json provider config for a local
OpenAI-compatible server, and known tool-calling compatibility issues
against self-hosted backends (llama.cpp --jinja requirement, empty
tool_calls handling) plus Qwen3.8-27B-specific caveats.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships
UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB
RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude-Bot