One script to bring the running stack in sync after any change: git pull,
validate compose config, rebuild the litellm-pgvector local build, re-pull
images, recreate what changed. Documented in README's quickstart, and
CLAUDE.md now tells agents to run it after touching compose/config/scripts
rather than just describing a change as done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- search_tools block in litellm-config.yaml (SearXNG as a first-class
search_provider, standalone /v1/search endpoint, not a model tool) plus
extra_hosts on the litellm service so it can resolve search.home.
- New embedding-server (nomic-embed-text-v1.5 on a second llama.cpp
instance), pgvector-db, and litellm-pgvector services — LiteLLM's native
knowledgebase feature has no Qdrant backend, so this is the only
self-hosted path (docs/research/litellm-knowledgebase.md).
- vector_store_registry + local-embedding model entry in
litellm-config.yaml, wiring it together.
- scripts/ingest-memory.sh to load data/memory.md and
data/claude-legacy-memory.md into the knowledgebase.
- docs/memory-knowledgebase.md documenting the whole setup; data/
gitignored (personal memory content, not meant to be committed).
- New .env vars (SEARXNG_LAN_IP, PGVECTOR_DB_PASSWORD,
LITELLM_PGVECTOR_API_KEY, LITELLM_PGVECTOR_EMBEDDING_KEY,
EMBEDDING_MODEL_FILE) and generate-secrets.sh support for the
auto-generatable ones.
Resolves#22 and #23 (wayfinder map #21). Not yet verified on real
hardware — see #24.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers whether SearXNG is natively supported by LiteLLM's /v1/search
feature, whether it's a model-tool or automatic retrieval, what
docker-compose.yml networking change is needed for the litellm container
to reach the LAN's search.home host, and how it interacts with this
project's known-flaky Qwen3.8-27B tool-calling.
Research only — litellm-config.yaml and docker-compose.yml are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Qwen3 spends output tokens on reasoning_content before writing content.
Open WebUI's default chat request doesn't set max_tokens, so it fell
through to llama.cpp's low default and the model ran out mid-thought,
returning finish_reason=length with empty content — no reply shown in
Open WebUI. Confirmed via a manual /v1/chat/completions call: works with
max_tokens=2000, fails without it.
litellm_params.max_tokens is a default, not a cap — any caller (or Open
WebUI's per-model Advanced Params) that sets its own max_tokens still
overrides it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Without these, LiteLLM's /ui falls back to username 'admin' and the
master key as the password, forcing the master key to be typed into
a browser. generate-secrets.sh now fills UI_PASSWORD like the other
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
The curlimages/curl image drops to non-root curl_user (uid 100) by
default, but the models named volume is created root-owned by Docker,
so writes into it failed with 'Permission denied'.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Answers whether the $20/mo Claude Pro subscription can be used as a
LiteLLM backend instead of a pay-per-token Anthropic API key. Finding:
no — Pro/Max OAuth credentials are contractually restricted to Claude
Code / native Anthropic apps (Consumer ToS + Usage Policy, actively
enforced server-side since March 2026). Documents the supported path:
a separate Anthropic Console API key as a model_list entry, same shape
as the existing llama.cpp entry. Same pattern applies to other
providers (ChatGPT Plus, Gemini subscriptions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified against the live GitHub repo/docs (not secondhand): real,
active, functioning AI gateway, ~5 months old, effectively
single-maintainer, BSL 1.1. Fails two of the requirements from
docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages
inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic
can't be routed through it) and no request-priority/queuing feature.
Recommendation: stay on LiteLLM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open WebUI now points at litellm instead of llama-server directly, using a
provisioned virtual key. llama-server's host port is dropped (internal-only
on the ai-stack network) since the proxy is the only intended entry point
now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with
per-CLI virtual keys instead of the old shared dummy key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with
custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a
priority-scheduling stub (per #16, needs real-hardware smoke test), and
required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real
hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp
directly, migration is #15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #8. The stack is reached over LAN/WAN hostnames,
not localhost:
- Open WebUI: ai.home (LAN) and ai.haylan.ch (external, via DMZ) routed
through the existing Nginx Proxy Manager instance to the published
WEBUI_PORT — no new reverse-proxy service needed in this repo.
- llama.cpp's raw API stays LAN-only and unregistered in NPM (no auth of
its own); coding CLIs reach it via the box's LAN IP or ai.home directly.
New docs/network-access.md documents the NPM proxy-host setup and the
LAN-only decision. coding-cli-setup.md and README updated to stop
referencing localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #6. Covers the two server endpoints (Anthropic
shim vs OpenAI-compatible), exact config for all three CLIs, and surfaces
the shared Qwen3.8-27B tool-calling risk plus OpenCode's own documented
local-backend brittleness up front rather than burying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers issue #7. Confirms sst/opencode moved to anomalyco/opencode,
documents install methods, opencode.json provider config for a local
OpenAI-compatible server, and known tool-calling compatibility issues
against self-hosted backends (llama.cpp --jinja requirement, empty
tool_calls handling) plus Qwen3.8-27B-specific caveats.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #4. Wires up the locked decisions from the map:
- llama.cpp (ghcr.io/ggml-org/llama.cpp:server-rocm, gfx1201) serving
Qwen3.8-27B-UD-Q4_K_XL.gguf, port published for direct Claude Code CLI /
Kimi CLI access alongside Open WebUI.
- Open WebUI with WEBUI_AUTH on, RAG+Memory wired to a standalone Qdrant
service.
- Lazytainer labels on llama-server for a 15 min idle-stop.
- Named Docker volumes only (models, qdrant-data, openwebui-data) — no host
bind-mounts.
- One-off 'downloader' compose profile instead of a host-side script with
its own dependencies, wrapped by scripts/download-model.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships
UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB
RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude-Bot