haylanandClaude-Bot
f508f5670a
- search_tools block in litellm-config.yaml (SearXNG as a first-class search_provider, standalone /v1/search endpoint, not a model tool) plus extra_hosts on the litellm service so it can resolve search.home. - New embedding-server (nomic-embed-text-v1.5 on a second llama.cpp instance), pgvector-db, and litellm-pgvector services — LiteLLM's native knowledgebase feature has no Qdrant backend, so this is the only self-hosted path (docs/research/litellm-knowledgebase.md). - vector_store_registry + local-embedding model entry in litellm-config.yaml, wiring it together. - scripts/ingest-memory.sh to load data/memory.md and data/claude-legacy-memory.md into the knowledgebase. - docs/memory-knowledgebase.md documenting the whole setup; data/ gitignored (personal memory content, not meant to be committed). - New .env vars (SEARXNG_LAN_IP, PGVECTOR_DB_PASSWORD, LITELLM_PGVECTOR_API_KEY, LITELLM_PGVECTOR_EMBEDDING_KEY, EMBEDDING_MODEL_FILE) and generate-secrets.sh support for the auto-generatable ones. Resolves #22 and #23 (wayfinder map #21). Not yet verified on real hardware — see #24. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3.2 KiB
LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
See the wayfinder map (issue #1) for the full architecture rationale and open questions.
Quickstart
cp .env.example .env
# set LITELLM_MASTER_KEY / LITELLM_SALT_KEY (openssl rand -hex 32), see .env.example
./scripts/download-model.sh
docker compose up -d litellm litellm-db llama-server qdrant # bring the proxy up first
Log into LiteLLM's Admin UI (http://<this-machine>:4000/ui), create an openwebui virtual key (see docs/proxy-key-onboarding.md), set OPENWEBUI_LITELLM_KEY in .env to it, then:
docker compose up -d
- Open WebUI:
http://<this-machine>:3000locally, orai.home/ai.haylan.chonce routed through Nginx Proxy Manager — seedocs/network-access.md. First signup becomes the admin account (WEBUI_AUTHis on). - llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see docs/coding-cli-setup.md.
Known risk: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see docs/research/qwen3.8-27b-tool-calling.md.
AI proxy (LiteLLM)
An AI gateway/proxy fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). Before docker compose up -d, set LITELLM_MASTER_KEY and LITELLM_SALT_KEY in .env (see .env.example).
- Proxy API:
http://<this-machine>:4000/v1locally, orproxy.ai.home/proxy.ai.haylan.chonce routed through NPM — seedocs/network-access.md. - Admin UI (
/ui, key/budget management): LAN-only — seedocs/network-access.md. - Issuing a key for a new workload:
docs/proxy-key-onboarding.md. - Request priority across workloads:
docs/proxy-request-priority.md.
Open WebUI and the coding CLIs (see docs/coding-cli-setup.md) route through the proxy now — llama-server has no published host port anymore. Not yet verified: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see docs/proxy-request-priority.md) — see issue #17.
Web search, knowledgebase, and memory
The gateway also fronts SearXNG-backed web search and a pgvector-backed knowledgebase (loaded with data/memory.md / data/claude-legacy-memory.md), wired at the LiteLLM layer so every client gets them, not just Open WebUI — see docs/memory-knowledgebase.md. Not yet verified on real hardware — see issue #24.