haylanandClaude-Bot
e7983f0710
This reverts commitabeadc49c8. Restores vendor/litellm-pgvector/ and the vector_store_registry wiring (in-band file_search tool-call support) at the user's request, after re-confirming against docs.litellm.ai/docs/completion/knowledgebase and litellm-pgvector's own README that pg_vector is still not an in-process vector_store_registry backend -- it requires this same standalone connector service either way, so there is no simpler 'native' path that was missed. Trading back in: 793 lines of vendored code, the untested Prisma migration, and the git-context build risk noted in VENDORED.md (all flagged as unverified against real hardware in issue #24), in exchange for the file_search in-band tool call memory-retrieval did not support. Conflicts resolved on top of later commits (Redis, update.sh key-minting fold-in): - .env.example / docs/memory-knowledgebase.md: kept the auto-mint-via- update.sh language, renamed MEMORY_RETRIEVAL_* back to LITELLM_PGVECTOR_*. - scripts/generate-secrets.sh: left deleted -- its job was folded into update.sh in24d749b, unrelated to this revert. - scripts/update.sh: renamed the MEMORY_RETRIEVAL_* secret/mint calls to LITELLM_PGVECTOR_* to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018WHfjWrSEcGhCoeu6dQfDa
LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
See the wayfinder map (issue #1) for the full architecture rationale and open questions.
Quickstart
./scripts/download-model.sh
./scripts/update.sh
update.sh creates .env from .env.example if missing, fills in every secret and per-workload virtual key it can generate itself (random secrets via openssl, OPENWEBUI_LITELLM_KEY/MEMORY_RETRIEVAL_EMBEDDING_KEY minted through LiteLLM's own /key/generate API, SEARXNG_LAN_IP resolved from search.home on this host), then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank and only recreates what changed. See docs/proxy-key-onboarding.md if a key mint fails and needs doing by hand.
- Open WebUI:
http://<this-machine>:3000locally, orai.home/ai.haylan.chonce routed through Nginx Proxy Manager — seedocs/network-access.md. First signup becomes the admin account (WEBUI_AUTHis on). - llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see docs/coding-cli-setup.md.
Known risk: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see docs/research/qwen3.8-27b-tool-calling.md.
AI proxy (LiteLLM)
An AI gateway/proxy fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). ./scripts/update.sh handles LITELLM_MASTER_KEY/LITELLM_SALT_KEY and every other secret (see .env.example).
- Proxy API:
http://<this-machine>:4000/v1locally, orproxy.ai.home/proxy.ai.haylan.chonce routed through NPM — seedocs/network-access.md. - Admin UI (
/ui, key/budget management): LAN-only — seedocs/network-access.md. - Issuing a key for a new workload:
docs/proxy-key-onboarding.md. - Request priority across workloads:
docs/proxy-request-priority.md.
Open WebUI and the coding CLIs (see docs/coding-cli-setup.md) route through the proxy now — llama-server has no published host port anymore. Not yet verified: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see docs/proxy-request-priority.md) — see issue #17.
Web search, knowledgebase, and memory
The gateway also fronts SearXNG-backed web search and a pgvector-backed knowledgebase (loaded with data/memory.md / data/claude-legacy-memory.md), wired at the LiteLLM layer so every client gets them, not just Open WebUI — see docs/memory-knowledgebase.md. Not yet verified on real hardware — see issue #24.