feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
+26
-33
@@ -30,48 +30,41 @@ LLAMA_CTX_SIZE=131072
|
||||
|
||||
# --- Open WebUI ---
|
||||
WEBUI_PORT=8008
|
||||
# Minted automatically by ./scripts/update.sh — leave blank. Manual fallback:
|
||||
# docs/proxy-key-onboarding.md.
|
||||
OPENWEBUI_LITELLM_KEY=
|
||||
# No scripted mint yet — set OPENWEBUI_OMNIROUTE_KEY below by hand instead.
|
||||
|
||||
# --- Lazytainer ---
|
||||
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
||||
LAZYTAINER_INACTIVE_TIMEOUT=900
|
||||
|
||||
# --- Embedding model (knowledgebase, see docs/memory-knowledgebase.md) ---
|
||||
EMBEDDING_MODEL_FILE=nomic-embed-text-v1.5.Q8_0.gguf
|
||||
|
||||
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
|
||||
# Resolved automatically by ./scripts/update.sh from search.home on this
|
||||
# host — leave blank. Only set by hand if that resolution fails (e.g.
|
||||
# search.home isn't a static DHCP reservation and its IP drifted).
|
||||
SEARXNG_LAN_IP=
|
||||
|
||||
# --- LiteLLM proxy (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
|
||||
LITELLM_PORT=4000
|
||||
# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
|
||||
# API_PORT is the only port published to the host/internet (reverse-proxied
|
||||
# by NPM) — the dashboard (DASHBOARD_PORT) is never published, see
|
||||
# docker-compose.yml's omniroute service comment.
|
||||
OMNIROUTE_API_PORT=20129
|
||||
OMNIROUTE_DASHBOARD_PORT=20128
|
||||
# Random values, filled in automatically by ./scripts/update.sh — leave
|
||||
# blank. LITELLM_SALT_KEY encrypts stored data; do not change it after the
|
||||
# first run (existing encrypted data becomes unreadable if you do).
|
||||
LITELLM_MASTER_KEY=
|
||||
LITELLM_SALT_KEY=
|
||||
LITELLM_DB_PASSWORD=
|
||||
# Backs litellm's router state/rate-limits/budgets/cache invalidation
|
||||
# (the redis service). Random value, filled in automatically — leave blank.
|
||||
REDIS_PASSWORD=
|
||||
# Admin UI login (https://<proxy>/ui). Without these, LiteLLM falls back to
|
||||
# username "admin" / password = LITELLM_MASTER_KEY — set these instead so the
|
||||
# master key never has to be typed into the browser. UI_PASSWORD is filled
|
||||
# in automatically by ./scripts/update.sh if blank.
|
||||
UI_USERNAME=admin
|
||||
UI_PASSWORD=
|
||||
|
||||
# --- Knowledgebase (pgvector + litellm-pgvector, see docs/memory-knowledgebase.md) ---
|
||||
# Random value, filled in automatically by ./scripts/update.sh — leave blank.
|
||||
PGVECTOR_DB_PASSWORD=
|
||||
# Auth key litellm-pgvector requires on its own API (its SERVER_API_KEY).
|
||||
# Random value, filled in automatically by ./scripts/update.sh — leave blank.
|
||||
LITELLM_PGVECTOR_API_KEY=
|
||||
# A virtual key litellm-pgvector uses to call back into litellm for
|
||||
# embeddings. Minted automatically by ./scripts/update.sh — leave blank.
|
||||
# Manual fallback: docs/proxy-key-onboarding.md.
|
||||
LITELLM_PGVECTOR_EMBEDDING_KEY=
|
||||
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
|
||||
# change it there afterwards — this is only the first-boot value):
|
||||
OMNIROUTE_INITIAL_PASSWORD=
|
||||
# Signs dashboard session cookies:
|
||||
OMNIROUTE_JWT_SECRET=
|
||||
# Encrypts API key values at rest in omniroute's SQLite DB:
|
||||
OMNIROUTE_API_KEY_SECRET=
|
||||
# Encrypts the whole SQLite DB at rest. Do not change after first run —
|
||||
# existing encrypted data becomes unreadable if you do (same caveat as
|
||||
# LiteLLM's old LITELLM_SALT_KEY):
|
||||
OMNIROUTE_STORAGE_ENCRYPTION_KEY=
|
||||
# Per-deployment salts — random is fine, just needs to be stable:
|
||||
OMNIROUTE_MACHINE_ID_SALT=
|
||||
OMNIROUTE_CLI_SALT=
|
||||
# Per-workload virtual keys — omniroute has no scripted /key/generate
|
||||
# equivalent yet (its key-creation endpoint needs a dashboard login session,
|
||||
# not a static bearer key — see issue #37), so mint these by hand in the
|
||||
# dashboard for now. See docs/proxy-key-onboarding.md.
|
||||
OPENWEBUI_OMNIROUTE_KEY=
|
||||
|
||||
@@ -10,4 +10,4 @@ Single-context: `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/do
|
||||
|
||||
### Deploying changes
|
||||
|
||||
The running stack lives on a separate box (the R9700 server), not wherever this repo is being edited. After **any** change to `docker-compose.yml`, `litellm-config.yaml`, `.env.example`, or a script under `scripts/`, commit/push it, then run `./scripts/update.sh` on the server to apply it — don't just describe the change as done. If this session doesn't have shell access to the server, say so explicitly and tell the user to run it themselves rather than leaving it unsaid.
|
||||
The running stack lives on a separate box (the R9700 server), not wherever this repo is being edited. After **any** change to `docker-compose.yml`, `.env.example`, or a script under `scripts/`, commit/push it, then run `./scripts/update.sh` on the server to apply it — don't just describe the change as done. If this session doesn't have shell access to the server, say so explicitly and tell the user to run it themselves rather than leaving it unsaid.
|
||||
|
||||
@@ -10,7 +10,7 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/
|
||||
./scripts/update.sh
|
||||
```
|
||||
|
||||
`update.sh` creates `.env` from `.env.example` if missing, fills in every secret and per-workload virtual key it can generate itself (random secrets via `openssl`, `OPENWEBUI_LITELLM_KEY`/`LITELLM_PGVECTOR_EMBEDDING_KEY` minted through LiteLLM's own `/key/generate` API, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads both model GGUFs into the `models` volume if they're not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips models already downloaded, and only recreates what changed. See [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md) if a key mint fails and needs doing by hand.
|
||||
`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Per-workload API keys (e.g. `OPENWEBUI_OMNIROUTE_KEY`) have no scripted mint yet — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md). Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
|
||||
|
||||
- Open WebUI: `http://<this-machine>:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on).
|
||||
- llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
|
||||
@@ -19,17 +19,20 @@ Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [
|
||||
|
||||
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
|
||||
|
||||
## AI proxy (LiteLLM)
|
||||
## AI gateway (OmniRoute)
|
||||
|
||||
An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). `./scripts/update.sh` handles `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` and every other secret (see `.env.example`).
|
||||
An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload API keys and usage tracking. As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) this is [OmniRoute](https://github.com/diegosouzapw/OmniRoute), replacing the original LiteLLM setup. `./scripts/update.sh` handles most of OmniRoute's secrets (see `.env.example`); per-workload API keys still need minting by hand in the dashboard — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
|
||||
|
||||
- Proxy API: `http://<this-machine>:4000/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
|
||||
- Admin UI (`/ui`, key/budget management): LAN-only — see `docs/network-access.md`.
|
||||
- Gateway API: `http://<this-machine>:${OMNIROUTE_API_PORT:-20129}/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
|
||||
- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
|
||||
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
|
||||
- Request priority across workloads: [`docs/proxy-request-priority.md`](docs/proxy-request-priority.md).
|
||||
|
||||
Open WebUI and the coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the proxy now — llama-server has no published host port anymore. **Not yet verified**: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).
|
||||
Open WebUI and the coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting, Redis integration).
|
||||
|
||||
### Web search, knowledgebase, and memory
|
||||
**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
|
||||
|
||||
The gateway also fronts SearXNG-backed web search and a pgvector-backed knowledgebase (loaded with `data/memory.md` / `data/claude-legacy-memory.md`), wired at the LiteLLM layer so every client gets them, not just Open WebUI — see [`docs/memory-knowledgebase.md`](docs/memory-knowledgebase.md). **Not yet verified on real hardware** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24).
|
||||
### Web search
|
||||
|
||||
The gateway also fronts SearXNG-backed web search, wired at the gateway layer so every client gets it, not just Open WebUI — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
|
||||
|
||||
The gateway-level knowledgebase/memory feature (litellm-pgvector, pgvector-db, a dedicated embedding model) that used to sit alongside this was removed — this stack doesn't use it. Open WebUI's own built-in memory/RAG (backed by `qdrant`) is unrelated and unaffected.
|
||||
|
||||
+34
-179
@@ -35,36 +35,6 @@ services:
|
||||
- "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
|
||||
- "lazytainer.group.llamaserver.minPacketThreshold=2"
|
||||
|
||||
embedding-server:
|
||||
image: ghcr.io/ggml-org/llama.cpp:server-rocm
|
||||
container_name: embedding-server
|
||||
devices:
|
||||
- /dev/kfd
|
||||
- /dev/dri
|
||||
group_add:
|
||||
- video
|
||||
- render
|
||||
security_opt:
|
||||
- seccomp=unconfined
|
||||
ipc: host
|
||||
volumes:
|
||||
- models:/models
|
||||
command: >
|
||||
-m /models/${EMBEDDING_MODEL_FILE:-nomic-embed-text-v1.5.Q8_0.gguf}
|
||||
--host 0.0.0.0
|
||||
--port 8080
|
||||
--embeddings
|
||||
--pooling mean
|
||||
--n-gpu-layers 999
|
||||
--ctx-size 8192
|
||||
# A dedicated embedding model — the chat model isn't embedding-trained
|
||||
# and llama.cpp serves one model per process, so this is a second small
|
||||
# instance, not a mode switch on llama-server. See
|
||||
# docs/research/litellm-knowledgebase.md. Small enough (~150MB Q8) to
|
||||
# run alongside the chat model's ~19.6GB in the R9700's 32GB VRAM.
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
# ponytail: one-off downloader, not a standing service — run via
|
||||
# `docker compose --profile tools run --rm downloader`. Folded into
|
||||
# scripts/update.sh, which runs this every time; the `test -f` guard is
|
||||
@@ -86,23 +56,6 @@ services:
|
||||
curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
|
||||
# ponytail: same one-off pattern as `downloader`, for the embedding model —
|
||||
# run via `docker compose --profile tools run --rm downloader-embedding`,
|
||||
# also folded into scripts/update.sh.
|
||||
downloader-embedding:
|
||||
image: curlimages/curl:latest
|
||||
profiles: ["tools"]
|
||||
user: root
|
||||
volumes:
|
||||
- models:/models
|
||||
entrypoint: ["sh", "-c"]
|
||||
command:
|
||||
- >
|
||||
test -f /models/${EMBEDDING_MODEL_FILE:-nomic-embed-text-v1.5.Q8_0.gguf} &&
|
||||
echo "already downloaded, skipping" ||
|
||||
curl -L --fail --create-dirs -o /models/${EMBEDDING_MODEL_FILE:-nomic-embed-text-v1.5.Q8_0.gguf}
|
||||
https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/${EMBEDDING_MODEL_FILE:-nomic-embed-text-v1.5.Q8_0.gguf}
|
||||
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
container_name: qdrant
|
||||
@@ -122,19 +75,20 @@ services:
|
||||
depends_on:
|
||||
qdrant:
|
||||
condition: service_healthy
|
||||
litellm:
|
||||
omniroute:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- openwebui-data:/app/backend/data
|
||||
env_file: .env
|
||||
environment:
|
||||
- WEBUI_AUTH=True
|
||||
# Routed through the litellm proxy, not llama-server directly — see issue #15.
|
||||
# OPENAI_API_KEY must be a virtual key created for Open WebUI per
|
||||
# docs/proxy-key-onboarding.md (name it "openwebui"), set as
|
||||
# OPENWEBUI_LITELLM_KEY in .env.
|
||||
- OPENAI_API_BASE_URL=http://litellm:4000/v1
|
||||
- OPENAI_API_KEY=${OPENWEBUI_LITELLM_KEY}
|
||||
# Routed through the omniroute gateway, not llama-server directly — see
|
||||
# issue #15 (original rationale) and #31 (litellm -> omniroute
|
||||
# migration). OPENAI_API_KEY must be a per-workload key created for
|
||||
# Open WebUI in the omniroute dashboard (Keys -> Create, label
|
||||
# "openwebui") — no scripted mint yet, see docs/proxy-key-onboarding.md.
|
||||
- OPENAI_API_BASE_URL=http://omniroute:${OMNIROUTE_API_PORT:-20129}/v1
|
||||
- OPENAI_API_KEY=${OPENWEBUI_OMNIROUTE_KEY}
|
||||
- VECTOR_DB=qdrant
|
||||
- QDRANT_URI=http://qdrant:6333
|
||||
ports:
|
||||
@@ -142,148 +96,50 @@ services:
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
litellm:
|
||||
image: ghcr.io/berriai/litellm:main-stable
|
||||
container_name: litellm
|
||||
# Replaces litellm — see issue #31 (wayfinder map) for the full migration
|
||||
# rationale/findings. No static config.yaml equivalent: provider routing
|
||||
# (llama-server, searxng-search) is registered once through the dashboard
|
||||
# or POST /api/providers after first boot, not checked into this repo —
|
||||
# see docs/proxy-key-onboarding.md.
|
||||
omniroute:
|
||||
image: diegosouzapw/omniroute:latest
|
||||
container_name: omniroute
|
||||
depends_on:
|
||||
litellm-db:
|
||||
condition: service_healthy
|
||||
llama-server:
|
||||
condition: service_started
|
||||
redis:
|
||||
condition: service_healthy
|
||||
volumes:
|
||||
- ./litellm-config.yaml:/app/config.yaml:ro
|
||||
# LITELLM_MASTER_KEY / LITELLM_SALT_KEY come straight from .env via env_file
|
||||
# (names match what litellm reads). LITELLM_SALT_KEY must not change after
|
||||
# first run — see .env.example.
|
||||
- omniroute-data:/app/data
|
||||
env_file: .env
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://litellm:${LITELLM_DB_PASSWORD}@litellm-db:5432/litellm
|
||||
# Setting these is all LiteLLM needs to use Redis for router state,
|
||||
# rate limits/budgets, and cache invalidation — no extra config.yaml
|
||||
# block required. See https://docs.litellm.ai/docs/proxy/caching.
|
||||
- REDIS_HOST=redis
|
||||
- REDIS_PORT=6379
|
||||
- REDIS_PASSWORD=${REDIS_PASSWORD}
|
||||
# The litellm container only joins the ai-stack bridge network, which has
|
||||
# no visibility into the LAN's mDNS/local-DNS names — search.home won't
|
||||
# resolve without this. Set SEARXNG_LAN_IP in .env to its stable LAN IP
|
||||
# (static DHCP reservation recommended). See docs/research/litellm-searxng-search.md.
|
||||
# Split-port mode: dashboard and API are fully separate ports (unlike
|
||||
# LiteLLM's single :4000 for both /v1 and /ui) — only API_PORT is
|
||||
# published below, so the dashboard has no network route in from
|
||||
# outside this container at all. No NPM path-deny rule needed.
|
||||
- API_HOST=0.0.0.0
|
||||
- API_PORT=${OMNIROUTE_API_PORT:-20129}
|
||||
- DASHBOARD_PORT=${OMNIROUTE_DASHBOARD_PORT:-20128}
|
||||
# Required to register llama-server/searxng-search as providers —
|
||||
# their base URLs are LAN/container-internal addresses, blocked by
|
||||
# default (SSRF guard against public-provider spoofing).
|
||||
- OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true
|
||||
- OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true
|
||||
# Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
|
||||
# network can't resolve search.home on its own.
|
||||
extra_hosts:
|
||||
- "search.home:${SEARXNG_LAN_IP}"
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
ports:
|
||||
# published for LAN access (proxy.ai.home) and, via NPM, proxy.ai.haylan.ch —
|
||||
# NPM must deny the /ui path on the external host. See docs/network-access.md.
|
||||
- "${LITELLM_PORT:-4000}:4000"
|
||||
- "${OMNIROUTE_API_PORT:-20129}:${OMNIROUTE_API_PORT:-20129}"
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"
|
||||
- python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:${OMNIROUTE_API_PORT:-20129}/healthz')"
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
litellm-db:
|
||||
image: postgres:16-alpine
|
||||
container_name: litellm-db
|
||||
env_file: .env
|
||||
environment:
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=${LITELLM_DB_PASSWORD}
|
||||
- POSTGRES_DB=litellm
|
||||
volumes:
|
||||
- litellm-db-data:/var/lib/postgresql/data
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -d litellm -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
# Backs litellm's router state, rate limits/budgets, and cache
|
||||
# invalidation (see the litellm service's REDIS_* env vars above).
|
||||
# ponytail: no persistence volume — everything litellm stores here is
|
||||
# cache/coordination state it's fine to lose on restart, not source data.
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: redis
|
||||
command: ["redis-server", "--requirepass", "${REDIS_PASSWORD}"]
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "redis-cli -a ${REDIS_PASSWORD} ping | grep -q PONG"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
# Separate Postgres instance (with the pgvector extension) for the
|
||||
# knowledgebase — NOT the same database as litellm-db, which is plain
|
||||
# postgres:16-alpine and has no vector extension installed. See
|
||||
# docs/research/litellm-knowledgebase.md.
|
||||
pgvector-db:
|
||||
image: pgvector/pgvector:pg16
|
||||
container_name: pgvector-db
|
||||
env_file: .env
|
||||
environment:
|
||||
- POSTGRES_USER=litellm_pgvector
|
||||
- POSTGRES_PASSWORD=${PGVECTOR_DB_PASSWORD}
|
||||
- POSTGRES_DB=litellm_pgvector
|
||||
volumes:
|
||||
- pgvector-db-data:/var/lib/postgresql/data
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -d litellm_pgvector -U litellm_pgvector"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 10
|
||||
|
||||
# LiteLLM's native knowledgebase/vector-store feature has no Qdrant backend
|
||||
# (the qdrant service above only serves Open WebUI's own RAG/Memory) — this
|
||||
# companion service (github.com/BerriAI/litellm-pgvector) is the only
|
||||
# self-hosted path. No published image exists yet, so this builds from a
|
||||
# vendored copy in vendor/litellm-pgvector/ (see that dir's README) rather
|
||||
# than a remote git build context — the server's Docker/BuildKit couldn't
|
||||
# do an authenticated-looking clone of a public github.com repo (fails
|
||||
# with "could not read Username ... terminal prompts disabled"), and
|
||||
# vendoring sidesteps needing that debugged. See
|
||||
# docs/research/litellm-knowledgebase.md.
|
||||
# ponytail: unverified against real hardware — Prisma migration behavior on
|
||||
# first boot and the exact vector_store_registry field names for the
|
||||
# pg_vector provider need a live smoke test. See issue #24.
|
||||
litellm-pgvector:
|
||||
build:
|
||||
context: ./vendor/litellm-pgvector
|
||||
container_name: litellm-pgvector
|
||||
depends_on:
|
||||
pgvector-db:
|
||||
condition: service_healthy
|
||||
litellm:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- DATABASE_URL=postgresql://litellm_pgvector:${PGVECTOR_DB_PASSWORD}@pgvector-db:5432/litellm_pgvector
|
||||
- SERVER_API_KEY=${LITELLM_PGVECTOR_API_KEY}
|
||||
# Calls back into litellm for embeddings, same pattern as any other
|
||||
# workload — see docs/proxy-key-onboarding.md for issuing this key.
|
||||
# openai/ prefix required — litellm.aembedding can't infer a provider
|
||||
# from a bare model name plus a custom api_base (raises "LLM Provider
|
||||
# NOT provided"), same reasoning as the openai/ prefix on
|
||||
# qwen3.8-27b-local and local-embedding in litellm-config.yaml.
|
||||
- EMBEDDING__MODEL=openai/local-embedding
|
||||
- EMBEDDING__BASE_URL=http://litellm:4000
|
||||
- EMBEDDING__API_KEY=${LITELLM_PGVECTOR_EMBEDDING_KEY}
|
||||
- EMBEDDING__DIMENSIONS=768
|
||||
expose:
|
||||
- "8000"
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
lazytainer:
|
||||
image: ghcr.io/vmorganp/lazytainer:master
|
||||
container_name: lazytainer
|
||||
@@ -311,5 +167,4 @@ volumes:
|
||||
models:
|
||||
qdrant-data:
|
||||
openwebui-data:
|
||||
litellm-db-data:
|
||||
pgvector-db-data:
|
||||
omniroute-data:
|
||||
|
||||
+12
-12
@@ -1,24 +1,24 @@
|
||||
# Pointing a coding-agent CLI at this stack
|
||||
|
||||
This stack routes through the [AI proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (LiteLLM) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The proxy exposes:
|
||||
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
|
||||
|
||||
- **OpenAI-compatible**: `http://<ai-box>:4000/v1` (or `${LITELLM_PORT}` if you changed it in `.env`)
|
||||
- **Anthropic Messages API** (LiteLLM's own unified `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:4000`
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}`
|
||||
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the proxy as `qwen3.8-27b-local` — behind whichever wire format the client speaks.
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
|
||||
|
||||
`<ai-box>` is this machine's LAN address, or `proxy.ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
|
||||
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (LiteLLM's Admin UI, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
|
||||
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
|
||||
|
||||
## Claude Code CLI
|
||||
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the proxy's unified endpoint, not llama.cpp directly:
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:4000
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_API_PORT:-20129}
|
||||
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
|
||||
claude
|
||||
```
|
||||
@@ -32,7 +32,7 @@ Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in
|
||||
```toml
|
||||
[providers.openai]
|
||||
type = "openai"
|
||||
base_url = "http://<ai-box>:4000/v1"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1"
|
||||
api_key = "<kimi-cli virtual key>"
|
||||
```
|
||||
|
||||
@@ -57,7 +57,7 @@ curl -fsSL https://opencode.ai/install | bash
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "AI proxy (local)",
|
||||
"options": {
|
||||
"baseURL": "http://<ai-box>:4000/v1",
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1",
|
||||
"apiKey": "<opencode-cli virtual key>"
|
||||
},
|
||||
"models": {
|
||||
@@ -84,8 +84,8 @@ Select the model with `aiproxy/qwen3.8-27b-local`.
|
||||
|
||||
| CLI | Wire format | Endpoint | Config |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:4000` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `opencode.json` provider block |
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `opencode.json` provider block |
|
||||
|
||||
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
# Knowledgebase, memory, and web search
|
||||
|
||||
Three gateway-level capabilities added on top of the [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9), so every client behind LiteLLM gets them — not just Open WebUI. See [issue #21](https://git.arthurerlich.de/haylan/LLM-Server/issues/21) for the rationale.
|
||||
|
||||
**Verified against a live deploy** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24), closed after smoke-testing found and fixed several bugs: a missing `api_key` in `vector_store_registry` (was silently falling through to the real `api.openai.com`), `litellm-pgvector`'s Prisma schema never actually being pushed to `pgvector-db` (now handled by `./scripts/update.sh`), a 1536- vs 768-dim vector column mismatch, and its create endpoint ignoring any caller-supplied store id (both fixed locally — see `vendor/litellm-pgvector/VENDORED.md`). `scripts/ingest-memory.sh` was also silently broken (posted chunks with no embedding attached) and has been fixed to embed via LiteLLM before inserting.
|
||||
|
||||
## Web search (SearXNG)
|
||||
|
||||
`litellm-config.yaml`'s `search_tools` block wires the LAN's SearXNG instance in as a **standalone REST endpoint**, not a model-callable tool — call it directly:
|
||||
|
||||
```bash
|
||||
curl http://<proxy>:4000/v1/search/searxng-search \
|
||||
-H "Authorization: Bearer <a virtual key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "...", "max_results": 5}'
|
||||
```
|
||||
|
||||
Because this doesn't ask the model to emit a tool call, it sidesteps Qwen3.8-27B's known-flaky tool-calling (`docs/research/qwen3.8-27b-tool-calling.md`) entirely. Open WebUI's own web-search setting can point at this endpoint the same way.
|
||||
|
||||
Requires `SEARXNG_LAN_IP` set in `.env` so the `litellm` container can resolve `search.home` via `extra_hosts` — `./scripts/update.sh` resolves and fills this in automatically from the host's own DNS if it's blank (use a static DHCP reservation for `search.home` so it doesn't drift). Full research: `docs/research/litellm-searxng-search.md`.
|
||||
|
||||
## Knowledgebase (vector store / RAG)
|
||||
|
||||
LiteLLM's native knowledgebase feature has **no Qdrant backend** — the `qdrant` service in this stack only serves Open WebUI's own separate RAG/Memory feature and is unrelated to this. The only self-hosted path is [litellm-pgvector](https://github.com/BerriAI/litellm-pgvector), a companion service backed by its own Postgres+pgvector database (`pgvector-db`), which this stack now runs alongside `litellm`. Full research: `docs/research/litellm-knowledgebase.md`.
|
||||
|
||||
New pieces:
|
||||
|
||||
- **`embedding-server`** — a second llama.cpp instance (small footprint, `nomic-embed-text-v1.5`) serving `/v1/embeddings`. The chat model isn't embedding-trained and llama.cpp serves one model per process, so this can't just be a flag on `llama-server`.
|
||||
- **`pgvector-db`** — Postgres with the pgvector extension, separate from `litellm-db`.
|
||||
- **`litellm-pgvector`** — the connector service; no published image exists, so it's built from a vendored copy of the upstream repo at `vendor/litellm-pgvector/` (see that dir's `VENDORED.md`) — a remote git build context failed on the server's Docker/BuildKit setup.
|
||||
- `litellm-config.yaml`'s `local-embedding` model entry and `vector_store_registry` block, tying it together.
|
||||
|
||||
### First-time setup
|
||||
|
||||
`./scripts/update.sh` fetches the embedding model automatically (skips it if already downloaded). To do it by hand instead:
|
||||
|
||||
```bash
|
||||
docker compose --profile tools run --rm downloader-embedding # fetch the embedding model
|
||||
docker compose up -d embedding-server pgvector-db litellm-pgvector
|
||||
```
|
||||
|
||||
`./scripts/update.sh` mints `LITELLM_PGVECTOR_EMBEDDING_KEY` automatically (a `litellm-pgvector` virtual key via LiteLLM's own API) if it's blank — it calls back into `litellm` for embeddings, same as any other workload. See `docs/proxy-key-onboarding.md` if a mint fails and it needs doing by hand.
|
||||
|
||||
### Loading memory into it
|
||||
|
||||
`data/memory.md` and `data/claude-legacy-memory.md` — Claude-memory-style fact files — get loaded via:
|
||||
|
||||
```bash
|
||||
./scripts/ingest-memory.sh
|
||||
```
|
||||
|
||||
One chunk per fact/paragraph line, tagged with `source`/`section` metadata. Re-run after editing either file (see the script's header comment for the no-dedup caveat).
|
||||
|
||||
### Querying it
|
||||
|
||||
Via the OpenAI Assistants-style `file_search` tool on a chat completion:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "qwen3.8-27b-local",
|
||||
"messages": [...],
|
||||
"tools": [{"type": "file_search", "vector_store_ids": ["memory-and-notes"]}]
|
||||
}
|
||||
```
|
||||
|
||||
or directly: `POST /v1/vector_stores/memory-and-notes/search` with `{"query": "..."}`.
|
||||
@@ -15,13 +15,11 @@ The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and
|
||||
|
||||
If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture (the raw API would need its own auth in front of it).
|
||||
|
||||
## The AI proxy (LiteLLM) — `proxy.ai.home` / `proxy.ai.haylan.ch`
|
||||
## The AI gateway (OmniRoute) — `proxy.ai.home` / `proxy.ai.haylan.ch`
|
||||
|
||||
Once the gateway from [issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) is deployed, it gets its own hostnames, same NPM pattern as Open WebUI above:
|
||||
As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute — same NPM pattern as Open WebUI above, but a cleaner split than LiteLLM's ever was:
|
||||
|
||||
- **`proxy.ai.home`** — internal only, fronts the full LiteLLM port (API + Admin UI).
|
||||
- **`proxy.ai.haylan.ch`** — external, via the DMZ/NPM. Fronts only the inference API paths.
|
||||
- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port.
|
||||
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
|
||||
|
||||
**Every proxy call already requires a valid virtual key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.
|
||||
|
||||
**LiteLLM's Admin UI (`/ui`) stays LAN-only**, same reasoning as llama.cpp's raw API: it manages every workload's keys and budgets, so it doesn't belong on the public internet. LiteLLM serves `/ui` on the same port as its API by default, so `proxy.ai.haylan.ch`'s NPM Proxy Host needs an explicit rule denying the `/ui` path (a "Deny" custom location, same UI as the "Advanced" tab used for other NPM hosts) — `proxy.ai.home` has no such restriction and reaches both the API and the Admin UI.
|
||||
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.
|
||||
|
||||
@@ -1,25 +1,17 @@
|
||||
# Onboarding a workload onto the AI proxy
|
||||
# Onboarding a workload onto the AI gateway
|
||||
|
||||
How to issue a new per-workload API key against the LiteLLM proxy (see [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
|
||||
How to issue a new per-workload API key against the OmniRoute gateway (see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) — the LiteLLM → OmniRoute migration; original gateway rationale in [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
|
||||
|
||||
`OPENWEBUI_LITELLM_KEY` and `LITELLM_PGVECTOR_EMBEDDING_KEY` — the two keys this stack's own services need — are minted automatically by `./scripts/update.sh` via the same API `curl` shows below; the steps here are for any other workload, or for those two if the automatic mint ever fails.
|
||||
`OPENWEBUI_OMNIROUTE_KEY` — the one key this stack's own services need — has no scripted mint yet: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key — including that one — by hand for now, via the dashboard steps below.
|
||||
|
||||
## Create the key
|
||||
|
||||
1. Log into LiteLLM's Admin UI (`/ui` on the proxy's deployed URL).
|
||||
2. Create a new virtual key ("Keys" → "Create Key").
|
||||
3. Name it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `openwebui`. This name is the ledger: LiteLLM's dashboard lists keys by name, so there's no separate tracking doc to keep in sync — name it clearly and the Usage tab tells you the rest (spend, last used, etc.).
|
||||
4. Leave budget and rate limits unset (unlimited) by default. This is a shadow-cost estimate for fun, not real accounting or resource protection — see `docs/research/proxy-shadow-pricing.md`. Only set a budget if a specific workload turns out to need a tripwire.
|
||||
1. Log into the omniroute dashboard (`http://<host>:${OMNIROUTE_DASHBOARD_PORT:-20128}` — LAN/host-only, never published publicly, see `docker-compose.yml`'s `omniroute` service).
|
||||
2. "Keys" → "Create API key".
|
||||
3. Label it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `openwebui`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync.
|
||||
4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.
|
||||
|
||||
Or the same thing over the API (what `update.sh` does):
|
||||
|
||||
```bash
|
||||
curl -sf -X POST "http://<proxy>:4000/key/generate" \
|
||||
-H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key_alias": "<workload>-<purpose>"}'
|
||||
# -> {"key": "sk-...", ...}
|
||||
```
|
||||
Once `POST /api/keys`'s session-auth flow is worked out (issue #37), the equivalent `curl` here can replace this manual step, the way `update.sh` used to automate LiteLLM's `/key/generate`.
|
||||
|
||||
## Hand it to the workload
|
||||
|
||||
@@ -27,7 +19,7 @@ Drop the key into that workload's own `.env` (or equivalent config) — never in
|
||||
|
||||
## Retiring or rotating a key
|
||||
|
||||
No scheduled rotation. Revoke the key by hand in the Admin UI ("Keys" → delete) only when:
|
||||
No scheduled rotation. Revoke the key by hand in the dashboard ("Keys" → delete) only when:
|
||||
- the workload is retired, or
|
||||
- the key is suspected leaked/compromised.
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Request priority on the AI proxy
|
||||
|
||||
One local model instance (llama.cpp on the single R9700) serves every workload through the LiteLLM proxy ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
|
||||
**Stale as of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (LiteLLM → OmniRoute migration)** — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
|
||||
|
||||
One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
|
||||
|
||||
## Tiers
|
||||
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
model_list:
|
||||
- model_name: qwen3.8-27b-local
|
||||
litellm_params:
|
||||
# Static name — llama.cpp serves whatever model it loaded regardless of
|
||||
# what's requested here; this string isn't shell-expanded (this file
|
||||
# isn't docker-compose.yml, .env vars don't reach it).
|
||||
model: openai/qwen3.8-27b-local
|
||||
api_base: http://llama-server:8080/v1
|
||||
api_key: local
|
||||
# Qwen3 is a reasoning model — it spends output tokens on
|
||||
# reasoning_content before ever writing content. Callers that don't
|
||||
# set their own max_tokens (Open WebUI's default request didn't) hit
|
||||
# llama.cpp's low default, so the model runs out mid-thought and
|
||||
# content comes back empty. This is a floor, not a cap — any caller
|
||||
# that passes its own max_tokens still overrides it.
|
||||
# Raised from 4096: confirmed in the wild (llama-server logs) that
|
||||
# 4096 wasn't enough — reasoning_content alone ate the whole budget on
|
||||
# a real request (n_gen = 4096 exactly, no answer ever written). At
|
||||
# ~26.7 t/s and a 65536-token context window, 16384 is a ~10-minute
|
||||
# worst case, not the full ~20-minute worst case 32768 would be.
|
||||
max_tokens: 16384
|
||||
model_info:
|
||||
# Shadow cloud-cost estimate — priced against Claude Sonnet 5's published
|
||||
# rate, not real spend (this proxy only ever routes to the local model).
|
||||
# Source: https://platform.claude.com/docs/en/about-claude/pricing,
|
||||
# checked 2026-08-25. Update these two numbers if that page changes.
|
||||
input_cost_per_token: 0.000002 # $2 / MTok
|
||||
output_cost_per_token: 0.00001 # $10 / MTok
|
||||
|
||||
- model_name: local-embedding
|
||||
litellm_params:
|
||||
# Served by the dedicated embedding-server (nomic-embed-text-v1.5), not
|
||||
# the chat model — see docker-compose.yml. Called by litellm-pgvector
|
||||
# to embed knowledgebase content, and available directly at
|
||||
# /v1/embeddings for anything else that wants it.
|
||||
model: openai/local-embedding
|
||||
api_base: http://embedding-server:8080/v1
|
||||
api_key: local
|
||||
model_info:
|
||||
mode: embedding
|
||||
|
||||
# SearXNG-backed web search — a standalone REST endpoint (/v1/search/searxng-search),
|
||||
# NOT a model-callable tool and not auto-injected into chat completions. See
|
||||
# docs/research/litellm-searxng-search.md. Requires the litellm container to
|
||||
# resolve search.home — see the `extra_hosts` entry in docker-compose.yml.
|
||||
search_tools:
|
||||
- search_tool_name: searxng-search
|
||||
litellm_params:
|
||||
search_provider: searxng
|
||||
api_base: http://search.home/
|
||||
|
||||
# Knowledgebase / RAG, backed by the litellm-pgvector companion service (NOT
|
||||
# Qdrant — LiteLLM's native vector-store feature has no Qdrant provider, see
|
||||
# docs/research/litellm-knowledgebase.md). vector_store_id is this proxy's
|
||||
# own identifier for the store, not assigned by a backend.
|
||||
# Smoke-tested end-to-end against a running deploy (issue #24): search via
|
||||
# both /v1/vector_stores/{id}/search directly and the file_search tool on a
|
||||
# chat completion. Needed several fixes beyond this block to work — a
|
||||
# missing api_key here, litellm-pgvector's Prisma schema never having been
|
||||
# pushed, a 1536- vs 768-dim mismatch, and its create endpoint ignoring any
|
||||
# caller-supplied id — see scripts/update.sh, scripts/ingest-memory.sh, and
|
||||
# vendor/litellm-pgvector/'s local patches (models.py, main.py,
|
||||
# prisma/schema.prisma).
|
||||
#
|
||||
# This block only seeds the store into litellm's in-memory registry at
|
||||
# boot — it does NOT make it appear on the Admin UI's Vector Stores page
|
||||
# (/ui/vector-stores). That page reads litellm's own DB
|
||||
# (LiteLLM_ManagedVectorStoresTable), a separate registration scripts/
|
||||
# update.sh also does via POST /vector_store/new. Keep both in sync by
|
||||
# hand if you change api_base/api_key here — see update.sh's comment on
|
||||
# why there's no automatic sync from this block to the DB row.
|
||||
vector_store_registry:
|
||||
- vector_store_name: memory-and-notes
|
||||
litellm_params:
|
||||
vector_store_id: "memory-and-notes"
|
||||
custom_llm_provider: pg_vector
|
||||
api_base: http://litellm-pgvector:8000
|
||||
# Required by litellm's pg_vector provider (see
|
||||
# PGVectorStoreConfig.validate_environment in litellm's source) — it's
|
||||
# the Bearer token litellm-pgvector's own API checks against its
|
||||
# SERVER_API_KEY. Was missing entirely, which is why every vector
|
||||
# store call was failing with "Incorrect API key provided: None"
|
||||
# before litellm-pgvector was ever reached. See issue #24.
|
||||
api_key: os.environ/LITELLM_PGVECTOR_API_KEY
|
||||
embedding_model: local-embedding
|
||||
|
||||
router_settings:
|
||||
# ponytail: LiteLLM's request-prioritization scheduler is beta (see
|
||||
# docs/proxy-request-priority.md) — exact settings key/shape must be
|
||||
# confirmed against LiteLLM's current docs and smoke-tested against
|
||||
# llama.cpp before workloads depend on it. Redis is available (see the
|
||||
# litellm service's REDIS_* env vars in docker-compose.yml) if the
|
||||
# scheduler needs shared state for it.
|
||||
enable_priority_scheduling: true
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
@@ -1,85 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Loads data/memory.md and data/claude-legacy-memory.md into the LiteLLM
|
||||
# knowledgebase (the "memory-and-notes" vector store, see litellm-config.yaml)
|
||||
# via litellm-pgvector's batch-embeddings endpoint. Both files are optional —
|
||||
# a file that doesn't exist yet is skipped, not an error.
|
||||
#
|
||||
# ponytail: one chunk per non-empty, non-heading line — both source files are
|
||||
# already one fact/paragraph per line (no hard-wrapping), so this needs no
|
||||
# real chunking logic. Re-run after editing either file; there's no dedup, so
|
||||
# this appends duplicates on a second run against unchanged content — clear
|
||||
# the store first (DELETE the vector_store_id's rows) if you need a clean
|
||||
# reload.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
[ -f .env ] && set -a && . ./.env && set +a
|
||||
|
||||
: "${LITELLM_PGVECTOR_API_KEY:?Set LITELLM_PGVECTOR_API_KEY in .env first}"
|
||||
: "${LITELLM_PGVECTOR_EMBEDDING_KEY:?Set LITELLM_PGVECTOR_EMBEDDING_KEY in .env first}"
|
||||
LITELLM_PGVECTOR_URL="${LITELLM_PGVECTOR_URL:-http://localhost:8000}"
|
||||
LITELLM_URL="${LITELLM_URL:-http://localhost:${LITELLM_PORT:-4000}}"
|
||||
VECTOR_STORE_ID="memory-and-notes"
|
||||
|
||||
# Must match litellm-config.yaml's vector_store_registry entry — the
|
||||
# registry just points at a store the backend must already know about.
|
||||
# Ignores failure if it already exists (no documented idempotency check).
|
||||
# id is a local addition to litellm-pgvector's create endpoint (see
|
||||
# vendor/litellm-pgvector/main.py) — without it, create always minted a
|
||||
# random UUID and this script's writes could never land on VECTOR_STORE_ID.
|
||||
curl -sf -X POST "${LITELLM_PGVECTOR_URL}/v1/vector_stores" \
|
||||
-H "Authorization: Bearer ${LITELLM_PGVECTOR_API_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"id\": \"${VECTOR_STORE_ID}\", \"name\": \"${VECTOR_STORE_ID}\"}" > /dev/null 2>&1 || true
|
||||
|
||||
ingest_file() {
|
||||
local file="$1"
|
||||
if [ ! -f "$file" ]; then
|
||||
echo "Skipping $file (not present)."
|
||||
return
|
||||
fi
|
||||
|
||||
local section="" contents="[]" metas="[]"
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
"#"*) section="${line#\# }"; section="${section#\#\# }"; continue ;;
|
||||
""|"---") continue ;;
|
||||
esac
|
||||
contents=$(jq --arg c "$line" '. += [$c]' <<<"$contents")
|
||||
metas=$(jq --arg content "$line" --arg source "$file" --arg section "$section" \
|
||||
'. += [{"content": $content, "metadata": {"source": $source, "section": $section}}]' <<<"$metas")
|
||||
done < "$file"
|
||||
|
||||
local n
|
||||
n=$(jq 'length' <<<"$contents")
|
||||
if [ "$n" -eq 0 ]; then
|
||||
echo "Nothing to ingest from $file (no fact/paragraph lines)."
|
||||
return
|
||||
fi
|
||||
|
||||
# litellm-pgvector's embeddings endpoints take a precomputed vector per
|
||||
# chunk — they don't call the embedding model themselves (only query-time
|
||||
# search does, via its own EMBEDDING__* config). So this has to embed
|
||||
# client-side first, via the same proxy every other workload uses.
|
||||
echo "Embedding $n chunks from $file via LiteLLM..."
|
||||
local embeddings
|
||||
embeddings=$(curl -sf "${LITELLM_URL}/v1/embeddings" \
|
||||
-H "Authorization: Bearer ${LITELLM_PGVECTOR_EMBEDDING_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$(jq -n --argjson input "$contents" '{"model": "local-embedding", "input": $input}')" \
|
||||
| jq '[.data[].embedding]')
|
||||
|
||||
local batch
|
||||
batch=$(jq -n --argjson metas "$metas" --argjson embeds "$embeddings" \
|
||||
'[range(0; ($metas | length)) as $i | $metas[$i] + {"embedding": $embeds[$i]}]')
|
||||
|
||||
echo "Ingesting $n chunks from $file..."
|
||||
curl -sf -X POST "${LITELLM_PGVECTOR_URL}/v1/vector_stores/${VECTOR_STORE_ID}/embeddings/batch" \
|
||||
-H "Authorization: Bearer ${LITELLM_PGVECTOR_API_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$(jq -n --argjson embeddings "$batch" '{"embeddings": $embeddings}')" > /dev/null
|
||||
}
|
||||
|
||||
ingest_file data/memory.md
|
||||
ingest_file data/claude-legacy-memory.md
|
||||
echo "Done."
|
||||
+29
-87
@@ -1,9 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
# The one command to run after any change to this repo (compose file,
|
||||
# litellm-config.yaml, .env, or a git pull) to bring the running stack in
|
||||
# sync. Ensures secrets/keys exist, pulls, validates, rebuilds/re-pulls
|
||||
# images, and recreates only what changed — safe to run any time, including
|
||||
# with nothing to do.
|
||||
# The one command to run after any change to this repo (compose file, .env,
|
||||
# or a git pull) to bring the running stack in sync. Ensures secrets/keys
|
||||
# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
|
||||
# changed — safe to run any time, including with nothing to do.
|
||||
#
|
||||
# omniroute's own routing/provider config (llama-server, search) lives in
|
||||
# its dashboard, not a checked-in file like the old litellm-config.yaml —
|
||||
# see issue #31 and docs/proxy-key-onboarding.md.
|
||||
#
|
||||
# ponytail: no rollback/backup logic — this is a single-user homelab box,
|
||||
# not a fleet. If a bad config lands, `git revert` + re-run is the recovery
|
||||
@@ -31,19 +34,20 @@ set_if_blank() {
|
||||
|
||||
echo "==> filling in missing secrets"
|
||||
# Random values — safe to re-run, never overwrites what's already set.
|
||||
# LITELLM_SALT_KEY especially: never change it after first run, existing
|
||||
# encrypted data becomes unreadable if you do.
|
||||
set_if_blank LITELLM_MASTER_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank LITELLM_SALT_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank LITELLM_DB_PASSWORD "$(openssl rand -hex 32)"
|
||||
set_if_blank REDIS_PASSWORD "$(openssl rand -hex 32)"
|
||||
set_if_blank UI_PASSWORD "$(openssl rand -hex 16)"
|
||||
set_if_blank PGVECTOR_DB_PASSWORD "$(openssl rand -hex 32)"
|
||||
set_if_blank LITELLM_PGVECTOR_API_KEY "$(openssl rand -hex 32)"
|
||||
# OMNIROUTE_STORAGE_ENCRYPTION_KEY especially: never change it after first
|
||||
# run, existing encrypted data becomes unreadable if you do (same caveat as
|
||||
# LiteLLM's old LITELLM_SALT_KEY).
|
||||
set_if_blank OMNIROUTE_INITIAL_PASSWORD "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_JWT_SECRET "$(openssl rand -base64 48)"
|
||||
set_if_blank OMNIROUTE_API_KEY_SECRET "$(openssl rand -hex 32)"
|
||||
set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
|
||||
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
|
||||
|
||||
echo "==> resolving SEARXNG_LAN_IP"
|
||||
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
|
||||
# not from inside the litellm container (see docs/research/litellm-searxng-search.md).
|
||||
# not from inside the omniroute container (see docs/research/litellm-searxng-search.md,
|
||||
# still the relevant background even though omniroute replaced litellm — see issue #31).
|
||||
searxng_ip="$(getent hosts search.home 2>/dev/null | awk '{print $1}' | head -1)"
|
||||
if [ -n "$searxng_ip" ]; then
|
||||
set_if_blank SEARXNG_LAN_IP "$searxng_ip"
|
||||
@@ -65,86 +69,24 @@ docker compose build --pull
|
||||
|
||||
echo "==> ensuring models are downloaded (skips already-present files)"
|
||||
docker compose --profile tools run --rm downloader
|
||||
docker compose --profile tools run --rm downloader-embedding
|
||||
|
||||
echo "==> bringing up litellm (needed to mint virtual keys below)"
|
||||
docker compose up -d --wait litellm-db litellm
|
||||
echo "==> bringing up omniroute"
|
||||
docker compose up -d --wait omniroute
|
||||
|
||||
# OPENWEBUI_LITELLM_KEY / LITELLM_PGVECTOR_EMBEDDING_KEY are per-workload
|
||||
# virtual keys, not random secrets — minted via LiteLLM's own API
|
||||
# (docs/proxy-key-onboarding.md documents the manual Admin UI route; this is
|
||||
# the same thing over the REST endpoint LITELLM_MASTER_KEY already
|
||||
# authenticates against).
|
||||
# ponytail: no scripted key-minting yet, unlike the old LiteLLM
|
||||
# /key/generate flow — omniroute's POST /api/keys needs a dashboard login
|
||||
# session (ManagementSessionAuth), not a static bearer key, and that flow
|
||||
# hasn't been verified against a live instance (see issue #37). Until then,
|
||||
# check for the workload keys and just remind rather than fail.
|
||||
set -a && . ./.env && set +a
|
||||
mint_key_if_blank() {
|
||||
local key="$1" alias="$2"
|
||||
if grep -qE "^${key}=.*[^[:space:]]" .env; then
|
||||
echo "${key}: already set, skipping."
|
||||
return
|
||||
fi
|
||||
local minted
|
||||
minted=$(curl -sf -X POST "http://localhost:${LITELLM_PORT:-4000}/key/generate" \
|
||||
-H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"key_alias\": \"${alias}\"}" | jq -r '.key')
|
||||
if [ -n "$minted" ] && [ "$minted" != "null" ]; then
|
||||
# Same missing-line-vs-blank-line handling as set_if_blank above.
|
||||
if grep -qE "^${key}=" .env; then
|
||||
sed -i "s|^${key}=.*|${key}=${minted}|" .env
|
||||
if grep -qE "^OPENWEBUI_OMNIROUTE_KEY=.*[^[:space:]]" .env; then
|
||||
echo "OPENWEBUI_OMNIROUTE_KEY: already set, skipping."
|
||||
else
|
||||
echo "${key}=${minted}" >> .env
|
||||
echo "OPENWEBUI_OMNIROUTE_KEY: blank — mint it by hand in the omniroute dashboard (http://localhost:\${OMNIROUTE_DASHBOARD_PORT:-20128}) and set it in .env. See docs/proxy-key-onboarding.md."
|
||||
fi
|
||||
echo "${key}: minted."
|
||||
else
|
||||
echo "${key}: mint failed, create it by hand per docs/proxy-key-onboarding.md."
|
||||
fi
|
||||
}
|
||||
mint_key_if_blank OPENWEBUI_LITELLM_KEY openwebui
|
||||
mint_key_if_blank LITELLM_PGVECTOR_EMBEDDING_KEY litellm-pgvector
|
||||
set -a && . ./.env && set +a
|
||||
|
||||
echo "==> recreating changed services"
|
||||
docker compose up -d --remove-orphans
|
||||
|
||||
# litellm-pgvector's Dockerfile only runs `prisma generate` (codegen) at
|
||||
# build time — nothing ever applied the schema to pgvector-db itself, so the
|
||||
# vector_stores/embeddings tables plain didn't exist until this was added
|
||||
# (see issue #24). --accept-data-loss is the same "no rollback/backup logic,
|
||||
# git revert is the recovery path" tradeoff as the rest of this script — a
|
||||
# schema-incompatible change here would need a manual look regardless.
|
||||
echo "==> syncing litellm-pgvector's database schema"
|
||||
docker compose up -d --wait pgvector-db litellm-pgvector
|
||||
docker compose exec -T litellm-pgvector prisma db push --accept-data-loss
|
||||
|
||||
# Registers memory-and-notes in litellm's own DB (LiteLLM_ManagedVectorStoresTable),
|
||||
# not just litellm-config.yaml's vector_store_registry block. Both matter for
|
||||
# different reasons: config.yaml seeds it into memory at boot (works even
|
||||
# before this script has ever run against a fresh DB); the DB row is what
|
||||
# /vector_store/list — and so the Admin UI's Vector Stores page — actually
|
||||
# shows, since that endpoint only auto-syncs a config-only entry into the DB
|
||||
# view once a DB row with the same id exists (see issue #24 follow-up).
|
||||
#
|
||||
# Must pass the real key, not the os.environ/... form used in
|
||||
# litellm-config.yaml — this hits the live management API, not the
|
||||
# config.yaml loader, so there's no env-substitution pass over the request
|
||||
# body. Ignores failure if the row already exists (no update-in-place: see
|
||||
# below).
|
||||
#
|
||||
# No update-if-changed path — the DB row is otherwise never touched once
|
||||
# created (/vector_store/update in this litellm version can't set
|
||||
# litellm_params at all — VectorStoreUpdateRequest has no such field, so an
|
||||
# update silently no-ops on it). If LITELLM_PGVECTOR_API_KEY ever rotates,
|
||||
# fix this row by hand: /vector_store/delete then re-run this script.
|
||||
echo "==> registering memory-and-notes vector store with litellm (for the Admin UI)"
|
||||
curl -sf -X POST "http://localhost:${LITELLM_PORT:-4000}/vector_store/new" \
|
||||
-H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$(jq -n --arg key "${LITELLM_PGVECTOR_API_KEY}" '{
|
||||
vector_store_id: "memory-and-notes",
|
||||
custom_llm_provider: "pg_vector",
|
||||
vector_store_name: "memory-and-notes",
|
||||
litellm_params: {api_base: "http://litellm-pgvector:8000", api_key: $key}
|
||||
}')" > /dev/null 2>&1 || echo "memory-and-notes: already registered (or registration failed — check by hand if this is a fresh deploy)."
|
||||
|
||||
echo "==> status"
|
||||
docker compose ps
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
.env
|
||||
__pycache__/*
|
||||
venv/*
|
||||
venv
|
||||
Vendored
-32
@@ -1,32 +0,0 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PYTHONPATH=/app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
build-essential \
|
||||
curl \
|
||||
postgresql-client \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set work directory
|
||||
WORKDIR /app
|
||||
|
||||
# Install Python dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy project
|
||||
COPY . .
|
||||
|
||||
# Generate Prisma client
|
||||
RUN prisma generate
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8000
|
||||
|
||||
# Command to run the application
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
Vendored
-21
@@ -1,21 +0,0 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Berri AI
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
Vendored
-388
@@ -1,388 +0,0 @@
|
||||
# OpenAI Vector Stores API with PGVector
|
||||
|
||||
A FastAPI application that provides OpenAI-compatible vector store endpoints using PGVector and LiteLLM proxy for embeddings.
|
||||
|
||||
## Features
|
||||
|
||||
- 🔌 OpenAI-compatible API endpoints
|
||||
- 🗄️ PGVector for efficient vector storage and similarity search
|
||||
- 🎛️ Configurable database field mappings
|
||||
- 🔄 LiteLLM proxy integration for any embedding model
|
||||
- 🐳 Docker support
|
||||
- ⚡ FastAPI with async support
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### 1. Create Vector Store
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Support FAQ"
|
||||
}'
|
||||
```
|
||||
|
||||
### 2. List Vector Stores
|
||||
```bash
|
||||
# List all vector stores
|
||||
curl -X GET \
|
||||
http://localhost:8000/v1/vector_stores \
|
||||
-H "Authorization: Bearer your-api-key"
|
||||
|
||||
# List with pagination (limit and after parameters)
|
||||
curl -X GET \
|
||||
"http://localhost:8000/v1/vector_stores?limit=10&after=vs_abc123" \
|
||||
-H "Authorization: Bearer your-api-key"
|
||||
```
|
||||
|
||||
### 3. Add Single Embedding to Vector Store
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores/vs_abc123/embeddings \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"content": "Our return policy allows returns within 30 days of purchase.",
|
||||
"embedding": [0.1, 0.2, 0.3, ...],
|
||||
"metadata": {
|
||||
"category": "returns",
|
||||
"source": "faq",
|
||||
"id": "return_policy_1"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. Add Multiple Embeddings (Batch)
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores/vs_abc123/embeddings/batch \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"embeddings": [
|
||||
{
|
||||
"content": "Our return policy allows returns within 30 days of purchase.",
|
||||
"embedding": [0.1, 0.2, 0.3, ...],
|
||||
"metadata": {"category": "returns"}
|
||||
},
|
||||
{
|
||||
"content": "Shipping is free for orders over $50.",
|
||||
"embedding": [0.4, 0.5, 0.6, ...],
|
||||
"metadata": {"category": "shipping"}
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### 5. Search Vector Store
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores/vs_abc123/search \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"query": "What is the return policy?",
|
||||
"limit": 20,
|
||||
"filters": {"category": "support"}
|
||||
}'
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Create a `.env` file with the following configuration:
|
||||
|
||||
```bash
|
||||
# Database Configuration
|
||||
DATABASE_URL="postgresql://username:password@localhost:5432/vectordb?schema=public"
|
||||
|
||||
# API Configuration
|
||||
SERVER_API_KEY="your-api-key-here"
|
||||
|
||||
# Server Configuration
|
||||
HOST="0.0.0.0"
|
||||
PORT=8000
|
||||
|
||||
# LiteLLM Proxy Configuration
|
||||
EMBEDDING__MODEL="text-embedding-ada-002"
|
||||
EMBEDDING__BASE_URL="http://localhost:4000"
|
||||
EMBEDDING__API_KEY="sk-1234"
|
||||
EMBEDDING__DIMENSIONS=1536
|
||||
|
||||
# Database Field Configuration (optional)
|
||||
DB_FIELDS__ID_FIELD="id"
|
||||
DB_FIELDS__CONTENT_FIELD="content"
|
||||
DB_FIELDS__METADATA_FIELD="metadata"
|
||||
DB_FIELDS__EMBEDDING_FIELD="embedding"
|
||||
DB_FIELDS__VECTOR_STORE_ID_FIELD="vector_store_id"
|
||||
DB_FIELDS__CREATED_AT_FIELD="created_at"
|
||||
```
|
||||
|
||||
### Database Field Mapping
|
||||
|
||||
You can customize the database field names by setting environment variables:
|
||||
|
||||
- `DB_FIELDS__ID_FIELD` - Primary key field (default: "id")
|
||||
- `DB_FIELDS__CONTENT_FIELD` - Text content field (default: "content")
|
||||
- `DB_FIELDS__METADATA_FIELD` - JSON metadata field (default: "metadata")
|
||||
- `DB_FIELDS__EMBEDDING_FIELD` - Vector embedding field (default: "embedding")
|
||||
- `DB_FIELDS__VECTOR_STORE_ID_FIELD` - Foreign key field (default: "vector_store_id")
|
||||
- `DB_FIELDS__CREATED_AT_FIELD` - Timestamp field (default: "created_at")
|
||||
|
||||
### LiteLLM Proxy Configuration
|
||||
|
||||
The application uses LiteLLM proxy for embeddings. Configure it with:
|
||||
|
||||
- `EMBEDDING__MODEL` - Model name (e.g., "text-embedding-ada-002")
|
||||
- `EMBEDDING__BASE_URL` - LiteLLM proxy URL (e.g., "http://localhost:4000")
|
||||
- `EMBEDDING__API_KEY` - LiteLLM proxy API key
|
||||
- `EMBEDDING__DIMENSIONS` - Embedding dimensions (default: 1536)
|
||||
|
||||
## Setup and Installation
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Database Setup
|
||||
|
||||
```bash
|
||||
# Generate Prisma client
|
||||
prisma generate
|
||||
|
||||
# Run database migrations
|
||||
prisma db push
|
||||
```
|
||||
|
||||
### 3. Set up LiteLLM Proxy
|
||||
|
||||
Start LiteLLM proxy pointing to your preferred embedding model:
|
||||
|
||||
```bash
|
||||
# Example: Start LiteLLM proxy for OpenAI
|
||||
litellm --model text-embedding-ada-002 --port 4000
|
||||
```
|
||||
|
||||
### 4. Run the Application
|
||||
|
||||
```bash
|
||||
python main.py
|
||||
```
|
||||
|
||||
Or using uvicorn directly:
|
||||
|
||||
```bash
|
||||
uvicorn main:app --host 0.0.0.0 --port 8000 --reload
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
### Build and run with Docker:
|
||||
|
||||
```bash
|
||||
# Build the image
|
||||
docker build -t vector-store-api .
|
||||
|
||||
# Run the container
|
||||
docker run -p 8000:8000 --env-file .env vector-store-api
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
The application uses two main tables:
|
||||
|
||||
### vector_stores
|
||||
- `id` (string, primary key)
|
||||
- `name` (string)
|
||||
- `file_counts` (json)
|
||||
- `status` (string)
|
||||
- `usage_bytes` (integer)
|
||||
- `created_at` (timestamp)
|
||||
- `expires_after` (json, optional)
|
||||
- `expires_at` (timestamp, optional)
|
||||
- `last_active_at` (timestamp, optional)
|
||||
- `metadata` (json, optional)
|
||||
|
||||
### embeddings
|
||||
- `id` (string, primary key)
|
||||
- `vector_store_id` (string, foreign key)
|
||||
- `content` (string)
|
||||
- `embedding` (vector(1536))
|
||||
- `metadata` (json, optional)
|
||||
- `created_at` (timestamp)
|
||||
|
||||
## Supported Models
|
||||
|
||||
Any embedding model supported by LiteLLM proxy can be used. Examples:
|
||||
|
||||
- OpenAI: `text-embedding-ada-002`, `text-embedding-3-small`, `text-embedding-3-large`
|
||||
- Cohere: `embed-english-v3.0`, `embed-multilingual-v3.0`
|
||||
- Voyage: `voyage-2`, `voyage-large-2`
|
||||
- And many more...
|
||||
|
||||
## API Response Format
|
||||
|
||||
### Vector Store Response
|
||||
```json
|
||||
{
|
||||
"id": "vs_abc123",
|
||||
"object": "vector_store",
|
||||
"created_at": 1699024800,
|
||||
"name": "Support FAQ",
|
||||
"usage_bytes": 0,
|
||||
"file_counts": {
|
||||
"in_progress": 0,
|
||||
"completed": 0,
|
||||
"failed": 0,
|
||||
"cancelled": 0,
|
||||
"total": 0
|
||||
},
|
||||
"status": "completed",
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Vector Store List Response
|
||||
```json
|
||||
{
|
||||
"object": "list",
|
||||
"data": [
|
||||
{
|
||||
"id": "vs_abc123",
|
||||
"object": "vector_store",
|
||||
"created_at": 1699024800,
|
||||
"name": "Support FAQ",
|
||||
"usage_bytes": 1024,
|
||||
"file_counts": {"completed": 5, "total": 5},
|
||||
"status": "completed",
|
||||
"metadata": {}
|
||||
}
|
||||
],
|
||||
"first_id": "vs_abc123",
|
||||
"last_id": "vs_def456",
|
||||
"has_more": false
|
||||
}
|
||||
```
|
||||
|
||||
### Search Response
|
||||
```json
|
||||
{
|
||||
"object": "vector_store.search",
|
||||
"data": [
|
||||
{
|
||||
"id": "emb_123",
|
||||
"content": "Return policy text...",
|
||||
"score": 0.95,
|
||||
"metadata": {"category": "support"}
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"total_tokens": 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Example Search Request
|
||||
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores/vs_support_faq/search \
|
||||
-H "Authorization: Bearer sk-1234" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"query": "How do I return an item?",
|
||||
"limit": 5,
|
||||
"return_metadata": true
|
||||
}'
|
||||
```
|
||||
|
||||
## Health Check
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
## Migrating Existing Data
|
||||
|
||||
If you have an existing database with embeddings and content, you can easily migrate using the embedding APIs:
|
||||
|
||||
### 1. Create Vector Store
|
||||
First, create a vector store for your data:
|
||||
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "Migrated Data",
|
||||
"metadata": {"source": "legacy_system"}
|
||||
}'
|
||||
```
|
||||
|
||||
### 2. Batch Insert Embeddings
|
||||
Use the batch endpoint to efficiently insert multiple embeddings:
|
||||
|
||||
```bash
|
||||
curl -X POST \
|
||||
http://localhost:8000/v1/vector_stores/vs_your_id/embeddings/batch \
|
||||
-H "Authorization: Bearer your-api-key" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"embeddings": [
|
||||
{
|
||||
"content": "Your text content here",
|
||||
"embedding": [0.1, 0.2, 0.3, ...1536 dimensions...],
|
||||
"metadata": {"source_id": "doc_123", "category": "support"}
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### 3. Migration Script Example
|
||||
|
||||
Here's a Python script example for migrating from an existing database:
|
||||
|
||||
```python
|
||||
import psycopg2
|
||||
import requests
|
||||
import json
|
||||
|
||||
# Connect to your existing database
|
||||
conn = psycopg2.connect("your_existing_db_url")
|
||||
cur = conn.cursor()
|
||||
|
||||
# Fetch existing data
|
||||
cur.execute("SELECT content, embedding, metadata FROM your_table")
|
||||
rows = cur.fetchall()
|
||||
|
||||
# Prepare batch data
|
||||
embeddings = []
|
||||
for content, embedding, metadata in rows:
|
||||
embeddings.append({
|
||||
"content": content,
|
||||
"embedding": embedding.tolist(), # Convert numpy array to list
|
||||
"metadata": metadata or {}
|
||||
})
|
||||
|
||||
# Send batch to API
|
||||
response = requests.post(
|
||||
"http://localhost:8000/v1/vector_stores/your_vector_store_id/embeddings/batch",
|
||||
headers={
|
||||
"Authorization": "Bearer your-api-key",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
json={"embeddings": embeddings}
|
||||
)
|
||||
|
||||
print(f"Migrated {len(embeddings)} embeddings")
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT License
|
||||
Vendored
-25
@@ -1,25 +0,0 @@
|
||||
Vendored from https://github.com/BerriAI/litellm-pgvector at commit
|
||||
`b553f84a32f580b4303297df5567f25912b59d93` (main, 2026-09-02). See
|
||||
`docker-compose.yml`'s `litellm-pgvector` service comment for why this is
|
||||
vendored instead of built from a remote git context.
|
||||
|
||||
**Local changes on top of that commit** (found smoke-testing issue #24 —
|
||||
without these, the store can never be searched or written to):
|
||||
|
||||
- `prisma/schema.prisma`: `Embedding.embedding` was `vector(1536)`
|
||||
(OpenAI ada-002's size); changed to `vector(768)` to match this stack's
|
||||
actual embedding model (nomic-embed-text-v1.5).
|
||||
- `models.py` / `main.py`: `POST /v1/vector_stores` always minted a random
|
||||
UUID for the new store's `id`, ignoring anything the caller asked for.
|
||||
Added an optional `id` field to `VectorStoreCreateRequest` and made
|
||||
`create_vector_store` use it when given — `litellm-config.yaml`'s
|
||||
`vector_store_registry` addresses this store by a fixed id
|
||||
(`memory-and-notes`), which never matched a real row otherwise.
|
||||
|
||||
Re-applying these after a re-vendor: diff this directory against upstream
|
||||
before overwriting, or just redo the three edits above.
|
||||
|
||||
To update: `git clone https://github.com/BerriAI/litellm-pgvector.git`
|
||||
somewhere, copy everything except `.git/` over this directory, re-apply the
|
||||
local changes above, update the commit hash above, and run
|
||||
`./scripts/update.sh`.
|
||||
Vendored
-60
@@ -1,60 +0,0 @@
|
||||
from typing import Dict, Optional
|
||||
from pydantic import BaseModel
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
|
||||
class DatabaseFieldConfig(BaseModel):
|
||||
"""Configuration for database field mappings"""
|
||||
id_field: str = "id"
|
||||
content_field: str = "content"
|
||||
metadata_field: str = "metadata"
|
||||
embedding_field: str = "embedding"
|
||||
vector_store_id_field: str = "vector_store_id"
|
||||
created_at_field: str = "created_at"
|
||||
|
||||
|
||||
class EmbeddingConfig(BaseModel):
|
||||
"""Configuration for embedding generation via LiteLLM proxy"""
|
||||
model: str = "text-embedding-ada-002"
|
||||
base_url: str = "http://localhost:4000" # LiteLLM proxy URL
|
||||
api_key: str = "sk-1234" # LiteLLM proxy API key
|
||||
dimensions: int = 1536
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application settings"""
|
||||
# Database configuration
|
||||
database_url: str = "postgresql://username:password@localhost:5432/vectordb?schema=public"
|
||||
|
||||
# API configuration
|
||||
server_api_key: str = "your-api-key-here"
|
||||
port: int = 8000
|
||||
host: str = "0.0.0.0"
|
||||
|
||||
# Database field mappings
|
||||
db_fields: DatabaseFieldConfig = DatabaseFieldConfig()
|
||||
|
||||
# Embedding configuration
|
||||
embedding: EmbeddingConfig = EmbeddingConfig()
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
env_nested_delimiter = "__"
|
||||
case_sensitive = False
|
||||
|
||||
# Allow environment variables like:
|
||||
# DB_FIELDS__ID_FIELD=custom_id
|
||||
# EMBEDDING__MODEL=text-embedding-3-small
|
||||
# EMBEDDING__API_BASE=https://api.openai.com/v1
|
||||
|
||||
@property
|
||||
def table_names(self) -> Dict[str, str]:
|
||||
"""Get table names"""
|
||||
return {
|
||||
"vector_stores": "vector_stores",
|
||||
"embeddings": "embeddings"
|
||||
}
|
||||
|
||||
|
||||
# Global settings instance
|
||||
settings = Settings()
|
||||
-90
@@ -1,90 +0,0 @@
|
||||
from typing import List, Optional
|
||||
from config import settings, EmbeddingConfig
|
||||
from litellm.types.utils import EmbeddingResponse
|
||||
import litellm
|
||||
import logging
|
||||
|
||||
class EmbeddingService:
|
||||
"""Service for generating embeddings using OpenAI SDK pointed at LiteLLM proxy"""
|
||||
|
||||
def __init__(self, config: Optional[EmbeddingConfig] = None):
|
||||
self.config = config or settings.embedding
|
||||
|
||||
|
||||
|
||||
async def generate_embedding(self, text: str) -> List[float]:
|
||||
"""
|
||||
Generate embedding for a single text using LiteLLM proxy
|
||||
|
||||
Args:
|
||||
text: Text to embed
|
||||
|
||||
Returns:
|
||||
List of floats representing the embedding vector
|
||||
"""
|
||||
try:
|
||||
response: EmbeddingResponse = await litellm.aembedding(
|
||||
model=self.config.model,
|
||||
input=[text],
|
||||
api_base=self.config.base_url,
|
||||
api_key=self.config.api_key
|
||||
)
|
||||
logging.debug(f"Embedding response: {response}")
|
||||
|
||||
# Extract embedding from response
|
||||
embedding = response.data[0]["embedding"]
|
||||
|
||||
# Validate embedding dimensions
|
||||
if len(embedding) != self.config.dimensions:
|
||||
raise ValueError(
|
||||
f"Expected embedding dimension {self.config.dimensions}, "
|
||||
f"got {len(embedding)}"
|
||||
)
|
||||
|
||||
return embedding
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to generate embedding: {str(e)}")
|
||||
|
||||
async def generate_embeddings(self, texts: List[str]) -> List[List[float]]:
|
||||
"""
|
||||
Generate embeddings for multiple texts
|
||||
|
||||
Args:
|
||||
texts: List of texts to embed
|
||||
|
||||
Returns:
|
||||
List of embedding vectors
|
||||
"""
|
||||
try:
|
||||
# Generate embeddings using LiteLLM
|
||||
response = await litellm.aembedding(
|
||||
model=self.config.model,
|
||||
input=texts,
|
||||
api_base=self.config.base_url,
|
||||
api_key=self.config.api_key
|
||||
)
|
||||
|
||||
# Extract embeddings from response
|
||||
embeddings = [item.embedding for item in response.data]
|
||||
|
||||
# Validate embedding dimensions
|
||||
for i, embedding in enumerate(embeddings):
|
||||
if len(embedding) != self.config.dimensions:
|
||||
raise ValueError(
|
||||
f"Expected embedding dimension {self.config.dimensions} for text {i}, "
|
||||
f"got {len(embedding)}"
|
||||
)
|
||||
|
||||
return embeddings
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to generate embeddings: {str(e)}")
|
||||
|
||||
def update_config(self, new_config: EmbeddingConfig):
|
||||
"""Update the embedding configuration"""
|
||||
self.config = new_config
|
||||
|
||||
|
||||
# Global embedding service instance
|
||||
embedding_service = EmbeddingService()
|
||||
Vendored
-530
@@ -1,530 +0,0 @@
|
||||
import os
|
||||
import asyncio
|
||||
import time
|
||||
from typing import List, Optional
|
||||
from fastapi import FastAPI, HTTPException, Depends, Header
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from prisma import Prisma
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from models import (
|
||||
VectorStoreCreateRequest,
|
||||
VectorStoreResponse,
|
||||
VectorStoreSearchRequest,
|
||||
VectorStoreSearchResponse,
|
||||
SearchResult,
|
||||
EmbeddingCreateRequest,
|
||||
EmbeddingResponse,
|
||||
EmbeddingBatchCreateRequest,
|
||||
EmbeddingBatchCreateResponse,
|
||||
VectorStoreListResponse,
|
||||
ContentChunk
|
||||
)
|
||||
from config import settings
|
||||
from embedding_service import embedding_service
|
||||
|
||||
load_dotenv()
|
||||
|
||||
app = FastAPI(
|
||||
title="OpenAI Vector Stores API",
|
||||
description="OpenAI-compatible Vector Stores API using PGVector",
|
||||
version="1.0.0"
|
||||
)
|
||||
|
||||
# CORS middleware
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Global Prisma client
|
||||
db = Prisma()
|
||||
|
||||
security = HTTPBearer()
|
||||
|
||||
|
||||
async def get_api_key(credentials: HTTPAuthorizationCredentials = Depends(security)):
|
||||
"""Validate API key from Authorization header"""
|
||||
expected_key = settings.server_api_key
|
||||
if credentials.credentials != expected_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return credentials.credentials
|
||||
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
"""Connect to database on startup"""
|
||||
await db.connect()
|
||||
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def shutdown():
|
||||
"""Disconnect from database on shutdown"""
|
||||
await db.disconnect()
|
||||
|
||||
|
||||
async def generate_query_embedding(query: str) -> List[float]:
|
||||
"""
|
||||
Generate an embedding for the query using LiteLLM
|
||||
"""
|
||||
return await embedding_service.generate_embedding(query)
|
||||
|
||||
|
||||
@app.post("/v1/vector_stores", response_model=VectorStoreResponse)
|
||||
async def create_vector_store(
|
||||
request: VectorStoreCreateRequest,
|
||||
api_key: str = Depends(get_api_key)
|
||||
):
|
||||
"""
|
||||
Create a new vector store.
|
||||
"""
|
||||
try:
|
||||
# Use raw SQL to insert the vector store with configurable table/field names
|
||||
vector_store_table = settings.table_names["vector_stores"]
|
||||
|
||||
# ponytail: honor a caller-supplied id (request.id) instead of
|
||||
# always minting one — litellm's vector_store_registry addresses
|
||||
# this store by a fixed id (see litellm-config.yaml), which never
|
||||
# matched anything when this always generated a random UUID.
|
||||
import uuid as _uuid
|
||||
vector_store_id = request.id or str(_uuid.uuid4())
|
||||
|
||||
result = await db.query_raw(
|
||||
f"""
|
||||
INSERT INTO {vector_store_table} (id, name, file_counts, status, usage_bytes, expires_after, metadata, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, NOW())
|
||||
RETURNING id, name, file_counts, status, usage_bytes, expires_after, expires_at, last_active_at, metadata,
|
||||
EXTRACT(EPOCH FROM created_at)::bigint as created_at_timestamp
|
||||
""",
|
||||
vector_store_id,
|
||||
request.name,
|
||||
{"in_progress": 0, "completed": 0, "failed": 0, "cancelled": 0, "total": 0},
|
||||
"completed",
|
||||
0,
|
||||
request.expires_after,
|
||||
request.metadata or {}
|
||||
)
|
||||
|
||||
if not result:
|
||||
raise HTTPException(status_code=500, detail="Failed to create vector store")
|
||||
|
||||
vector_store = result[0]
|
||||
|
||||
# Convert to response format
|
||||
created_at = int(vector_store["created_at_timestamp"])
|
||||
expires_at = int(vector_store["expires_at"].timestamp()) if vector_store.get("expires_at") else None
|
||||
last_active_at = int(vector_store["last_active_at"].timestamp()) if vector_store.get("last_active_at") else None
|
||||
|
||||
return VectorStoreResponse(
|
||||
id=vector_store["id"],
|
||||
created_at=created_at,
|
||||
name=vector_store["name"],
|
||||
usage_bytes=vector_store["usage_bytes"] or 0,
|
||||
file_counts=vector_store["file_counts"] or {"in_progress": 0, "completed": 0, "failed": 0, "cancelled": 0, "total": 0},
|
||||
status=vector_store["status"],
|
||||
expires_after=vector_store["expires_after"],
|
||||
expires_at=expires_at,
|
||||
last_active_at=last_active_at,
|
||||
metadata=vector_store["metadata"]
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create vector store: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/v1/vector_stores", response_model=VectorStoreListResponse)
|
||||
async def list_vector_stores(
|
||||
limit: Optional[int] = 20,
|
||||
after: Optional[str] = None,
|
||||
before: Optional[str] = None,
|
||||
api_key: str = Depends(get_api_key)
|
||||
):
|
||||
"""
|
||||
List vector stores with optional pagination.
|
||||
"""
|
||||
try:
|
||||
limit = min(limit or 20, 100) # Cap at 100 results
|
||||
|
||||
vector_store_table = settings.table_names["vector_stores"]
|
||||
|
||||
# Build base query
|
||||
base_query = f"""
|
||||
SELECT id, name, file_counts, status, usage_bytes, expires_after, expires_at, last_active_at, metadata,
|
||||
EXTRACT(EPOCH FROM created_at)::bigint as created_at_timestamp
|
||||
FROM {vector_store_table}
|
||||
"""
|
||||
|
||||
# Add pagination conditions
|
||||
conditions = []
|
||||
params = []
|
||||
param_count = 1
|
||||
|
||||
if after:
|
||||
conditions.append(f"id > ${param_count}")
|
||||
params.append(after)
|
||||
param_count += 1
|
||||
|
||||
if before:
|
||||
conditions.append(f"id < ${param_count}")
|
||||
params.append(before)
|
||||
param_count += 1
|
||||
|
||||
if conditions:
|
||||
base_query += " WHERE " + " AND ".join(conditions)
|
||||
|
||||
# Add ordering and limit
|
||||
final_query = base_query + f" ORDER BY created_at DESC LIMIT {limit + 1}"
|
||||
|
||||
# Execute query
|
||||
results = await db.query_raw(final_query, *params)
|
||||
|
||||
# Check if there are more results
|
||||
has_more = len(results) > limit
|
||||
if has_more:
|
||||
results = results[:limit] # Remove extra result
|
||||
|
||||
# Convert to response format
|
||||
vector_stores = []
|
||||
for row in results:
|
||||
created_at = int(row["created_at_timestamp"])
|
||||
expires_at = int(row["expires_at"].timestamp()) if row.get("expires_at") else None
|
||||
last_active_at = int(row["last_active_at"].timestamp()) if row.get("last_active_at") else None
|
||||
|
||||
vector_store = VectorStoreResponse(
|
||||
id=row["id"],
|
||||
created_at=created_at,
|
||||
name=row["name"],
|
||||
usage_bytes=row["usage_bytes"] or 0,
|
||||
file_counts=row["file_counts"] or {"in_progress": 0, "completed": 0, "failed": 0, "cancelled": 0, "total": 0},
|
||||
status=row["status"],
|
||||
expires_after=row["expires_after"],
|
||||
expires_at=expires_at,
|
||||
last_active_at=last_active_at,
|
||||
metadata=row["metadata"]
|
||||
)
|
||||
vector_stores.append(vector_store)
|
||||
|
||||
# Determine first_id and last_id
|
||||
first_id = vector_stores[0].id if vector_stores else None
|
||||
last_id = vector_stores[-1].id if vector_stores else None
|
||||
|
||||
return VectorStoreListResponse(
|
||||
data=vector_stores,
|
||||
first_id=first_id,
|
||||
last_id=last_id,
|
||||
has_more=has_more
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise HTTPException(status_code=500, detail=f"Failed to list vector stores: {str(e)}")
|
||||
|
||||
|
||||
@app.post("/v1/vector_stores/{vector_store_id}/search", response_model=VectorStoreSearchResponse)
|
||||
@app.post("/vector_stores/{vector_store_id}/search", response_model=VectorStoreSearchResponse)
|
||||
async def search_vector_store(
|
||||
vector_store_id: str,
|
||||
request: VectorStoreSearchRequest,
|
||||
api_key: str = Depends(get_api_key)
|
||||
):
|
||||
"""
|
||||
Search a vector store for similar content.
|
||||
"""
|
||||
try:
|
||||
# Check if vector store exists
|
||||
vector_store_table = settings.table_names["vector_stores"]
|
||||
vector_store_result = await db.query_raw(
|
||||
f"SELECT id FROM {vector_store_table} WHERE id = $1",
|
||||
vector_store_id
|
||||
)
|
||||
if not vector_store_result:
|
||||
raise HTTPException(status_code=404, detail="Vector store not found")
|
||||
|
||||
# Generate embedding for query
|
||||
query_embedding = await generate_query_embedding(request.query)
|
||||
query_vector_str = "[" + ",".join(map(str, query_embedding)) + "]"
|
||||
|
||||
# Build the raw SQL query for vector similarity search
|
||||
limit = min(request.limit or 20, 100) # Cap at 100 results
|
||||
|
||||
# Base query with vector similarity using cosine distance
|
||||
# Use configurable field names
|
||||
fields = settings.db_fields
|
||||
table_name = settings.table_names["embeddings"]
|
||||
|
||||
# Build query with proper parameter placeholders for Prisma
|
||||
param_count = 1
|
||||
query_params = [query_vector_str, vector_store_id]
|
||||
|
||||
base_query = f"""
|
||||
SELECT
|
||||
{fields.id_field},
|
||||
{fields.content_field},
|
||||
{fields.metadata_field},
|
||||
({fields.embedding_field} <=> ${param_count}::vector) as distance
|
||||
FROM {table_name}
|
||||
WHERE {fields.vector_store_id_field} = ${param_count + 1}
|
||||
"""
|
||||
param_count += 2
|
||||
|
||||
# Add metadata filters if provided
|
||||
filter_conditions = []
|
||||
|
||||
if request.filters:
|
||||
for key, value in request.filters.items():
|
||||
filter_conditions.append(f"{fields.metadata_field}->>${param_count} = ${param_count + 1}")
|
||||
query_params.extend([key, str(value)])
|
||||
param_count += 2
|
||||
|
||||
if filter_conditions:
|
||||
base_query += " AND " + " AND ".join(filter_conditions)
|
||||
|
||||
# Add ordering and limit
|
||||
final_query = base_query + f" ORDER BY distance ASC LIMIT {limit}"
|
||||
|
||||
# Execute the query
|
||||
results = await db.query_raw(final_query, *query_params)
|
||||
|
||||
# Convert results to SearchResult objects
|
||||
search_results = []
|
||||
for row in results:
|
||||
# Convert distance to similarity score (1 - normalized_distance)
|
||||
# Cosine distance ranges from 0 (identical) to 2 (opposite)
|
||||
similarity_score = max(0, 1 - (row['distance'] / 2))
|
||||
|
||||
# Extract filename from metadata or use a default
|
||||
metadata = row[fields.metadata_field] or {}
|
||||
filename = metadata.get('filename', 'document.txt')
|
||||
|
||||
content_chunks = [ContentChunk(type="text", text=row[fields.content_field])]
|
||||
|
||||
result = SearchResult(
|
||||
file_id=row[fields.id_field],
|
||||
filename=filename,
|
||||
score=similarity_score,
|
||||
attributes=metadata if request.return_metadata else None,
|
||||
content=content_chunks
|
||||
)
|
||||
search_results.append(result)
|
||||
|
||||
return VectorStoreSearchResponse(
|
||||
search_query=request.query,
|
||||
data=search_results,
|
||||
has_more=False, # TODO: Implement pagination
|
||||
next_page=None
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise HTTPException(status_code=500, detail=f"Search failed: {str(e)}")
|
||||
|
||||
|
||||
@app.post("/v1/vector_stores/{vector_store_id}/embeddings", response_model=EmbeddingResponse)
|
||||
async def create_embedding(
|
||||
vector_store_id: str,
|
||||
request: EmbeddingCreateRequest,
|
||||
api_key: str = Depends(get_api_key)
|
||||
):
|
||||
"""
|
||||
Add a single embedding to a vector store.
|
||||
"""
|
||||
try:
|
||||
# Check if vector store exists
|
||||
vector_store_table = settings.table_names["vector_stores"]
|
||||
vector_store_result = await db.query_raw(
|
||||
f"SELECT id FROM {vector_store_table} WHERE id = $1",
|
||||
vector_store_id
|
||||
)
|
||||
if not vector_store_result:
|
||||
raise HTTPException(status_code=404, detail="Vector store not found")
|
||||
|
||||
# Convert embedding to vector string format
|
||||
embedding_vector_str = "[" + ",".join(map(str, request.embedding)) + "]"
|
||||
|
||||
# Insert embedding using configurable field names
|
||||
fields = settings.db_fields
|
||||
table_name = settings.table_names["embeddings"]
|
||||
|
||||
result = await db.query_raw(
|
||||
f"""
|
||||
INSERT INTO {table_name} ({fields.id_field}, {fields.vector_store_id_field}, {fields.content_field},
|
||||
{fields.embedding_field}, {fields.metadata_field}, {fields.created_at_field})
|
||||
VALUES (gen_random_uuid(), $1, $2, $3::vector, $4, NOW())
|
||||
RETURNING {fields.id_field}, {fields.vector_store_id_field}, {fields.content_field},
|
||||
{fields.metadata_field}, EXTRACT(EPOCH FROM {fields.created_at_field})::bigint as created_at_timestamp
|
||||
""",
|
||||
vector_store_id,
|
||||
request.content,
|
||||
embedding_vector_str,
|
||||
request.metadata or {}
|
||||
)
|
||||
|
||||
if not result:
|
||||
raise HTTPException(status_code=500, detail="Failed to create embedding")
|
||||
|
||||
embedding = result[0]
|
||||
|
||||
# Update vector store statistics
|
||||
await db.query_raw(
|
||||
f"""
|
||||
UPDATE {vector_store_table}
|
||||
SET
|
||||
file_counts = jsonb_set(
|
||||
jsonb_set(
|
||||
COALESCE(file_counts, '{{"in_progress": 0, "completed": 0, "failed": 0, "cancelled": 0, "total": 0}}'::jsonb),
|
||||
'{{completed}}',
|
||||
(COALESCE(file_counts->>'completed', '0')::int + 1)::text::jsonb
|
||||
),
|
||||
'{{total}}',
|
||||
(COALESCE(file_counts->>'total', '0')::int + 1)::text::jsonb
|
||||
),
|
||||
usage_bytes = COALESCE(usage_bytes, 0) + LENGTH($2),
|
||||
last_active_at = NOW()
|
||||
WHERE id = $1
|
||||
""",
|
||||
vector_store_id,
|
||||
request.content
|
||||
)
|
||||
|
||||
return EmbeddingResponse(
|
||||
id=embedding[fields.id_field],
|
||||
vector_store_id=embedding[fields.vector_store_id_field],
|
||||
content=embedding[fields.content_field],
|
||||
metadata=embedding[fields.metadata_field],
|
||||
created_at=int(embedding["created_at_timestamp"])
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create embedding: {str(e)}")
|
||||
|
||||
|
||||
@app.post("/v1/vector_stores/{vector_store_id}/embeddings/batch", response_model=EmbeddingBatchCreateResponse)
|
||||
async def create_embeddings_batch(
|
||||
vector_store_id: str,
|
||||
request: EmbeddingBatchCreateRequest,
|
||||
api_key: str = Depends(get_api_key)
|
||||
):
|
||||
"""
|
||||
Add multiple embeddings to a vector store in batch.
|
||||
"""
|
||||
try:
|
||||
# Check if vector store exists
|
||||
vector_store_table = settings.table_names["vector_stores"]
|
||||
vector_store_result = await db.query_raw(
|
||||
f"SELECT id FROM {vector_store_table} WHERE id = $1",
|
||||
vector_store_id
|
||||
)
|
||||
if not vector_store_result:
|
||||
raise HTTPException(status_code=404, detail="Vector store not found")
|
||||
|
||||
if not request.embeddings:
|
||||
raise HTTPException(status_code=400, detail="No embeddings provided")
|
||||
|
||||
# Prepare batch insert
|
||||
fields = settings.db_fields
|
||||
table_name = settings.table_names["embeddings"]
|
||||
|
||||
# Build VALUES clause for batch insert
|
||||
values_clauses = []
|
||||
params = []
|
||||
param_count = 1
|
||||
|
||||
for embedding_req in request.embeddings:
|
||||
embedding_vector_str = "[" + ",".join(map(str, embedding_req.embedding)) + "]"
|
||||
values_clauses.append(f"(gen_random_uuid(), ${param_count}, ${param_count + 1}, ${param_count + 2}::vector, ${param_count + 3}, NOW())")
|
||||
params.extend([
|
||||
vector_store_id,
|
||||
embedding_req.content,
|
||||
embedding_vector_str,
|
||||
embedding_req.metadata or {}
|
||||
])
|
||||
param_count += 4
|
||||
|
||||
values_clause = ", ".join(values_clauses)
|
||||
|
||||
# Execute batch insert
|
||||
result = await db.query_raw(
|
||||
f"""
|
||||
INSERT INTO {table_name} ({fields.id_field}, {fields.vector_store_id_field}, {fields.content_field},
|
||||
{fields.embedding_field}, {fields.metadata_field}, {fields.created_at_field})
|
||||
VALUES {values_clause}
|
||||
RETURNING {fields.id_field}, {fields.vector_store_id_field}, {fields.content_field},
|
||||
{fields.metadata_field}, EXTRACT(EPOCH FROM {fields.created_at_field})::bigint as created_at_timestamp
|
||||
""",
|
||||
*params
|
||||
)
|
||||
|
||||
if not result:
|
||||
raise HTTPException(status_code=500, detail="Failed to create embeddings")
|
||||
|
||||
# Calculate total content length for usage bytes update
|
||||
total_content_length = sum(len(emb.content) for emb in request.embeddings)
|
||||
|
||||
# Update vector store statistics
|
||||
await db.query_raw(
|
||||
f"""
|
||||
UPDATE {vector_store_table}
|
||||
SET
|
||||
file_counts = jsonb_set(
|
||||
jsonb_set(
|
||||
COALESCE(file_counts, '{{"in_progress": 0, "completed": 0, "failed": 0, "cancelled": 0, "total": 0}}'::jsonb),
|
||||
'{{completed}}',
|
||||
(COALESCE(file_counts->>'completed', '0')::int + $2)::text::jsonb
|
||||
),
|
||||
'{{total}}',
|
||||
(COALESCE(file_counts->>'total', '0')::int + $2)::text::jsonb
|
||||
),
|
||||
usage_bytes = COALESCE(usage_bytes, 0) + $3,
|
||||
last_active_at = NOW()
|
||||
WHERE id = $1
|
||||
""",
|
||||
vector_store_id,
|
||||
len(request.embeddings),
|
||||
total_content_length
|
||||
)
|
||||
|
||||
# Convert results to response format
|
||||
embeddings = []
|
||||
for row in result:
|
||||
embeddings.append(EmbeddingResponse(
|
||||
id=row[fields.id_field],
|
||||
vector_store_id=row[fields.vector_store_id_field],
|
||||
content=row[fields.content_field],
|
||||
metadata=row[fields.metadata_field],
|
||||
created_at=int(row["created_at_timestamp"])
|
||||
))
|
||||
|
||||
return EmbeddingBatchCreateResponse(
|
||||
data=embeddings,
|
||||
created=int(time.time())
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create embeddings batch: {str(e)}")
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
"""Health check endpoint"""
|
||||
return {"status": "healthy", "timestamp": int(time.time())}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run("main:app", host=settings.host, port=settings.port, reload=True)
|
||||
Vendored
-92
@@ -1,92 +0,0 @@
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pydantic import BaseModel
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
class VectorStoreCreateRequest(BaseModel):
|
||||
name: str
|
||||
# ponytail: not part of upstream litellm-pgvector — added locally so
|
||||
# callers (litellm's vector_store_registry, scripts/ingest-memory.sh)
|
||||
# can pin a human-readable id instead of getting a random UUID back.
|
||||
# litellm-config.yaml's vector_store_registry addresses stores by a
|
||||
# fixed vector_store_id, which only works if creation can honor it.
|
||||
id: Optional[str] = None
|
||||
file_ids: Optional[List[str]] = None
|
||||
expires_after: Optional[Dict[str, Any]] = None
|
||||
chunking_strategy: Optional[Dict[str, Any]] = None
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class VectorStoreResponse(BaseModel):
|
||||
id: str
|
||||
object: str = "vector_store"
|
||||
created_at: int
|
||||
name: str
|
||||
usage_bytes: int
|
||||
file_counts: Dict[str, int]
|
||||
status: str
|
||||
expires_after: Optional[Dict[str, Any]] = None
|
||||
expires_at: Optional[int] = None
|
||||
last_active_at: Optional[int] = None
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class VectorStoreSearchRequest(BaseModel):
|
||||
query: str
|
||||
limit: Optional[int] = 20
|
||||
filters: Optional[Dict[str, Any]] = None
|
||||
return_metadata: Optional[bool] = True
|
||||
|
||||
|
||||
class ContentChunk(BaseModel):
|
||||
type: str = "text"
|
||||
text: str
|
||||
|
||||
|
||||
class SearchResult(BaseModel):
|
||||
file_id: str
|
||||
filename: str
|
||||
score: float
|
||||
attributes: Optional[Dict[str, Any]] = None
|
||||
content: List[ContentChunk]
|
||||
|
||||
|
||||
class VectorStoreSearchResponse(BaseModel):
|
||||
object: str = "vector_store.search_results.page"
|
||||
search_query: str
|
||||
data: List[SearchResult]
|
||||
has_more: bool = False
|
||||
next_page: Optional[str] = None
|
||||
|
||||
|
||||
class EmbeddingCreateRequest(BaseModel):
|
||||
content: str
|
||||
embedding: List[float]
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class EmbeddingResponse(BaseModel):
|
||||
id: str
|
||||
object: str = "embedding"
|
||||
vector_store_id: str
|
||||
content: str
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
created_at: int
|
||||
|
||||
|
||||
class EmbeddingBatchCreateRequest(BaseModel):
|
||||
embeddings: List[EmbeddingCreateRequest]
|
||||
|
||||
|
||||
class EmbeddingBatchCreateResponse(BaseModel):
|
||||
object: str = "embedding.batch"
|
||||
data: List[EmbeddingResponse]
|
||||
created: int
|
||||
|
||||
|
||||
class VectorStoreListResponse(BaseModel):
|
||||
object: str = "list"
|
||||
data: List[VectorStoreResponse]
|
||||
first_id: Optional[str] = None
|
||||
last_id: Optional[str] = None
|
||||
has_more: bool = False
|
||||
-45
@@ -1,45 +0,0 @@
|
||||
// This is your Prisma schema file,
|
||||
// learn more about it in the docs: https://pris.ly/d/prisma-schema
|
||||
|
||||
generator client {
|
||||
provider = "prisma-client-py"
|
||||
}
|
||||
|
||||
datasource db {
|
||||
provider = "postgresql"
|
||||
url = env("DATABASE_URL")
|
||||
}
|
||||
|
||||
model VectorStore {
|
||||
id String @id @default(cuid())
|
||||
name String
|
||||
file_counts Json?
|
||||
status String @default("completed")
|
||||
usage_bytes Int? @default(0)
|
||||
created_at DateTime @default(now())
|
||||
expires_after Json?
|
||||
expires_at DateTime?
|
||||
last_active_at DateTime?
|
||||
metadata Json?
|
||||
embeddings Embedding[]
|
||||
|
||||
@@map("vector_stores")
|
||||
}
|
||||
|
||||
model Embedding {
|
||||
id String @id @default(cuid())
|
||||
vector_store_id String
|
||||
content String
|
||||
// 768, not OpenAI's ada-002-sized 1536 — this stack's embedding_model is
|
||||
// nomic-embed-text-v1.5 (see docker-compose.yml's embedding-server and
|
||||
// litellm-config.yaml's local-embedding entry), confirmed 768-dim live
|
||||
// against /v1/embeddings. A push with the wrong size here makes every
|
||||
// insert fail on a pgvector dimension mismatch.
|
||||
embedding Unsupported("vector(768)")
|
||||
metadata Json?
|
||||
created_at DateTime @default(now())
|
||||
|
||||
vector_store VectorStore @relation(fields: [vector_store_id], references: [id], onDelete: Cascade)
|
||||
|
||||
@@map("embeddings")
|
||||
}
|
||||
-10
@@ -1,10 +0,0 @@
|
||||
fastapi==0.104.1
|
||||
uvicorn[standard]==0.24.0
|
||||
prisma==0.11.0
|
||||
python-dotenv==1.0.0
|
||||
pydantic>=2.5.0
|
||||
psycopg2-binary==2.9.7
|
||||
pgvector==0.2.4
|
||||
python-multipart==0.0.6
|
||||
litellm==1.74.3
|
||||
pydantic-settings==2.1.0
|
||||
Reference in New Issue
Block a user