feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
+12
-12
@@ -1,24 +1,24 @@
|
||||
# Pointing a coding-agent CLI at this stack
|
||||
|
||||
This stack routes through the [AI proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (LiteLLM) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The proxy exposes:
|
||||
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
|
||||
|
||||
- **OpenAI-compatible**: `http://<ai-box>:4000/v1` (or `${LITELLM_PORT}` if you changed it in `.env`)
|
||||
- **Anthropic Messages API** (LiteLLM's own unified `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:4000`
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}`
|
||||
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the proxy as `qwen3.8-27b-local` — behind whichever wire format the client speaks.
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
|
||||
|
||||
`<ai-box>` is this machine's LAN address, or `proxy.ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
|
||||
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (LiteLLM's Admin UI, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
|
||||
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
|
||||
|
||||
## Claude Code CLI
|
||||
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the proxy's unified endpoint, not llama.cpp directly:
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:4000
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_API_PORT:-20129}
|
||||
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
|
||||
claude
|
||||
```
|
||||
@@ -32,7 +32,7 @@ Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in
|
||||
```toml
|
||||
[providers.openai]
|
||||
type = "openai"
|
||||
base_url = "http://<ai-box>:4000/v1"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1"
|
||||
api_key = "<kimi-cli virtual key>"
|
||||
```
|
||||
|
||||
@@ -57,7 +57,7 @@ curl -fsSL https://opencode.ai/install | bash
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "AI proxy (local)",
|
||||
"options": {
|
||||
"baseURL": "http://<ai-box>:4000/v1",
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1",
|
||||
"apiKey": "<opencode-cli virtual key>"
|
||||
},
|
||||
"models": {
|
||||
@@ -84,8 +84,8 @@ Select the model with `aiproxy/qwen3.8-27b-local`.
|
||||
|
||||
| CLI | Wire format | Endpoint | Config |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:4000` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `opencode.json` provider block |
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `opencode.json` provider block |
|
||||
|
||||
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
# Knowledgebase, memory, and web search
|
||||
|
||||
Three gateway-level capabilities added on top of the [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9), so every client behind LiteLLM gets them — not just Open WebUI. See [issue #21](https://git.arthurerlich.de/haylan/LLM-Server/issues/21) for the rationale.
|
||||
|
||||
**Verified against a live deploy** — see [issue #24](https://git.arthurerlich.de/haylan/LLM-Server/issues/24), closed after smoke-testing found and fixed several bugs: a missing `api_key` in `vector_store_registry` (was silently falling through to the real `api.openai.com`), `litellm-pgvector`'s Prisma schema never actually being pushed to `pgvector-db` (now handled by `./scripts/update.sh`), a 1536- vs 768-dim vector column mismatch, and its create endpoint ignoring any caller-supplied store id (both fixed locally — see `vendor/litellm-pgvector/VENDORED.md`). `scripts/ingest-memory.sh` was also silently broken (posted chunks with no embedding attached) and has been fixed to embed via LiteLLM before inserting.
|
||||
|
||||
## Web search (SearXNG)
|
||||
|
||||
`litellm-config.yaml`'s `search_tools` block wires the LAN's SearXNG instance in as a **standalone REST endpoint**, not a model-callable tool — call it directly:
|
||||
|
||||
```bash
|
||||
curl http://<proxy>:4000/v1/search/searxng-search \
|
||||
-H "Authorization: Bearer <a virtual key>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "...", "max_results": 5}'
|
||||
```
|
||||
|
||||
Because this doesn't ask the model to emit a tool call, it sidesteps Qwen3.8-27B's known-flaky tool-calling (`docs/research/qwen3.8-27b-tool-calling.md`) entirely. Open WebUI's own web-search setting can point at this endpoint the same way.
|
||||
|
||||
Requires `SEARXNG_LAN_IP` set in `.env` so the `litellm` container can resolve `search.home` via `extra_hosts` — `./scripts/update.sh` resolves and fills this in automatically from the host's own DNS if it's blank (use a static DHCP reservation for `search.home` so it doesn't drift). Full research: `docs/research/litellm-searxng-search.md`.
|
||||
|
||||
## Knowledgebase (vector store / RAG)
|
||||
|
||||
LiteLLM's native knowledgebase feature has **no Qdrant backend** — the `qdrant` service in this stack only serves Open WebUI's own separate RAG/Memory feature and is unrelated to this. The only self-hosted path is [litellm-pgvector](https://github.com/BerriAI/litellm-pgvector), a companion service backed by its own Postgres+pgvector database (`pgvector-db`), which this stack now runs alongside `litellm`. Full research: `docs/research/litellm-knowledgebase.md`.
|
||||
|
||||
New pieces:
|
||||
|
||||
- **`embedding-server`** — a second llama.cpp instance (small footprint, `nomic-embed-text-v1.5`) serving `/v1/embeddings`. The chat model isn't embedding-trained and llama.cpp serves one model per process, so this can't just be a flag on `llama-server`.
|
||||
- **`pgvector-db`** — Postgres with the pgvector extension, separate from `litellm-db`.
|
||||
- **`litellm-pgvector`** — the connector service; no published image exists, so it's built from a vendored copy of the upstream repo at `vendor/litellm-pgvector/` (see that dir's `VENDORED.md`) — a remote git build context failed on the server's Docker/BuildKit setup.
|
||||
- `litellm-config.yaml`'s `local-embedding` model entry and `vector_store_registry` block, tying it together.
|
||||
|
||||
### First-time setup
|
||||
|
||||
`./scripts/update.sh` fetches the embedding model automatically (skips it if already downloaded). To do it by hand instead:
|
||||
|
||||
```bash
|
||||
docker compose --profile tools run --rm downloader-embedding # fetch the embedding model
|
||||
docker compose up -d embedding-server pgvector-db litellm-pgvector
|
||||
```
|
||||
|
||||
`./scripts/update.sh` mints `LITELLM_PGVECTOR_EMBEDDING_KEY` automatically (a `litellm-pgvector` virtual key via LiteLLM's own API) if it's blank — it calls back into `litellm` for embeddings, same as any other workload. See `docs/proxy-key-onboarding.md` if a mint fails and it needs doing by hand.
|
||||
|
||||
### Loading memory into it
|
||||
|
||||
`data/memory.md` and `data/claude-legacy-memory.md` — Claude-memory-style fact files — get loaded via:
|
||||
|
||||
```bash
|
||||
./scripts/ingest-memory.sh
|
||||
```
|
||||
|
||||
One chunk per fact/paragraph line, tagged with `source`/`section` metadata. Re-run after editing either file (see the script's header comment for the no-dedup caveat).
|
||||
|
||||
### Querying it
|
||||
|
||||
Via the OpenAI Assistants-style `file_search` tool on a chat completion:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "qwen3.8-27b-local",
|
||||
"messages": [...],
|
||||
"tools": [{"type": "file_search", "vector_store_ids": ["memory-and-notes"]}]
|
||||
}
|
||||
```
|
||||
|
||||
or directly: `POST /v1/vector_stores/memory-and-notes/search` with `{"query": "..."}`.
|
||||
@@ -15,13 +15,11 @@ The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and
|
||||
|
||||
If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture (the raw API would need its own auth in front of it).
|
||||
|
||||
## The AI proxy (LiteLLM) — `proxy.ai.home` / `proxy.ai.haylan.ch`
|
||||
## The AI gateway (OmniRoute) — `proxy.ai.home` / `proxy.ai.haylan.ch`
|
||||
|
||||
Once the gateway from [issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) is deployed, it gets its own hostnames, same NPM pattern as Open WebUI above:
|
||||
As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute — same NPM pattern as Open WebUI above, but a cleaner split than LiteLLM's ever was:
|
||||
|
||||
- **`proxy.ai.home`** — internal only, fronts the full LiteLLM port (API + Admin UI).
|
||||
- **`proxy.ai.haylan.ch`** — external, via the DMZ/NPM. Fronts only the inference API paths.
|
||||
- **`proxy.ai.home`** and **`proxy.ai.haylan.ch`** both point only at `${OMNIROUTE_API_PORT:-20129}` — the API port.
|
||||
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
|
||||
|
||||
**Every proxy call already requires a valid virtual key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.
|
||||
|
||||
**LiteLLM's Admin UI (`/ui`) stays LAN-only**, same reasoning as llama.cpp's raw API: it manages every workload's keys and budgets, so it doesn't belong on the public internet. LiteLLM serves `/ui` on the same port as its API by default, so `proxy.ai.haylan.ch`'s NPM Proxy Host needs an explicit rule denying the `/ui` path (a "Deny" custom location, same UI as the "Advanced" tab used for other NPM hosts) — `proxy.ai.home` has no such restriction and reaches both the API and the Admin UI.
|
||||
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.
|
||||
|
||||
@@ -1,25 +1,17 @@
|
||||
# Onboarding a workload onto the AI proxy
|
||||
# Onboarding a workload onto the AI gateway
|
||||
|
||||
How to issue a new per-workload API key against the LiteLLM proxy (see [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
|
||||
How to issue a new per-workload API key against the OmniRoute gateway (see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) — the LiteLLM → OmniRoute migration; original gateway rationale in [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
|
||||
|
||||
`OPENWEBUI_LITELLM_KEY` and `LITELLM_PGVECTOR_EMBEDDING_KEY` — the two keys this stack's own services need — are minted automatically by `./scripts/update.sh` via the same API `curl` shows below; the steps here are for any other workload, or for those two if the automatic mint ever fails.
|
||||
`OPENWEBUI_OMNIROUTE_KEY` — the one key this stack's own services need — has no scripted mint yet: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key — including that one — by hand for now, via the dashboard steps below.
|
||||
|
||||
## Create the key
|
||||
|
||||
1. Log into LiteLLM's Admin UI (`/ui` on the proxy's deployed URL).
|
||||
2. Create a new virtual key ("Keys" → "Create Key").
|
||||
3. Name it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `openwebui`. This name is the ledger: LiteLLM's dashboard lists keys by name, so there's no separate tracking doc to keep in sync — name it clearly and the Usage tab tells you the rest (spend, last used, etc.).
|
||||
4. Leave budget and rate limits unset (unlimited) by default. This is a shadow-cost estimate for fun, not real accounting or resource protection — see `docs/research/proxy-shadow-pricing.md`. Only set a budget if a specific workload turns out to need a tripwire.
|
||||
1. Log into the omniroute dashboard (`http://<host>:${OMNIROUTE_DASHBOARD_PORT:-20128}` — LAN/host-only, never published publicly, see `docker-compose.yml`'s `omniroute` service).
|
||||
2. "Keys" → "Create API key".
|
||||
3. Label it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `openwebui`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync.
|
||||
4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.
|
||||
|
||||
Or the same thing over the API (what `update.sh` does):
|
||||
|
||||
```bash
|
||||
curl -sf -X POST "http://<proxy>:4000/key/generate" \
|
||||
-H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key_alias": "<workload>-<purpose>"}'
|
||||
# -> {"key": "sk-...", ...}
|
||||
```
|
||||
Once `POST /api/keys`'s session-auth flow is worked out (issue #37), the equivalent `curl` here can replace this manual step, the way `update.sh` used to automate LiteLLM's `/key/generate`.
|
||||
|
||||
## Hand it to the workload
|
||||
|
||||
@@ -27,7 +19,7 @@ Drop the key into that workload's own `.env` (or equivalent config) — never in
|
||||
|
||||
## Retiring or rotating a key
|
||||
|
||||
No scheduled rotation. Revoke the key by hand in the Admin UI ("Keys" → delete) only when:
|
||||
No scheduled rotation. Revoke the key by hand in the dashboard ("Keys" → delete) only when:
|
||||
- the workload is retired, or
|
||||
- the key is suspected leaked/compromised.
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Request priority on the AI proxy
|
||||
|
||||
One local model instance (llama.cpp on the single R9700) serves every workload through the LiteLLM proxy ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
|
||||
**Stale as of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (LiteLLM → OmniRoute migration)** — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
|
||||
|
||||
One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
|
||||
|
||||
## Tiers
|
||||
|
||||
|
||||
Reference in New Issue
Block a user