feat(gateway): migrate LiteLLM to OmniRoute, drop the memory/knowledgebase feature
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
This commit is contained in:
+12
-12
@@ -1,24 +1,24 @@
|
||||
# Pointing a coding-agent CLI at this stack
|
||||
|
||||
This stack routes through the [AI proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (LiteLLM) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The proxy exposes:
|
||||
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
|
||||
|
||||
- **OpenAI-compatible**: `http://<ai-box>:4000/v1` (or `${LITELLM_PORT}` if you changed it in `.env`)
|
||||
- **Anthropic Messages API** (LiteLLM's own unified `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:4000`
|
||||
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1`
|
||||
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}`
|
||||
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the proxy as `qwen3.8-27b-local` — behind whichever wire format the client speaks.
|
||||
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
|
||||
|
||||
`<ai-box>` is this machine's LAN address, or `proxy.ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
|
||||
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (LiteLLM's Admin UI, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
|
||||
|
||||
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
|
||||
|
||||
## Claude Code CLI
|
||||
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the proxy's unified endpoint, not llama.cpp directly:
|
||||
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
|
||||
|
||||
```bash
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:4000
|
||||
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_API_PORT:-20129}
|
||||
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
|
||||
claude
|
||||
```
|
||||
@@ -32,7 +32,7 @@ Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in
|
||||
```toml
|
||||
[providers.openai]
|
||||
type = "openai"
|
||||
base_url = "http://<ai-box>:4000/v1"
|
||||
base_url = "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1"
|
||||
api_key = "<kimi-cli virtual key>"
|
||||
```
|
||||
|
||||
@@ -57,7 +57,7 @@ curl -fsSL https://opencode.ai/install | bash
|
||||
"npm": "@ai-sdk/openai-compatible",
|
||||
"name": "AI proxy (local)",
|
||||
"options": {
|
||||
"baseURL": "http://<ai-box>:4000/v1",
|
||||
"baseURL": "http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1",
|
||||
"apiKey": "<opencode-cli virtual key>"
|
||||
},
|
||||
"models": {
|
||||
@@ -84,8 +84,8 @@ Select the model with `aiproxy/qwen3.8-27b-local`.
|
||||
|
||||
| CLI | Wire format | Endpoint | Config |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:4000` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `opencode.json` provider block |
|
||||
| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}` | `ANTHROPIC_BASE_URL` env var |
|
||||
| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `config.toml` provider block |
|
||||
| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_API_PORT:-20129}/v1` | `opencode.json` provider block |
|
||||
|
||||
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
|
||||
|
||||
Reference in New Issue
Block a user