haylanandClaude-Bot
472e3a4738
LiteLLM -> OmniRoute (issue #31, wayfinder map + research tickets #32-37): replace the litellm/litellm-db services with omniroute, split-port mode (API_PORT published/reverse-proxied, DASHBOARD_PORT never published - tighter than litellm's old /ui NPM path-deny rule), 5 new secrets in place of LITELLM_MASTER_KEY/LITELLM_SALT_KEY, llama-server/searxng registered as omniroute providers post-boot (no static config.yaml equivalent). No scripted per-workload key minting yet - omniroute's POST /api/keys needs a dashboard session, not a static bearer key - so OPENWEBUI_OMNIROUTE_KEY is a manual step for now (docs/proxy-key-onboarding.md). Caveat carried into the map and README: OmniRoute's own docs (docs/security/STEALTH_GUIDE.md, MITM-TPROXY-DECRYPT.md, PUBLIC_CREDS.md on its release/v3.8.51 branch) describe shipped features for AI-provider client-detection evasion, system-wide HTTPS interception via a locally installed root CA, and hiding credentials from secret scanners. Proceeding anyway was an explicit, informed user decision. Also drops the gateway-level memory/knowledgebase feature entirely (user: "I don't need it") - litellm-pgvector, pgvector-db, embedding-server, scripts/ingest-memory.sh, vendor/litellm-pgvector/, docs/memory- knowledgebase.md. Open WebUI's own qdrant-backed memory/RAG is unrelated and untouched. litellm-config.yaml deleted (was kept as a rollback reference, but there's no rollback path to a feature being deliberately removed). Not yet verified against real hardware - see issue #31's open tickets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
29 lines
2.8 KiB
Markdown
29 lines
2.8 KiB
Markdown
# Request priority on the AI proxy
|
|
|
|
**Stale as of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (LiteLLM → OmniRoute migration)** — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
|
|
|
|
One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
|
|
|
|
## Tiers
|
|
|
|
Two tiers, assigned per workload's virtual key (per `docs/proxy-key-onboarding.md`):
|
|
|
|
- **High priority** (interactive — someone's waiting): Open WebUI chat, coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review.
|
|
- **Low priority** (batch — nobody's watching a spinner): Paperless OCR/tagging, Nextcloud Memories face-recognition, AI watermark removal.
|
|
|
|
## Mechanism
|
|
|
|
Use LiteLLM's built-in request-prioritization scheduler ([docs](https://docs.litellm.ai/docs/scheduler)) — callers pass a `priority` value, LiteLLM's router queues and dispatches by priority. Per `docs/research/proxy-tool-choice.md`, this feature is real but **beta**: there's a known, closed-as-not-planned bug where the `priority` field can leak into the provider request. Treat it as unproven, not settled:
|
|
|
|
- **#14 (compose authoring) must smoke-test the scheduler against llama.cpp specifically** before this is relied on — confirm the `priority` field doesn't leak into llama.cpp's request and actually reorders dispatch under load.
|
|
- **If it's broken in practice**, fall back to a lightweight queuing shim in front of the proxy (a small sidecar) rather than reworking the gateway tool choice. Don't build this shim speculatively — only if the smoke test fails.
|
|
- Single-instance deployment (this stack) doesn't need Redis for virtual keys/spend, but LiteLLM's scheduler does use Redis for cross-instance state — if the scheduler needs it even single-instance, add a `redis` service to docker-compose.yml at that point, not before.
|
|
|
|
## Timeout behavior
|
|
|
|
A request queued too long (burst of batch jobs, or the model just being slow) times out and returns an error to the caller — no indefinite waiting. Use LiteLLM's default request timeout unless testing shows it needs tuning.
|
|
|
|
## Lazytainer interaction
|
|
|
|
No new risk: queueing happens inside LiteLLM *before* it dispatches to llama.cpp. Lazytainer watches actual traffic reaching the llama.cpp container, so it still sees the first dispatched request and wakes the container normally — priority ordering only changes which queued request gets dispatched first, not whether Lazytainer sees traffic.
|