haylanandClaude-Bot
3bbda098b3
No longer needed - every client is a coding CLI behind the OmniRoute
gateway, not a chat UI. Drops the open-webui and qdrant services,
WEBUI_PORT/OPENWEBUI_OMNIROUTE_KEY env vars, and the openwebui-data/
qdrant-data volumes. Qdrant only ever served Open WebUI's own built-in
memory/RAG (unrelated to the gateway-level knowledgebase removed in
472e3a4), so it goes too rather than sit unused.
Docs updated: README, docs/network-access.md (ai.home/ai.haylan.ch
section was entirely about Open WebUI, rewritten around the gateway),
docs/proxy-key-onboarding.md, docs/proxy-request-priority.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VPZ6TogJiYxG8E4EQBB197
2.8 KiB
Request priority on the AI proxy
Stale as of issue #31 (LiteLLM → OmniRoute migration) — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway (issue #9). Interactive usage shouldn't get stuck behind a batch job.
Tiers
Two tiers, assigned per workload's virtual key (per docs/proxy-key-onboarding.md):
- High priority (interactive — someone's waiting): coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review.
- Low priority (batch — nobody's watching a spinner): Paperless OCR/tagging, Nextcloud Memories face-recognition, AI watermark removal.
Mechanism
Use LiteLLM's built-in request-prioritization scheduler (docs) — callers pass a priority value, LiteLLM's router queues and dispatches by priority. Per docs/research/proxy-tool-choice.md, this feature is real but beta: there's a known, closed-as-not-planned bug where the priority field can leak into the provider request. Treat it as unproven, not settled:
- #14 (compose authoring) must smoke-test the scheduler against llama.cpp specifically before this is relied on — confirm the
priorityfield doesn't leak into llama.cpp's request and actually reorders dispatch under load. - If it's broken in practice, fall back to a lightweight queuing shim in front of the proxy (a small sidecar) rather than reworking the gateway tool choice. Don't build this shim speculatively — only if the smoke test fails.
- Single-instance deployment (this stack) doesn't need Redis for virtual keys/spend, but LiteLLM's scheduler does use Redis for cross-instance state — if the scheduler needs it even single-instance, add a
redisservice to docker-compose.yml at that point, not before.
Timeout behavior
A request queued too long (burst of batch jobs, or the model just being slow) times out and returns an error to the caller — no indefinite waiting. Use LiteLLM's default request timeout unless testing shows it needs tuning.
Lazytainer interaction
No new risk: queueing happens inside LiteLLM before it dispatches to llama.cpp. Lazytainer watches actual traffic reaching the llama.cpp container, so it still sees the first dispatched request and wakes the container normally — priority ordering only changes which queued request gets dispatched first, not whether Lazytainer sees traffic.