haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

How should the proxy queue and prioritize concurrent requests across workloads? #16

New Issue
Closed
opened 2026-08-25 04:41:13 +00:00 by haylan · 1 comment
haylan commented 2026-08-25 04:41:13 +00:00
Owner
Copy Link
Copy Source

Part of #9

Question

Only one local model instance/GPU serves every workload through this proxy — interactive usage (Open WebUI chat, coding CLIs) can collide with batch/background jobs (Paperless OCR/tagging, Nextcloud Memories, watermark removal) firing at the same time. Per grilling on #9: a priority queue is wanted, not plain FIFO, so interactive requests aren't stuck behind batch work.

Resolve: what priority tiers exist and which workload maps to which tier (e.g. interactive > batch); whether the tool chosen in #10 has native request queuing/priority support or a separate queuing component is needed in front of it; what happens to a request that's queued too long (timeout, reject, notify); how this interacts with Lazytainer's idle-suspend on the llama.cpp container (a queued request shouldn't get dropped because the container just spun down).

Part of #9 ## Question Only one local model instance/GPU serves every workload through this proxy — interactive usage (Open WebUI chat, coding CLIs) can collide with batch/background jobs (Paperless OCR/tagging, Nextcloud Memories, watermark removal) firing at the same time. Per grilling on #9: a **priority queue** is wanted, not plain FIFO, so interactive requests aren't stuck behind batch work. Resolve: what priority tiers exist and which workload maps to which tier (e.g. interactive > batch); whether the tool chosen in #10 has native request queuing/priority support or a separate queuing component is needed in front of it; what happens to a request that's queued too long (timeout, reject, notify); how this interacts with Lazytainer's idle-suspend on the llama.cpp container (a queued request shouldn't get dropped because the container just spun down).
haylan added the wayfinder:grilling label 2026-08-25 04:41:13 +00:00
haylan added a new dependency 2026-08-25 04:41:14 +00:00
#14 Author the docker-compose service for the chosen proxy
haylan self-assigned this 2026-08-25 05:05:34 +00:00
haylan commented 2026-08-25 05:07:28 +00:00
Author
Owner
Copy Link
Copy Source

Resolved via grilling.

Two tiers, assigned per workload's virtual key: high priority (Open WebUI chat, coding CLIs, Gitea code review — interactive) vs. low priority (Paperless OCR/tagging, Nextcloud Memories, watermark removal — batch). Mechanism: LiteLLM's built-in scheduler as the default plan, but it's beta (known closed-as-not-planned priority-leak bug) — #14 must smoke-test it against llama.cpp before relying on it; fall back to a lightweight queuing shim only if that fails, not built speculatively. Queued-too-long requests time out and error back to the caller, no indefinite waits. Confirmed no new Lazytainer risk: queueing happens inside LiteLLM before dispatch, so Lazytainer still sees the first dispatched request and wakes the container as it always has.

Documented in docs/proxy-request-priority.md, committed on main.

Resolved via grilling. Two tiers, assigned per workload's virtual key: high priority (Open WebUI chat, coding CLIs, Gitea code review — interactive) vs. low priority (Paperless OCR/tagging, Nextcloud Memories, watermark removal — batch). Mechanism: LiteLLM's built-in scheduler as the default plan, but it's beta (known closed-as-not-planned priority-leak bug) — #14 must smoke-test it against llama.cpp before relying on it; fall back to a lightweight queuing shim only if that fails, not built speculatively. Queued-too-long requests time out and error back to the caller, no indefinite waits. Confirmed no new Lazytainer risk: queueing happens inside LiteLLM before dispatch, so Lazytainer still sees the first dispatched request and wakes the container as it always has. Documented in docs/proxy-request-priority.md, committed on main.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:grilling
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Blocks
#14 Author the docker-compose service for the chosen proxy
haylan/LLM-Server
Depends on
#10 Which self-hosted AI gateway/proxy tool fits this effort's needs?
haylan/LLM-Server
Reference: haylan/LLM-Server#16