fix(omniroute): raise stream-idle timeout to 180s

Contended prefill (see LLAMA_PARALLEL) can outrun the ~95s dashboard
value before first token, causing OmniRoute to cancel still-working
requests. Track it in git via STREAM_IDLE_TIMEOUT_MS instead of a
dashboard-only setting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-05 11:29:12 +02:00
co-authored by Claude-Bot
parent 23e90fe8fb
commit 633292b291
2 changed files with 12 additions and 0 deletions
+8
View File
@@ -92,6 +92,14 @@ services:
# Docker guide — every client here is a coding CLI, which needs the
# larger figure the guide recommends. Paired with mem_limit below.
- OMNIROUTE_MEMORY_MB=8192
# Default 300000 (5 min) per OmniRoute's own docs, but this deployment
# had it dialed down elsewhere (dashboard) to ~95s — too tight for a
# contended local llama-server: large-context prefill under multiple
# concurrent slots can outrun that before the first SSE token arrives,
# so OmniRoute cancels a request that was actually still working (see
# LLAMA_PARALLEL above for the other half of this fix). Raised here so
# it's tracked in git instead of a dashboard-only setting.
- STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
# Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
# network can't resolve search.home on its own.
extra_hosts: