haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(omniroute): raise stream-idle timeout to 180s

Browse Source 
Contended prefill (see LLAMA_PARALLEL) can outrun the ~95s dashboard
value before first token, causing OmniRoute to cancel still-working
requests. Track it in git via STREAM_IDLE_TIMEOUT_MS instead of a
dashboard-only setting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-09-05 11:29:12 +02:00
co-authored by Claude-Bot
parent 23e90fe8fb
commit 633292b291
2 changed files with 12 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+4
View File
@@ -61,6 +61,10 @@ SEARXNG_LAN_IP=
# published at all — see docker-compose.yml's omniroute service comment.
OMNIROUTE_API_PORT=20129
OMNIROUTE_DASHBOARD_PORT=20128
# SSE inactivity timeout before OmniRoute gives up on a streaming request and
# cancels it (which cancels the matching llama-server task too). 180s gives
# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
# Random values, filled in automatically by ./scripts/update.sh — leave
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
# change it there afterwards — this is only the first-boot value):