haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(omniroute): raise REQUEST_TIMEOUT_MS and enable --cache-reuse to stop non-ping SSE stream aborts

Browse Source 
STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).

Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
  as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
  margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
  so every request reprefilled its full prompt from scratch even when most
  of a conversation's prefix was unchanged. This is the actual fix for why
  compact-prompt prefill was slow enough to hit the timeout in the first
  place.

Documents the distinction and root cause in docs/research/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
This commit is contained in:
haylan
2026-09-10 10:12:02 +02:00
co-authored by Claude-Bot
parent 930e407053
commit e31647812a
3 changed files with 84 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+5
View File
@@ -71,6 +71,11 @@ OMNIROUTE_DASHBOARD_PORT=20128
# cancels it (which cancels the matching llama-server task too). 180s gives
# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
# Wait budget for the *first* SSE token specifically (distinct from the
# inter-chunk timeout above) — see
# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a
# contended, large-context prefill even after retries eat into the budget.
OMNIROUTE_REQUEST_TIMEOUT_MS=1800000
# Random values, filled in automatically by ./scripts/update.sh — leave
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
# change it there afterwards — this is only the first-boot value):