STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).
Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
so every request reprefilled its full prompt from scratch even when most
of a conversation's prefix was unchanged. This is the actual fix for why
compact-prompt prefill was slow enough to hit the timeout in the first
place.
Documents the distinction and root cause in docs/research/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
