LLM-Server/docs/research/omniroute-non-ping-sse-stream-timeout.md

OmniRoute's "non-ping SSE" first-token deadline — a different timer than STREAM_IDLE_TIMEOUT_MS

Date: 2026-09-10

STREAM_IDLE_TIMEOUT_MS was raised to 180000 on 2026-09-09 (see docker-compose.yml's omniroute service) specifically to give contended llama-server prefill room to produce a first token. It didn't work: the very next morning, qwen-code sessions against qwen3.8-27b-local still hit repeated

Stream produced no non-ping SSE event within 95000ms

(and once at 115000ms) — both well under the 180s the compose fix set, and well under the connection's own providerSpecificData.timeoutMs: 1200000 (confirmed live via GET /api/providers/<id>). Neither of those settings bounds this failure.

Root cause

Per OmniRoute's own docs (docs/reference/ENVIRONMENT.md, "Timeout Settings" section) and a maintainer reply in diegosouzapw/OmniRoute#10602:

Variable	Default	Governs
REQUEST_TIMEOUT_MS	600000 (10 min)	Overall upstream request budget. The first non-ping SSE event's deadline inherits this one.
STREAM_IDLE_TIMEOUT_MS	120000 (2 min)	Max gap between successive SSE chunks once streaming has already started — does not govern the wait for the first chunk.
STREAM_PING_INTERVAL_MS	30000 (30s)	How often OmniRoute emits its own keepalive pings on the stream — these explicitly do not count as "non-ping" events, so they can't rescue a request against the first deadline.

So the 2026-09-09 fix tuned the wrong timer for this failure mode: STREAM_IDLE_TIMEOUT_MS only matters once llama-server has already emitted something. The "no token at all yet" case — exactly what a large compact-prompt prefill on a contended local model produces — is bounded by REQUEST_TIMEOUT_MS instead.

The observed 95000ms/115000ms figures are also not REQUEST_TIMEOUT_MS's raw 600000ms default: OmniRoute computes the first-event deadline as remaining budget, not a flat timer — REQUEST_TIMEOUT_MS minus time already spent in OmniRoute's own request-queue/retry/cooldown cycle (requestRetry: 3, connectionCooldown.apikey.baseCooldownMs, provider breaker) before the request was actually dispatched to llama-server. Confirmed live via GET /api/settings → resilienceSettings on this deployment. Most of the 10-minute default budget was being burned by retries before the final attempt even started.

Fix

Set REQUEST_TIMEOUT_MS explicitly, generously — applied in docker-compose.yml as OMNIROUTE_REQUEST_TIMEOUT_MS (default 1800000 / 30 min), same pattern as OMNIROUTE_STREAM_IDLE_TIMEOUT_MS. This doesn't replace the 2026-09-09 STREAM_IDLE_TIMEOUT_MS fix — that one still matters for mid-stream stalls after generation has started — it addresses the separate "nothing has arrived yet" case that fix didn't cover.

Raising REQUEST_TIMEOUT_MS buys headroom; it doesn't address why prefill on a 50K+ token compact prompt can take that long in the first place. llama-server had no --cache-reuse flag set — every request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged from the previous turn. Added --cache-reuse 256 (docker-compose.yml) so llama.cpp reuses cached KV for any matching ≥256-token chunk via KV-shift instead of reprocessing it, which is the actual fix for compact-prompt prefill time — the timeout bump above is a safety margin around it, not a substitute.

Sources

• diegosouzapw/OmniRoute — docs/reference/ENVIRONMENT.md ("Timeout Settings"), Discussion #10602
• Live GET /api/providers/<connectionId>, GET /api/settings, GET /api/resilience against this deployment's OmniRoute instance (2026-09-10)
• docs/research/omniroute-account-semaphore-timeout.md — the related-but-distinct 30s semaphore/429 investigation