From e31647812add389f486d571f762fe83a57870702 Mon Sep 17 00:00:00 2001 From: Haylan Date: Thu, 10 Sep 2026 10:10:40 +0200 Subject: [PATCH] fix(omniroute): raise REQUEST_TIMEOUT_MS and enable --cache-reuse to stop non-ping SSE stream aborts STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended llama-server prefill room to produce a first token, but qwen-code sessions kept hitting "Stream produced no non-ping SSE event within 95000ms" the very next morning. Per OmniRoute's own docs, that's the wrong timer: the first non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min, computed as remaining budget after retries/cooldowns), not STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming has already started). Two changes: - Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety margin, not the root-cause fix. - Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured, so every request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged. This is the actual fix for why compact-prompt prefill was slow enough to hit the timeout in the first place. Documents the distinction and root cause in docs/research/. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc --- .env.example | 5 ++ docker-compose.yml | 19 ++++++ .../omniroute-non-ping-sse-stream-timeout.md | 60 +++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100644 docs/research/omniroute-non-ping-sse-stream-timeout.md diff --git a/.env.example b/.env.example index df4c744..3b30a16 100644 --- a/.env.example +++ b/.env.example @@ -71,6 +71,11 @@ OMNIROUTE_DASHBOARD_PORT=20128 # cancels it (which cancels the matching llama-server task too). 180s gives # contended prefill (see LLAMA_PARALLEL above) room to produce a first token. OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000 +# Wait budget for the *first* SSE token specifically (distinct from the +# inter-chunk timeout above) — see +# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a +# contended, large-context prefill even after retries eat into the budget. +OMNIROUTE_REQUEST_TIMEOUT_MS=1800000 # Random values, filled in automatically by ./scripts/update.sh — leave # blank. Bootstrap dashboard admin password (log in at the dashboard port, # change it there afterwards — this is only the first-boot value): diff --git a/docker-compose.yml b/docker-compose.yml index 2c6b3d8..6f4307c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -30,7 +30,16 @@ services: --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 + --cache-reuse 256 --jinja + # --cache-reuse 256: reuse cached KV for any matching prompt chunk of at + # least 256 tokens (KV-shift, no reprocessing) instead of reprefilling + # from scratch every request. Directly targets the actual root cause + # behind the OmniRoute non-ping SSE timeout, not just the symptom — see + # docs/research/omniroute-non-ping-sse-stream-timeout.md. Pairs with + # OmniRoute's promptCacheAffinityEnabled (dashboard default), which keeps + # a conversation's requests pinned to the same slot so there's a matching + # prefix to reuse. # No published host port: llama-server is reached only via the omniroute # gateway on the ai-stack docker network now — see issue #15. Its # unauthenticated API no longer needs to be LAN-reachable directly. @@ -283,6 +292,16 @@ services: # LLAMA_PARALLEL above for the other half of this fix). Raised here so # it's tracked in git instead of a dashboard-only setting. - STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000} + # Different timer than STREAM_IDLE_TIMEOUT_MS above — that one only + # bounds gaps *between* SSE chunks once streaming has started. + # REQUEST_TIMEOUT_MS bounds the wait for the *first* non-ping SSE + # event, and it's what was still firing ("Stream produced no non-ping + # SSE event within 95000ms") the morning after the timeout above was + # raised — see docs/research/omniroute-non-ping-sse-stream-timeout.md. + # Default 600000 (10 min) per OmniRoute's own docs, but the effective + # deadline is remaining budget after retries/cooldowns eat into it, not + # a flat timer, so raised well past the default for headroom. + - REQUEST_TIMEOUT_MS=${OMNIROUTE_REQUEST_TIMEOUT_MS:-1800000} # Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge # network can't resolve search.home on its own. extra_hosts: diff --git a/docs/research/omniroute-non-ping-sse-stream-timeout.md b/docs/research/omniroute-non-ping-sse-stream-timeout.md new file mode 100644 index 0000000..2f0cbef --- /dev/null +++ b/docs/research/omniroute-non-ping-sse-stream-timeout.md @@ -0,0 +1,60 @@ +# OmniRoute's "non-ping SSE" first-token deadline — a different timer than `STREAM_IDLE_TIMEOUT_MS` + +**Date:** 2026-09-10 + +`STREAM_IDLE_TIMEOUT_MS` was raised to 180000 on 2026-09-09 (see docker-compose.yml's `omniroute` +service) specifically to give contended `llama-server` prefill room to produce a first token. It didn't +work: the very next morning, qwen-code sessions against `qwen3.8-27b-local` still hit repeated + +``` +Stream produced no non-ping SSE event within 95000ms +``` + +(and once at 115000ms) — both well under the 180s the compose fix set, and well under the connection's +own `providerSpecificData.timeoutMs: 1200000` (confirmed live via `GET /api/providers/`). Neither of +those settings bounds this failure. + +## Root cause + +Per OmniRoute's own docs (`docs/reference/ENVIRONMENT.md`, "Timeout Settings" section) and a maintainer +reply in [diegosouzapw/OmniRoute#10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602): + +| Variable | Default | Governs | +|---|---|---| +| `REQUEST_TIMEOUT_MS` | 600000 (10 min) | Overall upstream request budget. **The first non-ping SSE event's deadline inherits this one.** | +| `STREAM_IDLE_TIMEOUT_MS` | 120000 (2 min) | Max gap between *successive* SSE chunks once streaming has already started — does not govern the wait for the first chunk. | +| `STREAM_PING_INTERVAL_MS` | 30000 (30s) | How often OmniRoute emits its own keepalive pings on the stream — these explicitly do not count as "non-ping" events, so they can't rescue a request against the first deadline. | + +So the 2026-09-09 fix tuned the wrong timer for this failure mode: `STREAM_IDLE_TIMEOUT_MS` only matters +once `llama-server` has already emitted something. The "no token at all yet" case — exactly what a large +compact-prompt prefill on a contended local model produces — is bounded by `REQUEST_TIMEOUT_MS` instead. + +The observed 95000ms/115000ms figures are also *not* `REQUEST_TIMEOUT_MS`'s raw 600000ms default: OmniRoute +computes the first-event deadline as **remaining budget**, not a flat timer — `REQUEST_TIMEOUT_MS` minus +time already spent in OmniRoute's own request-queue/retry/cooldown cycle (`requestRetry: 3`, +`connectionCooldown.apikey.baseCooldownMs`, provider breaker) before the request was actually dispatched +to `llama-server`. Confirmed live via `GET /api/settings` → `resilienceSettings` on this deployment. Most +of the 10-minute default budget was being burned by retries before the final attempt even started. + +## Fix + +Set `REQUEST_TIMEOUT_MS` explicitly, generously — applied in docker-compose.yml as +`OMNIROUTE_REQUEST_TIMEOUT_MS` (default 1800000 / 30 min), same pattern as +`OMNIROUTE_STREAM_IDLE_TIMEOUT_MS`. This doesn't replace the 2026-09-09 `STREAM_IDLE_TIMEOUT_MS` fix — +that one still matters for mid-stream stalls after generation has started — it addresses the separate +"nothing has arrived yet" case that fix didn't cover. + +Raising `REQUEST_TIMEOUT_MS` buys headroom; it doesn't address *why* prefill on a 50K+ token compact +prompt can take that long in the first place. `llama-server` had no `--cache-reuse` flag set — every +request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged +from the previous turn. Added `--cache-reuse 256` (docker-compose.yml) so llama.cpp reuses cached KV for +any matching ≥256-token chunk via KV-shift instead of reprocessing it, which is the actual fix for +compact-prompt prefill time — the timeout bump above is a safety margin around it, not a substitute. + +## Sources + +- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `docs/reference/ENVIRONMENT.md` + ("Timeout Settings"), [Discussion #10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602) +- Live `GET /api/providers/`, `GET /api/settings`, `GET /api/resilience` against this + deployment's OmniRoute instance (2026-09-10) +- [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the related-but-distinct 30s semaphore/429 investigation