fix(omniroute): raise REQUEST_TIMEOUT_MS and enable --cache-reuse to stop non-ping SSE stream aborts
STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended llama-server prefill room to produce a first token, but qwen-code sessions kept hitting "Stream produced no non-ping SSE event within 95000ms" the very next morning. Per OmniRoute's own docs, that's the wrong timer: the first non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min, computed as remaining budget after retries/cooldowns), not STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming has already started). Two changes: - Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety margin, not the root-cause fix. - Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured, so every request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged. This is the actual fix for why compact-prompt prefill was slow enough to hit the timeout in the first place. Documents the distinction and root cause in docs/research/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
This commit is contained in:
@@ -71,6 +71,11 @@ OMNIROUTE_DASHBOARD_PORT=20128
|
||||
# cancels it (which cancels the matching llama-server task too). 180s gives
|
||||
# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
|
||||
OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
|
||||
# Wait budget for the *first* SSE token specifically (distinct from the
|
||||
# inter-chunk timeout above) — see
|
||||
# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a
|
||||
# contended, large-context prefill even after retries eat into the budget.
|
||||
OMNIROUTE_REQUEST_TIMEOUT_MS=1800000
|
||||
# Random values, filled in automatically by ./scripts/update.sh — leave
|
||||
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
|
||||
# change it there afterwards — this is only the first-boot value):
|
||||
|
||||
@@ -30,7 +30,16 @@ services:
|
||||
--flash-attn on
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
--cache-reuse 256
|
||||
--jinja
|
||||
# --cache-reuse 256: reuse cached KV for any matching prompt chunk of at
|
||||
# least 256 tokens (KV-shift, no reprocessing) instead of reprefilling
|
||||
# from scratch every request. Directly targets the actual root cause
|
||||
# behind the OmniRoute non-ping SSE timeout, not just the symptom — see
|
||||
# docs/research/omniroute-non-ping-sse-stream-timeout.md. Pairs with
|
||||
# OmniRoute's promptCacheAffinityEnabled (dashboard default), which keeps
|
||||
# a conversation's requests pinned to the same slot so there's a matching
|
||||
# prefix to reuse.
|
||||
# No published host port: llama-server is reached only via the omniroute
|
||||
# gateway on the ai-stack docker network now — see issue #15. Its
|
||||
# unauthenticated API no longer needs to be LAN-reachable directly.
|
||||
@@ -283,6 +292,16 @@ services:
|
||||
# LLAMA_PARALLEL above for the other half of this fix). Raised here so
|
||||
# it's tracked in git instead of a dashboard-only setting.
|
||||
- STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
|
||||
# Different timer than STREAM_IDLE_TIMEOUT_MS above — that one only
|
||||
# bounds gaps *between* SSE chunks once streaming has started.
|
||||
# REQUEST_TIMEOUT_MS bounds the wait for the *first* non-ping SSE
|
||||
# event, and it's what was still firing ("Stream produced no non-ping
|
||||
# SSE event within 95000ms") the morning after the timeout above was
|
||||
# raised — see docs/research/omniroute-non-ping-sse-stream-timeout.md.
|
||||
# Default 600000 (10 min) per OmniRoute's own docs, but the effective
|
||||
# deadline is remaining budget after retries/cooldowns eat into it, not
|
||||
# a flat timer, so raised well past the default for headroom.
|
||||
- REQUEST_TIMEOUT_MS=${OMNIROUTE_REQUEST_TIMEOUT_MS:-1800000}
|
||||
# Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
|
||||
# network can't resolve search.home on its own.
|
||||
extra_hosts:
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# OmniRoute's "non-ping SSE" first-token deadline — a different timer than `STREAM_IDLE_TIMEOUT_MS`
|
||||
|
||||
**Date:** 2026-09-10
|
||||
|
||||
`STREAM_IDLE_TIMEOUT_MS` was raised to 180000 on 2026-09-09 (see docker-compose.yml's `omniroute`
|
||||
service) specifically to give contended `llama-server` prefill room to produce a first token. It didn't
|
||||
work: the very next morning, qwen-code sessions against `qwen3.8-27b-local` still hit repeated
|
||||
|
||||
```
|
||||
Stream produced no non-ping SSE event within 95000ms
|
||||
```
|
||||
|
||||
(and once at 115000ms) — both well under the 180s the compose fix set, and well under the connection's
|
||||
own `providerSpecificData.timeoutMs: 1200000` (confirmed live via `GET /api/providers/<id>`). Neither of
|
||||
those settings bounds this failure.
|
||||
|
||||
## Root cause
|
||||
|
||||
Per OmniRoute's own docs (`docs/reference/ENVIRONMENT.md`, "Timeout Settings" section) and a maintainer
|
||||
reply in [diegosouzapw/OmniRoute#10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602):
|
||||
|
||||
| Variable | Default | Governs |
|
||||
|---|---|---|
|
||||
| `REQUEST_TIMEOUT_MS` | 600000 (10 min) | Overall upstream request budget. **The first non-ping SSE event's deadline inherits this one.** |
|
||||
| `STREAM_IDLE_TIMEOUT_MS` | 120000 (2 min) | Max gap between *successive* SSE chunks once streaming has already started — does not govern the wait for the first chunk. |
|
||||
| `STREAM_PING_INTERVAL_MS` | 30000 (30s) | How often OmniRoute emits its own keepalive pings on the stream — these explicitly do not count as "non-ping" events, so they can't rescue a request against the first deadline. |
|
||||
|
||||
So the 2026-09-09 fix tuned the wrong timer for this failure mode: `STREAM_IDLE_TIMEOUT_MS` only matters
|
||||
once `llama-server` has already emitted something. The "no token at all yet" case — exactly what a large
|
||||
compact-prompt prefill on a contended local model produces — is bounded by `REQUEST_TIMEOUT_MS` instead.
|
||||
|
||||
The observed 95000ms/115000ms figures are also *not* `REQUEST_TIMEOUT_MS`'s raw 600000ms default: OmniRoute
|
||||
computes the first-event deadline as **remaining budget**, not a flat timer — `REQUEST_TIMEOUT_MS` minus
|
||||
time already spent in OmniRoute's own request-queue/retry/cooldown cycle (`requestRetry: 3`,
|
||||
`connectionCooldown.apikey.baseCooldownMs`, provider breaker) before the request was actually dispatched
|
||||
to `llama-server`. Confirmed live via `GET /api/settings` → `resilienceSettings` on this deployment. Most
|
||||
of the 10-minute default budget was being burned by retries before the final attempt even started.
|
||||
|
||||
## Fix
|
||||
|
||||
Set `REQUEST_TIMEOUT_MS` explicitly, generously — applied in docker-compose.yml as
|
||||
`OMNIROUTE_REQUEST_TIMEOUT_MS` (default 1800000 / 30 min), same pattern as
|
||||
`OMNIROUTE_STREAM_IDLE_TIMEOUT_MS`. This doesn't replace the 2026-09-09 `STREAM_IDLE_TIMEOUT_MS` fix —
|
||||
that one still matters for mid-stream stalls after generation has started — it addresses the separate
|
||||
"nothing has arrived yet" case that fix didn't cover.
|
||||
|
||||
Raising `REQUEST_TIMEOUT_MS` buys headroom; it doesn't address *why* prefill on a 50K+ token compact
|
||||
prompt can take that long in the first place. `llama-server` had no `--cache-reuse` flag set — every
|
||||
request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged
|
||||
from the previous turn. Added `--cache-reuse 256` (docker-compose.yml) so llama.cpp reuses cached KV for
|
||||
any matching ≥256-token chunk via KV-shift instead of reprocessing it, which is the actual fix for
|
||||
compact-prompt prefill time — the timeout bump above is a safety margin around it, not a substitute.
|
||||
|
||||
## Sources
|
||||
|
||||
- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `docs/reference/ENVIRONMENT.md`
|
||||
("Timeout Settings"), [Discussion #10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602)
|
||||
- Live `GET /api/providers/<connectionId>`, `GET /api/settings`, `GET /api/resilience` against this
|
||||
deployment's OmniRoute instance (2026-09-10)
|
||||
- [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the related-but-distinct 30s semaphore/429 investigation
|
||||
Reference in New Issue
Block a user