# OmniRoute's "non-ping SSE" first-token deadline — a different timer than `STREAM_IDLE_TIMEOUT_MS`

**Date:** 2026-09-10

`STREAM_IDLE_TIMEOUT_MS` was raised to 180000 on 2026-09-09 (see docker-compose.yml's `omniroute`
service) specifically to give contended `llama-server` prefill room to produce a first token. It didn't
work: the very next morning, qwen-code sessions against `qwen3.8-27b-local` still hit repeated

```
Stream produced no non-ping SSE event within 95000ms
```

(and once at 115000ms) — both well under the 180s the compose fix set, and well under the connection's
own `providerSpecificData.timeoutMs: 1200000` (confirmed live via `GET /api/providers/<id>`). Neither of
those settings bounds this failure.

## Root cause

Per OmniRoute's own docs (`docs/reference/ENVIRONMENT.md`, "Timeout Settings" section) and a maintainer
reply in [diegosouzapw/OmniRoute#10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602):

| Variable | Default | Governs |
|---|---|---|
| `REQUEST_TIMEOUT_MS` | 600000 (10 min) | Overall upstream request budget. **The first non-ping SSE event's deadline inherits this one.** |
| `STREAM_IDLE_TIMEOUT_MS` | 120000 (2 min) | Max gap between *successive* SSE chunks once streaming has already started — does not govern the wait for the first chunk. |
| `STREAM_PING_INTERVAL_MS` | 30000 (30s) | How often OmniRoute emits its own keepalive pings on the stream — these explicitly do not count as "non-ping" events, so they can't rescue a request against the first deadline. |

So the 2026-09-09 fix tuned the wrong timer for this failure mode: `STREAM_IDLE_TIMEOUT_MS` only matters
once `llama-server` has already emitted something. The "no token at all yet" case — exactly what a large
compact-prompt prefill on a contended local model produces — is bounded by `REQUEST_TIMEOUT_MS` instead.

The observed 95000ms/115000ms figures are also *not* `REQUEST_TIMEOUT_MS`'s raw 600000ms default: OmniRoute
computes the first-event deadline as **remaining budget**, not a flat timer — `REQUEST_TIMEOUT_MS` minus
time already spent in OmniRoute's own request-queue/retry/cooldown cycle (`requestRetry: 3`,
`connectionCooldown.apikey.baseCooldownMs`, provider breaker) before the request was actually dispatched
to `llama-server`. Confirmed live via `GET /api/settings` → `resilienceSettings` on this deployment. Most
of the 10-minute default budget was being burned by retries before the final attempt even started.

## Fix

Set `REQUEST_TIMEOUT_MS` explicitly, generously — applied in docker-compose.yml as
`OMNIROUTE_REQUEST_TIMEOUT_MS` (default 1800000 / 30 min), same pattern as
`OMNIROUTE_STREAM_IDLE_TIMEOUT_MS`. This doesn't replace the 2026-09-09 `STREAM_IDLE_TIMEOUT_MS` fix —
that one still matters for mid-stream stalls after generation has started — it addresses the separate
"nothing has arrived yet" case that fix didn't cover.

Raising `REQUEST_TIMEOUT_MS` buys headroom; it doesn't address *why* prefill on a 50K+ token compact
prompt can take that long in the first place. `llama-server` had no `--cache-reuse` flag set — every
request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged
from the previous turn. Added `--cache-reuse 256` (docker-compose.yml) so llama.cpp reuses cached KV for
any matching ≥256-token chunk via KV-shift instead of reprocessing it, which is the actual fix for
compact-prompt prefill time — the timeout bump above is a safety margin around it, not a substitute.

## Sources

- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `docs/reference/ENVIRONMENT.md`
  ("Timeout Settings"), [Discussion #10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602)
- Live `GET /api/providers/<connectionId>`, `GET /api/settings`, `GET /api/resilience` against this
  deployment's OmniRoute instance (2026-09-10)
- [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the related-but-distinct 30s semaphore/429 investigation