LLM-Server/docs/research/omniroute-account-semaphore-timeout.md

# OmniRoute's per-connection semaphore timeout — hardcoded, not a setting

**Date:** 2026-09-09

Any OmniRoute connection whose upstream can only handle a small, fixed number of concurrent requests
(this repo's `llama-server`/`qwen-classifier`, both effectively single-GPU-slot-limited) can hit a hard
30-second reject once more requests are in flight than the connection's `maxConcurrent` allows — even
though the request would have succeeded fine if it had just waited its turn. This surfaced first as the
`pr-agent`/`CodersPlacePI` 429/504 investigation (see the issue tracker), then again while sizing
`qwen-classifier`. Recorded here so it doesn't have to be re-diagnosed from scratch next time.

## The error

```
{"error":{"message":"Semaphore timeout after 30000ms for <provider>:<connectionId>","type":"rate_limit_error","code":"rate_limit_exceeded"}}
```

## Root cause (confirmed against OmniRoute's own source, [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute))

`open-sse/services/accountSemaphore.ts`:

```ts
const DEFAULT_TIMEOUT_MS = 30_000;
...
function createSemaphoreTimeoutError(semaphoreKey, timeoutMs) {
  const error = new Error(`Semaphore timeout after ${timeoutMs}ms for ${semaphoreKey}`);
  error.code = "SEMAPHORE_TIMEOUT"; // classified upstream as HTTP 429 rate_limit_exceeded
  return error;
}
```

Called from `open-sse/handlers/chatCore.ts`:

```ts
await acquireAccountSemaphore(accountSemaphoreKey, {
  maxConcurrency: accountSemaphoreMaxConcurrency, // = the connection's maxConcurrent
  signal: streamController.signal,
  // no timeoutMs passed → always falls back to the hardcoded 30_000 default
})
```

This is **not** the same thing as OmniRoute's documented quota-share concurrency gate
(`open-sse/services/combo/quotaShareConcurrency.ts`, key prefix `qsconn:`), which is deliberately
fail-open per its own doc comment ("a saturated queue or timeout proceeds without a slot rather than
ever rejecting a dispatchable request") — that one only matters for quota-share combos. The account
semaphore above is a *different*, always-on gate keyed `provider:connectionId`, has no fail-open path,
and its 30-second timeout is a bare `await` with nothing passed to override it — not exposed via
`/api/resilience`, not an env var, not a dashboard toggle, not documented anywhere in
`docs/reference/ENVIRONMENT.md`. It's a hardcoded constant in vendored code.

Also **not** the same as `requestQueue.maxWaitMs` (visible via `GET /api/resilience`, this deployment
already has it at `86400000`) — that one bounds a Bottleneck-managed *execution* timer that starts only
after dispatch, surfaces as HTTP 504 `RATE_LIMIT_EXECUTION_TIMEOUT`, and is unrelated to the 429 above.

## What actually fixes it

The 30s ceiling itself cannot be raised — no config surface reaches it in the current OmniRoute build.
Two real options:

1. **Bypass the semaphore, let the upstream's own queue absorb concurrency instead.**
   `maxConcurrency == null || maxConcurrency <= 0` fully bypasses `accountSemaphore.ts` (see
   `isBypassed()`) — no gate, no 30s timer, requests pass straight through to the upstream. This only
   works if the upstream itself queues gracefully with no reject-timeout of its own — confirmed true for
   llama.cpp's server (`tools/server/server-queue.cpp` has no queue-wait timeout; excess requests just
   wait for a free slot). If you do this, also raise the connection's own
   `providerSpecificData.timeoutMs` (bounded 1ms–24h, `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS`) generously —
   that's the timer that now matters: "did the upstream return response headers in time," which on
   llama.cpp means the full queue-wait-then-generate time, since llama.cpp sends **zero bytes, not even
   headers**, while a request sits queued (confirmed in `server-context.cpp`: `res->status = 200` is only
   set after the first generated token exists).
2. **Reduce how often more than `maxConcurrent` requests actually stack up** — e.g. the
   `pr-agent`/Gitea webhook fix (narrowing the subscribed event list so one PR action doesn't fire 3+
   near-simultaneous AI calls). Doesn't remove the ceiling, just makes it less likely to be hit.

Applied in this repo: `llama-server`'s OmniRoute connection has `maxConcurrent: null` and
`providerSpecificData.timeoutMs: 1200000` (20 min — matches worst-case 2-slots-busy + queued + own
generation time). `qwen-classifier` uses a much shorter `timeoutMs: 120000` since it isn't
GPU-contended the same way — see `docs/coding-cli-setup/qwen-code.md`.

## Sources

- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `open-sse/services/accountSemaphore.ts`, `open-sse/handlers/chatCore.ts`, `open-sse/services/combo/quotaShareConcurrency.ts`, `open-sse/services/rateLimitManager.ts`, `docs/architecture/RESILIENCE_GUIDE.md`, `docs/reference/ENVIRONMENT.md`
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `tools/server/server-queue.cpp`, `tools/server/server-context.cpp`
- `src/shared/validation/providerSpecificData.ts` (OmniRoute) — `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS` bound