LLM-Server/docs/research/omniroute-account-semaphore-timeout.md

OmniRoute's per-connection semaphore timeout — hardcoded, not a setting

Date: 2026-09-09

Any OmniRoute connection whose upstream can only handle a small, fixed number of concurrent requests (this repo's llama-server/qwen-classifier, both effectively single-GPU-slot-limited) can hit a hard 30-second reject once more requests are in flight than the connection's maxConcurrent allows — even though the request would have succeeded fine if it had just waited its turn. This surfaced first as the pr-agent/CodersPlacePI 429/504 investigation (see the issue tracker), then again while sizing qwen-classifier. Recorded here so it doesn't have to be re-diagnosed from scratch next time.

The error

{"error":{"message":"Semaphore timeout after 30000ms for <provider>:<connectionId>","type":"rate_limit_error","code":"rate_limit_exceeded"}}

Root cause (confirmed against OmniRoute's own source, diegosouzapw/OmniRoute)

open-sse/services/accountSemaphore.ts:

const DEFAULT_TIMEOUT_MS = 30_000;
...
function createSemaphoreTimeoutError(semaphoreKey, timeoutMs) {
  const error = new Error(`Semaphore timeout after ${timeoutMs}ms for ${semaphoreKey}`);
  error.code = "SEMAPHORE_TIMEOUT"; // classified upstream as HTTP 429 rate_limit_exceeded
  return error;
}

Called from open-sse/handlers/chatCore.ts:

await acquireAccountSemaphore(accountSemaphoreKey, {
  maxConcurrency: accountSemaphoreMaxConcurrency, // = the connection's maxConcurrent
  signal: streamController.signal,
  // no timeoutMs passed → always falls back to the hardcoded 30_000 default
})

This is not the same thing as OmniRoute's documented quota-share concurrency gate (open-sse/services/combo/quotaShareConcurrency.ts, key prefix qsconn:), which is deliberately fail-open per its own doc comment ("a saturated queue or timeout proceeds without a slot rather than ever rejecting a dispatchable request") — that one only matters for quota-share combos. The account semaphore above is a different, always-on gate keyed provider:connectionId, has no fail-open path, and its 30-second timeout is a bare await with nothing passed to override it — not exposed via /api/resilience, not an env var, not a dashboard toggle, not documented anywhere in docs/reference/ENVIRONMENT.md. It's a hardcoded constant in vendored code.

Also not the same as requestQueue.maxWaitMs (visible via GET /api/resilience, this deployment already has it at 86400000) — that one bounds a Bottleneck-managed execution timer that starts only after dispatch, surfaces as HTTP 504 RATE_LIMIT_EXECUTION_TIMEOUT, and is unrelated to the 429 above.

What actually fixes it

The 30s ceiling itself cannot be raised — no config surface reaches it in the current OmniRoute build. Two real options:

1. Bypass the semaphore, let the upstream's own queue absorb concurrency instead. maxConcurrency == null || maxConcurrency <= 0 fully bypasses accountSemaphore.ts (see isBypassed()) — no gate, no 30s timer, requests pass straight through to the upstream. This only works if the upstream itself queues gracefully with no reject-timeout of its own — confirmed true for llama.cpp's server (tools/server/server-queue.cpp has no queue-wait timeout; excess requests just wait for a free slot). If you do this, also raise the connection's own providerSpecificData.timeoutMs (bounded 1ms–24h, MAX_PROVIDER_SPECIFIC_TIMEOUT_MS) generously — that's the timer that now matters: "did the upstream return response headers in time," which on llama.cpp means the full queue-wait-then-generate time, since llama.cpp sends zero bytes, not even headers, while a request sits queued (confirmed in server-context.cpp: res->status = 200 is only set after the first generated token exists).
2. Reduce how often more than maxConcurrent requests actually stack up — e.g. the pr-agent/Gitea webhook fix (narrowing the subscribed event list so one PR action doesn't fire 3+ near-simultaneous AI calls). Doesn't remove the ceiling, just makes it less likely to be hit.

Applied in this repo: llama-server's OmniRoute connection has maxConcurrent: null and providerSpecificData.timeoutMs: 1200000 (20 min — matches worst-case 2-slots-busy + queued + own generation time). qwen-classifier uses a much shorter timeoutMs: 120000 since it isn't GPU-contended the same way — see docs/coding-cli-setup/qwen-code.md.

Sources

• diegosouzapw/OmniRoute — open-sse/services/accountSemaphore.ts, open-sse/handlers/chatCore.ts, open-sse/services/combo/quotaShareConcurrency.ts, open-sse/services/rateLimitManager.ts, docs/architecture/RESILIENCE_GUIDE.md, docs/reference/ENVIRONMENT.md
• ggml-org/llama.cpp — tools/server/server-queue.cpp, tools/server/server-context.cpp
• src/shared/validation/providerSpecificData.ts (OmniRoute) — MAX_PROVIDER_SPECIFIC_TIMEOUT_MS bound