docs(llm): document qwen-classifier reality, add --reasoning off safety net
- docker-compose.yml: add --reasoning off to qwen-classifier per fast-model-choice.md's own recommendation (ggml-org/llama.cpp#20809 safety net) — missed in the original rollout, caught while writing this up. - docs/coding-cli-setup/qwen-code.md: rewritten to match what's actually deployed (qwen-classifier, partial GPU offload, 65536 ctx, Q4_K_XL) — previously described an unimplemented llama-server-fast/8192-ctx plan. Documents the non-interactive MCP tool allow-list gap found live-testing. - docs/coding-cli-setup/opencode.md: fix stale 65536 example that didn't match its own documented LLAMA_CTX_SIZE/LLAMA_PARALLEL formula (131072). - docs/research/fast-model-choice.md: implementation note recording where the actual rollout diverged from this doc's original recommendations (service name, quant, context size, CPU-first-then-GPU path). - docs/research/omniroute-account-semaphore-timeout.md: new — the hardcoded 30s per-connection semaphore timeout found during the pr-agent investigation, root-caused against OmniRoute's own source, and the maxConcurrent:null + providerSpecificData.timeoutMs fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -195,6 +195,39 @@ comfortably affords the higher-precision quant.
|
||||
- [docs/research/qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) (this repo — cross-referenced
|
||||
for the 27B model's own, still-open, tool-calling parser bugs)
|
||||
|
||||
## Implementation note (2026-09-09) — what actually shipped, and why it differs
|
||||
|
||||
The model pick (`Qwen3-4B-Instruct-2507`) held up and is what's deployed. Several sizing assumptions in
|
||||
this doc didn't survive contact with the real deployment, though — worth recording so the next person
|
||||
tuning this doesn't re-derive the same corrections from scratch:
|
||||
|
||||
- **Service name is `qwen-classifier`, not `llama-server-fast`** — this doc's proposed name never got
|
||||
used. There's no `LLAMA_FAST_CTX_SIZE`/`LLAMA_FAST_PARALLEL` in `.env.example` either; the real config
|
||||
lives inline in `docker-compose.yml`'s `qwen-classifier` command.
|
||||
- **CPU-only was tried first and rejected** — this doc's VRAM budget analysis (§5) assumed GPU
|
||||
residency from the start, but the actual rollout path tried CPU-only first (to sidestep VRAM
|
||||
contention entirely) and found it too slow: real classification calls blew past OmniRoute's request
|
||||
timeout and retry-looped. Moved to GPU after that, which is what §5's math was for all along.
|
||||
- **Q4_K_XL weights, not Q8_0** — §5's "~2.4GB headroom" case assumed Q8_0 (4.28GB). In practice, fitting
|
||||
the classifier onto the R9700 *alongside* the 27B model (not in an assumed-empty 7GB budget) left only
|
||||
~6.1GB free VRAM total, and even Q4_K_XL (2.37GB) plus full-context KV cache didn't leave enough real
|
||||
margin at full GPU offload — see the "measured live" numbers in `docker-compose.yml`'s `qwen-classifier`
|
||||
comment block. Landed on **partial GPU offload (28/36 layers)** instead of full offload, which is not a
|
||||
case this doc considered at all.
|
||||
- **65536 context, not 8192** — §5 sized the context "in the low thousands," reasoning from qwen-code's
|
||||
two-stage classifier description alone. Directly reading qwen-code's actual source
|
||||
(`packages/core/src/permissions/classifier-transcript.ts`: `MAX_TRANSCRIPT_MESSAGES=40`,
|
||||
`MAX_HISTORICAL_ACTION_CHARS=4000`/message) puts the real worst case at ~40-50K tokens — confirmed
|
||||
live, a real classifier call during testing hit 15,116 prompt tokens. 8192 would have been undersized
|
||||
for real usage; 65536 gives margin without the original setting.json value (131072, copied from the
|
||||
main model's entry, not a real qwen-code requirement) wasting VRAM for no reason.
|
||||
- **§4's `--reasoning off` recommendation was initially missed** in the first deployment pass and added
|
||||
only once this doc was re-read while writing this note. It's now in `docker-compose.yml`'s
|
||||
`qwen-classifier` command, per this doc's own "add it regardless, no-cost safety net" reasoning — still
|
||||
unconfirmed whether the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build actually reproduces
|
||||
#20809 (nothing in testing so far surfaced `reasoning_content` where `tool_calls` was expected, but
|
||||
that wasn't specifically probed for either).
|
||||
|
||||
## Confidence/uncertainty summary
|
||||
|
||||
- **High confidence:** Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote);
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# OmniRoute's per-connection semaphore timeout — hardcoded, not a setting
|
||||
|
||||
**Date:** 2026-09-09
|
||||
|
||||
Any OmniRoute connection whose upstream can only handle a small, fixed number of concurrent requests
|
||||
(this repo's `llama-server`/`qwen-classifier`, both effectively single-GPU-slot-limited) can hit a hard
|
||||
30-second reject once more requests are in flight than the connection's `maxConcurrent` allows — even
|
||||
though the request would have succeeded fine if it had just waited its turn. This surfaced first as the
|
||||
`pr-agent`/`CodersPlacePI` 429/504 investigation (see the issue tracker), then again while sizing
|
||||
`qwen-classifier`. Recorded here so it doesn't have to be re-diagnosed from scratch next time.
|
||||
|
||||
## The error
|
||||
|
||||
```
|
||||
{"error":{"message":"Semaphore timeout after 30000ms for <provider>:<connectionId>","type":"rate_limit_error","code":"rate_limit_exceeded"}}
|
||||
```
|
||||
|
||||
## Root cause (confirmed against OmniRoute's own source, [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute))
|
||||
|
||||
`open-sse/services/accountSemaphore.ts`:
|
||||
|
||||
```ts
|
||||
const DEFAULT_TIMEOUT_MS = 30_000;
|
||||
...
|
||||
function createSemaphoreTimeoutError(semaphoreKey, timeoutMs) {
|
||||
const error = new Error(`Semaphore timeout after ${timeoutMs}ms for ${semaphoreKey}`);
|
||||
error.code = "SEMAPHORE_TIMEOUT"; // classified upstream as HTTP 429 rate_limit_exceeded
|
||||
return error;
|
||||
}
|
||||
```
|
||||
|
||||
Called from `open-sse/handlers/chatCore.ts`:
|
||||
|
||||
```ts
|
||||
await acquireAccountSemaphore(accountSemaphoreKey, {
|
||||
maxConcurrency: accountSemaphoreMaxConcurrency, // = the connection's maxConcurrent
|
||||
signal: streamController.signal,
|
||||
// no timeoutMs passed → always falls back to the hardcoded 30_000 default
|
||||
})
|
||||
```
|
||||
|
||||
This is **not** the same thing as OmniRoute's documented quota-share concurrency gate
|
||||
(`open-sse/services/combo/quotaShareConcurrency.ts`, key prefix `qsconn:`), which is deliberately
|
||||
fail-open per its own doc comment ("a saturated queue or timeout proceeds without a slot rather than
|
||||
ever rejecting a dispatchable request") — that one only matters for quota-share combos. The account
|
||||
semaphore above is a *different*, always-on gate keyed `provider:connectionId`, has no fail-open path,
|
||||
and its 30-second timeout is a bare `await` with nothing passed to override it — not exposed via
|
||||
`/api/resilience`, not an env var, not a dashboard toggle, not documented anywhere in
|
||||
`docs/reference/ENVIRONMENT.md`. It's a hardcoded constant in vendored code.
|
||||
|
||||
Also **not** the same as `requestQueue.maxWaitMs` (visible via `GET /api/resilience`, this deployment
|
||||
already has it at `86400000`) — that one bounds a Bottleneck-managed *execution* timer that starts only
|
||||
after dispatch, surfaces as HTTP 504 `RATE_LIMIT_EXECUTION_TIMEOUT`, and is unrelated to the 429 above.
|
||||
|
||||
## What actually fixes it
|
||||
|
||||
The 30s ceiling itself cannot be raised — no config surface reaches it in the current OmniRoute build.
|
||||
Two real options:
|
||||
|
||||
1. **Bypass the semaphore, let the upstream's own queue absorb concurrency instead.**
|
||||
`maxConcurrency == null || maxConcurrency <= 0` fully bypasses `accountSemaphore.ts` (see
|
||||
`isBypassed()`) — no gate, no 30s timer, requests pass straight through to the upstream. This only
|
||||
works if the upstream itself queues gracefully with no reject-timeout of its own — confirmed true for
|
||||
llama.cpp's server (`tools/server/server-queue.cpp` has no queue-wait timeout; excess requests just
|
||||
wait for a free slot). If you do this, also raise the connection's own
|
||||
`providerSpecificData.timeoutMs` (bounded 1ms–24h, `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS`) generously —
|
||||
that's the timer that now matters: "did the upstream return response headers in time," which on
|
||||
llama.cpp means the full queue-wait-then-generate time, since llama.cpp sends **zero bytes, not even
|
||||
headers**, while a request sits queued (confirmed in `server-context.cpp`: `res->status = 200` is only
|
||||
set after the first generated token exists).
|
||||
2. **Reduce how often more than `maxConcurrent` requests actually stack up** — e.g. the
|
||||
`pr-agent`/Gitea webhook fix (narrowing the subscribed event list so one PR action doesn't fire 3+
|
||||
near-simultaneous AI calls). Doesn't remove the ceiling, just makes it less likely to be hit.
|
||||
|
||||
Applied in this repo: `llama-server`'s OmniRoute connection has `maxConcurrent: null` and
|
||||
`providerSpecificData.timeoutMs: 1200000` (20 min — matches worst-case 2-slots-busy + queued + own
|
||||
generation time). `qwen-classifier` uses a much shorter `timeoutMs: 120000` since it isn't
|
||||
GPU-contended the same way — see `docs/coding-cli-setup/qwen-code.md`.
|
||||
|
||||
## Sources
|
||||
|
||||
- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `open-sse/services/accountSemaphore.ts`, `open-sse/handlers/chatCore.ts`, `open-sse/services/combo/quotaShareConcurrency.ts`, `open-sse/services/rateLimitManager.ts`, `docs/architecture/RESILIENCE_GUIDE.md`, `docs/reference/ENVIRONMENT.md`
|
||||
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `tools/server/server-queue.cpp`, `tools/server/server-context.cpp`
|
||||
- `src/shared/validation/providerSpecificData.ts` (OmniRoute) — `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS` bound
|
||||
Reference in New Issue
Block a user