docs(research): OmniRoute memory tools hijack classifier tool-call, x-omniroute-no-memory fix

docs(research): trace 48-min outage to third OmniRoute timeout and open ROCm#6630 GPU hang
docs(research): qwen-classifier needs no context-window match; 8B/30B won't fit VRAM
2026-09-15 23:31:58 +02:00 · 2026-09-15 23:31:58 +02:00 · 2026-09-15 23:31:58 +02:00
3 changed files with 862 additions and 0 deletions
@@ -0,0 +1,249 @@
 # Does the qwen-classifier need to match the main model's context window, and would upgrading it to Qwen3-8B or Qwen3-Coder-30B-A3B-Instruct fit on the current R9700?
 **Date:** 2026-09-15
 **Question raised:** should `qwen-classifier` be resized to `LLAMA_CTX_SIZE / LLAMA_PARALLEL` (131072, matching the
 main model's per-slot context), should the classifier model itself move up to Qwen3-8B or
 Qwen3-Coder-30B-A3B-Instruct, and does that need a second GPU?
 **Answer: No, no, and not for this reason.** qwen-code's own docs state no context-size requirement for
 `fastModel` at all — the "match the main model" premise doesn't come from any primary source. Separately, and
 independently of context size: neither Qwen3-8B nor Qwen3-Coder-30B-A3B-Instruct fits in the ~6.1GiB of VRAM
 actually free on the card today, at *any* context length, once weights alone are counted — this is a raw-VRAM
 problem, not a context-window problem, exactly matching the "qwen8b needs to offload more to the RAM" intuition
 in the request. A second GPU would solve the VRAM problem (and incidentally remove this repo's own
 `GPU_MAX_HW_QUEUES=1` ROCm#5706 workaround from applying to this pair), but isn't deployed hardware today —
 it's a rack-build/acquisition question, not a config change.
 ## 1. Does qwen-code require the fast/classifier model to match the main model's context window?
 No — checked against the user's own three linked pages, fetched directly:
 - **`fastModel` settings docs**: "Model used for generating prompt suggestions and speculative execution,"
  configurable via `inherit` (main model), `fast`, a model ID, or `authType:model-id`; "Leave empty to use the
  main model." The docs recommend "a smaller/faster model (e.g., `qwen3-coder-flash`) reduces latency and
  cost" — **no context-window size or capacity requirement is stated anywhere on this page.**
  Source: [qwen-code docs — Configuration / Settings, `#fastmodel`](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/settings/#fastmodel)
 - **Approval Mode / Auto Mode classifier docs**: describes what the classifier evaluates (shell commands,
  network calls, out-of-workspace edits) and its allow/block behavior, but **does not name a specific model or
  state any context-window requirement** — the only operational note is that "when the classifier API is
  unreachable, the action is blocked rather than allowed."
  Source: [qwen-code docs — Approval Mode, `#4-auto-mode---classifier-driven-approval`](https://qwenlm.github.io/qwen-code-docs/en/users/features/approval-mode/#4-auto-mode---classifier-driven-approval)
 - **Auto Mode "How it works"**: confirms the two-stage design (Stage 1: ~300ms, `{shouldBlock}` only; Stage 2:
  chain-of-thought reconsideration, only on a Stage-1 block) and what data reaches the classifier — user text,
  assistant tool-use calls, and tool-specific projections (truncated edit content, fetch URLs, shell command
  text). **Tool results are explicitly never sent to the classifier.** It "uses your configured fast model
  (`/model --fast`)," falling back to the main session model only if none is set. **No statement anywhere
  requires or implies the fast model's context window match the main model's.**
  Source: [qwen-code docs — Auto Mode, `#how-it-works`](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/#how-it-works)
 This confirms and sharpens what this repo's own `fast-model-choice.md` already found by reading qwen-code's
 source directly (`packages/core/src/permissions/classifier-transcript.ts`: `MAX_TRANSCRIPT_MESSAGES=40`,
 `MAX_HISTORICAL_ACTION_CHARS=4000`/message, worst case ~40-50K tokens, live-tested at 15,116 prompt tokens) —
 that doc already called the original `131072` in `settings.json` "copied from the main model's entry, not a
 real qwen-code requirement." The three docs pages fetched here add nothing that contradicts that: **there is no
 primary-source basis for `LLAMA_CTX_SIZE / LLAMA_PARALLEL` symmetry between the two models.** The current
 `65536` classifier ctx already carries ~1.5x margin over the real worst case.
 ## 2. VRAM math for Qwen3-8B and Qwen3-Coder-30B-A3B-Instruct as classifier candidates
 Same method this repo already uses (`qwen3.8-27b-quant.md`, `fast-model-choice.md` §5): per-token KV cache =
 `layers × 2(K+V) × kv_heads × head_dim × bytes`, read directly from each model's own `config.json`.
 ### Qwen3-8B
 - Architecture (`Qwen/Qwen3-8B` `config.json`): `num_hidden_layers: 36`, `num_key_value_heads: 8`,
  `num_attention_heads: 32`, `head_dim: 128`, `hidden_size: 4096`, `max_position_embeddings: 40960`,
  `rope_scaling: null`.
  Source: [Qwen/Qwen3-8B `config.json`](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)
 - **Native context is 32,768 tokens**, not the 262,144 the user's brief assumed (that number belongs to
  Qwen3-4B-Instruct-2507, a different, non-reasoning 2507-refresh model — Qwen3-8B is the earlier,
  thinking-capable Qwen3 architecture with a materially smaller native window). Extending past 32K needs YaRN:
  > "Qwen3 natively supports context lengths of up to 32,768 tokens. For conversations where the total length
  > (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling
  > techniques to handle long texts effectively."
  and llama.cpp-specific YaRN invocation is given explicitly:
  `./llama-cli ... -c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768`, with a documented
  caveat that "all the notable open-source frameworks implement **static** YaRN, which means the scaling
  factor remains constant regardless of input length, potentially impacting performance on shorter texts."
  Source: [Qwen/Qwen3-8B-GGUF — Processing Long Texts](https://huggingface.co/Qwen/Qwen3-8B-GGUF#processing-long-texts)
 - **Weights** (official Qwen quants, fetched from the GGUF repo file list): Q5_K_M = 5.85 GB, Q8_0 = 8.71 GB.
  Source: [Qwen/Qwen3-8B-GGUF](https://huggingface.co/Qwen/Qwen3-8B-GGUF)
  *(Not independently verified: unsloth's equivalent `UD-Q4_K_XL` quant, which is what this repo's
  `docker-compose.yml`/`.env.example` actually download for every model deployed so far — the unsloth file
  size wasn't fetched, only the official Qwen quants above. Treat Q5_K_M/Q8_0 as a reasonable bound, not the
  exact file this repo would pull.)*
 - Per-token KV cache: `36 × 2 × 8 × 128 × 2 bytes = 144 KiB/token` fp16 — identical to Qwen3-4B-Instruct-2507's
  figure in `fast-model-choice.md` §5, since both share the same `layers/kv_heads/head_dim` triple.
 | Context | KV (fp16) | KV (q8_0) | KV (q4_0, current classifier setting) |
 |---|---|---|---|
 | 65,536 (current classifier ctx) | 9.0 GiB | 4.5 GiB | **2.25 GiB** |
 | 131,072 (user's proposed "match main model") | 18.0 GiB | 9.0 GiB | **4.5 GiB** |
 Weights + KV (q4_0, smallest realistic combo):
 | Context | Q5_K_M weights + q4_0 KV | Q8_0 weights + q4_0 KV |
 |---|---|---|
 | 65,536 | 5.85 + 2.25 = **8.1 GB** | 8.71 + 2.25 = **10.96 GB** |
 | 131,072 | 5.85 + 4.5 = **10.35 GB** | 8.71 + 4.5 = **13.21 GB** |
 ### Qwen3-Coder-30B-A3B-Instruct
 - Architecture (fetched from the shared Qwen3-30B-A3B-family `config.json`): `num_hidden_layers: 48`,
  `num_key_value_heads: 4`, `num_attention_heads: 32`, `head_dim: 128`, `hidden_size: 2048`, **MoE**:
  `num_experts: 128`, `num_experts_per_tok: 8` (8 of 128 experts active per token — confirms this is a sparse
  MoE model, not a dense one like the 27B or 8B candidates; the "active params" figure describes *compute*
  per token, not memory footprint — **all 128 experts' weights still have to be resident** wherever the model
  is loaded, GPU or RAM).
  Source: [Qwen/Qwen3-30B-A3B-family `config.json`](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)
 - **UD-Q4_K_XL file size (the exact quant/quantizer this repo already standardizes on): 17.7 GB.** 30.5B total /
  3.3B activated parameters. Native context "262,144 natively... can be extended further using Yarn to reach
  1M tokens."
  Source: [unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF?show_file_info=Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf)
 - Per-token KV cache: `48 × 2 × 4 × 128 × 2 bytes = 96 KiB/token` fp16 (smaller per-token than the 8B/4B
  candidates, since `num_key_value_heads` is 4 here vs. 8 — but this saving is irrelevant given the weights
  size below).
 | Context | KV (fp16) | KV (q8_0) | KV (q4_0) |
 |---|---|---|---|
 | 65,536 | 6.0 GiB | 3.0 GiB | **1.5 GiB** |
 | 131,072 | 12.0 GiB | 6.0 GiB | **3.0 GiB** |
 Weights + KV (q4_0):
 | Context | Total |
 |---|---|
 | 65,536 | 17.7 + 1.5 = **19.2 GB** |
 | 131,072 | 17.7 + 3.0 = **20.7 GB** |
 ## 3. Does either candidate fit the ~6.1 GiB actually free on the card today?
 **No — neither does, at either context size, even at the smallest quant/KV-quant combination tested.**
 - Qwen3-8B's cheapest realistic combination (Q5_K_M weights + q4_0 KV at the *current* 65536 ctx, not even
  the proposed 131072) is **8.1 GB — already ~2 GB over the measured 6.1 GiB free budget**, before accounting
  for compute-buffer/batch overhead that `fast-model-choice.md` §"Implementation note" already found could add
  meaningfully on top of the naive weights+KV estimate (that's exactly why the 4B classifier ended up needing
  `--flash-attn on` and partial 28/36-layer offload instead of the originally-predicted comfortable full-GPU
  fit).
 - Qwen3-Coder-30B-A3B-Instruct isn't close at any setting tested — its weights alone (17.7 GB) are triple the
  entire free budget, and this doesn't change with context size since the weights term dominates.
 - This is a **VRAM-capacity problem, not a context-window problem** — directly confirming the "qwen8b needs to
  offload more to the RAM" intuition in the original request. Reducing context doesn't fix it; the weights
  don't fit regardless.
 **CPU/RAM offload mechanics:** llama.cpp's `--n-gpu-layers` is documented as "max. number of layers to store in
 VRAM, either an exact number, `'auto'`, or `'all'`" — the layers not selected are computed on CPU, with the
 model loaded via mmap by default (same mechanism this repo's own `.env.example` already documents for
 `LLAMA_GPU_LAYERS`: "if GPU+RAM ever can't hold the working set, the OS pages the rest in from disk
 automatically"). For the MoE Coder-30B-A3B model specifically, this repo's own `.env.example` already flags the
 more targeted alternative — `--n-cpu-moe`/`--cpu-moe`/`--override-tensor "exps"` — as the flags that "target
 Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B)," i.e. exactly this model's architecture: these offload only
 expert-tensor weights to CPU while keeping attention/shared layers and KV cache on GPU, which is the
 mechanically correct lever for an MoE model, unlike the blunt `--n-gpu-layers` used for the dense 8B/27B/4B
 models. **Neither llama.cpp's own README nor the Qwen model cards fetched here document a quantified
 performance cost for partial offload** — no primary source gives a "N layers offloaded = X% slower" figure.
 Source: [llama.cpp `tools/server/README.md`](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md)
 What *is* directly measured, in this repo's own deployment history: CPU-only was tried first for the current,
 much smaller 4B classifier and rejected — "too slow in practice: real classification calls blew past
 OmniRoute's 60s timeout and retry-looped (504→499→504)" (`docker-compose.yml`'s `qwen-classifier` comment
 block). An 8B dense model has roughly double the compute of the 4B model per token; a 30B-A3B model's *routing*
 overhead on CPU (choosing 8 of 128 experts per token, each a separate weight lookup) adds a different kind of
 cost that neither this repo nor the sources fetched here have measured. **Given the classifier's Stage 1 has an
 explicit ~300ms latency budget** (`fast-model-choice.md` §1, from qwen-code's own docs), and this repo already
 has one concrete data point that CPU offload breaks that budget at a smaller model size, extending either
 candidate onto significant CPU offload carries real, unquantified latency risk — the same failure mode already
 observed once, at a favorable (smaller) model size.
 ## 4. Does a second GPU solve this, and is one actually available?
 **Not today.** `docs/server-planing.md` is a rack-build plan for "3-4x AMD Radeon AI PRO R9700 (32GB) GPUs" —
 a future-state document, not present inventory. Every GPU-facing comment in this repo's own
 `docker-compose.yml`/`.env.example`/`rocm-gpu-pin-and-render-group.md` consistently refers to "the single 32GB
 R9700" and measures the "~6.1GiB free" budget against one physical card holding both `llama-server` and
 `qwen-classifier`. Adding a second GPU is a hardware-acquisition and rack-build question — physically sourcing,
 installing, and power/PCIe-provisioning a card per `server-planing.md`'s own build plan — not a
 `docker-compose.yml`/`.env.example` change.
 **If a second GPU were added**, it would directly remove one already-documented risk for this specific pair:
 this repo's own `rocm-gpu-pin-and-render-group.md` traced the GPU-pinned-at-100%/ROCm#5706 bug to its precise
 trigger condition —
 > "The pin only appears with two concurrent HIP-context-holding processes **on the same GPU**... Root cause: an
 > AMD MES (Micro Engine Scheduler) firmware bug triggered by HIP hardware-queue creation."
 Source: [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706), via this repo's own
 [`rocm-gpu-pin-and-render-group.md`](rocm-gpu-pin-and-render-group.md)
 Since the confirmed trigger is *two HIP contexts sharing one physical card*, moving the classifier to its own,
 second GPU would put each service on a single-HIP-context card — the condition that trips the bug wouldn't
 exist for this pair anymore, and the `GPU_MAX_HW_QUEUES=1` workaround currently applied to both services
 specifically because they share one card would no longer be load-bearing for *this* pair (it would still apply
 if any future third service shared a card with either model). This wasn't independently re-verified across two
 *separate* physical cards by any source fetched in this pass — it's a direct extrapolation from the confirmed
 root cause, same category of caveat that doc's own author already flagged for its within-one-card claim.
 ## Bottom line / recommendation
 1. **Don't apply `LLAMA_CTX_SIZE / LLAMA_PARALLEL` symmetry to the classifier.** No qwen-code primary source
   states or implies the fast/classifier model needs to match the main model's context window. The real
   requirement (§1, already established in `fast-model-choice.md`) is ~40-50K tokens worst case; the current
   `65536` already has margin. Doubling to 131072 would only double VRAM spent on KV cache for a model that
   won't otherwise fit anyway (§2-3).
 2. **Don't upgrade the classifier to Qwen3-8B or Qwen3-Coder-30B-A3B-Instruct on the current single-GPU setup.**
   Neither fits the ~6.1GiB actually free, at any context size — this is a weights-size problem, not a
   context-window problem. Forcing it would mean either (a) shrinking the main 27B model's own VRAM footprint
   to make room (a real trade-off against the primary model, not evaluated here), or (b) CPU/partial offload,
   which this repo has direct, measured evidence already breaks the classifier's latency budget at a *smaller*
   model size than either candidate.
 3. **A second GPU is the clean fix for VRAM contention and would also retire the ROCm#5706 workaround's
   relevance for this pair — but it isn't deployed hardware today.** `server-planing.md` is a future build
   plan; this is an acquisition/rack-build decision, not something achievable via a config change right now.
 4. If the actual underlying motivation is classifier *quality* (not context capacity), that's a separate,
   legitimate question this doc doesn't answer — worth its own research pass rather than solving it via a
   bigger model that doesn't fit the hardware.
 ## Sources
 - [qwen-code docs — Configuration / Settings, `#fastmodel`](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/settings/#fastmodel)
 - [qwen-code docs — Approval Mode, `#4-auto-mode---classifier-driven-approval`](https://qwenlm.github.io/qwen-code-docs/en/users/features/approval-mode/#4-auto-mode---classifier-driven-approval)
 - [qwen-code docs — Auto Mode, `#how-it-works`](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/#how-it-works)
 - [Qwen/Qwen3-8B `config.json`](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)
 - [Qwen/Qwen3-8B-GGUF](https://huggingface.co/Qwen/Qwen3-8B-GGUF)
 - [Qwen/Qwen3-8B-GGUF — Processing Long Texts](https://huggingface.co/Qwen/Qwen3-8B-GGUF#processing-long-texts)
 - [Qwen/Qwen3-30B-A3B-family `config.json`](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)
 - [unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF?show_file_info=Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf)
 - [llama.cpp `tools/server/README.md`](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md)
 - [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706)
 - [docs/research/fast-model-choice.md](fast-model-choice.md) (this repo — classifier transcript sizing,
  qwen-classifier's real deployment history)
 - [docs/research/qwen3.8-27b-quant.md](qwen3.8-27b-quant.md) (this repo — KV-cache-from-config.json method
  reused here)
 - [docs/research/rocm-gpu-pin-and-render-group.md](rocm-gpu-pin-and-render-group.md) (this repo — ROCm#5706
  trigger condition and `GPU_MAX_HW_QUEUES` scoping)
 - [docs/server-planing.md](../server-planing.md) (this repo — confirms only 1 of a planned 4 GPUs is deployed)
 - `docker-compose.yml`, `.env.example` (this repo — current `qwen-classifier`/`llama-server` config and the
  measured "~6.1GiB free" VRAM figure)
 ## Confidence / uncertainty summary
 - **High confidence:** qwen-code's `fastModel`/Auto-Mode docs state no context-window requirement (direct
  quotes from all three linked pages); Qwen3-8B's native 32,768 context and YaRN caveat (direct model-card
  quote); Qwen3-Coder-30B-A3B-Instruct's MoE architecture and 17.7GB Q4_K_XL file size (direct from the
  quantizer's own repo page); the KV-cache-per-token math for both candidates (computed directly from each
  model's own `config.json`, same method already validated in this repo's prior research); the ROCm#5706
  trigger condition being scoped to two HIP contexts on the *same* GPU (direct quote from this repo's own
  prior research, itself sourced from the upstream issue).
 - **Medium confidence:** the exact unsloth `UD-Q4_K_XL`-equivalent file size for Qwen3-8B — only the official
  Qwen quants (Q5_K_M/Q8_0) were fetched, not unsloth's own repo, so the real number this repo would actually
  download wasn't directly verified (bounded reasonably by the Q5_K_M figure, which is already the smallest
  realistic option and still doesn't fit). The claim that CPU/partial-offload latency risk scales unfavorably
  for larger/MoE models is a reasoned extrapolation from this repo's one measured data point (4B CPU-only
  rejected) plus general MoE-routing-overhead reasoning, not a directly measured benchmark for either candidate.
 - **Low confidence / not independently verified:** whether `GPU_MAX_HW_QUEUES=1`/ROCm#5706 genuinely has zero
  relevance across two *separate* physical GPUs (extrapolated from the confirmed same-GPU trigger condition,
  same caveat this repo's own prior research already flagged for its own claim); no primary source found that
  quantifies llama.cpp's actual inference-speed penalty for partial `--n-gpu-layers` or `--n-cpu-moe` offload
  in general — this is a documented gap in the sources checked, not a guessed number.
@@ -0,0 +1,397 @@
 # A 48-minute total outage on `qwen3.8-27b-local`, and the third OmniRoute timeout mechanism this repo hadn't documented yet
 **Date:** 2026-09-15
 **Verdict:** The error — `"[504]: Direct response did not start within 30000ms — retrying on a fresh socket"` —
 comes from a **third, previously-undocumented OmniRoute timeout mechanism** (`OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`,
 default 30s), distinct from both timeouts already recorded in
 [`omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) and
 [`omniroute-non-ping-sse-stream-timeout.md`](./omniroute-non-ping-sse-stream-timeout.md). It exists specifically to
 recover from a *stale pooled TCP socket* by retrying once on a brand-new connection — but in the incident analyzed
 here, **both** the original attempt and the fresh-socket retry timed out, repeatedly, for 46 requests over 48
 straight minutes with zero successes. That pattern rules out a stale-socket explanation (a fresh socket bypasses
 the pool entirely) and points instead at the upstream itself — `llama-server`, or the R9700 GPU underneath it —
 being genuinely unresponsive for the whole window. The best primary-source match for that symptom is an **open,
 still-unresolved AMD ROCm bug specific to this exact GPU** ([ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630)):
 an MES-firmware hang during generation on `gfx1201`/R9700 that leaves the process alive but stuck, sometimes for
 no logged reason at all. No config change fixes this — raising the 30s timeout only makes each failed attempt
 take longer to give up, it doesn't un-wedge a hung GPU.
 ## The evidence
 **Source:** `omniroute-request-logs-6h-2026-09-15.json`, a 339-entry OmniRoute request-log export the user pulled
 from the dashboard, covering `2026-09-15T12:49:52Z`–`18:44:54Z`. Every entry with a non-200 status (66 of them)
 is a `POST /v1/chat/completions` against `qwen3.8-27b-local` (`/models/Qwen3.8-27B-UD-Q4_K_XL.gguf`), all on the
 same `connectionId` (`649a2d3e-7527-488e-9b8a-dc4ac2624176`) and the same `provider`
 (`openai-compatible-chat-a7bda643-6687-41f4-b75d-fd2cab746874`) — a single upstream connection, not a fan-out
 artifact.
 Sorting every error by timestamp shows **one continuous outage**, not scattered slow requests:
 - Last successful `/v1/chat/completions` before the outage: `15:18:12.285Z`
 - **First failure:** `15:25:06.415Z` — status 504, `"[504]: Direct response did not start within 30000ms —
  retrying on a fresh socket"`, duration `60186ms`
 - **Every single `/v1/chat/completions` attempt** from `15:25:06.415Z` through `16:13:29.502Z` failed — 46× 504
  (all clustered `60021`-`60324ms`, i.e. two back-to-back 30s attempts, both failing) interleaved with 20× 499
  (`"Request aborted"` / `"Client disconnected: request_signal_aborted"`, durations `1.4s`-`99.97s` — these are
  qwen-code giving up client-side while OmniRoute was still mid-retry)
 - **First successful recovery:** `16:13:50.388Z`, `20873ms` — 21 minutes after the last failure attempt cluster,
  i.e. the very next attempt after the outage window succeeded normally
 - No successful `/v1/chat/completions` call appears anywhere inside the `15:25:06Z`-`16:13:29Z` window — confirmed
  by filtering all 154 `/v1/chat/completions` log entries in that range: every one is 504 or 499.
 `git log --since=2026-09-14 --until=2026-09-16` shows **zero commits** in this repo on 2026-09-15 — the outage
 correlates with no deploy, `scripts/update.sh` run, or config change on this end.
 **A second, independent data point** (see caveat below): the user also pasted a large raw text table, copied
 directly from OmniRoute's dashboard UI rather than the JSON export, showing the **`qwen-classifier`** connection
 (`qwen3-4b`, both the `-UD-Q4_K_XL.gguf` file this repo's `docker-compose.yml` currently defaults to, and a
 `-UD-Q8_K_XL.gguf` variant that appears **nowhere** in this repo's checked-in `docker-compose.yml`/`.env.example`
 — either tested by hand against `LLAMA_CLASSIFIER_MODEL_FILE` outside version control, or evidence of drift worth
 checking directly on the server) failing repeatedly across two accounts (`Haylan`, `qwen-cli-main`), with the
 same signature: `TI: 0|TO: 0` (zero tokens either direction — failed before generating anything) and durations
 clustering at exactly `60.0`-`60.4s` for the 504s. That's the identical two-attempts-at-30s-each shape as the 27B
 outage above, strongly suggesting the same underlying mechanism, though — important caveat — **this table is not
 present in the 6-hour JSON export and covers a different, longer time range** (timestamps back to `01:21` and
 `23:54` on unspecified dates), so it cannot be directly time-correlated against the 27B outage above. Treat it as
 corroborating evidence that this failure mode recurs on both local model connections, not as proof they failed at
 the same moment.
 ## Root cause: `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`, a mechanism built for a different problem
 Confirmed directly against OmniRoute's own source, [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
 (same repo the two existing timeout docs already cite) — `open-sse/utils/directResponseStartTimeout.ts`:
 ```ts
 const DEFAULT_DIRECT_HEADERS_TIMEOUT_MS = 30_000;
 const DIRECT_RESPONSE_START_TIMEOUT_CODE = "DIRECT_RESPONSE_START_TIMEOUT";
 export function resolveDirectHeadersTimeoutMs(
  env: Record<string, string | undefined> = process.env
 ): number {
  const raw = env.OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS;
  if (raw == null || raw.trim() === "") return DEFAULT_DIRECT_HEADERS_TIMEOUT_MS;
  ...
 }
 function createDirectResponseStartTimeout(timeoutMs: number): Error & { code: string } {
  const err = new Error(
    `Direct response did not start within ${timeoutMs}ms — retrying on a fresh socket`
  ) as Error & { code: string };
  ...
 }
 ```
 "Direct" here means **direct (no-proxy) egress** — confirmed in `open-sse/utils/proxyFetch.ts`, which routes any
 connection with no configured upstream HTTP proxy through this path (as opposed to OmniRoute's separate
 proxy/relay egress paths). Every local connection in this repo (`llama-server`, `qwen-classifier`, both reached
 over the `ai-stack` Docker network with no proxy) is "direct" — so this timeout mechanism governs **every**
 request to either local model, streaming or non-streaming alike, not just non-streaming JSON responses as the
 name might suggest.
 ### Why it exists (and why it didn't help here)
 Two OmniRoute issues, both with dedicated regression tests in the repo, explain the actual design intent:
 - **#4252** (`tests/unit/proxyfetch-retry-fresh-socket-4252.test.ts`): "Undici dispatcher fails on direct provider
  requests in 502 bursts" — the default direct dispatcher pools keep-alive sockets; some upstreams silently close
  idle pooled sockets, so the next request reusing one fails with `UND_ERR_SOCKET`. Fix: retry once on a **fresh,
  no-keep-alive dispatcher** (`getRetryDispatcher()`, a different instance from `getDefaultDispatcher()`) so the
  retry can't grab another already-dead pooled socket.
 - **#10214** (`tests/unit/proxyfetch-direct-response-start-timeout-10214.test.ts`): "Direct (no-proxy) requests
  stall on a silently-dropped pooled keep-alive socket until the caller's deadline or a service restart" — the
  harder case: a pooled socket that dies **without even an error**, just silence. Undici's `headersTimeout`
  default (600s) is far too slow to catch this in practice, and the existing #4252 retry never fires because no
  error is thrown to trigger it. The fix bounds each direct attempt's response-start wait to
  `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` (30s default) via `directFetchWithBoundedResponseStart`, and retries
  **once** on the same fresh no-keep-alive dispatcher from #4252 when that bound is hit.
 Both fixes assume the *socket* is the problem, not the upstream. `directFetchWithBoundedResponseStart`'s own
 implementation (`open-sse/utils/directResponseStartTimeout.ts`) is exactly two attempts: pooled, then fresh. When
 attempt 2 — a brand-new socket that cannot possibly be a zombie pooled connection — **also** times out at 30s,
 the retry logic has nothing left to try and the request fails with `DIRECT_RESPONSE_START_TIMEOUT_CODE`
 (surfaced as the 504 seen in the logs). A fresh socket succeeding to *connect* but the *server* never sending a
 response is exactly what "the upstream process is alive but stuck" looks like from OmniRoute's side — it can't
 distinguish "GPU is wedged mid-generation" from "stale pooled socket," because both present as "nothing came
 back in 30s." 46 consecutive both-attempts-failed cycles over 48 minutes is far outside what a transient stale-socket
 burst (the scenario #4252/#10214 were built for) would produce; it's consistent with a sustained upstream
 outage instead.
 **Not currently configured in this repo**: `grep`-ing `docker-compose.yml` and `.env.example` for
 `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` finds nothing — this deployment runs on the unmodified 30s default. (This
 is also, notably, a *fourth* data point alongside the two already-documented mechanisms and the `requestQueue.maxWaitMs`/
 `RATE_LIMIT_EXECUTION_TIMEOUT` timer mentioned in passing in `omniroute-account-semaphore-timeout.md` — OmniRoute
 has at least four independent timeout knobs guarding different stages of a request's life, three of them
 30-second-flavored by default, which is worth keeping in mind the next time an unfamiliar timeout string shows up.)
 ## Why the upstream itself was likely unresponsive: ROCm/legacy-rocm-build#6630
 This session has **no SSH/shell access to the actual R9700 server** — there's no SSH config, and nothing in
 `scripts/` does remote exec, confirmed by inspecting `scripts/update.sh` and the absence of any `~/.ssh/config`
 entry for the box. So none of the following is confirmed against this specific incident's `dmesg`/`rocm-smi`/
 `docker logs` output — it's the closest primary-source match to the *symptom*, not a diagnosis of *this* outage.
 [ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630) ("gfx1201 R9700 ROCm 7.14
 llama.cpp generation hang, MES queue failure and PSP reset -62; Vulkan passes") is an **open**, actively
 investigated issue (created 2026-08-19, most recent update 2026-08-28, no fix landed) that reproduces on **the
 exact same GPU this repo runs on** — Radeon AI PRO R9700, `gfx1201` — running **llama.cpp with `-fa on`** (this
 repo's `llama-server` also runs `--flash-attn on`), with a controlled Vulkan-vs-ROCm A/B: Vulkan completes
 normally, ROCm hangs during token generation. Direct quotes:
 > "During the ROCm generation stall: GPU busy reached 100%, memory busy remained 0%, VRAM use was only about 1.1
 > GB, **the container remained alive but made no output progress**."
 > "when MES stops responding, **the driver can stay unaware of it indefinitely** — the failure only surfaces when
 > something happens to send the next MES message... if a run is left alone after it stops making progress, the
 > kernel prints nothing at all, so a hang can look like a slow workload rather than a fault."
 > "the failure is probabilistic, not deterministic... a single passing run on this host does not indicate a
 > healthy configuration."
 The thread (12 comments as of this research pass, an AMD engineer `harkgill-amd` participating) has ruled out,
 one at a time, `uni_mes=0`, `mes_log_enable=1`, ROCm 6.4.4, ROCm 7.14, ROCm 10.0.0 stable, the latest TheRock
 nightly, and GFXOFF-disable — **no confirmed fix or workaround exists in the thread as of this research pass**.
 A related comment on the same issue (`chrisfranson`) reports the identical MES `REMOVE_QUEUE`/MODE1-reset
 signature from a **completely unrelated workload** (headless LibreOffice with OpenCL) on the same `gfx1201`
 silicon, reinforcing that this is a driver/firmware-level fault under general GPU load, not something specific
 to llama.cpp's request pattern.
 **This is a different bug from the one already mitigated in this repo.** `docs/research/rocm-gpu-pin-and-render-group.md`
 already documents and works around `ROCm/ROCm#5706` (clock/power pinned at boost whenever two concurrent HIP
 contexts share the GPU — fixed via `GPU_MAX_HW_QUEUES=1`, already set on both `llama-server` and `qwen-classifier`
 in `docker-compose.yml`). #5706's symptom is elevated power draw with the GPU still working; #6630's symptom is
 generation fully halting with `gpu_busy=100%`/`mem_busy=0%` and MES no longer responding at all — a real hang, not
 a clock-pin inefficiency. `GPU_MAX_HW_QUEUES=1` targets #5706's specific trigger (hardware-queue oversubscription
 across concurrent HIP processes) and has no evidence in #6630's thread of affecting that bug — #6630 reproduces
 in single-GPU, single-process benchmarks with no second HIP context involved at all, so the already-applied fix
 should not be assumed to help here.
 ## What would actually resolve this vs. what wouldn't
 - **Raising `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`** — not recommended as a fix. It would make each failed attempt
  take longer before giving up (worse latency during a real hang), without addressing why the GPU stopped
  responding. It's the right lever only if future evidence shows genuinely-slow-but-working responses being
  mistaken for hangs (the same shape as the already-fixed `REQUEST_TIMEOUT_MS` issue in
  `omniroute-non-ping-sse-stream-timeout.md`) — this incident's 46-for-46 both-attempts-failed pattern over 48
  minutes doesn't fit that shape.
 - **Concrete next step for whoever has server access when this recurs**: check `dmesg | grep -i amdgpu` and
  `journalctl -k` on the R9700 host for `MES(...) failed to respond`, `GPU reset begin`, or `PSP resume failed`
  lines matching #6630's signature, and `docker logs llama-server`/`docker logs qwen-classifier` to see whether
  the process was alive-but-stuck (consistent with #6630) versus crashed/restarted (which would point elsewhere).
  Capturing this during a live incident is the only way to move this from "best primary-source match" to
  "confirmed root cause."
 - **Monitor [ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630)** for a fix —
  it's open and active (AMD engineer engaged as of 2026-08-26); no released ROCm version as of this research pass
  is confirmed clean.
 - **The `-UD-Q8_K_XL.gguf` classifier variant in the pasted table but absent from version control** is worth a
  direct look on the server (`cat .env` / `docker inspect qwen-classifier` for the actual `LLAMA_CLASSIFIER_MODEL_FILE`
  in effect) — outside this research pass's reach without server access, flagged here so it isn't lost.
 ## Recovery: how the hang actually clears (or doesn't) — addendum, 2026-09-15
 Follow-up question: what actually recovers the socket once this hits, given it's been observed to stay wedged
 for days at a time? Pulled the full comment thread on
 [ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630) directly via the GitHub API
 (12 comments, `angelhalo` as primary reporter, `harkgill-amd` as the responding AMD engineer, plus one
 corroborating report from `chrisfranson` on unrelated hardware/workload) — the earlier research pass's source list
 cited this issue but hadn't read the full thread. Three things fall directly out of it:
 **There is no reliable in-band recovery.** The driver doesn't notice the hang on its own — direct quote:
 "when MES stops responding, the driver can stay unaware of it indefinitely — the failure only surfaces when
 something happens to send the next MES message." In a captured live hang, "every driver-managed ring is
 completely idle while the GPU reports 100% busy," `dmesg` has zero amdgpu lines, and no task is in D-state —
 so from the OS's perspective nothing is wrong; only sending the GPU another command (which killing/restarting
 the stuck process does) triggers the driver to discover the wedge and attempt its own MODE1 reset.
 **Once triggered, that reset itself is a coin flip across three documented outcomes**, not a guaranteed fix:
 1. **Clean recovery** — `GPU reset succeeded, trying to resume`, PSP resumes, the card comes back (VRAM is
   wiped — "VRAM is lost due to GPU reset!" — so the container needs a real restart to reload the model, a plain
   process respawn isn't enough even when the reset itself works).
 2. **Failed resume** — `PSP resume failed`, `GPU reset end with ret = -62` (the original report's own outcome) —
   the reset attempt itself fails, leaving the GPU in a worse state than before.
 3. **Full kernel soft-lockup** — `chrisfranson`'s independent report (different workload — headless LibreOffice
   OpenCL, different card — RX 9070 XT, same `gfx1201` silicon) hit outcome 2 or 3 twice out of three times:
   "the whole system hard-locked (kernel soft lockup pegging a CPU at ~90-100% softirq, requiring a physical
   power cycle)."
 `angelhalo` deliberately left one hang untouched rather than killing the process, to observe it without
 contaminating the state with a reset: "I recovered only with a subsequent cold power cycle" — no reset was ever
 triggered because nothing sent the GPU another message. Their standard test procedure between every single run
 in this thread is "a cold power cycle (AC removed, ≥30 s)," specifically **not** a warm/soft reboot — stated
 reason: "on this card a MODE1 reset takes the host down with it," meaning even the OS's own reboot path can't
 be trusted to come back cleanly once this GPU is in a bad state. This is the practical answer to "why does it
 stay stuck for days": nothing about the hang self-clears, `docker`'s `restart: unless-stopped` policy never
 fires because the container process is alive and never exits (confirmed: this repo's `llama-server` and
 `qwen-classifier` services have no `healthcheck` block at all — only `omniroute` itself does, a plain TCP
 connect check on its own dashboard port, which says nothing about whether `llama-server`/`qwen-classifier` are
 responding) — so a hang persists until a human notices the symptom (requests failing) and manually intervenes,
 and "days" is just however long that takes to notice on a homelab box, not a property of the hang itself.
 **No fix or reliable mitigation exists as of this reading (2026-08-28, the thread's latest comment).**
 `harkgill-amd` (AMD) could not reproduce locally and asked for a nightly-driver retest; `angelhalo` retested and
 it still failed. Every other variable tested still hangs: ROCm 6.4.4 through 10.0.0 stable, TheRock nightlies,
 `amdgpu.uni_mes=0`, `cwsr_enable=0`, `mes_log_enable=1`, GFXOFF disabled, two different physical R9700 cards, both
 llama.cpp and vLLM. The thread's own conclusion, as of the last comment: "a probabilistic lost-completion event"
 with no known trigger to avoid and no known driver/firmware combination that's clean.
 **Practical takeaway for this repo, given no upstream fix exists:**
 - A restart *might* recover it, *might* make it worse (failed PSP resume), and *might* take the whole host down
  requiring a physical power cycle — there's no way to know in advance which outcome a given hang will produce.
 - Nothing currently watches for this automatically. Docker's `restart: unless-stopped` is the wrong tool (process
  doesn't exit) — recovering automatically would need a `healthcheck` against `llama-server`'s own `/health`
  endpoint (llama.cpp's built-in liveness endpoint) paired with something that acts on an `unhealthy` status,
  since Docker itself doesn't restart on failed healthchecks without an external watcher (e.g. `willfarrell/autoheal`
  or equivalent) — not evaluated here, flagged as a real gap, not a recommendation to implement blind: an
  automated restart during a hang that's about to fail its PSP resume and lock the host could turn a
  "requests are failing" incident into "the box needs a physical power cycle" automatically and unattended,
  which is a real downside worth weighing against faster detection.
 - Given the reset outcome is unpredictable, the safest manual recovery when this is caught live is: restart the
  affected container, then immediately check `dmesg | grep -i amdgpu` for `PSP resume failed` or a soft-lockup
  signature before assuming it's fixed — if either appears, a full reboot (and per this thread's own testing
  practice, possibly a genuine AC power cycle rather than a warm reboot) is the next step, not a second restart
  attempt.
 ## Caveats and open questions
 - **JSON export vs. pasted table are two different, non-overlapping captures.** The JSON file is a precise 6-hour
  window with full per-request detail; the pasted table is a longer, dashboard-UI-copied range with less
  structure and no verifiable overlap with the JSON file's timestamps. A fresh multi-day JSON export (same
  `request-logs` endpoint used to produce the file analyzed here) would let a future pass check whether the
  classifier's failures and the 27B model's outage are literally simultaneous (strong evidence for a shared
  GPU-level cause) or independent recurrences of the same mechanism on separate schedules.
 - **Why the outage self-recovered after ~48 minutes with no observed restart is unexplained.** #6630's thread
  describes hangs resolving via an explicit GPU reset (sometimes failing, requiring reboot) — not a case of a
  hang clearing on its own after a fixed interval. Nothing in the available data (no server access) confirms
  whether a restart happened that isn't visible from OmniRoute's logs, or whether this specific hang genuinely
  self-cleared, which would be a data point *against* the #6630 hypothesis worth capturing next time.
 - **Live reproduction was not attempted.** The user suggested testing tool-calls against the classifier via the
  Windows-side qwen-code CLI (`C:\Users\aerli\AppData\Local\qwen-code\bin\qwen.cmd`) to try to reproduce a
  "Direct response did not start" failure live. Skipped for this pass: qwen-code requires an interactive/
  already-authenticated session to drive meaningfully, and deliberately trying to reproduce a GPU hang against
  the shared production classifier risked a genuine 60s+ stall on infrastructure other work depends on, for
  uncertain diagnostic payoff given the strength of the log-based and source-based evidence already gathered.
  Worth doing deliberately, with server access on hand to capture `rocm-smi`/`dmesg` simultaneously, rather than
  as a quick check from this pass.
 ## Sources
 - `omniroute-request-logs-6h-2026-09-15.json` — OmniRoute dashboard request-log export provided by the user
  (2026-09-15, 339 entries, `12:49:52Z`-`18:44:54Z`)
 - User-pasted OmniRoute dashboard table (`qwen-classifier`/`qwen3-4b` failures, separate capture window)
 - [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) —
  `open-sse/utils/directResponseStartTimeout.ts`, `open-sse/utils/proxyFetch.ts`, `open-sse/utils/proxyDispatcher.ts`,
  `tests/unit/proxyfetch-direct-response-start-timeout-10214.test.ts`, `tests/unit/proxyfetch-retry-fresh-socket-4252.test.ts`,
  `open-sse/handlers/chatCore/upstreamTimeouts.ts`
 - [ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630) — open R9700/gfx1201
  llama.cpp generation-hang issue, full comment thread (2026-08-19 through 2026-08-28)
 - [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the
  first already-documented 30s OmniRoute timeout (account semaphore, 429, hardcoded)
 - [`docs/research/omniroute-non-ping-sse-stream-timeout.md`](./omniroute-non-ping-sse-stream-timeout.md) — the
  second already-documented timeout (first-SSE-event deadline, `REQUEST_TIMEOUT_MS`-derived)
 - [`docs/research/rocm-gpu-pin-and-render-group.md`](./rocm-gpu-pin-and-render-group.md) — the already-mitigated,
  *different* R9700/gfx1201 MES bug ([ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706), clock-pin/power,
  not a hang)
 - Local `docker-compose.yml`, `.env.example` (grepped directly, confirming `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`
  is unset / on the 30s default, and that `GPU_MAX_HW_QUEUES=1` is already applied to both GPU services)
 - `git log --since=2026-09-14 --until=2026-09-16` (this repo, confirming zero commits during the outage window)
 ## Confidence / uncertainty summary
 - **High confidence**: the exact mechanism and semantics of `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` /
  `directFetchWithBoundedResponseStart` (read directly from OmniRoute's own source and its two regression test
  files, which spell out the intent in comments referencing the originating issues); the JSON log's chronology,
  single-connection scope, and the "two ~30s attempts, both failing" shape of every 504 (computed directly from
  the log file); that this timeout is unconfigured in this repo (direct grep); that no commits landed in this
  repo during the outage window (direct `git log`).
 - **Medium confidence**: that ROCm/legacy-rocm-build#6630 is the actual root cause of *this specific* outage.
  The GPU model, driver-family symptom shape (`gpu_busy=100%`/`mem_busy=0%`, alive-but-stuck, sometimes
  logged/sometimes silent), and `-fa on` usage all match closely, and it's an open/unresolved/actively-discussed
  issue as of this research pass — but nothing from this incident's own `dmesg`/`rocm-smi` output was available
  to confirm it directly (no SSH access), so this is the best primary-source match to the symptom, not a
  confirmed diagnosis.
 - **Low confidence / open**: why the outage recovered on its own after ~48 minutes with no observed restart;
  whether the classifier's pasted-table failures share the exact same triggering event as the 27B outage
  analyzed here (same failure signature, but no verified time-overlap between the two data sources); the
  provenance of the `-UD-Q8_K_XL.gguf` classifier variant seen in the pasted table but absent from version
  control.
 ## Live test, 2026-09-15: the classifier is measurably too slow at its own documented worst case — independent of any hang
 Before scoping a fix, tested the live `qwen-classifier` backend directly against realistic worst-case load,
 per the user's request to gather fresh evidence rather than design blind. Two attempts to reproduce this through
 qwen-code itself first surfaced an unrelated, separately-useful finding; the direct backend test below is what
 actually answered the question.
 ### qwen-code's own headless mode never reaches the classifier
 Ran `qwen --approval-mode auto <prompt>` (positional/one-shot, non-interactive) from the Windows-side install
 (`C:\Users\aerli\AppData\Local\qwen-code\bin\qwen.cmd`, which has both `fastModel` and the `omniroute-search` MCP
 server already configured), asking it to run a shell `dir` and use the web-search MCP tool. Both attempts hit a
 wall before any classifier request was even sent:
 - The MCP tool call was refused outright: `Warning: Tool "mcp__omniroute-search__search" requires user approval
  but cannot execute in non-interactive mode. ... use the -y flag (YOLO mode)`.
 - The shell tool: the model itself reported `run_shell_command` as "not registered" in this session and silently
  substituted a read-only `glob` call instead — no approval prompt, no classifier call, no system warning printed
  (unlike the MCP case), across two separate clean runs.
 **Conclusion: one-shot headless `qwen <prompt>` invocations don't exercise Auto Mode's classifier at all for
 approval-requiring tools** — they're declined or silently rerouted before the classifier ever gets a request.
 The classifier only fires in a genuinely interactive session, where it substitutes for the human's live approval
 decision. This wasn't previously documented anywhere in this repo and is worth keeping in mind: headless qwen-code
 testing is not a valid way to probe classifier behavior, live or otherwise. (Not investigated further: whether
 `qwen serve`/`--input-format stream-json` headless-agent modes behave differently — plausible, since they're
 built for exactly this kind of automation, but out of scope for this pass.)
 ### Direct backend test: real classifier latency at realistic token counts
 Given headless qwen-code couldn't drive this, sent shaped classifier requests straight to
 `http://proxy-ai.home/v1/chat/completions` (model `qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`,
 confirmed present in `GET /v1/models`) using a temporary scoped API key, mimicking the `{shouldBlock}`
 JSON-verdict shape and the `MAX_TRANSCRIPT_MESSAGES=40` / `MAX_HISTORICAL_ACTION_CHARS=4000` structure this
 repo's own `fast-model-choice.md` already read out of qwen-code's `classifier-transcript.ts` source. Five calls,
 in order:
 | # | Prompt tokens | `cached_tokens` | Wall time | Notes |
 |---|---|---|---|---|
 | 1 | 1,000 | 3 | 2.24s | Small prompt, genuinely fresh — healthy baseline. |
 | 2 | 51,234 | 51,233 | **64.21s** | First send of a large synthetic worst-case transcript (~40 highly self-similar 4K-char "historical action" blocks). Near-total cache hit reported, yet still the slowest call — see caveat below. |
 | 3 | 51,234 | 51,233 | 0.07s | **Identical repeat of #2.** Same `chatcmpl-...` id as #2 came back — this is OmniRoute short-circuiting an exact-duplicate request via a proxy-level response cache, not fresh inference. Confirms #2/#3's `cached_tokens` field is not a reliable proxy for wall-clock latency on its own. |
 | 4 | 1,000 | 3 | 0.03s | Identical repeat of #1 — same id, same response-cache short-circuit. |
 | 5 | 30,958 | 919 | **28.66s** | Fresh, non-repeated worst-case-shaped transcript (different random content, no internal self-similarity to trigger cache effects). Mostly-uncached (919/30,958) — this is the clean data point. |
 Call #5 is the one to trust: **~31K genuinely-fresh prompt tokens took 28.7 seconds** on the current
 `--n-gpu-layers 28` (of 36) / `--cache-type-k/v q4_0` / `--parallel 1` configuration, with the backend otherwise
 idle and healthy (no hang in progress). Extrapolating that rate to the repo's own documented worst case (a real
 15,116-token call observed live per `fast-model-choice.md`, and a theoretical ceiling around 40-50K tokens per
 `classifier-transcript.ts`'s limits) puts a genuine worst-case classifier call at **roughly 30-65 seconds of
 normal, non-hung processing time** — consistent with call #2's 64.21s, even though that call's own cache
 metadata is too muddied by internal prompt self-similarity to use as a second clean sample.
 **This directly overlaps both binding timeouts**: qwen-code's own client-side classifier stage timeout
 (`stage1Ms`/`stage2Ms`, `60000` each in the Windows-side `settings.json` observed this session, `30000`/`60000`
 in the WSL-side one) and `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`'s 30-second-per-attempt window documented above.
 A worst-case classifier call landing in the 30-65s range will **routinely** trip one or both of these timeouts
 on its own, with the backend never having hung at all — the exact same 499/504/"Request aborted" shape as the
 GPU-hang hypothesis produces, but from an entirely mundane, deterministic cause: **the classifier's current
 configuration is simply too slow for the request sizes qwen-code's own classifier-transcript design allows.**
 ### What this changes
 This doesn't rule out ROCm/legacy-rocm-build#6630 — the 27B model's 48-minute total outage (zero successes, not
 just slow ones) doesn't fit a "just slow" explanation, and remains best matched by a real GPU hang. But it does
 mean **the classifier's own recurring failures (the pasted-table evidence) very plausibly have a second,
 independent, non-probabilistic cause that a healthcheck/restart-on-hang design wouldn't fix at all** — restarting
 a classifier that's merely slow-but-working at worst-case load just interrupts a call that would have succeeded,
 and would fire repeatedly under normal peak usage, not just during a rare hang. Any fix that only targets "detect
 and recover from an unresponsive GPU" leaves this second failure mode untouched. Two independent levers worth
 weighing before finalizing a scope: raising the classifier's own timeouts to match its real worst-case latency
 (cheap, immediate, but does nothing for actual hangs), and/or speeding up the classifier itself (full GPU offload
 if VRAM allows, a faster quant, or capping the transcript size client-side) to bring worst-case latency back
 under the existing timeouts.
 **Not investigated in this pass**: whether call #2's 64.21s (vs. call #5's extrapolated ~45-48s at a similar
 token count) reflects genuine non-linear slowdown at the very largest context sizes, real concurrent contention
 from other production traffic sharing the same `--parallel 1` slot during the test, or is just noise from a
 single sample each — worth a few more clean, uniquely-content, worst-case-sized calls at different times of day
 before treating either number as precise.
@@ -0,0 +1,216 @@
 # OmniRoute's builtin memory tools silently hijack qwen-code's classifier tool-call, not a model or GPU problem
 **Date:** 2026-09-15
 **Verdict:** The `"Classifier stage 1 unavailable"` / `"Auto Mode couldn't classify this action"` failures are **not**
 a GPU hang, not a timeout, and not a Qwen3-4B quality problem. Confirmed directly from a live debug log: the fast
 model (`qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`) *is* being used (`stage=fast` in every classifier
 log line), and it responds well within its timeout (18.7s against a 30-60s budget). The failure is
 `"Error: Invalid side query response: params must have required property 'shouldBlock'"` — a **schema-validation
 failure on an in-time response**. Root cause, confirmed directly against OmniRoute's own source
 (`open-sse/handlers/chatCore/memorySkillsInjection.ts`): **OmniRoute silently appends its own builtin memory tools
 (`memory_save`/`update`/`search`/`delete`) to every non-streaming chat completion's `tools` array**, whenever
 memory is enabled for the calling key — regardless of what tools the caller declared. qwen-code's classifier forces
 `tool_choice: ANY` (call *some* tool, not a specific one) so it can get a structured `respond_in_schema` JSON
 response. With OmniRoute's extra tools spliced in, the small model sometimes picks the injected `memory_save` tool
 instead — and since qwen-code's client only extracts the classifier's answer from a `respond_in_schema` function
 call (not from a stray `memory_save` call, even if the answer also happens to be present as plain text), the
 result validated against `STAGE1_SCHEMA` is empty, producing exactly the observed error.
 ## The debug-log evidence
 Captured directly from a `-d` (debug) qwen-code run, `C:\Users\aerli\.qwen\debug\886d00eb-...txt`:
 ```
 21:09:56 [DEBUG] [CLASSIFIER] ALLOW stage=fast tool=mcp__omniroute-search__search durationMs=15412
 21:10:28 [WARN]  [CLASSIFIER] failUnavailable stage=fast durationMs=18727 reason="Classifier stage 1 unavailable" cause="Error: Invalid side query response: params must have required property 'shouldBlock'"
 ```
 Both lines are tagged `stage=fast` — qwen-code's own internal label confirming the classifier used the configured
 fast model both times, settling a live question this session raised about whether the classifier was silently
 falling back to the main 27B model. It wasn't. The second call's 18.7s duration is comfortably inside this
 deployment's configured `stage1Ms` (30000-60000ms depending on install) — this was not a timeout.
 ## Reproduced directly
 Sent the exact same request shape qwen-code's `classifyAction()`/`generateJson()` builds — a forced tool call to a
 synthetic `respond_in_schema` function with `STAGE1_SCHEMA` (`{required: ["shouldBlock"], properties: {shouldBlock:
 {type: "boolean"}}}`) as its only declared tool, `tool_choice: "required"` — directly against
 `POST /v1/chat/completions` on this deployment (model `qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`).
 The very first response:
 ```json
 {
  "message": {
    "content": "{\"shouldBlock\": true}",
    "tool_calls": [{
      "type": "function",
      "function": {
        "name": "memory_save",
        "arguments": "{\"key\": \"tool_call_review\", \"content\": \"web_fetch call to https://www.swfte.com/ai/leaderboard was blocked due to potential data exfiltration or unauthorized external access.\", \"type\": \"factual\"}"
      }
    }]
  },
  "tool_results": [{ "output": "{\"success\":true,\"memory\":{\"id\":\"84389721-...\", ...}, \"message\":\"Memory saved successfully\"}" }]
 }
 ```
 `memory_save` was never declared in the request's `tools` array — only `respond_in_schema` was. OmniRoute added it
 server-side, the model called it instead of (or alongside) the intended tool, and OmniRoute **auto-executed it**
 (`tool_results` shows a real memory record was created, with a UUID and timestamp — this isn't inert). The correct
 `{"shouldBlock": true}` answer is sitting right there in `content`, but qwen-code's `generateJson()` only ever
 looks for it inside a `respond_in_schema` function call (`functionCalls.find(call => call.name ===
 "respond_in_schema")`); when that's absent it falls through to a raw-text JSON parse of `getResponseText(result)`
 — and OpenAI-client conventions generally don't populate a response's plain-text field when `finish_reason` is
 `tool_calls` rather than `stop`, so that fallback likely returns nothing, yielding `{}` against a schema that
 requires `shouldBlock`. This is a single confirmed repro, not a guaranteed-every-time reproduction — the injected
 tool doesn't win every time (an immediate identical follow-up request returned an OmniRoute-cached copy of the same
 response, not a fresh sample — see the cache caveat in
 [`omniroute-direct-response-timeout-outage-2026-09-15.md`](./omniroute-direct-response-timeout-outage-2026-09-15.md)),
 but it reproduces the *exact* failure shape from the live debug log on the first genuine attempt.
 ## Root cause, confirmed in OmniRoute's own source
 [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) —
 `open-sse/handlers/chatCore/memorySkillsInjection.ts`:
 ```ts
 if (memoryOwnerId && memorySettings?.enabled && body.stream !== true) {
  // Server-side builtin memory tools (memory_save/update/search/delete) are
  // executed by the gateway's tool-call interception, which runs only on the
  // non-stream path. Stream clients (opencode etc.) execute tools client-side,
  // so for them these tools would be announced but never executed; they should
  // use the MCP memory tools (omniroute_memory_*) instead.
  const existingTools = Array.isArray(body.tools) ? body.tools : [];
  ...
  const memoryTools = buildMemoryToolsForProvider(...).filter(tool => !existingToolNames.has(name));
  if (memoryTools.length > 0) {
    body = { ...body, tools: [...existingTools, ...memoryTools] };
  }
 }
 ```
 This runs unconditionally for any non-streaming request from a key with memory enabled — there is no exemption
 for a caller that already set `tool_choice` to force a *specific* tool. qwen-code's classifier is exactly this
 case: a single-purpose, forced-`ANY`, non-streaming tool call, which is precisely the shape this injection logic
 was not written to avoid interfering with.
 `src/lib/memory/settings.ts` confirms `enabled: false` is the *default* — memory is off by default in a fresh
 OmniRoute install specifically because of injected-context cost, per its own comment:
 > "Off by default: enabling memory injects up to `maxTokens` (~2k) of retrieved context into every chat request,
 > which is billed — a surprising cost for new installs... Opt in explicitly via Settings → Memory... Per-request
 > opt-out is also available via the `x-omniroute-no-memory` header."
 This deployment has memory enabled (confirmed live by the reproduction above), which is presumably a deliberate
 choice for other workflows (chat memory across sessions) — but it has an undocumented-to-this-repo side effect on
 any caller using forced-tool-call classification.
 ## What would fix this
 Two per-target exclusions were checked live against this deployment and confirmed **not to exist**:
 - **Per-API-key memory override**: `GET /api/keys` was fetched directly (the temporary key handed to this session
  turned out to carry admin access, well beyond the plain `/v1` workload scope its name implied). Every key's full
  field list was inspected — `noLog`, `scopes`, `allowedModels`, `rateLimits`, `disableNonPublicModels`, etc. — with
  no memory-related field anywhere.
 - **Per-model/connection override**: `GET /api/providers/<id>` for the classifier's own connection
  (`qwen3-4b`, id `b78ceb4c-52f8-47ae-b245-483baa6e3fc2`) was fetched directly. `providerSpecificData` (`prefix`,
  `apiType`, `baseUrl`, `nodeName`, `timeoutMs`, `apiKeyHealth`) has no memory field either — consistent with the
  source: `memoryOwnerId` is resolved purely from the *calling key* (`resolveMemoryOwnerId(apiKeyInfo)`), before
  OmniRoute has even picked a provider, so it can't know or care that this particular request targets the
  classifier model specifically.
 **The fix that was actually available and is now applied**: `x-omniroute-no-memory`, OmniRoute's own per-*request*
 opt-out (not per-key or per-model), confirmed end-to-end and traced through both sides:
 - OmniRoute's handling, confirmed directly in `open-sse/handlers/chatCore.ts` and its own test suite
  (`tests/unit/no-memory-header.test.ts`): `memoryOwnerId = isNoMemoryRequested(headers) ? null : resolveMemoryOwnerId(...)`
  — a null owner id short-circuits *both* branches in `injectMemoryAndSkills` (context injection and tool
  injection). The test suite gives the exact accepted values: `"true"`, `"1"`, `"yes"` (case-insensitive on both
  the header name and value); `"false"`/`"0"`/`"no"`/empty do not trigger it.
 - qwen-code's support for sending it, confirmed against the installed bundle, *not* just the docs: `modelProviders.
  openai[].generationConfig.customHeaders` (documented at
  [model-providers](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/model-providers/)) flows into
  `DefaultOpenAICompatibleProvider.buildClient()` (`chunk-CXTPVBFA.js`), which passes it straight into the
  underlying `OpenAI` SDK client as `defaultHeaders` — applied to every request made through that one model entry,
  and *only* that entry (confirmed this is the generic OpenAI-compatible-chat client, the same one the classifier's
  `apiType: "chat"` connection uses — not Anthropic- or Responses-API-specific plumbing).
 **Applied**, 2026-09-15: added to the `qwen3-4b-classifier` entry in the Windows-side `~/.qwen/settings.json`
 (`C:\Users\aerli\.qwen\settings.json`), inside its `generationConfig`, alongside the existing `contextWindowSize`
 and `extra_body`:
 ```json
 "customHeaders": { "x-omniroute-no-memory": "true" }
 ```
 Scoped to this one model entry only — the main `qwen3.8-27b-local` connection's `generationConfig` is untouched,
 so its own memory-context behavior (if any is relied on elsewhere) is unaffected. qwen-code's own
 `[MODEL_PROVIDERS_HOT_RELOAD]` settings watcher (confirmed present in this session's debug log) should pick this
 up on the already-running session without a restart. **Not yet verified live** — the next classifier failure (or a
 deliberate repro, per the "Reproduced directly" section above) should confirm no `memory_save`-shaped tool call
 appears in the response once this is in effect.
 - **Remaining fallback, if the header approach doesn't hold up**: disable memory globally for this deployment
  (`PATCH /api/settings/memory`, `enabled: false`, or Settings → Memory in the dashboard) — blunt, but confirmed to
  work by definition since `enabled: false` is every fresh install's default.
 - **Also worth doing regardless**: file this upstream with OmniRoute. Their own code already special-cases one
  caller type (streaming clients) right next to this injection logic; a similar exemption for a caller that already
  set `tool_choice` to force one specific tool would be a clean fix on their end that doesn't depend on every
  client remembering to send an opt-out header.
 - **Not a fix, and not the problem**: nothing on the classifier-model or llama.cpp side. Qwen3-4B-Instruct-2507
  correctly produced the right answer (`{"shouldBlock": true}`) in the one reproduction captured here — the model
  was never at fault.
 ## Scope note
 This session's earlier hypothesis that the 27B model's 48-minute total outage
 ([`omniroute-direct-response-timeout-outage-2026-09-15.md`](./omniroute-direct-response-timeout-outage-2026-09-15.md))
 was caused by a ROCm/gfx1201 GPU hang is set aside here per explicit direction, not retracted — that was a
 different incident (zero successes for 48 straight minutes, a shape this memory-injection bug doesn't produce) and
 this finding doesn't bear on it either way.
 ## Sources
 - Live debug log, `C:\Users\aerli\.qwen\debug\886d00eb-5b2b-4d84-b1ef-60909f75eec2.txt` (this session, 2026-09-15)
 - Direct reproduction against this deployment's `POST /v1/chat/completions` (this session, 2026-09-15)
 - Live `GET /api/keys`, `GET /api/providers`, `GET /api/providers/b78ceb4c-52f8-47ae-b245-483baa6e3fc2`,
  `GET /api/settings/memory` against this deployment's OmniRoute instance (this session, 2026-09-15) — confirmed no
  per-key or per-connection memory field exists in either schema
 - [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) —
  `open-sse/handlers/chatCore/memorySkillsInjection.ts`, `open-sse/handlers/chatCore.ts` (the
  `isNoMemoryRequested`/`resolveMemoryOwnerId` branch), `src/lib/memory/settings.ts`, `src/lib/memory/injection.ts`,
  `open-sse/mcp-server/tools/memoryTools.ts`, `tests/unit/no-memory-header.test.ts` (exact accepted header
  name/value set)
 - [Qwen Code docs — Model Providers](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/model-providers/)
  (`customHeaders` field, documented under `generationConfig`)
 - Installed qwen-code bundle — `chunk-N7VWZDWW.js`, `chunk-HBU7EKY4.js` (`classifyAction`, `runSideQuery`,
  `resolveDefaultModel`, `generateJson`, `resolveFastModelSelector`, `getFastModel`) and `chunk-CXTPVBFA.js`
  (`DefaultOpenAICompatibleProvider.buildHeaders()`/`buildClient()`, confirming `customHeaders` reaches the actual
  OpenAI SDK client as `defaultHeaders` for the plain chat-completions path the classifier uses) — all read
  directly from the bundled (unminified variable names) source, not inferred from docs alone
 - Applied fix: `C:\Users\aerli\.qwen\settings.json`, `qwen3-4b-classifier` entry's `generationConfig.customHeaders`
  (this session, 2026-09-15)
 ## Confidence / uncertainty summary
 - **High confidence**: the fast model is genuinely used for classification (`stage=fast` in qwen-code's own debug
  log, both on success and failure); the failure is a schema-validation error on an in-time response, not a
  timeout (18.7s duration, explicit error text); OmniRoute's `memorySkillsInjection.ts` unconditionally injects
  builtin memory tools into non-streaming completions for any memory-enabled key, with no exemption for
  forced-single-tool callers (read directly from source); no per-key or per-model/connection memory override
  exists in this OmniRoute version (confirmed by reading the complete live schema of both, not by absence of
  documentation); `x-omniroute-no-memory: true` is a real, working per-request opt-out on OmniRoute's side (its
  own test suite) and is reachable from qwen-code via `modelProviders.openai[].generationConfig.customHeaders`,
  traced to the exact HTTP client the classifier's connection type uses (not inferred from docs alone — confirmed
  against the bundled source's actual header-merging code).
 - **Medium confidence**: that this exact tool-injection mechanism explains the *specific* production failures seen
  earlier in this session's testing (the reproduction matches the failure shape and the source confirms the
  mechanism exists and applies to this call pattern, but the live debug-log failure itself wasn't captured
  mid-flight with response inspection — only its aftermath, the error message).
 - **Low confidence / not verified**: the exact conditions under which the model picks the injected tool over the
  intended one (one clean reproduction on the first attempt, not a characterized hit rate — the failure may not be
  deterministic, so the `customHeaders` fix should still be watched rather than assumed to have fully resolved it
  on the strength of this write-up alone); whether the applied `customHeaders` fix has been confirmed live yet
  (not as of this writing — see "Applied" above).
Author	SHA1	Message	Date
haylan	a3f1099bfc	docs(research): OmniRoute memory tools hijack classifier tool-call, x-omniroute-no-memory fix	2026-09-15 23:31:58 +02:00
haylan	ed90256f43	docs(research): trace 48-min outage to third OmniRoute timeout and open ROCm#6630 GPU hang	2026-09-15 23:31:58 +02:00
haylan	fa852c8fb4	docs(research): qwen-classifier needs no context-window match; 8B/30B won't fit VRAM	2026-09-15 23:31:58 +02:00