From a3f1099bfcb59606a943c11d1d82bc57c4d204a7 Mon Sep 17 00:00:00 2001 From: ArthurErlich Date: Tue, 15 Sep 2026 23:31:58 +0200 Subject: [PATCH] docs(research): OmniRoute memory tools hijack classifier tool-call, x-omniroute-no-memory fix --- ...memory-tool-injection-breaks-classifier.md | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 docs/research/omniroute-memory-tool-injection-breaks-classifier.md diff --git a/docs/research/omniroute-memory-tool-injection-breaks-classifier.md b/docs/research/omniroute-memory-tool-injection-breaks-classifier.md new file mode 100644 index 0000000..67950bb --- /dev/null +++ b/docs/research/omniroute-memory-tool-injection-breaks-classifier.md @@ -0,0 +1,216 @@ +# OmniRoute's builtin memory tools silently hijack qwen-code's classifier tool-call, not a model or GPU problem + +**Date:** 2026-09-15 + +**Verdict:** The `"Classifier stage 1 unavailable"` / `"Auto Mode couldn't classify this action"` failures are **not** +a GPU hang, not a timeout, and not a Qwen3-4B quality problem. Confirmed directly from a live debug log: the fast +model (`qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`) *is* being used (`stage=fast` in every classifier +log line), and it responds well within its timeout (18.7s against a 30-60s budget). The failure is +`"Error: Invalid side query response: params must have required property 'shouldBlock'"` — a **schema-validation +failure on an in-time response**. Root cause, confirmed directly against OmniRoute's own source +(`open-sse/handlers/chatCore/memorySkillsInjection.ts`): **OmniRoute silently appends its own builtin memory tools +(`memory_save`/`update`/`search`/`delete`) to every non-streaming chat completion's `tools` array**, whenever +memory is enabled for the calling key — regardless of what tools the caller declared. qwen-code's classifier forces +`tool_choice: ANY` (call *some* tool, not a specific one) so it can get a structured `respond_in_schema` JSON +response. With OmniRoute's extra tools spliced in, the small model sometimes picks the injected `memory_save` tool +instead — and since qwen-code's client only extracts the classifier's answer from a `respond_in_schema` function +call (not from a stray `memory_save` call, even if the answer also happens to be present as plain text), the +result validated against `STAGE1_SCHEMA` is empty, producing exactly the observed error. + +## The debug-log evidence + +Captured directly from a `-d` (debug) qwen-code run, `C:\Users\aerli\.qwen\debug\886d00eb-...txt`: + +``` +21:09:56 [DEBUG] [CLASSIFIER] ALLOW stage=fast tool=mcp__omniroute-search__search durationMs=15412 +21:10:28 [WARN] [CLASSIFIER] failUnavailable stage=fast durationMs=18727 reason="Classifier stage 1 unavailable" cause="Error: Invalid side query response: params must have required property 'shouldBlock'" +``` + +Both lines are tagged `stage=fast` — qwen-code's own internal label confirming the classifier used the configured +fast model both times, settling a live question this session raised about whether the classifier was silently +falling back to the main 27B model. It wasn't. The second call's 18.7s duration is comfortably inside this +deployment's configured `stage1Ms` (30000-60000ms depending on install) — this was not a timeout. + +## Reproduced directly + +Sent the exact same request shape qwen-code's `classifyAction()`/`generateJson()` builds — a forced tool call to a +synthetic `respond_in_schema` function with `STAGE1_SCHEMA` (`{required: ["shouldBlock"], properties: {shouldBlock: +{type: "boolean"}}}`) as its only declared tool, `tool_choice: "required"` — directly against +`POST /v1/chat/completions` on this deployment (model `qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`). +The very first response: + +```json +{ + "message": { + "content": "{\"shouldBlock\": true}", + "tool_calls": [{ + "type": "function", + "function": { + "name": "memory_save", + "arguments": "{\"key\": \"tool_call_review\", \"content\": \"web_fetch call to https://www.swfte.com/ai/leaderboard was blocked due to potential data exfiltration or unauthorized external access.\", \"type\": \"factual\"}" + } + }] + }, + "tool_results": [{ "output": "{\"success\":true,\"memory\":{\"id\":\"84389721-...\", ...}, \"message\":\"Memory saved successfully\"}" }] +} +``` + +`memory_save` was never declared in the request's `tools` array — only `respond_in_schema` was. OmniRoute added it +server-side, the model called it instead of (or alongside) the intended tool, and OmniRoute **auto-executed it** +(`tool_results` shows a real memory record was created, with a UUID and timestamp — this isn't inert). The correct +`{"shouldBlock": true}` answer is sitting right there in `content`, but qwen-code's `generateJson()` only ever +looks for it inside a `respond_in_schema` function call (`functionCalls.find(call => call.name === +"respond_in_schema")`); when that's absent it falls through to a raw-text JSON parse of `getResponseText(result)` +— and OpenAI-client conventions generally don't populate a response's plain-text field when `finish_reason` is +`tool_calls` rather than `stop`, so that fallback likely returns nothing, yielding `{}` against a schema that +requires `shouldBlock`. This is a single confirmed repro, not a guaranteed-every-time reproduction — the injected +tool doesn't win every time (an immediate identical follow-up request returned an OmniRoute-cached copy of the same +response, not a fresh sample — see the cache caveat in +[`omniroute-direct-response-timeout-outage-2026-09-15.md`](./omniroute-direct-response-timeout-outage-2026-09-15.md)), +but it reproduces the *exact* failure shape from the live debug log on the first genuine attempt. + +## Root cause, confirmed in OmniRoute's own source + +[diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — +`open-sse/handlers/chatCore/memorySkillsInjection.ts`: + +```ts +if (memoryOwnerId && memorySettings?.enabled && body.stream !== true) { + // Server-side builtin memory tools (memory_save/update/search/delete) are + // executed by the gateway's tool-call interception, which runs only on the + // non-stream path. Stream clients (opencode etc.) execute tools client-side, + // so for them these tools would be announced but never executed; they should + // use the MCP memory tools (omniroute_memory_*) instead. + const existingTools = Array.isArray(body.tools) ? body.tools : []; + ... + const memoryTools = buildMemoryToolsForProvider(...).filter(tool => !existingToolNames.has(name)); + if (memoryTools.length > 0) { + body = { ...body, tools: [...existingTools, ...memoryTools] }; + } +} +``` + +This runs unconditionally for any non-streaming request from a key with memory enabled — there is no exemption +for a caller that already set `tool_choice` to force a *specific* tool. qwen-code's classifier is exactly this +case: a single-purpose, forced-`ANY`, non-streaming tool call, which is precisely the shape this injection logic +was not written to avoid interfering with. + +`src/lib/memory/settings.ts` confirms `enabled: false` is the *default* — memory is off by default in a fresh +OmniRoute install specifically because of injected-context cost, per its own comment: + +> "Off by default: enabling memory injects up to `maxTokens` (~2k) of retrieved context into every chat request, +> which is billed — a surprising cost for new installs... Opt in explicitly via Settings → Memory... Per-request +> opt-out is also available via the `x-omniroute-no-memory` header." + +This deployment has memory enabled (confirmed live by the reproduction above), which is presumably a deliberate +choice for other workflows (chat memory across sessions) — but it has an undocumented-to-this-repo side effect on +any caller using forced-tool-call classification. + +## What would fix this + +Two per-target exclusions were checked live against this deployment and confirmed **not to exist**: + +- **Per-API-key memory override**: `GET /api/keys` was fetched directly (the temporary key handed to this session + turned out to carry admin access, well beyond the plain `/v1` workload scope its name implied). Every key's full + field list was inspected — `noLog`, `scopes`, `allowedModels`, `rateLimits`, `disableNonPublicModels`, etc. — with + no memory-related field anywhere. +- **Per-model/connection override**: `GET /api/providers/` for the classifier's own connection + (`qwen3-4b`, id `b78ceb4c-52f8-47ae-b245-483baa6e3fc2`) was fetched directly. `providerSpecificData` (`prefix`, + `apiType`, `baseUrl`, `nodeName`, `timeoutMs`, `apiKeyHealth`) has no memory field either — consistent with the + source: `memoryOwnerId` is resolved purely from the *calling key* (`resolveMemoryOwnerId(apiKeyInfo)`), before + OmniRoute has even picked a provider, so it can't know or care that this particular request targets the + classifier model specifically. + +**The fix that was actually available and is now applied**: `x-omniroute-no-memory`, OmniRoute's own per-*request* +opt-out (not per-key or per-model), confirmed end-to-end and traced through both sides: + +- OmniRoute's handling, confirmed directly in `open-sse/handlers/chatCore.ts` and its own test suite + (`tests/unit/no-memory-header.test.ts`): `memoryOwnerId = isNoMemoryRequested(headers) ? null : resolveMemoryOwnerId(...)` + — a null owner id short-circuits *both* branches in `injectMemoryAndSkills` (context injection and tool + injection). The test suite gives the exact accepted values: `"true"`, `"1"`, `"yes"` (case-insensitive on both + the header name and value); `"false"`/`"0"`/`"no"`/empty do not trigger it. +- qwen-code's support for sending it, confirmed against the installed bundle, *not* just the docs: `modelProviders. + openai[].generationConfig.customHeaders` (documented at + [model-providers](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/model-providers/)) flows into + `DefaultOpenAICompatibleProvider.buildClient()` (`chunk-CXTPVBFA.js`), which passes it straight into the + underlying `OpenAI` SDK client as `defaultHeaders` — applied to every request made through that one model entry, + and *only* that entry (confirmed this is the generic OpenAI-compatible-chat client, the same one the classifier's + `apiType: "chat"` connection uses — not Anthropic- or Responses-API-specific plumbing). + +**Applied**, 2026-09-15: added to the `qwen3-4b-classifier` entry in the Windows-side `~/.qwen/settings.json` +(`C:\Users\aerli\.qwen\settings.json`), inside its `generationConfig`, alongside the existing `contextWindowSize` +and `extra_body`: + +```json +"customHeaders": { "x-omniroute-no-memory": "true" } +``` + +Scoped to this one model entry only — the main `qwen3.8-27b-local` connection's `generationConfig` is untouched, +so its own memory-context behavior (if any is relied on elsewhere) is unaffected. qwen-code's own +`[MODEL_PROVIDERS_HOT_RELOAD]` settings watcher (confirmed present in this session's debug log) should pick this +up on the already-running session without a restart. **Not yet verified live** — the next classifier failure (or a +deliberate repro, per the "Reproduced directly" section above) should confirm no `memory_save`-shaped tool call +appears in the response once this is in effect. + +- **Remaining fallback, if the header approach doesn't hold up**: disable memory globally for this deployment + (`PATCH /api/settings/memory`, `enabled: false`, or Settings → Memory in the dashboard) — blunt, but confirmed to + work by definition since `enabled: false` is every fresh install's default. +- **Also worth doing regardless**: file this upstream with OmniRoute. Their own code already special-cases one + caller type (streaming clients) right next to this injection logic; a similar exemption for a caller that already + set `tool_choice` to force one specific tool would be a clean fix on their end that doesn't depend on every + client remembering to send an opt-out header. +- **Not a fix, and not the problem**: nothing on the classifier-model or llama.cpp side. Qwen3-4B-Instruct-2507 + correctly produced the right answer (`{"shouldBlock": true}`) in the one reproduction captured here — the model + was never at fault. + +## Scope note + +This session's earlier hypothesis that the 27B model's 48-minute total outage +([`omniroute-direct-response-timeout-outage-2026-09-15.md`](./omniroute-direct-response-timeout-outage-2026-09-15.md)) +was caused by a ROCm/gfx1201 GPU hang is set aside here per explicit direction, not retracted — that was a +different incident (zero successes for 48 straight minutes, a shape this memory-injection bug doesn't produce) and +this finding doesn't bear on it either way. + +## Sources + +- Live debug log, `C:\Users\aerli\.qwen\debug\886d00eb-5b2b-4d84-b1ef-60909f75eec2.txt` (this session, 2026-09-15) +- Direct reproduction against this deployment's `POST /v1/chat/completions` (this session, 2026-09-15) +- Live `GET /api/keys`, `GET /api/providers`, `GET /api/providers/b78ceb4c-52f8-47ae-b245-483baa6e3fc2`, + `GET /api/settings/memory` against this deployment's OmniRoute instance (this session, 2026-09-15) — confirmed no + per-key or per-connection memory field exists in either schema +- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — + `open-sse/handlers/chatCore/memorySkillsInjection.ts`, `open-sse/handlers/chatCore.ts` (the + `isNoMemoryRequested`/`resolveMemoryOwnerId` branch), `src/lib/memory/settings.ts`, `src/lib/memory/injection.ts`, + `open-sse/mcp-server/tools/memoryTools.ts`, `tests/unit/no-memory-header.test.ts` (exact accepted header + name/value set) +- [Qwen Code docs — Model Providers](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/model-providers/) + (`customHeaders` field, documented under `generationConfig`) +- Installed qwen-code bundle — `chunk-N7VWZDWW.js`, `chunk-HBU7EKY4.js` (`classifyAction`, `runSideQuery`, + `resolveDefaultModel`, `generateJson`, `resolveFastModelSelector`, `getFastModel`) and `chunk-CXTPVBFA.js` + (`DefaultOpenAICompatibleProvider.buildHeaders()`/`buildClient()`, confirming `customHeaders` reaches the actual + OpenAI SDK client as `defaultHeaders` for the plain chat-completions path the classifier uses) — all read + directly from the bundled (unminified variable names) source, not inferred from docs alone +- Applied fix: `C:\Users\aerli\.qwen\settings.json`, `qwen3-4b-classifier` entry's `generationConfig.customHeaders` + (this session, 2026-09-15) + +## Confidence / uncertainty summary + +- **High confidence**: the fast model is genuinely used for classification (`stage=fast` in qwen-code's own debug + log, both on success and failure); the failure is a schema-validation error on an in-time response, not a + timeout (18.7s duration, explicit error text); OmniRoute's `memorySkillsInjection.ts` unconditionally injects + builtin memory tools into non-streaming completions for any memory-enabled key, with no exemption for + forced-single-tool callers (read directly from source); no per-key or per-model/connection memory override + exists in this OmniRoute version (confirmed by reading the complete live schema of both, not by absence of + documentation); `x-omniroute-no-memory: true` is a real, working per-request opt-out on OmniRoute's side (its + own test suite) and is reachable from qwen-code via `modelProviders.openai[].generationConfig.customHeaders`, + traced to the exact HTTP client the classifier's connection type uses (not inferred from docs alone — confirmed + against the bundled source's actual header-merging code). +- **Medium confidence**: that this exact tool-injection mechanism explains the *specific* production failures seen + earlier in this session's testing (the reproduction matches the failure shape and the source confirms the + mechanism exists and applies to this call pattern, but the live debug-log failure itself wasn't captured + mid-flight with response inspection — only its aftermath, the error message). +- **Low confidence / not verified**: the exact conditions under which the model picks the injected tool over the + intended one (one clean reproduction on the first attempt, not a characterized hit rate — the failure may not be + deterministic, so the `customHeaders` fix should still be watched rather than assumed to have fully resolved it + on the strength of this write-up alone); whether the applied `customHeaders` fix has been confirmed live yet + (not as of this writing — see "Applied" above).