docs(research): OmniRoute memory tools hijack classifier tool-call, x-omniroute-no-memory fix

STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).

Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
  as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
  margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
  so every request reprefilled its full prompt from scratch even when most
  of a conversation's prefix was unchanged. This is the actual fix for why
  compact-prompt prefill was slow enough to hit the timeout in the first
  place.

Documents the distinction and root cause in docs/research/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc

.env.example

@@ -1,35 +1,132 @@
-# Copy to .env and adjust. All values below are defaults baked into
-# docker-compose.yml — only uncomment/change what you actually want to override.
+# Copy to .env and adjust — or just run ./scripts/update.sh, which creates
+# .env from this file and fills in every secret/key below it can generate
+# itself (see each var's comment). All values below are defaults baked into
+# docker-compose.yml — only uncomment/change what you actually want to
+# override.
 
 # --- llama.cpp / model ---
 LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
+# 999 = every layer on GPU (this model is dense, not MoE, and already fits
+# fully in 32GB VRAM — see docs/research/qwen3.8-27b-quant.md). Lower this
+# to leave that many fewer layers on GPU and push the rest to CPU/system RAM
+# if something else is contending for VRAM — llama.cpp has no separate
+# "RAM offload" flag, --n-gpu-layers *is* the RAM-offload knob for a dense
+# model. Don't reach for --n-cpu-moe/--cpu-moe/--override-tensor "exps" —
+# those target Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B), not this
+# one, and are no-ops here.
+# There's no separate "then SSD" tier to enable either: llama.cpp mmaps the
+# model file by default (no --no-mmap here), so if GPU+RAM ever can't hold
+# the working set, the OS pages the rest in from disk automatically — an
+# implicit, slow last resort, not a config knob. An explicit tiered SSD
+# offload has been an open llama.cpp feature request since 2025 (still
+# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
 LLAMA_GPU_LAYERS=999
-# 65536 (64K) fits comfortably in 32GB VRAM alongside the model weights.
-# Raise toward 131072 if you need more context; see docs/research/qwen3.8-27b-quant.md
-# for the VRAM math at larger context sizes.
-LLAMA_CTX_SIZE=65536
+# 262144 = this model's true max (max_position_embeddings in Qwen/Qwen3.8-27B's
+# config.json) — the largest --ctx-size llama.cpp will even accept for it.
+# fp16 KV cache at full context would be ~16GB, on top of 17.6GB weights =
+# ~33.6GB, which does NOT fit the 32GB R9700 on its own. docker-compose.yml
+# now runs --cache-type-k/v q8_0, which roughly halves KV memory (~8GB at
+# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
+# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
+LLAMA_CTX_SIZE=262144
+# Concurrent request slots — the real hardware ceiling for this GPU, not a
+# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
+# default; dropped to 2 because more contended prefill was blowing requests
+# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
+# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
+# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
+# the timeout that matters moved to OmniRoute's per-connection
+# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
+# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
+# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
+# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
+# comfortably above observed usage.
+LLAMA_PARALLEL=2
 
-# --- Open WebUI ---
-WEBUI_PORT=8008
-# Required — create an "openwebui" virtual key in LiteLLM's Admin UI first
-# (see docs/proxy-key-onboarding.md), then paste it here.
-OPENWEBUI_LITELLM_KEY=
+# Dedicated GPU-resident backend for qwen-code's tool-call harmfulness
+# classifier (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
+# qwen-classifier service comment for the why and the VRAM/context math.
+LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
 
 # --- Lazytainer ---
 # Seconds of inactivity before llama-server is stopped. 900 = 15 min.
 LAZYTAINER_INACTIVE_TIMEOUT=900
 
-# --- LiteLLM proxy (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
-LITELLM_PORT=4000
-# Required — generate real random values before first run, e.g. `openssl rand -hex 32`.
-# LITELLM_SALT_KEY encrypts stored data; do not change it after the first run
-# (existing encrypted data becomes unreadable if you do).
-# Required — generate a real random value before first run, e.g. `openssl rand -hex 32`.
-LITELLM_MASTER_KEY=
-LITELLM_SALT_KEY=
-LITELLM_DB_PASSWORD=
-# Admin UI login (https://<proxy>/ui). Without these, LiteLLM falls back to
-# username "admin" / password = LITELLM_MASTER_KEY — set these instead so the
-# master key never has to be typed into the browser.
-UI_USERNAME=admin
-UI_PASSWORD=
+# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
+# Resolved automatically by ./scripts/update.sh from search.home on this
+# host — leave blank. Only set by hand if that resolution fails (e.g.
+# search.home isn't a static DHCP reservation and its IP drifted).
+SEARXNG_LAN_IP=
+
+# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
+# OMNIROUTE_PORT is the host-published port (reverse-proxied by NPM) — kept
+# at 4000, same as the old LiteLLM setup, so existing NPM/firewall config
+# doesn't need to change. It's mapped via plain Docker port publishing onto
+# API_PORT, omniroute's own container-internal port (left at its default,
+# not reconfigured to match). The dashboard (DASHBOARD_PORT) is never
+# published at all — see docker-compose.yml's omniroute service comment.
+OMNIROUTE_API_PORT=20129
+OMNIROUTE_DASHBOARD_PORT=20128
+# SSE inactivity timeout before OmniRoute gives up on a streaming request and
+# cancels it (which cancels the matching llama-server task too). 180s gives
+# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
+OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
+# Wait budget for the *first* SSE token specifically (distinct from the
+# inter-chunk timeout above) — see
+# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a
+# contended, large-context prefill even after retries eat into the budget.
+OMNIROUTE_REQUEST_TIMEOUT_MS=1800000
+# Random values, filled in automatically by ./scripts/update.sh — leave
+# blank. Bootstrap dashboard admin password (log in at the dashboard port,
+# change it there afterwards — this is only the first-boot value):
+OMNIROUTE_INITIAL_PASSWORD=
+# Signs dashboard session cookies:
+OMNIROUTE_JWT_SECRET=
+# Encrypts API key values at rest in omniroute's SQLite DB:
+OMNIROUTE_API_KEY_SECRET=
+# Encrypts the whole SQLite DB at rest. Do not change after first run —
+# existing encrypted data becomes unreadable if you do (same caveat as
+# LiteLLM's old LITELLM_SALT_KEY):
+OMNIROUTE_STORAGE_ENCRYPTION_KEY=
+# Per-deployment salts — random is fine, just needs to be stable:
+OMNIROUTE_MACHINE_ID_SALT=
+OMNIROUTE_CLI_SALT=
+# Required (production) — shared secret for the internal Codex Responses
+# WebSocket bridge. Random value, filled in automatically:
+OMNIROUTE_WS_BRIDGE_SECRET=
+# Per-workload virtual keys (one per client that calls the gateway) have no
+# scripted /key/generate equivalent yet — omniroute's key-creation endpoint
+# needs a dashboard login session, not a static bearer key (see issue #37).
+# Mint them by hand in the dashboard, add a KEY=value line here per workload
+# as you onboard one. See docs/proxy-key-onboarding.md.
+
+# --- ComfyUI (local image generation, see issue #38 wayfinder map) ---
+# yurisasc/comfyui-rocm7.1 manages GPU-group access via these GID/UID env
+# vars rather than relying solely on docker-compose.yml's group_add.
+# Resolved automatically from the host by ./scripts/update.sh — leave blank.
+COMFYUI_PUID=
+COMFYUI_PGID=
+
+# Shared by every GPU-touching service (llama-server, llama-server-fast,
+# comfyui) for group_add: — resolved to real host GIDs by ./scripts/update.sh
+# rather than left as plain group names in docker-compose.yml, because Docker
+# resolves a *named* group_add entry against the container's own /etc/group,
+# not the host's, and fails unpredictably when the image doesn't define one
+# (worse with multiple GPU services racing on the same lookup at once — see
+# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
+HOST_VIDEO_GID=
+HOST_RENDER_GID=
+
+# --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
+# image-generation-model-choice.md and issue #42) ---
+# Three files: diffusion weights, text encoder, VAE — all from the official
+# Comfy-Org FP8 split, chosen specifically because it's the only candidate
+# with a ComfyUI workflow pre-validated on this exact GPU (gfx1201/R9700).
+COMFYUI_DIFFUSION_MODEL_FILE=qwen_image_fp8_e4m3fn.safetensors
+COMFYUI_TEXT_ENCODER_FILE=qwen_2.5_vl_7b_fp8_scaled.safetensors
+COMFYUI_VAE_FILE=qwen_image_vae.safetensors
+
+# --- RAG databases (qdrant + neo4j, see wayfinder notes) ---
+# No auth on qdrant (its default) — same trust boundary as llama-server:
+# ai-stack is not exposed off-box. Random, filled in automatically:
+NEO4J_PASSWORD=

.gitignore

@@ -1,2 +1,8 @@
 .claude/
-.env
+.env
+# Personal memory content ingested by scripts/ingest-memory.sh — not meant
+# to be committed to this repo.
+data/
+.leankg/
+.cache/
+.qwen/tmp

AGENTS.md

@@ -0,0 +1 @@
+@CLAUDE.md

CLAUDE.md

@@ -7,3 +7,7 @@ Issues live as Gitea issues on `git.arthurerlich.de` (repo `haylan/LLM-Server`);
 ### Domain docs
 
 Single-context: `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
+
+### Deploying changes
+
+The running stack lives on a separate box (the R9700 server), not wherever this repo is being edited. After **any** change to `docker-compose.yml`, `.env.example`, or a script under `scripts/`, commit/push it, then run `./scripts/update.sh` on the server to apply it — don't just describe the change as done. If this session doesn't have shell access to the server, say so explicitly and tell the user to run it themselves rather than leaving it unsaid.

QWEN.md

@@ -0,0 +1 @@
+@CLAUDE.md

README.md

@@ -1,38 +1,35 @@
 # LLM-Server
 
-Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
+Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle.
 
 See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
 
 ## Quickstart
 
 ```bash
-cp .env.example .env
-# set LITELLM_MASTER_KEY / LITELLM_SALT_KEY (openssl rand -hex 32), see .env.example
-./scripts/download-model.sh
-docker compose up -d litellm litellm-db llama-server qdrant   # bring the proxy up first
+./scripts/update.sh
 ```
 
-Log into LiteLLM's Admin UI (`http://<this-machine>:4000/ui`), create an `openwebui` virtual key (see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md)), set `OPENWEBUI_LITELLM_KEY` in `.env` to it, then:
+`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
 
-```bash
-docker compose up -d
-```
+llama.cpp's own API is internal-only — everything routes through the AI gateway below.
 
-- Open WebUI: `http://<this-machine>:3000` locally, or `ai.home` / `ai.haylan.ch` once routed through Nginx Proxy Manager — see [`docs/network-access.md`](docs/network-access.md). First signup becomes the admin account (`WEBUI_AUTH` is on).
-- llama.cpp's own API is internal-only now — everything routes through the AI proxy below.
-
-Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
+Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).
 
 **Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
 
-## AI proxy (LiteLLM)
+## AI gateway (OmniRoute)
 
-An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload virtual keys, usage tracking, and a shadow cost estimate ("what this would have cost on Claude Sonnet 5"). Before `docker compose up -d`, set `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` in `.env` (see `.env.example`).
+An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload API keys and usage tracking. As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) this is [OmniRoute](https://github.com/diegosouzapw/OmniRoute), replacing the original LiteLLM setup. `./scripts/update.sh` handles most of OmniRoute's secrets (see `.env.example`); per-workload API keys still need minting by hand in the dashboard — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
 
-- Proxy API: `http://<this-machine>:4000/v1` locally, or `proxy.ai.home` / `proxy.ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
-- Admin UI (`/ui`, key/budget management): LAN-only — see `docs/network-access.md`.
+- Gateway API: `http://<this-machine>:${OMNIROUTE_PORT:-4000}/v1` locally, or `proxy-ai.home` / `proxy-ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
+- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
 - Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
-- Request priority across workloads: [`docs/proxy-request-priority.md`](docs/proxy-request-priority.md).
 
-Open WebUI and the coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the proxy now — llama-server has no published host port anymore. **Not yet verified**: none of this has been smoke-tested on real hardware (LiteLLM's priority scheduler in particular is beta — see `docs/proxy-request-priority.md`) — see [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17).
+Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
+
+**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
+
+### Web search
+
+The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).

docker-compose.yml

@@ -5,12 +5,19 @@ services:
     devices:
       - /dev/kfd
       - /dev/dri
+    # Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
+    # .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
     group_add:
-      - video
-      - render
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
     security_opt:
       - seccomp=unconfined
     ipc: host
+    # Caps this process's HIP hardware-queue allocation — works around
+    # ROCm/ROCm#5706 (GPU pinned at 100%/boost-clock whenever two
+    # concurrent HIP contexts touch this card). See the research doc above.
+    environment:
+      - GPU_MAX_HW_QUEUES=1
     volumes:
       - models:/models
     command: >
@@ -18,10 +25,23 @@ services:
       --host 0.0.0.0
       --port 8080
       --n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
-      --ctx-size ${LLAMA_CTX_SIZE:-65536}
+      --ctx-size ${LLAMA_CTX_SIZE:-262144}
+      --parallel ${LLAMA_PARALLEL:-2}
+      --flash-attn on
+      --cache-type-k q8_0
+      --cache-type-v q8_0
+      --cache-reuse 256
       --jinja
-    # No published host port: llama-server is reached only via the litellm
-    # proxy on the ai-stack docker network now — see issue #15. Its
+    # --cache-reuse 256: reuse cached KV for any matching prompt chunk of at
+    # least 256 tokens (KV-shift, no reprocessing) instead of reprefilling
+    # from scratch every request. Directly targets the actual root cause
+    # behind the OmniRoute non-ping SSE timeout, not just the symptom — see
+    # docs/research/omniroute-non-ping-sse-stream-timeout.md. Pairs with
+    # OmniRoute's promptCacheAffinityEnabled (dashboard default), which keeps
+    # a conversation's requests pinned to the same slot so there's a matching
+    # prefix to reuse.
+    # No published host port: llama-server is reached only via the omniroute
+    # gateway on the ai-stack docker network now — see issue #15. Its
     # unauthenticated API no longer needs to be LAN-reachable directly.
     expose:
       - "8080"
@@ -35,9 +55,87 @@ services:
       - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
       - "lazytainer.group.llamaserver.minPacketThreshold=2"
 
+  # Dedicated backend for qwen-code's tool-call harmfulness classifier
+  # (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
+  # 27B connection — every classification call then queued behind whatever
+  # heavy generation was already running on that model's 2 GPU slots (issue
+  # tracker: OmniRoute semaphore/pr-agent investigation).
+  #
+  # Tried CPU-only first (own process avoids the GPU queue entirely) — too
+  # slow in practice: real classification calls blew past OmniRoute's 60s
+  # timeout and retry-looped (504→499→504...). Moved to GPU instead.
+  #
+  # Context sizing: qwen-code's classifier transcript is hard-capped in its
+  # own source (MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000
+  # per message, packages/core/src/permissions/classifier-transcript.ts) —
+  # worst case is ~40-50K tokens, nowhere near the 131072 originally set in
+  # settings.json (that number was copied from the main model's entry, not
+  # a real qwen-code requirement). 65536 ctx gives ~1.5x margin over that
+  # worst case. Qwen3-4B-Instruct-2507 is still the model choice — smallest
+  # Qwen3 with long native context (262144) without RoPE-scaling, in case
+  # that margin ever needs to grow.
+  #
+  # VRAM: weights+KV math (~4.9GiB) predicted comfortable headroom in the
+  # ~6.1GiB free on the R9700, but measured live it actually used ~5.85GiB —
+  # left only ~700MB free, too tight. Dropping --batch-size/--ubatch-size
+  # barely moved it (~768MB free) — wrong lever. Actual cause: llama-server
+  # runs with --flash-attn on but this service was missing it — without
+  # flash attention the unfused attention compute buffer at 65536 ctx is
+  # much larger (roughly O(n^2) intermediate buffers vs flash-attn's fused,
+  # near-linear workspace), dwarfing the naive weights+KV estimate. Added
+  # --flash-attn on to match llama-server; re-verify with rocm-smi after
+  # deploy before trusting any of these numbers again. GPU_MAX_HW_QUEUES=1
+  # carried over from llama-server's comment above — same ROCm/ROCm#5706
+  # clock-pinning bug applies now that two HIP contexts (this +
+  # llama-server) share the card.
+  #
+  # --reasoning off is a no-cost safety net, not a confirmed-needed fix:
+  # ggml-org/llama.cpp#20809 (closed) documents some server builds
+  # misdetecting Qwen3-Instruct-2507 models as thinking models, routing
+  # tool-call output into reasoning_content instead of tool_calls — exactly
+  # the failure mode that ruled out the 27B model for this role in the
+  # first place. Whether the current image build still has it was never
+  # independently confirmed (see docs/research/fast-model-choice.md §4/§6).
+  qwen-classifier:
+    image: ghcr.io/ggml-org/llama.cpp:server-rocm
+    container_name: qwen-classifier
+    devices:
+      - /dev/kfd
+      - /dev/dri
+    group_add:
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
+    security_opt:
+      - seccomp=unconfined
+    ipc: host
+    environment:
+      - GPU_MAX_HW_QUEUES=1
+    volumes:
+      - models:/models
+    command: >
+      -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+      --host 0.0.0.0
+      --port 8080
+      --n-gpu-layers 28
+      --ctx-size 65536
+      --parallel 1
+      --batch-size 512
+      --ubatch-size 128
+      --flash-attn on
+      --reasoning off
+      --cache-type-k q4_0
+      --cache-type-v q4_0
+      --jinja
+    expose:
+      - "8080"
+    restart: unless-stopped
+    networks: [ai-stack]
+
   # ponytail: one-off downloader, not a standing service — run via
-  # `docker compose --profile tools run --rm downloader` (see scripts/download-model.sh).
-  # Keeps the model file inside the named `models` volume instead of a host bind-mount.
+  # `docker compose --profile tools run --rm downloader`. Folded into
+  # scripts/update.sh, which runs this every time; the `test -f` guard is
+  # what makes that safe to re-run without re-downloading. Keeps the model
+  # files inside the named `models` volume instead of a host bind-mount.
   downloader:
     image: curlimages/curl:latest
     profiles: ["tools"]
@@ -49,113 +147,244 @@ services:
     entrypoint: ["sh", "-c"]
     command:
       - >
+        test -f /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} &&
+        echo "already downloaded, skipping" ||
         curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
         https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
 
-  qdrant:
-    image: qdrant/qdrant:latest
-    container_name: qdrant
+  # Same pattern as downloader above, separate service so this one small
+  # file doesn't get re-checked/re-pulled by the big model's job.
+  downloader-classifier:
+    image: curlimages/curl:latest
+    profiles: ["tools"]
+    user: root
     volumes:
-      - qdrant-data:/qdrant/storage
-    restart: unless-stopped
-    networks: [ai-stack]
-    healthcheck:
-      test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/localhost/6333'"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
+      - models:/models
+    entrypoint: ["sh", "-c"]
+    command:
+      - >
+        test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
+        echo "already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
 
-  open-webui:
-    image: ghcr.io/open-webui/open-webui:main
-    container_name: open-webui
-    depends_on:
-      qdrant:
-        condition: service_healthy
-      litellm:
-        condition: service_healthy
+  # Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
+  # text encoder, VAE) — same test -f guard pattern as downloader above.
+  # See docs/research/image-generation-model-choice.md and issue #42.
+  #
+  # ponytail: target paths assume ComfyUI's standard models/ layout under
+  # BASE_STORAGE_PATH (/storage) — same "not independently confirmed against
+  # the image's Dockerfile" caveat already flagged on the comfyui service
+  # below. If ComfyUI doesn't pick these up, check its actual models root
+  # first.
+  downloader-comfyui:
+    image: curlimages/curl:latest
+    profiles: ["tools"]
+    user: root
     volumes:
-      - openwebui-data:/app/backend/data
-    env_file: .env
+      - comfyui-data:/storage
+    entrypoint: ["sh", "-c"]
+    command:
+      - >
+        mkdir -p /storage/models/diffusion_models /storage/models/text_encoders /storage/models/vae &&
+        (test -f /storage/models/diffusion_models/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen_image_fp8_e4m3fn.safetensors} &&
+        echo "diffusion model already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /storage/models/diffusion_models/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen_image_fp8_e4m3fn.safetensors}
+        https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen_image_fp8_e4m3fn.safetensors}) &&
+        (test -f /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen_2.5_vl_7b_fp8_scaled.safetensors} &&
+        echo "text encoder already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen_2.5_vl_7b_fp8_scaled.safetensors}
+        https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen_2.5_vl_7b_fp8_scaled.safetensors}) &&
+        (test -f /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_vae.safetensors} &&
+        echo "vae already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_vae.safetensors}
+        https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/${COMFYUI_VAE_FILE:-qwen_image_vae.safetensors})
+
+  # Local image generation — see issue #38 (wayfinder map). yurisasc's image
+  # is gfx1201-tuned specifically (R9700's arch), unlike the official/AMD
+  # ComfyUI image which doesn't pin RDNA4 support — see
+  # docs/research/image-generation-options.md.
+  comfyui:
+    image: yurisasc/comfyui-rocm7.1:latest
+    container_name: comfyui
+    devices:
+      - /dev/kfd
+      - /dev/dri
+    # Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
+    # .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
+    group_add:
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
+    security_opt:
+      - seccomp=unconfined
+    ipc: host
     environment:
-      - WEBUI_AUTH=True
-      # Routed through the litellm proxy, not llama-server directly — see issue #15.
-      # OPENAI_API_KEY must be a virtual key created for Open WebUI per
-      # docs/proxy-key-onboarding.md (name it "openwebui"), set as
-      # OPENWEBUI_LITELLM_KEY in .env.
-      - OPENAI_API_BASE_URL=http://litellm:4000/v1
-      - OPENAI_API_KEY=${OPENWEBUI_LITELLM_KEY}
-      - VECTOR_DB=qdrant
-      - QDRANT_URI=http://qdrant:6333
+      - HSA_OVERRIDE_GFX_VERSION=12.0.1
+      - PYTORCH_ROCM_ARCH=gfx1201
+      # This image also wants GID env vars directly (its own README asks
+      # for both these and group_add above) — same HOST_VIDEO_GID/
+      # HOST_RENDER_GID resolved by scripts/update.sh, shared with
+      # llama-server now instead of comfyui-only vars.
+      - PUID=${COMFYUI_PUID}
+      - PGID=${COMFYUI_PGID}
+      - VIDEO_GID=${HOST_VIDEO_GID}
+      - RENDER_GID=${HOST_RENDER_GID}
+      - BASE_STORAGE_PATH=/storage
+    volumes:
+      - comfyui-data:/storage
+    # ponytail: exact internal storage path taken from the image's own
+    # BASE_STORAGE_PATH env var, not independently confirmed against its
+    # Dockerfile — if models/workflows don't persist across a recreate,
+    # check this against the image's actual entrypoint first.
+    #
+    # Published host port (unlike llama-server's ai-stack-only pattern):
+    # ComfyUI's own UI is meant to be reachable directly too, for a planned
+    # external nginx reverse-proxy route to comfy.home — not just through
+    # OmniRoute. Still also reachable at http://comfyui:8188 internally on
+    # ai-stack, which is the URL to register as OmniRoute's comfyui
+    # provider (dashboard or POST /api/providers, per docs/proxy-key-onboarding.md
+    # — same undocumented-in-repo manual flow already used for llama-server).
     ports:
-      - "${WEBUI_PORT:-8008}:8080"
+      - "8138:8188"
     restart: unless-stopped
     networks: [ai-stack]
 
-  litellm:
-    image: ghcr.io/berriai/litellm:main-stable
-    container_name: litellm
+  # Replaces litellm — see issue #31 (wayfinder map) for the full migration
+  # rationale/findings. No static config.yaml equivalent: provider routing
+  # (llama-server, searxng-search) is registered once through the dashboard
+  # or POST /api/providers after first boot, not checked into this repo —
+  # see docs/proxy-key-onboarding.md.
+  omniroute:
+    image: diegosouzapw/omniroute:latest
+    container_name: omniroute
     depends_on:
-      litellm-db:
-        condition: service_healthy
       llama-server:
         condition: service_started
     volumes:
-      - ./litellm-config.yaml:/app/config.yaml:ro
-    # LITELLM_MASTER_KEY / LITELLM_SALT_KEY come straight from .env via env_file
-    # (names match what litellm reads). LITELLM_SALT_KEY must not change after
-    # first run — see .env.example.
+      - omniroute-data:/app/data
     env_file: .env
     environment:
-      - DATABASE_URL=postgresql://litellm:${LITELLM_DB_PASSWORD}@litellm-db:5432/litellm
-    command: ["--config", "/app/config.yaml", "--port", "4000"]
+      # Split-port mode: dashboard and API are fully separate ports (unlike
+      # LiteLLM's single :4000 for both /v1 and /ui) — both published
+      # directly below, unlike the old :4000-only host mapping.
+      - API_HOST=0.0.0.0
+      - API_PORT=${OMNIROUTE_API_PORT:-20129}
+      - DASHBOARD_PORT=${OMNIROUTE_DASHBOARD_PORT:-20128}
+      # Required to register llama-server/searxng-search as providers —
+      # their base URLs are LAN/container-internal addresses, blocked by
+      # default (SSRF guard against public-provider spoofing).
+      - OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true
+      - OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true
+      # Required (production) per docs/reference/ENVIRONMENT.md — shared
+      # secret for the internal Codex Responses WebSocket bridge. Missed on
+      # first pass; docker-compose config validated fine without it, but
+      # the docs are explicit this one's required, not optional.
+      - OMNIROUTE_WS_BRIDGE_SECRET=${OMNIROUTE_WS_BRIDGE_SECRET}
+      # Default heap (1024MB) is dashboard-only sized per OmniRoute's own
+      # Docker guide — every client here is a coding CLI, which needs the
+      # larger figure the guide recommends. Paired with mem_limit below.
+      - OMNIROUTE_MEMORY_MB=8192
+      # Default 300000 (5 min) per OmniRoute's own docs, but this deployment
+      # had it dialed down elsewhere (dashboard) to ~95s — too tight for a
+      # contended local llama-server: large-context prefill under multiple
+      # concurrent slots can outrun that before the first SSE token arrives,
+      # so OmniRoute cancels a request that was actually still working (see
+      # LLAMA_PARALLEL above for the other half of this fix). Raised here so
+      # it's tracked in git instead of a dashboard-only setting.
+      - STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
+      # Different timer than STREAM_IDLE_TIMEOUT_MS above — that one only
+      # bounds gaps *between* SSE chunks once streaming has started.
+      # REQUEST_TIMEOUT_MS bounds the wait for the *first* non-ping SSE
+      # event, and it's what was still firing ("Stream produced no non-ping
+      # SSE event within 95000ms") the morning after the timeout above was
+      # raised — see docs/research/omniroute-non-ping-sse-stream-timeout.md.
+      # Default 600000 (10 min) per OmniRoute's own docs, but the effective
+      # deadline is remaining budget after retries/cooldowns eat into it, not
+      # a flat timer, so raised well past the default for headroom.
+      - REQUEST_TIMEOUT_MS=${OMNIROUTE_REQUEST_TIMEOUT_MS:-1800000}
+    # Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
+    # network can't resolve search.home on its own.
+    extra_hosts:
+      - "search.home:${SEARXNG_LAN_IP}"
     ports:
-      # published for LAN access (proxy.ai.home) and, via NPM, proxy.ai.haylan.ch —
-      # NPM must deny the /ui path on the external host. See docs/network-access.md.
-      - "${LITELLM_PORT:-4000}:4000"
+      - "${OMNIROUTE_API_PORT:-20129}:${OMNIROUTE_API_PORT:-20129}"
+      - "${OMNIROUTE_DASHBOARD_PORT:-20128}:${OMNIROUTE_DASHBOARD_PORT:-20128}"
+    # 10+ GiB ceiling per OmniRoute's Docker guide, matching
+    # OMNIROUTE_MEMORY_MB=8192 above.
+    mem_limit: 10g
+    # SQLite WAL needs time to checkpoint back into the main DB file on
+    # shutdown — the Docker guide's --stop-timeout 40 equivalent.
+    stop_grace_period: 40s
     restart: unless-stopped
     networks: [ai-stack]
+    # ponytail: TCP-connect check, not an HTTP /healthz GET — the image has
+    # no python3/curl/wget (confirmed live, `which` found only node), and
+    # OmniRoute's own Docker guide already treats a bare TCP probe on this
+    # port as an acceptable liveness check, not just the HTTP one. Simpler
+    # and avoids depending on /healthz's exact path/response shape.
     healthcheck:
       test:
         - CMD-SHELL
-        - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"
+        - node -e "require('net').connect(${OMNIROUTE_API_PORT:-20129},'localhost').on('connect',function(){this.end();process.exit(0)}).on('error',()=>process.exit(1))"
       interval: 30s
       timeout: 10s
       retries: 3
       start_period: 40s
 
-  litellm-db:
-    image: postgres:16-alpine
-    container_name: litellm-db
-    env_file: .env
-    environment:
-      - POSTGRES_USER=litellm
-      - POSTGRES_PASSWORD=${LITELLM_DB_PASSWORD}
-      - POSTGRES_DB=litellm
-    volumes:
-      - litellm-db-data:/var/lib/postgresql/data
-    restart: unless-stopped
-    networks: [ai-stack]
-    healthcheck:
-      test: ["CMD-SHELL", "pg_isready -d litellm -U litellm"]
-      interval: 5s
-      timeout: 5s
-      retries: 10
-
   lazytainer:
     image: ghcr.io/vmorganp/lazytainer:master
     container_name: lazytainer
-    network_mode: host
+    # NOT network_mode: host — lazytainer identifies its own container by
+    # matching os.Hostname() against the Docker container-ID list
+    # (vmorganp/Lazytainer, configureFromLabels()); under host networking the
+    # container inherits the host's hostname instead of its own ID, so that
+    # match always fails and it panics with "Could not determine container ID
+    # of lazytainer" on every start. Host networking also can't see traffic
+    # to llama-server:8080 anyway — that port only exists on the ai-stack
+    # bridge network (no host port published, see issue #15 above). Joining
+    # ai-stack instead fixes both: hostname becomes the real container ID,
+    # and it's on the same network as the traffic it's watching.
+    networks: [ai-stack]
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
     restart: unless-stopped
     depends_on:
       - llama-server
 
+  # RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
+  # directly like comfyui above, not gatewayed through omniroute (it isn't an
+  # LLM provider).
+  qdrant:
+    image: qdrant/qdrant:latest
+    container_name: qdrant
+    volumes:
+      - qdrant-data:/qdrant/storage
+    ports:
+      - "6333:6333"
+    restart: unless-stopped
+    networks: [ai-stack]
+
+  # RAG graph store, native vector index too (can absorb qdrant's job later
+  # if the two-DB split proves unnecessary — see wayfinder notes).
+  neo4j:
+    image: neo4j:5-community
+    container_name: neo4j
+    environment:
+      - NEO4J_AUTH=neo4j/${NEO4J_PASSWORD:?run scripts/update.sh first to resolve this}
+    volumes:
+      - neo4j-data:/data
+    ports:
+      - "7474:7474" # browser UI
+      - "7687:7687" # bolt
+    restart: unless-stopped
+    networks: [ai-stack]
+
 networks:
   ai-stack:
 
 volumes:
   models:
+  omniroute-data:
+  comfyui-data:
   qdrant-data:
-  openwebui-data:
-  litellm-db-data:
+  neo4j-data:

docs/agents/issue-tracker.md

@@ -9,7 +9,7 @@ Use the **`tea` CLI** (already installed and authenticated as `haylan` via `tea
 - **Create an issue**: `tea issues create --title "..." --description "..." --labels "..."`
 - **Read an issue**: `tea issues <index> --comments`
 - **List issues**: `tea issues list --state open --labels "..."` (add `-f` to control which fields print)
-- **Comment on an issue**: `tea comments create <index> --description "..."` (check `tea comments -h` for exact flags)
+- **Comment on an issue**: `tea comment <index> -d "..."` (check `tea comments -h` for exact flags — `tea comments create` is invalid, `add`/`a` is the subcommand)
 - **Apply / remove labels**: `tea issues edit <index> --add-labels "..."` / `--remove-labels "..."`
 - **Close**: `tea issues close <index>`
 - **Labels**: `tea labels create --name "..." --color "#hex" --description "..."`; `tea labels list`
@@ -48,4 +48,4 @@ Used by `/wayfinder`. This Gitea instance (1.27.2) has **no native sub-issue/par
 - **Blocking**: native issue dependencies via the raw API calls above. A ticket is unblocked when every dependency (`GET .../dependencies`) is closed.
 - **Frontier query**: `tea issues list --state open --labels "wayfinder:<type1>,wayfinder:<type2>,..."` scoped to the map's children (cross-check against the map's task list), drop any with an open dependency or an assignee.
 - **Claim**: `tea issues edit <n> --add-assignees haylan` — the session's first write.
-- **Resolve**: `tea comments create <n> --description "<answer>"`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far, and check off its line in the map's task list.
+- **Resolve**: `tea comment <n> -d "<answer>"`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far, and check off its line in the map's task list. Map edits are full-body replaces (`tea issues edit` has no append) — concurrent resolutions racing on the same map issue can clobber each other's Decisions-so-far lines; re-fetch the map immediately before editing it, not from an earlier read.

docs/coding-cli-setup.md

@@ -1,91 +0,0 @@
-# Pointing a coding-agent CLI at this stack
-
-This stack routes through the [AI proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (LiteLLM) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The proxy exposes:
-
-- **OpenAI-compatible**: `http://<ai-box>:4000/v1` (or `${LITELLM_PORT}` if you changed it in `.env`)
-- **Anthropic Messages API** (LiteLLM's own unified `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:4000`
-
-Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the proxy as `qwen3.8-27b-local` — behind whichever wire format the client speaks.
-
-`<ai-box>` is this machine's LAN address, or `proxy.ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
-
-**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (LiteLLM's Admin UI, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
-
-> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
-
-## Claude Code CLI
-
-Claude Code speaks the **Anthropic Messages API** — point it at the proxy's unified endpoint, not llama.cpp directly:
-
-```bash
-export ANTHROPIC_BASE_URL=http://<ai-box>:4000
-export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
-claude
-```
-
-Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
-
-## Kimi CLI
-
-Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
-
-```toml
-[providers.openai]
-type = "openai"
-base_url = "http://<ai-box>:4000/v1"
-api_key = "<kimi-cli virtual key>"
-```
-
-If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
-
-## OpenCode CLI
-
-Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
-
-**Install**:
-```bash
-curl -fsSL https://opencode.ai/install | bash
-```
-
-**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
-
-```json
-{
-  "$schema": "https://opencode.ai/config.json",
-  "provider": {
-    "aiproxy": {
-      "npm": "@ai-sdk/openai-compatible",
-      "name": "AI proxy (local)",
-      "options": {
-        "baseURL": "http://<ai-box>:4000/v1",
-        "apiKey": "<opencode-cli virtual key>"
-      },
-      "models": {
-        "qwen3.8-27b-local": {
-          "name": "Qwen3.8-27B",
-          "limit": { "context": 65536, "output": 8192 }
-        }
-      }
-    }
-  }
-}
-```
-
-Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.
-
-Select the model with `aiproxy/qwen3.8-27b-local`.
-
-**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above):
-- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
-- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
-- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
-
-## Summary
-
-| CLI | Wire format | Endpoint | Config |
-|---|---|---|---|
-| Claude Code | Anthropic Messages | `http://<ai-box>:4000` | `ANTHROPIC_BASE_URL` env var |
-| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `config.toml` provider block |
-| OpenCode | OpenAI Chat Completions | `http://<ai-box>:4000/v1` | `opencode.json` provider block |
-
-Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.

docs/coding-cli-setup/claude-code.md

@@ -0,0 +1,13 @@
+# Claude Code CLI
+
+[← back to overview](index.md)
+
+Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
+
+```bash
+export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
+export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
+claude
+```
+
+Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.

docs/coding-cli-setup/index.md

@@ -0,0 +1,34 @@
+# Pointing a coding-agent CLI at this stack
+
+[← back to README](../../README.md)
+
+This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
+
+- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
+- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
+
+Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
+
+`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
+
+**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
+
+> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
+
+## Per-CLI setup
+
+- [Claude Code CLI](claude-code.md)
+- [Kimi CLI](kimi-cli.md)
+- [OpenCode CLI](opencode.md)
+- [Qwen Code CLI](qwen-code.md)
+
+## Summary
+
+| CLI | Wire format | Endpoint | Config |
+|---|---|---|---|
+| [Claude Code](claude-code.md) | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
+| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
+| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
+| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` |
+
+Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`, `docs/research/omniroute-account-semaphore-timeout.md` (a connection that can only handle a few concurrent requests — like `llama-server` or `qwen-classifier` — hits a hardcoded 30s reject once more requests queue up than its `maxConcurrent`, unless configured around it).

docs/coding-cli-setup/kimi-cli.md

@@ -0,0 +1,14 @@
+# Kimi CLI
+
+[← back to overview](index.md)
+
+Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
+
+```toml
+[providers.openai]
+type = "openai"
+base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
+api_key = "<kimi-cli virtual key>"
+```
+
+If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.

docs/coding-cli-setup/opencode.md

@@ -0,0 +1,45 @@
+# OpenCode CLI
+
+[← back to overview](index.md)
+
+Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
+
+**Install**:
+```bash
+curl -fsSL https://opencode.ai/install | bash
+```
+
+**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "aiproxy": {
+      "npm": "@ai-sdk/openai-compatible",
+      "name": "AI proxy (local)",
+      "options": {
+        "baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "apiKey": "<opencode-cli virtual key>"
+      },
+      "models": {
+        "qwen3.8-27b-local": {
+          "name": "Qwen3.8-27B",
+          "limit": { "context": 131072, "output": 8192 }
+        }
+      }
+    }
+  }
+}
+```
+
+Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server.
+
+Select the model with `aiproxy/qwen3.8-27b-local`.
+
+**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)):
+- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
+- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
+- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
+
+Further reading: `docs/research/opencode-cli-setup.md`.

docs/coding-cli-setup/qwen-code.md

@@ -0,0 +1,130 @@
+# Qwen Code CLI
+
+[← back to overview](index.md)
+
+Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier. Both are registered as separate providers in OmniRoute but reachable through the same gateway URL.
+
+## Why a second model exists
+
+Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's per-tool-call safety gate — it decides whether to auto-approve or block a shell command / tool call before it runs. It was originally aliased onto the main 27B model's own OmniRoute connection. That broke two ways in practice (see `docs/research/fast-model-choice.md` for the model research, and the issue-tracker history for the full incident):
+
+- **Queued behind heavy work.** Every classification call competed for the main model's 2 GPU slots with whatever real generation was already running, so a classifier check could sit blocked for minutes.
+- **CPU-only was tried first and was too slow.** Isolating the classifier onto its own CPU-only llama.cpp instance avoided the GPU queue entirely, but real classification calls (which can carry a non-trivial conversation transcript, not just the bare tool call) blew past OmniRoute's request timeout and retry-looped.
+
+The fix: a dedicated, GPU-resident `qwen-classifier` service (`docker-compose.yml`) running a small model (`Qwen3-4B-Instruct-2507`) on its own **partial** GPU offload — enough layers on the R9700 to be fast, sized to leave real VRAM headroom next to the 27B model rather than trusting a naive weights+KV estimate (see that service's comment block in `docker-compose.yml` for the actual measured numbers and the two wrong turns — batch-size tuning, then flash-attn — before partial offload turned out to be the real lever).
+
+## `~/.qwen/settings.json`
+
+```json
+{
+  "modelProviders": {
+    "openai": [
+      {
+        "id": "<main-model-provider-id-in-omniroute>",
+        "name": "qwen3.8-27b-local",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "generationConfig": { "contextWindowSize": 131072 }
+      },
+      {
+        "id": "<classifier-provider-id-in-omniroute>",
+        "name": "qwen3-4b-classifier",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "generationConfig": {
+          "contextWindowSize": 65536,
+          "extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
+        }
+      }
+    ]
+  },
+  "security": { "auth": { "selectedType": "openai" } },
+  "model": {
+    "name": "<main-model-provider-id-in-omniroute>",
+    "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
+  },
+  "fastModel": "<classifier-provider-id-in-omniroute>"
+}
+```
+
+- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=<qwen-code-cli virtual key>` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
+- **`contextWindowSize` for the main model is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it from `.env`: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**.
+- **The classifier's `contextWindowSize` (65536) is not per-slot math** — `qwen-classifier` runs `--parallel 1`, so its whole `--ctx-size` belongs to the one slot. 65536 isn't a guess either: qwen-code's own source hard-caps the classifier transcript (`MAX_TRANSCRIPT_MESSAGES=40`, `MAX_HISTORICAL_ACTION_CHARS=4000`/message in `packages/core/src/permissions/classifier-transcript.ts`) — worst case is ~40-50K tokens, so 65536 gives real margin without wasting VRAM the way the original 131072 (copied from the main model's entry, not an actual qwen-code requirement) would have.
+- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see `fast-model-choice.md` §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
+- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
+
+## Web search via OmniRoute
+
+Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
+
+```js
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+
+const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home";
+const API_KEY = process.env.OMNIROUTE_API_KEY;
+
+if (!API_KEY) {
+  console.error("OMNIROUTE_API_KEY is not set in the environment.");
+  process.exit(1);
+}
+
+const server = new McpServer({ name: "omniroute-search", version: "1.0.0" });
+
+server.registerTool(
+  "search",
+  {
+    description: "Web/news search via OmniRoute's /v1/search endpoint.",
+    inputSchema: { query: z.string().describe("Search query") },
+  },
+  async ({ query }) => {
+    const res = await fetch(`${BASE_URL}/v1/search`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` },
+      body: JSON.stringify({ query }),
+    });
+    const text = await res.text();
+    if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true };
+    return { content: [{ type: "text", text }] };
+  }
+);
+
+await server.connect(new StdioServerTransport());
+```
+
+Register it in `~/.qwen/settings.json`:
+
+```json
+{
+  "mcpServers": {
+    "omniroute-search": { "command": "node", "args": ["<path-to>/index.mjs"] }
+  },
+  "tools": { "webSearch": { "enabled": false } }
+}
+```
+
+It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.
+
+**Non-interactive mode (`qwen -p ...`) needs this tool explicitly allow-listed.** MCP tools require interactive confirmation by default; `--approval-mode auto` alone doesn't bypass that for a non-interactive run — pass `--allowed-tools mcp__omniroute-search__search` (or `-y` for full YOLO) alongside `-p`, or the search call never reaches the classifier at all and silently no-ops. Confirmed live: without the allow-list, only the tool calls the CLI's non-interactive gate lets through end up as classifier requests.
+
+## Auto Mode tuning
+
+Auto Mode's action classifier calls the fast model above. Even on the dedicated GPU-resident instance, give it real timeout headroom rather than trusting OmniRoute's default — and since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:
+
+```json
+{
+  "permissions": {
+    "autoMode": {
+      "classifier": { "timeouts": { "stage1Ms": 600000 } },
+      "hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] }
+    }
+  }
+}
+```
+
+`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each.
+
+Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see `docs/network-access.md` for reaching the dashboard API). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.
+
+Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.

docs/network-access.md

@@ -1,27 +1,24 @@
-# Network access: ai.home / ai.haylan.ch
+# Network access: proxy-ai.home / proxy-ai.haylan.ch
 
-Open WebUI is meant to be reachable as **`ai.home`** on the LAN and **`ai.haylan.ch`** from outside. This stack doesn't run its own reverse proxy — it publishes Open WebUI's port to the host (`${WEBUI_PORT:-3000}`, see `docker-compose.yml`) and relies on the **existing Nginx Proxy Manager (NPM)** instance already fronting other self-hosted services on this network.
-
-## What to set up in NPM
-
-Two Proxy Hosts, both pointing at this machine's LAN IP on port `${WEBUI_PORT:-3000}` (Open WebUI):
-
-- **`ai.home`** — internal only, no external DNS/TLS needed unless you want it.
-- **`ai.haylan.ch`** — external, reachable via the DMZ already forwarding it to NPM; let NPM issue/manage the TLS cert as it does for other services.
+This stack has no chat UI — every client is a coding CLI reaching the AI gateway (OmniRoute). It doesn't run its own reverse proxy — it publishes the gateway's API port to the host and relies on the **existing Nginx Proxy Manager (NPM)** instance already fronting other self-hosted services on this network.
 
 ## llama.cpp's raw API stays LAN-only — deliberately
 
-The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable via `ai.haylan.ch`. It has no authentication of its own (unlike Open WebUI, which has login enabled) — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) reach it directly over the LAN, using this machine's LAN IP or `ai.home` if your local DNS resolves that hostname straight to the box (bypassing NPM, which only fronts ports 80/443).
+The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else.
 
-If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture (the raw API would need its own auth in front of it).
+If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.
 
-## The AI proxy (LiteLLM) — `proxy.ai.home` / `proxy.ai.haylan.ch`
+## The AI gateway (OmniRoute) — `proxy-ai.home` / `proxy-ai.haylan.ch`
 
-Once the gateway from [issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) is deployed, it gets its own hostnames, same NPM pattern as Open WebUI above:
+As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute:
 
-- **`proxy.ai.home`** — internal only, fronts the full LiteLLM port (API + Admin UI).
-- **`proxy.ai.haylan.ch`** — external, via the DMZ/NPM. Fronts only the inference API paths.
+- **`proxy-ai.home`** and **`proxy-ai.haylan.ch`** both point only at `${OMNIROUTE_PORT:-4000}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `proxy-ai.home` internal-only, `proxy-ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual).
+- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
 
-**Every proxy call already requires a valid virtual key** (Bearer token, see `docs/proxy-key-onboarding.md`) — the same bar Open WebUI clears with its own login — so no extra NPM-level auth is needed for the external hostname.
+**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname.
 
-**LiteLLM's Admin UI (`/ui`) stays LAN-only**, same reasoning as llama.cpp's raw API: it manages every workload's keys and budgets, so it doesn't belong on the public internet. LiteLLM serves `/ui` on the same port as its API by default, so `proxy.ai.haylan.ch`'s NPM Proxy Host needs an explicit rule denying the `/ui` path (a "Deny" custom location, same UI as the "Advanced" tab used for other NPM hosts) — `proxy.ai.home` has no such restriction and reaches both the API and the Admin UI.
+## RAG knowledge graph (Neo4j) — `knowledge.proxy-ai.home`
+
+Set up as an NPM Proxy Host pointing at this machine's LAN IP on Neo4j's Browser port (`7474`, see `docker-compose.yml`'s `neo4j` service, [PR #50](https://git.arthurerlich.de/haylan/LLM-Server/pulls/50)). Internal-only, same as `proxy-ai.home` — no DMZ/external route, this is admin/dev tooling, not a client-facing endpoint. Bolt (`7687`, the actual query protocol) isn't proxied through NPM at all — clients on the LAN reach it directly at `<this-machine>:7687`.
+
+Qdrant's dashboard (`6333`) stays on its raw LAN IP/port for now — no hostname assigned yet.

docs/proxy-key-onboarding.md

@@ -1,13 +1,17 @@
-# Onboarding a workload onto the AI proxy
+# Onboarding a workload onto the AI gateway
 
-How to issue a new per-workload API key against the LiteLLM proxy (see [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
+How to issue a new per-workload API key against the OmniRoute gateway (see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) — the LiteLLM → OmniRoute migration; original gateway rationale in [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
+
+No workload in this stack itself needs a key right now — every client is external (a coding CLI, or another self-hosted service). There's no scripted mint yet either way: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key by hand for now, via the dashboard steps below.
 
 ## Create the key
 
-1. Log into LiteLLM's Admin UI (`/ui` on the proxy's deployed URL).
-2. Create a new virtual key ("Keys" → "Create Key").
-3. Name it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `openwebui`. This name is the ledger: LiteLLM's dashboard lists keys by name, so there's no separate tracking doc to keep in sync — name it clearly and the Usage tab tells you the rest (spend, last used, etc.).
-4. Leave budget and rate limits unset (unlimited) by default. This is a shadow-cost estimate for fun, not real accounting or resource protection — see `docs/research/proxy-shadow-pricing.md`. Only set a budget if a specific workload turns out to need a tripwire.
+1. Log into the omniroute dashboard. `DASHBOARD_PORT` (20128) is never published to the host (see `docker-compose.yml`'s `omniroute` service) — from the R9700 box itself, find the container's own address (`docker inspect -f '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`) and browse to `http://<that-ip>:20128` (the host can reach a container's bridge-network IP directly, published port or not). From elsewhere, SSH port-forward instead: `ssh -L 20128:<container-ip>:20128 <host>`, then browse `http://localhost:20128`.
+2. "Keys" → "Create API key".
+3. Label it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `claude-code-cli`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync.
+4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.
+
+Once `POST /api/keys`'s session-auth flow is worked out (issue #37), the equivalent `curl` here can replace this manual step, the way `update.sh` used to automate LiteLLM's `/key/generate`.
 
 ## Hand it to the workload
 
@@ -15,7 +19,7 @@ Drop the key into that workload's own `.env` (or equivalent config) — never in
 
 ## Retiring or rotating a key
 
-No scheduled rotation. Revoke the key by hand in the Admin UI ("Keys" → delete) only when:
+No scheduled rotation. Revoke the key by hand in the dashboard ("Keys" → delete) only when:
 - the workload is retired, or
 - the key is suspected leaked/compromised.
 

docs/proxy-request-priority.md

@@ -1,12 +1,14 @@
 # Request priority on the AI proxy
 
-One local model instance (llama.cpp on the single R9700) serves every workload through the LiteLLM proxy ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
+**Stale as of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (LiteLLM → OmniRoute migration)** — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
+
+One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
 
 ## Tiers
 
 Two tiers, assigned per workload's virtual key (per `docs/proxy-key-onboarding.md`):
 
-- **High priority** (interactive — someone's waiting): Open WebUI chat, coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review.
+- **High priority** (interactive — someone's waiting): coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review.
 - **Low priority** (batch — nobody's watching a spinner): Paperless OCR/tagging, Nextcloud Memories face-recognition, AI watermark removal.
 
 ## Mechanism

docs/research/500k-context-2-parallel-agents-gpu-roadmap.md

@@ -0,0 +1,737 @@
+# Research: hardware roadmap to 500k-token context × 2 parallel agents (1M stretch)
+
+**Date:** 2026-09-11
+
+**Question:** What VRAM does 500k-token context × 2 parallel llama-server slots (and a 1M-token
+stretch goal) actually cost for the Qwen3 family, and what hardware roadmap gets there from the
+current single-R9700 setup — given the user's stated plan to add an older (PCIe 4.0) Threadripper
+for lane count, reuse existing RAM/PSU (~200W headroom / one spare 8-pin), mix in already-owned
+NVIDIA cards (GTX 1080 8GB, RTX 2080 8GB, GT 710 1GB) for the classifier role, and price used GPUs
+at roughly $20-30/GB VRAM?
+
+**Answer, short version:** The two goals ("500k × 2 parallel" and "1M stretch") turn out to need
+**the same total VRAM budget** — because of how llama-server's `--ctx-size` and `--parallel` interact
+(§2), 500k × 2 slots and a single 1M-token slot both require setting `--ctx-size 1000000`. At
+`q4_0`-quantized KV cache that's **~33 GB** (weights + KV) for Qwen3.8-27B, at `q8_0` it's **~48 GB**,
+at fp16 it's **~79 GB** — before compute-buffer overhead. That does not fit on the current single
+32GB R9700 at any KV precision, and comfortably fits on two 32GB-class cards only at `q8_0`/`q4_0`.
+The user's $20-30/GB pricing intuition holds for last-gen used consumer cards (RTX 3060 12GB) but
+**not** for RTX 3090 24GB (~$44/GB currently) or a second R9700 (~$41/GB, new — no used market yet
+for a card released mid-2026). The stated Threadripper plan needs to specifically target the
+**non-PRO Threadripper 3000 series on sTRX4** (64 lanes, PCIe 4.0) — older Threadripper on the
+original TR4 socket (1000/2000 series) is PCIe 3.0 only, which doesn't match the user's own PCIe 4.0
+requirement. The power budget (~200W / one spare 8-pin) is exhausted by a *single* mid-tier used GPU
+addition — a PSU upgrade is not optional past the very first stage. See §7 for the roadmap.
+
+---
+
+## 1. Current state (from this repo)
+
+From `docker-compose.yml` and `.env.example` at the repo root:
+
+- **Main model:** `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights), `--ctx-size 262144`,
+  `--parallel 2`, `--flash-attn on`, `--cache-type-k q8_0 --cache-type-v q8_0`, `--n-gpu-layers 999`,
+  on one AMD Radeon AI PRO R9700 (32GB, ROCm/HIP, `gfx1201`).
+- **Classifier ("fast") model:** `Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`, `--ctx-size 65536`,
+  `--parallel 1`, `--n-gpu-layers 28` (partial offload), `--cache-type-k/v q4_0`, its own container on
+  the *same* R9700, sharing VRAM with the main model — see
+  [`docker-compose.yml`](../../docker-compose.yml) lines ~58-99 and
+  [`fast-model-choice.md`](fast-model-choice.md).
+- `.env.example` already documents the exact fact this research turns on: *"Each slot gets
+  `LLAMA_CTX_SIZE / LLAMA_PARALLEL` tokens of context"* — i.e. today's 262144 ctx-size ÷ 2 parallel
+  slots means each real request only gets **~131K tokens**, not the full 262144, confirmed in-repo
+  before any external source was checked.
+- Prior research already worked out the KV-cache formula for this exact model
+  ([`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md)) — this doc reuses and extends that math for the
+  500k/1M targets rather than re-deriving it.
+
+`docs/server-planing.md` describes a **different, earlier plan**: a 4× AMD Radeon AI PRO R9700 rig
+on a Gigabyte MZ32-AR0 (single-socket SP3/EPYC, 128 PCIe 4.0 lanes), fully AMD/ROCm. The user's plan
+in this ticket is not that — it pivots toward an older **Threadripper** (SP3's sibling desktop-HEDT
+socket family, not SP3 itself) and explicitly wants to mix in already-owned **NVIDIA** cards. These
+two plans are **not the same build** and, per §6, ROCm and CUDA cards cannot share one llama.cpp
+process — they can only coexist as separate containers on separate cards. Treat `server-planing.md`
+as superseded context, not the active plan, unless the user says otherwise.
+
+---
+
+## 2. llama-server parallelism: does each slot get its own full `--ctx-size`, or is it divided?
+
+**Divided.** This is the single fact that changes the whole budget by 2×, confirmed from three
+independent primary sources:
+
+1. **This repo's own `.env.example`** (quoted above) already documents it for the current deployment.
+2. **llama.cpp's own server README**, fetched directly
+   ([`tools/server/README.md`](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)):
+   `--ctx-size (-c)`: *"size of the prompt context (default: 0, 0 = loaded from model)"*;
+   `--parallel (-np)`: *"number of server slots (default: -1, -1 = auto)"* — the docs list these as
+   independent flags, but don't spell out the division themselves.
+3. **A real user's server log**, quoted verbatim in
+   [ggml-org/llama.cpp#11681](https://github.com/ggml-org/llama.cpp/issues/11681), is the actual proof:
+   running with `--ctx-size 327680 --parallel 6` produces `n_ctx = 327680`,
+   `n_ctx_per_seq = 54613` — i.e. `327680 / 6 ≈ 54613`. The reporter explicitly asked for a
+   `--ctx-size-per-seq`-style flag to *avoid* this division; no such flag exists as of the fetch date.
+
+Practical consequence: **to get 500,000 usable tokens on each of 2 parallel slots, `--ctx-size` must
+be set to 1,000,000, not 500,000.** The KV cache is sized off the *total* `--ctx-size`
+(`--kv-unified`, on by default when slots are auto per the README's `-kvu` entry, uses one shared
+pool sized to the full `n_ctx`) — so the VRAM cost of "500k × 2 parallel" and "one 1M-token slot"
+is **identical**: both require `--ctx-size 1000000`. This is a genuinely useful finding for the
+roadmap — reaching the 500k×2 target and the 1M stretch goal cost the same VRAM; the only difference
+is `--parallel 1` vs `--parallel 2` at deploy time, a config change with zero extra hardware cost.
+
+`--cache-type-k` / `--cache-type-v` accept `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`
+(default `f16`), per the same README fetch. The repo already uses `q8_0` on the main model and `q4_0`
+on the classifier, so both quantization tiers used in the math below are already-proven-working
+configurations in this stack, not hypothetical flags.
+
+---
+
+## 3. KV-cache math per model
+
+### Qwen3.8-27B (hybrid Gated-DeltaNet / attention)
+
+Reusing the architecture params already pulled from
+[`Qwen/Qwen3.8-27B/config.json`](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json) in
+[`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md), re-verified directly for this doc: `num_hidden_layers=64`,
+`full_attention_interval=4` → **16 of 64 layers are standard KV-caching attention**, the other 48 are
+Gated DeltaNet linear-attention layers with a small, context-length-*independent* recurrent state
+(tens of MB total, negligible next to the attention KV cache — ignored below).
+`num_key_value_heads=4` (GQA), `head_dim=256`. Native context `max_position_embeddings=262144`
+(YaRN-extensible to 1M per the model card — **both the 500k and 1M targets exceed native context and
+require RoPE/YaRN scaling**, which is a real quality caveat, not just a memory one — Qwen has not
+published independent long-context quality benchmarks past native length that this research found).
+
+Per-token KV cache, fp16, both K and V, across the 16 full-attention layers:
+
+```
+16 layers × 2 (K+V) × 4 kv_heads × 256 head_dim × 2 bytes = 64 KiB/token
+```
+
+| Total ctx-size | KV cache, fp16 | KV cache, q8_0 | KV cache, q4_0 |
+|---|---|---|---|
+| 262,144 (current) | ~16.0 GiB | ~8.0 GiB | ~4.0 GiB |
+| 500,000 | ~30.5 GiB | ~15.3 GiB | ~7.6 GiB |
+| **1,000,000 (500k×2, or 1M stretch)** | **~61.0 GiB** | **~30.5 GiB** | **~15.3 GiB** |
+
+(`q8_0` is 8-bit vs. fp16's 16-bit → exactly half; `q4_0` is 4-bit → exactly quarter, per llama.cpp's
+own cache-type byte widths.)
+
+### Qwen3-4B-Instruct-2507 (plain GQA transformer, classifier role)
+
+From [`Qwen/Qwen3-4B-Instruct-2507/config.json`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
+(already pulled in [`fast-model-choice.md`](fast-model-choice.md)): `num_hidden_layers=36` — every
+layer is standard attention here (no hybrid split), `num_key_value_heads=8`, `head_dim=128`.
+
+```
+36 layers × 2 (K+V) × 8 kv_heads × 128 head_dim × 2 bytes = 144 KiB/token
+```
+
+The classifier's real transcript ceiling is ~40-50K tokens (qwen-code's own
+`MAX_TRANSCRIPT_MESSAGES=40` × `MAX_HISTORICAL_ACTION_CHARS=4000`, per `fast-model-choice.md` §"what
+actually shipped") — nowhere near 500k/1M, so the classifier does **not** need to grow for this
+roadmap; it stays exactly as deployed today, on its own small allocation. Per-token cost is included
+here only because it feeds the "does the classifier's dedicated GPU need to change" question in §6.
+
+---
+
+## 4. Total VRAM budget: 500k × 2 parallel, and the 1M stretch
+
+Weights: `Qwen3.8-27B-UD-Q4_K_XL.gguf` is **17.6 GB**, confirmed directly from the
+[unsloth/Qwen3.8-27B-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main)
+(already verified in `qwen3.8-27b-quant.md`).
+
+Per §2, both "500k × 2 parallel" and "1M stretch" require `--ctx-size 1000000` — same KV budget:
+
+| KV precision | KV cache | + weights (17.6 GB) | + est. compute-buffer/runtime overhead* | **Realistic total** |
+|---|---|---|---|---|
+| fp16 (default) | 61.0 GiB | 78.6 GiB | +3-6 GiB | **~82-85 GB** |
+| q8_0 (proven in this stack today) | 30.5 GiB | 48.1 GiB | +3-6 GiB | **~51-54 GB** |
+| q4_0 (proven in this stack today, on the classifier) | 15.3 GiB | 32.9 GiB | +3-6 GiB | **~36-39 GB** |
+
+\* *Estimate, not a cited figure* — llama.cpp's flash-attention compute buffer scales closer to
+linear than the unfused-attention path, per this repo's own measured note in `docker-compose.yml`'s
+`qwen-classifier` comment (unfused attention buffers ballooned unexpectedly at 65536 ctx; flash-attn
+fixed it). `--flash-attn on` is already the deployed default for the main model, so the linear-ish
+regime applies, but no primary source gives an exact formula for this buffer size at 1M context — the
++3-6 GiB band is this doc's estimate based on the ratio observed in that in-repo incident, not a
+llama.cpp-documented number. Budget for the high end of that range when sizing hardware.
+
+**Bottom line:** at `q4_0` KV (the most aggressive, already-proven-in-this-repo tier), 500k×2 /
+1M needs **~36-39 GB** total VRAM for the 27B model alone. That does not fit one 32GB card at any
+precision — it needs at least two 32GB-class cards, or one ≥40GB card. At `q8_0` (the precision this
+repo already runs in production for quality reasons), budget **~51-54 GB** — two 32GB cards (64GB
+pooled) clears this with room to spare; a single 48GB-class card would not.
+
+(§11 below extends this table to higher weight-quant tiers — Q6_K_XL, Q8_0, BF16 — for users who want
+better output quality than `Q4_K_XL`, and to a Flash-Next alternative architecture; see §11.6-§11.7.)
+
+---
+
+## 5. CPU/motherboard: which Threadripper generations give PCIe 4.0, and how many lanes for GPUs
+
+AMD's own product/chipset pages, cross-checked against the launch reviews that quote them directly:
+
+| Platform | Socket | PCIe generation | Total CPU-provided lanes |
+|---|---|---|---|
+| Threadripper 1000/2000 series ("1920X", "2950X", etc.) | **TR4** | **PCIe 3.0 only** | 60-64 |
+| Threadripper 3000 series (3960X/3970X/3990X) | **sTRX4** | **PCIe 4.0** | 64 |
+| Threadripper 7000 series (non-PRO) | sTR5 | PCIe 5.0 (48 lanes) + PCIe 4.0 (24-32 lanes) | ~72-80 |
+| Threadripper PRO 3000WX/5000WX | sWRX8 | PCIe 4.0 | **128** |
+| Threadripper PRO 7000WX | sTR5 (WRX90) | PCIe 5.0 (128 lanes) + a few PCIe 3.0 | **128** |
+
+Sources: [AMD chipsets product page](https://www.amd.com/en/products/processors/chipsets.html)
+(sWRX8/socket listing), corroborated by
+[Tom's Hardware — Threadripper 3960X/3970X, sTRX4/TRX40 launch coverage](https://www.tomshardware.com/news/amd-unveils-threadripper-3960x-and-3970x-ryzen-9-3950x-details-and-athlon-3000g/2)
+(*"the 3rd Gen TR CPUs carry the same 64 PCIe lanes but double bandwidth by moving from Gen 3.0 to
+Gen 4.0"* — explicit confirmation TR4/1000-2000-series is PCIe 3.0 while sTRX4/3000-series is PCIe
+4.0), [PCWorld — Threadripper PRO launch](https://www.pcworld.com/article/393181/amd-threadripper-pro-has-64-cores-128-pcie-lanes-and-8-channel-memory-support.html)
+(*"128 PCIe lanes"* for PRO).
+
+**This directly matters for the user's plan.** "An older Threadripper... for more PCIe lanes" is
+ambiguous between two real, very different chips:
+
+- **TR4 (1000/2000 series)** — cheapest used option, but **PCIe 3.0** — does not meet the user's own
+  stated PCIe 4.0 requirement, and PCIe 3.0 x8 per GPU roughly halves inter-GPU/host transfer
+  bandwidth (matters more for training/tensor-parallel than for llama.cpp's inference-time layer
+  splitting, but still a real downgrade vs. the R9700's native PCIe 5.0).
+- **sTRX4 (3000 series, non-PRO)** — the correct "older Threadripper with PCIe 4.0" target: 64 lanes,
+  4-5 years old, real used-market availability, no PRO price premium.
+- **Threadripper PRO (3000WX/5000WX)** doubles the lane count to 128 but at meaningfully higher used
+  cost (workstation-tier, lower volume, sWRX8 boards are pricier than sTRX4/TRX40 boards) — worth it
+  only if 6 full-bandwidth (x16) GPU slots are actually needed; at x8-per-card (adequate for inference)
+  64 lanes already covers 6 GPUs with lanes to spare for NVMe/chipset.
+
+**Lane budget for 6 GPUs on sTRX4 (64 lanes), estimated (no vendor spec gives a topology this
+specific — treat this bullet as an estimate):** typical sTRX4 boards reserve ~4 lanes for the
+chipset uplink and commonly wire 1-2 M.2 slots directly to the CPU (4 lanes each) — so realistic
+GPU-available lanes land around 44-52 of the 64, i.e. **6 GPUs at x8 electrical each (48 lanes) is
+plausible but board-model-dependent**; x16-each for 6 cards is not possible on 64 lanes regardless of
+board. x8 electrical is not a meaningful inference-speed penalty for llama.cpp (weights are loaded
+once; the ongoing per-token traffic across PCIe is small compared to compute), so this is an
+acceptable tradeoff, not a real bottleneck for this workload.
+
+---
+
+## 6. Power budget vs. the ~200W / one spare 8-pin headroom
+
+Official/vendor TDPs:
+
+| Card | TDP | Source |
+|---|---|---|
+| GTX 1080 (owned) | 180W, one 8-pin | [confirmed 180W, PCIe 3.0 x16, 1× 8-pin](https://buildmyserver.com/products/zotac-nvidia-geforce-gtx-1080-8gb-gddr5-180w-pcie-3-0-x16-double-wide-gpu) — spec matches NVIDIA's own launch figures reported across multiple outlets incl. Tom's Hardware |
+| RTX 2080 (owned) | 215W | Cross-checked across gpuzoo/cputronic/notebookcheck spec pages, consistent at 215W |
+| GT 710 (owned) | ~19W, **no external power connector** (slot power only) | [MSI/EVGA/Zotac GT 710 spec pages](https://www.msi.com/Graphics-Card/GT-710-1GD5-LP/Specification) |
+| RTX 3060 12GB (candidate purchase) | 170W, one 8-pin | [NVIDIA-confirmed 170W TDP, one 8-pin connector](https://www.lowyat.net/2021/232659/nvidia-geforce-rtx-3060-specifications-now-official-includes-3584-cuda-cores-and-170w-tdp/) |
+| RTX 3090 24GB (candidate purchase) | 350W, two 8-pin, [NVIDIA's own RTX 3090 product page](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090/) lists 350W and a 750W PSU minimum | NVIDIA official |
+| R9700 32GB (already deployed / "more of the same") | 300W (per this repo's `server-planing.md`, consistent with AMD's own R9700 product page framing it as a 300W-class card) | in-repo prior research |
+
+**Against the stated ~200W / one spare 8-pin budget:**
+
+- Adding **one RTX 3060 12GB** (170W, one 8-pin) is the *only* candidate in this list that fits the
+  stated headroom as-is — it uses the one spare connector and stays under 200W.
+- Adding the already-owned **GTX 1080** (180W) as the classifier's dedicated card also just barely
+  fits (180W ≤ 200W, one 8-pin) — this is a genuinely free option since the card is already owned and
+  its power draw is within budget, unlike every purchase candidate below.
+- Adding the already-owned **RTX 2080** (215W) **exceeds** the stated 200W headroom by 15W — technically
+  over budget on paper, though real-world draw is usually a bit under rated TDP; flag it as marginal,
+  not safely fitting.
+- Adding a **second R9700** (300W) or an **RTX 3090** (350W, needs two 8-pin — the user has only one
+  spare) both blow well past the current power budget on both watts and connector count.
+- **The GT 710 draws no meaningful power (~19W, no PCIe power connector at all)** — it is free from a
+  power-budget standpoint regardless of what else is added.
+
+**PSU upgrade trigger:** the very first stage that adds *any* GPU beyond a GTX 1080-class card (180W,
+one 8-pin) or an RTX 3060 12GB (170W, one 8-pin) exhausts the stated headroom. Any stage that reaches
+for a second 32GB-class card (R9700 or equivalent) or any 300W+ card **requires a PSU upgrade before
+that stage**, not after — see the roadmap table in §7 for exactly which stage that is.
+
+---
+
+## 7. Mixed-GPU feasibility: ROCm + CUDA, and is the GT 710 usable at all
+
+**ROCm and CUDA are different llama.cpp builds, but that's exactly the pattern already in this
+repo.** `ghcr.io/ggml-org/llama.cpp` publishes both `server-rocm` and `server-cuda` as separate,
+independently-built image tags (confirmed present on the [ggml-org container registry](https://github.com/orgs/ggml-org/packages/container/llama.cpp)
+and documented in [`docs/docker.md`](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/docs/docker.md)
+— *"server-cuda: Same as `server` but compiled with CUDA support"*, *"server-rocm: Same as `server`
+but compiled with ROCm support"*). You cannot mix backends inside one process/container, but you
+**can** run one `server-rocm` container pinned to the R9700 and a separate `server-cuda` container
+pinned to an NVIDIA card, simultaneously, on the same host — this is architecturally identical to
+today's `llama-server` + `qwen-classifier` two-container split in `docker-compose.yml`, just with a
+different image tag for the NVIDIA-backed service and NVIDIA's container runtime (`nvidia-container-toolkit`
++ `--gpus` / device reservation, the CUDA-world equivalent of this repo's `/dev/kfd`+`/dev/dri`+
+numeric-GID ROCm pattern documented in
+[`rocm-gpu-pin-and-render-group.md`](rocm-gpu-pin-and-render-group.md)). None of that doc's ROCm-specific
+findings (the `GPU_MAX_HW_QUEUES=1` MES firmware workaround, the numeric-GID `group_add` fix) apply to
+an NVIDIA/CUDA container — those are ROCm-stack-specific bugs, not general multi-GPU-container issues.
+
+**Is this an implicit AMD→NVIDIA rebuild, or additive?** Worth surfacing explicitly since the two
+source plans conflict on this: `server-planing.md` is an AMD-only, ROCm-only 4×R9700 plan. This
+ticket's plan is **additive/mixed** — keep the R9700 running the main model under ROCm, and bolt on
+NVIDIA cards under CUDA for secondary roles (classifier, or a second inference GPU for the big model
+if going the "more of the same type" route means buying NVIDIA instead of more R9700s). Both are
+internally consistent, but they are different end-states — flag this choice back to the user rather
+than assuming one.
+
+**Splitting the *main* 27B model itself across mixed AMD+NVIDIA silicon in one process is not
+possible** — llama.cpp's multi-GPU tensor-split only works within a single backend build. To use
+both an R9700 and an NVIDIA card for the *same* model's layers, all the compute-hosting cards need to
+be the same backend (all-ROCm or all-CUDA) in that one process. This is why §5's roadmap treats "add
+GPU capacity to the main model" and "add a GPU for the classifier" as separable purchases with
+different backend constraints, not a single mixed pool.
+
+**Is the GT 710 usable for anything in this pipeline? No.** Reasoning:
+
+- 1GB VRAM cannot hold any meaningful fraction of either model's weights (17.6 GB / 2.4-4.3 GB) —
+  even a handful of transformer layers at Q4 quantization exceeds 1GB.
+- It's Kepler-generation silicon (192 CUDA cores, no tensor cores) — llama.cpp's CUDA backend
+  technically supports pre-Turing cards, but at this VRAM size there's nothing to usefully offload.
+- It draws power from the PCIe slot only, no external connector — genuinely free to keep installed.
+- **Plausible actual use: dedicate it as the box's display-output card**, so every compute-capable
+  GPU (R9700, and whichever NVIDIA cards get added) can be fully headless/compute-only with none of
+  their VRAM or a display output tied up driving a monitor — a real, if minor, use for it. This is
+  this doc's own inference from the spec facts above, not a claim found in any primary source.
+
+---
+
+## 8. GPU market pricing vs. the $20-30/GB assumption
+
+| Card | VRAM | Backend | Current used-market price (estimate — see caveat) | $/GB |
+|---|---|---|---|---|
+| RTX 3060 12GB | 12GB | CUDA | ~$240-300 used (eBay listings, [gpupoet.com tracker](https://gpupoet.com/gpu/shop/nvidia-geforce-rtx-3060): *"from $239"*, [eBay live listings](https://www.ebay.com/shop/rtx-3060-12gb) averaging ~$488 asking but with a $239 floor) | **~$20-25/GB** — matches the stated assumption |
+| RTX 3090 24GB | 24GB | CUDA | ~$1,010-1,050 used ([bestvaluegpu.com Sep 2026 tracker](https://bestvaluegpu.com/history/new-and-used-rtx-3090-price-history-and-specs/), [xda-developers coverage](https://www.xda-developers.com/used-rtx-3090-still-best-for-local-ai-in-value/)) | **~$42-44/GB** — well above the stated assumption |
+| R9700 32GB ("more of the same type") | 32GB | ROCm | **New only — $1,299 MSRP**, street price $1,400-1,585 as of this research ([overclock3d](https://overclock3d.net/news/gpu-displays/amd-unveils-its-1299-radeon-ai-pro-r9700-32gb-workstation-gpu/), [pricehistory.app tracker](https://pricehistory.app/p/powercolor-amd-radeon-ai-pro-r9700-32gb-BFcRhGIm)) — too recent a release (2026) for a used market to exist yet | **~$41-50/GB, and not a used-market price at all** |
+
+**Caveat on all three price figures:** these are live marketplace asking-price snapshots pulled via
+web search on 2026-09-11, not sold-price data or a vendor spec sheet — treat as directional, not
+exact. eBay asking prices in particular run above realized sale prices.
+
+**Correction to the user's stated assumption:** $20-30/GB is a good estimate specifically for
+**last-generation mainstream used cards** (RTX 3060 12GB fits it almost exactly) but **not** for
+high-VRAM flagship cards like the RTX 3090 (~1.5-2× that rate) or for "more of the same type" R9700
+units, which aren't used-market at all yet and sit even higher per GB than the 3090. If the plan is
+"cheapest path to more VRAM," multiple RTX 3060 12GB cards (or similar mid-tier used cards) beat one
+RTX 3090 on $/GB, at the cost of needing more PCIe slots and more total wattage/connectors to reach
+the same aggregate VRAM — which is exactly the tradeoff the Threadripper lane-count plan in §5 is
+for.
+
+---
+
+## 9. Step-by-step roadmap
+
+All "resulting max context" figures assume `--parallel 2` and the KV precision stated; per §2, the
+`--ctx-size` value shown is the *total* (pre-division) value to pass to llama-server.
+
+| Stage | Hardware change | Est. cost | Backend | Usable VRAM (main-model pool) | Max context @ parallel=2 (`q4_0` KV) | PSU upgrade triggered? |
+|---|---|---|---|---|---|---|
+| **0 (current)** | 1× R9700 32GB, in production | $0 | ROCm | 32GB (shared with classifier) | ~131K/slot today at `q8_0` KV (262144 total ÷ 2) | No |
+| **1 — classifier isolation** | Move classifier onto the already-owned **GTX 1080** (180W, own container, `server-cuda`), freeing the R9700 entirely for the main model. Matches the existing dual-model pattern qwen-code's own docs describe (§10) and this repo's `qwen-classifier` service already implements, just on separate silicon instead of a shared card. | $0 (already owned) | ROCm (main) + CUDA (classifier) | R9700's full 32GB now available to the main model alone | ~262K/slot @ `q8_0` (unchanged ctx-size, no more classifier contention) | **No** — 180W GTX 1080 fits the stated ~200W/one-8-pin headroom |
+| **2 — second big-model GPU** | Add **one more 32GB-class card** for the main model. Cheapest correct-backend option: a second R9700 (~$1,300-1,585 new, ROCm, same backend as the first — required if tensor-splitting one model across two cards) | ~$1,300-1,585 | ROCm | 64GB pooled | `--ctx-size 500000 --parallel 1` fits at `q4_0` (~33GB) or `q8_0` (~48GB, tight but fits in 64GB) — **not yet 500k×2** | **Yes** — 300W card, no spare 8-pin left after stage 1 |
+| **3 — reach 500k × 2 / 1M stretch** | No further hardware if stage 2's 64GB pool is used with `--cache-type-k/v q4_0`: `--ctx-size 1000000 --parallel 2` needs ~33-39GB (§4), fits inside 64GB with real headroom for the compute buffer. If `q8_0` KV is required instead (this repo's current quality bar for the main model), the ~51-54GB need is tight-to-marginal on 64GB — a **third** 32GB card (~96GB pool) removes the risk. | $0 (reuses stage 2) or +$1,300-1,585 for a 3rd card if `q8_0` KV is required | ROCm | 64GB (q4_0 case) or 96GB (q8_0 case) | **500k×2 parallel achieved**, and the 1M stretch goal is the *same config* with `--parallel 1` instead of 2 (§2) | Already upgraded at stage 2 |
+| **4 — optional CPU/lane platform swap** | Only needed if the plan is to keep scaling past 2-3 big cards, or to add several small used cards (RTX 3060 12GB) for extra headroom/throughput rather than raw ctx-size. Swap to **non-PRO Threadripper 3000-series (sTRX4)** — 64 PCIe 4.0 lanes, ~x8-per-slot for up to 6 GPUs (§5). Threadripper PRO 3000WX/5000WX (128 lanes) only if x16-per-card matters or 6+ full-bandwidth slots are wanted. | Used sTRX4 CPU+board: roughly $400-800 combined on the used market (not independently priced in this pass — **estimate**, not cited) | n/a (platform only) | n/a | n/a | Independent of GPU wattage — driven by whatever GPU count/wattage stage 5+ adds |
+| **5+ — scale-out via small used cards** | Add RTX 3060 12GB units (~$20-25/GB, the assumption that actually holds, §8) instead of more 32GB flagship cards, once lane count (stage 4) supports it — useful for extra parallel slots / throughput beyond the 500k×2 target rather than for raising ctx-size further (500k×2/1M is already met at stage 3). | ~$240-300/card | CUDA (separate container per §6) | +12GB pooled per card, but on a *different backend* from the ROCm main model — usable for extra classifier/small-model capacity or a separate CUDA-backend llama-server instance, not as additional tensor-split VRAM for the ROCm main model | Unchanged for the main model; adds parallel capacity elsewhere | Yes, cumulative — each additional 170W card needs PSU headroom stage 2 already consumed |
+
+**Where the existing dual-model pattern sits in this roadmap:** it's stage 1, and it's free. The
+qwen-code docs pattern (main model + a small, always-resident, non-thinking fast/classifier model —
+see §10) is already implemented in this repo; the only roadmap-relevant change is *which GPU* the
+classifier sits on, moving it off the R9700 entirely onto an already-owned NVIDIA card frees the
+R9700's full 32GB for the 500k×2/1M push instead of splitting it with the classifier as happens
+today.
+
+### 9.1 Upgrade path, as diagrams
+
+Diagram form of the same §9 table and §11.9's dense-vs-Flash-Next call — nothing new is claimed here,
+this is a visual index back into the cited sections above.
+
+**Stage-by-stage hardware path** (PSU-upgrade triggers and target reached called out inline):
+
+```mermaid
+flowchart TD
+    S0["Stage 0 — today<br/>1x R9700 32GB, ROCm<br/>classifier shares the card<br/>$0"]
+    S1["Stage 1 — classifier isolation<br/>+ GTX 1080 (owned, 180W, CUDA)<br/>R9700 freed for main model<br/>$0 · PSU OK (180W fits ~200W headroom)"]
+    S2["Stage 2 — 2nd big-model GPU<br/>+1x R9700 32GB (ROCm)<br/>64GB pooled<br/>~$1,300-1,585 · PSU UPGRADE REQUIRED (300W, no 8-pin left)"]
+    S3q4["Stage 3a — q4_0 KV<br/>--ctx-size 1,000,000 --parallel 2<br/>~33-39GB, fits in 64GB<br/>$0 (reuses stage 2)"]
+    S3q8["Stage 3b — q8_0 KV (current prod quality)<br/>~51-54GB, tight on 64GB<br/>+1x R9700 -> 96GB removes risk<br/>+~$1,300-1,585"]
+    TARGET(["500k x2 parallel reached<br/>= 1M stretch goal, same VRAM<br/>(--parallel 1 vs 2 is a config flag, §2)"])
+    S4["Stage 4 — platform swap (optional)<br/>sTRX4 Threadripper 3000, 64 PCIe4 lanes<br/>only needed past 2-3 big cards<br/>~$400-800 (estimate, §9)"]
+    S5["Stage 5+ — scale out<br/>+RTX 3060 12GB cards (CUDA, separate backend)<br/>extra parallel/throughput, not more ctx-size<br/>~$240-300/card · PSU upgrade each card"]
+
+    S0 --> S1 --> S2
+    S2 --> S3q4 --> TARGET
+    S2 --> S3q8 --> TARGET
+    TARGET -.->|"only if scaling past this"| S4 --> S5
+
+    style TARGET fill:#2e7d32,color:#fff,stroke:#1b5e20
+    style S2 fill:#8a5a00,color:#fff,stroke:#5c3d00
+    style S5 fill:#8a5a00,color:#fff,stroke:#5c3d00
+```
+
+**Model choice, and the one open question that could change it** (§11.9):
+
+```mermaid
+flowchart TD
+    Q{"Goal: 500k-1M ctx<br/>within a $20-30/GB VRAM budget?"}
+    D["Dense Qwen3.8-27B<br/>17.6-54.7GB weights (Q4_K_XL-BF16)<br/>reaches 500k x2 on 2-3 cards,<br/>500k x4 on 2-6 cards depending on quant<br/>(§11.7 tables)"]
+    F{"Try --n-cpu-moe:<br/>offload MoE experts to system RAM?<br/>(untested for this model, §11.8)"}
+    FBAD["Flash-Next, all-GPU weights<br/>111-354GB just for weights<br/>needs 4-13 cards before any KV cost<br/>NOT recommended at this budget (§11.9)"]
+    FGOOD["Flash-Next, experts in system RAM<br/>GPU VRAM could shrink a lot<br/>2.67x cheaper KV/token becomes relevant<br/>UNVERIFIED — prototype on real server first"]
+    CAVEAT["+ real caveat either way:<br/>PR #27742 flags unverified conv branch,<br/>3% QSA divergence, prefill-pos-0-only PLE<br/>(§11.1) — dense model carries no such flag"]
+
+    Q --> D
+    Q -->|"considering Flash-Next instead"| F
+    F -->|"works well"| FGOOD
+    F -->|"doesn't help / untested"| FBAD
+    FGOOD --> CAVEAT
+    FBAD --> CAVEAT
+
+    style D fill:#2e7d32,color:#fff,stroke:#1b5e20
+    style FBAD fill:#8a1c1c,color:#fff,stroke:#5c1212
+    style FGOOD fill:#8a5a00,color:#fff,stroke:#5c3d00
+```
+
+---
+
+## 10. Qwen-code's own docs on the fast-model/classifier pattern
+
+Fetched directly per the user's link:
+[qwenlm.github.io/qwen-code-docs/en/users/overview/](https://qwenlm.github.io/qwen-code-docs/en/users/overview/)
+— **the overview page itself does not describe the dual-model/classifier pattern**; it only covers
+single-model-provider setup (Alibaba ModelStudio / third-party / custom provider), one model at a
+time. The actual fast-model/classifier documentation lives on the **Auto Mode** page instead, which
+this repo's own `fast-model-choice.md` already fetched and cited in detail:
+[qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/) —
+summary (see `fast-model-choice.md` §1 for the full quote): a two-stage classifier gate, both stages
+using "your configured fast model (`/model --fast`)", Stage 1 a ~300ms `{shouldBlock}`-only check,
+Stage 2 a ~3-5s chain-of-thought review that only runs on a Stage-1 block. Nothing in either page
+gives a recommended *context size* or *model size* for the fast model beyond what's implied by that
+latency budget — this repo's own prior research (`fast-model-choice.md`) derived the actual context
+requirement from qwen-code's source code instead (`packages/core/src/permissions/classifier-transcript.ts`),
+since the docs pages don't state one. No new information changes that prior doc's conclusion; this
+section exists to confirm the overview page was checked directly as instructed and doesn't contradict
+or add to it.
+
+---
+
+## 11. Alternative: Qwen3.8-Flash-Next (MoE, hybrid attention)
+
+The user also wants to weigh switching (or adding) **Qwen3.8-Flash-Next** — a 125B-total/6B-active MoE
+with a hybrid recurrent-attention architecture — against staying on dense Qwen3.8-27B, and separately
+wants this section to cover **going up in weight quant** (Q4_K_XL → Q6_K_XL → Q8_0 → BF16/fp16) for
+*both* models, not just Q4. Feasibility first, since it gates everything else.
+
+### 11.1 Feasibility verdict: supported, but immature — read before trusting any number below
+
+Checked directly against the primary sources the task named:
+
+- **llama.cpp mainline support exists.** [PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742)
+  ("model: add Qwen3.8-Flash-Next (qwen4exp)") was **merged into `master` on 2026-08-27** by ngxson.
+  It adds the full architecture: Gated DeltaNet layers (sigmoid-gated linear attention), QSA
+  ("Qwen Sparse Attention", operating at micro-block granularity), hyper-connections, and the PLE
+  n-gram embedding table. `llama.cpp`'s own docs list CPU/CUDA/Metal/ROCm as supported backends for
+  it — this is not a CUDA-only feature.
+- **This repo's pinned image is a floating tag, not a version pin.** `docker-compose.yml` runs
+  `ghcr.io/ggml-org/llama.cpp:server-rocm` with no date/digest suffix — a rolling "latest ROCm server
+  build" tag, not a release version. The merge is from 2026-08-27, and today is 2026-09-11 (~2 weeks
+  later), so a **fresh pull** of `server-rocm` should include it — but whatever image is already
+  cached/running on the R9700 box may predate the merge. **Action before touching this model on the
+  server: `docker compose pull llama-server` and check the startup log's build/commit banner is dated
+  on/after 2026-08-27**, not just "the tag says server-rocm."
+- **Real, primary-source-flagged immaturity — this is the part that should temper enthusiasm.** The
+  PR's own description/review discussion states: *"The conv branch itself is still numerically
+  unverified because the fixture zeroes its weights"*; QSA sparse attention *"diverges on 3 percent of
+  positions"* above its budget threshold; the PLE depthwise convolution *"is exact only for a prefill
+  that starts at position 0"* (i.e. correctness is not guaranteed once `--cache-reuse`/prompt-caching
+  is in play — a flag this repo already turns on for the dense model per the latest commit). None of
+  that is disqualifying, but it is a primary-source admission that this is a fresh, not-fully-verified
+  implementation, not a mature, widely-battle-tested one like the dense Qwen3.8-27B path.
+- **Multi-slot serving needs an explicit new flag.** The same PR states: *"`set_input_qsa` asserted
+  `n_stream == 1`, so llama-server could not serve this model with more than one slot unless `-kvu`
+  was passed."* Per the server README (§2), `--kv-unified`/`-kvu` defaults to enabled **only when slot
+  count is auto** (`-1`). This repo's compose file sets `--parallel ${LLAMA_PARALLEL:-2}` **explicitly**
+  (not auto) — so adopting Flash-Next with `--parallel` > 1 requires **adding `--kv-unified` (or
+  `-kvu`) to the launch flags**, a real deploy-time change, not something that "just works" by copying
+  today's flag set onto a new model file.
+
+**Verdict: yes, runnable** on this repo's backend (ROCm, mainline, no dev branch needed) as long as the
+image is pulled after 2026-08-27 and `-kvu` is added for multi-slot use — but treat it as
+**usable-with-caution**, not a drop-in swap, given the PR author's own unresolved-correctness notes.
+
+### 11.2 Architecture, verified against `config.json` directly
+
+Fetched from `Qwen/Qwen3.8-Flash-Next`'s `config.json` (unsloth's GGUF repo repackages the same base
+model): `num_hidden_layers=48`, `hidden_size=2560`, `num_attention_heads=24`, `num_key_value_heads=2`,
+`head_dim=256`, `max_position_embeddings=262144` (same native/extensible-to-1M framing as the dense
+model — same YaRN quality caveat from §3 applies here too, unverified past native length), `num_experts=512`,
+`num_experts_per_tok=10`, and the linear-attention head config: `linear_num_key_heads=16`,
+`linear_num_value_heads=48`, `linear_key_head_dim=128`, `linear_value_head_dim=128`.
+
+Layer pattern (confirmed both from the model card's own description and `config.json`'s
+`full_attention_interval=4`): every 4th layer is full/QSA attention, the other 3 are Gated DeltaNet —
+**12 of 48 layers grow a real KV cache; the other 36 have a fixed-size recurrent state that does not
+grow with context length.** (24% full-attention layers vs. the dense model's 16-of-64 = 25% — similar
+ratio, but the *absolute* per-layer KV cost differs because `num_key_value_heads` is 2 here vs. 4 on
+the dense model — see below.)
+
+### 11.3 Per-token growing-KV-cache cost
+
+```
+12 full-attention layers × 2 (K+V) × 2 kv_heads × 256 head_dim × 2 bytes (fp16) = 24 KiB/token
+```
+
+| Total ctx-size | KV cache, fp16 | KV cache, q8_0 | KV cache, q4_0 |
+|---|---|---|---|
+| 262,144 (native) | ~6.0 GiB | ~3.0 GiB | ~1.5 GiB |
+| 500,000 | ~11.4 GiB | ~5.7 GiB | ~2.9 GiB |
+| **1,000,000 (500k×2, or 1M stretch)** | **~22.9 GiB** | **~11.4 GiB** | **~5.7 GiB** |
+| **2,000,000 (500k×4)** | **~45.8 GiB** | **~22.9 GiB** | **~11.4 GiB** |
+
+`--cache-type-k/v` are the same generic llama.cpp KV-cache-quantization flags used elsewhere in this
+doc; nothing in the PR or the server README suggests they're handled differently for the 12
+full-attention layers of a hybrid model — they quantize the same growing K/V buffers as on a plain
+transformer. (No primary source explicitly confirms this for *this* architecture specifically — flagged
+as a reasonable extrapolation, not a directly-cited fact, same caveat class as this doc's other
+estimates.)
+
+### 11.4 Fixed (non-growing) recurrent state — Gated DeltaNet layers
+
+The 36 Gated DeltaNet layers each keep a fixed-size recurrent state (an outer-product-style
+key×value matrix per head) that does **not** scale with context length — only with slot/sequence
+count. Sized from `config.json`'s linear-attention head params:
+
+```
+36 layers × linear_num_value_heads(48) × linear_key_head_dim(128) × linear_value_head_dim(128) × 4 bytes (fp32 state)
+≈ 36 × 48 × 128 × 128 × 4 bytes ≈ 108 MiB per slot
+```
+
+This is **this doc's own derivation from the published head-dimension params, not a value pulled
+directly from llama.cpp source or docs** — the PR text confirms the state exists per-stream/per-slot
+but doesn't publish an exact byte formula, so treat the ~108 MiB/slot figure as an estimate, medium
+confidence. Even at 4 parallel slots that's under half a gigabyte — **negligible** next to both the
+growing KV cache (GBs) and the weights (tens to hundreds of GB) computed below. The headline
+implication holds regardless of the exact multiplier: Flash-Next's "big memory line item" is the MoE
+weights, not the attention state of any kind.
+
+### 11.5 Magnitude vs. the dense model — how much cheaper is KV, really
+
+At the same total ctx-size, Flash-Next's growing KV cache is **24 KiB/token vs. the dense model's
+64 KiB/token — 2.67× smaller**, i.e. Flash-Next's KV budget is **37.5%** of the dense model's at
+identical context length. This is a real, significant win *for the KV-cache line item specifically* —
+but see §11.9: it's a much smaller slice of a much bigger total, because the weights move the other
+way by a far larger factor.
+
+### 11.6 Weight sizes — verified from each unsloth GGUF repo's actual file listing
+
+Fetched directly from the HF file trees (not estimated from ratios), current as of this research pass:
+
+| Quant tier | Qwen3.8-27B (dense) | Qwen3.8-Flash-Next (MoE) |
+|---|---|---|
+| Q4_K_XL (`UD-Q4_K_XL`) | **17.6 GB** (existing baseline) | **111.4 GB** (4 parts: 10.9MB + 49.9GB + 49.4GB + 12.1GB) |
+| Q6_K_XL (`UD-Q6_K_XL`) | **25.3 GB** | **169 GB** (6 parts) |
+| Q8_0 | **29 GB** | **188 GB** (6 parts) |
+| BF16/fp16 | **54.67 GB** (50GB + 4.67GB, 2 parts) | **354 GB** (8 parts) |
+
+Sources: [unsloth/Qwen3.8-27B-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main)
+and its `BF16/` subfolder; [unsloth/Qwen3.8-Flash-Next-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main)
+and its `UD-Q4_K_XL/`, `UD-Q6_K_XL/`, `Q8_0/`, `BF16/` subfolders (per-file sizes summed). The
+preliminary Q8_0 figure floated before this research pass (~192GB) was slightly high — the real
+listing sums to **188 GB**; everything else in the preliminary list was accurate to within rounding.
+
+**The weight-quant axis and the KV-cache-quant axis are independent knobs.** Raising weight quality
+(Q4_K_XL → BF16) does not require raising `--cache-type-k/v` — the two flags are unrelated, and this
+repo already proves that pattern works (`q8_0` KV cache is deployed today against `Q4_K_XL` weights).
+A user chasing **maximum output quality** can run e.g. **BF16 weights + `q4_0` KV cache** — full-precision
+weights for quality, still-compressed KV for context budget — or any other combination in the tables
+below; nothing about picking a higher weight quant forces a matching KV precision.
+
+### 11.7 Total VRAM: does it fit, across quant tiers and both parallelism targets
+
+All totals = weights + growing KV cache + an estimated **+3-6 GB** compute-buffer/runtime overhead
+(same estimate band as §4, carried over — not re-derived for this architecture; flagged medium
+confidence there too). "Cards" = ceil(total ÷ 32GB), i.e. how many R9700-class 32GB cards it takes.
+
+#### Dense Qwen3.8-27B — 500k × 2 parallel / 1M stretch (`--ctx-size 1,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| Q4_K_XL (17.6GB) | ~79-85GB (**3**) | ~48-54GB (**2**) | ~33-39GB (**2**) |
+| Q6_K_XL (25.3GB) | ~89-92GB (**3**) | ~59-62GB (**2**, tight) | ~44-47GB (**2**) |
+| Q8_0 (29GB) | ~93-96GB (**3**, edge) | ~63-66GB (**2-3**, edge) | ~47-50GB (**2**) |
+| BF16 (54.67GB) | ~119-122GB (**4**) | ~88-91GB (**3**) | ~73-76GB (**3**) |
+
+#### Dense Qwen3.8-27B — 500k × 4 parallel (`--ctx-size 2,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| Q4_K_XL (17.6GB) | ~143-146GB (**5**) | ~82-85GB (**3**) | ~51-54GB (**2**) |
+| Q6_K_XL (25.3GB) | ~150-153GB (**5**) | ~89-92GB (**3**) | ~59-62GB (**2**, tight) |
+| Q8_0 (29GB) | ~154-157GB (**5**) | ~93-96GB (**3**, edge) | ~63-66GB (**2-3**, edge) |
+| BF16 (54.67GB) | ~180-183GB (**6**) | ~119-122GB (**4**) | ~88-91GB (**3**) |
+
+#### Qwen3.8-Flash-Next — 500k × 2 parallel / 1M stretch (`--ctx-size 1,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| UD-Q4_K_XL (111.4GB) | ~137-140GB (**5**) | ~126-129GB (**4**, edge) | ~120-123GB (**4**) |
+| UD-Q6_K_XL (169GB) | ~195-198GB (**7**) | ~183-186GB (**6**) | ~178-181GB (**6**) |
+| Q8_0 (188GB) | ~214-217GB (**7**) | ~202-205GB (**7**) | ~197-200GB (**7**) |
+| BF16 (354GB) | ~357-360GB (**12**) | ~357-360GB (**12**) | ~357-360GB (**12**) |
+
+#### Qwen3.8-Flash-Next — 500k × 4 parallel (`--ctx-size 2,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| UD-Q4_K_XL (111.4GB) | ~160-163GB (**6**) | ~137-140GB (**5**) | ~126-129GB (**4**, edge) |
+| UD-Q6_K_XL (169GB) | ~218-221GB (**7**) | ~195-198GB (**7**) | ~183-186GB (**6**) |
+| Q8_0 (188GB) | ~234-237GB (**8**) | ~211-214GB (**7**) | ~199-202GB (**7**) |
+| BF16 (354GB) | ~397-400GB (**13**) | ~377-380GB (**12**) | ~366-369GB (**12**) |
+
+(Flash-Next's KV precision barely moves the total at any weight quant above `UD-Q6_K_XL` — the weights
+so dominate the budget that KV quantization stops mattering for the "how many cards" question. This
+is the clearest signal in this whole section: for Flash-Next, the weight-quant choice is the entire
+hardware-sizing decision; for the dense model, KV precision still matters a lot.)
+
+### 11.8 CPU MoE-expert offload — the one lever that could change this calculus
+
+Flash-Next is a 512-expert/10-active-per-token MoE, and llama.cpp has a purpose-built flag for exactly
+this shape of model, confirmed directly from the server README: **`--n-cpu-moe`** — *"keep the Mixture
+of Experts (MoE) weights of the first N layers in the CPU"* — plus the more general
+**`--override-tensor`** (*"override tensor buffer type"*, pattern-matched by tensor name) that the same
+flag is built on top of. Both are generic, architecture-agnostic llama.cpp mechanisms (they match on
+tensor name patterns, not model type), so there's no reason to expect them not to apply to Flash-Next's
+MoE tensors specifically — but this pass found **no primary source that has actually tested
+`--n-cpu-moe` against this specific qwen4exp architecture**, so treat "it works here" as plausible,
+not confirmed.
+
+If it does work as expected, this changes the whole weight-VRAM picture in §11.7: the ~90-95% of
+Flash-Next's weight footprint that's MoE expert tensors could live in system RAM while attention
+projections, the shared/non-expert tensors, and the full KV cache stay on GPU — meaning a much smaller
+GPU-VRAM number than the "all weights on GPU" tables above, at the cost of PCIe/RAM-bandwidth-bound
+inference speed for whichever experts get selected per token (this repo has no benchmark of that
+tradeoff, and it's highly system-RAM-bandwidth-dependent, so no number is given here — flagged as an
+escape hatch worth prototyping directly on the server, not something this research values responsibly
+without a real test run).
+
+### 11.9 Net recommendation: dense Qwen3.8-27B vs. Flash-Next, for this user's stated goal
+
+**Net loss for this user's goal, as things stand — stay on dense Qwen3.8-27B.** Reasoning:
+
+- The user's target (500k×2 or 500k×4, on a $20-30/GB-VRAM budget, GPUs in 32GB increments) is a
+  **VRAM-budget-constrained** goal, and §11.7 shows Flash-Next's *weights alone* (111-354GB depending
+  on quant) dwarf the entire dense-model total-VRAM figure from §4/§11.7 (33-183GB depending on quant)
+  at every parallelism target. Flash-Next's much cheaper per-token KV cache (§11.5, real and verified)
+  is a rounding error next to that weight-size gap — the "2.67× cheaper KV" win doesn't come close to
+  offsetting a "6-20× larger weight footprint," so at $20-30/GB-VRAM the *dense* model reaches 500k×2
+  or 500k×4 for a fraction of the card count and dollar cost that Flash-Next needs even at its lowest
+  usable quant (`UD-Q4_K_XL`, 4-5 cards minimum) — before even factoring in §11.1's immaturity flags.
+- The one scenario that could flip this verdict is `--n-cpu-moe` actually working well for this
+  architecture (§11.8) — if most of those 111-354GB of expert weights can sit in system RAM at
+  acceptable throughput, Flash-Next's GPU-VRAM number could shrink dramatically and its real KV-cache
+  advantage would start to matter. That is untested here and shouldn't be assumed; it's the one
+  concrete next step worth trying on the actual server before ruling Flash-Next out permanently.
+- Independent of VRAM: §11.1's primary-source-flagged correctness caveats (unverified conv branch,
+  3%-divergence QSA, prefill-position-0-only PLE exactness) are a real quality/stability risk on a
+  production coding-agent stack that dense Qwen3.8-27B simply doesn't carry, since it's been running
+  in this repo already.
+
+---
+
+## 12. 4-parallel × 500k scenario — all four combinations side by side
+
+Per §2's already-established, cited rule (`n_ctx_per_seq = n_ctx / n_parallel`,
+[ggml-org/llama.cpp#11681](https://github.com/ggml-org/llama.cpp/issues/11681)), the same division
+applies at 4 slots: **500k tokens on each of 4 parallel slots requires `--ctx-size 2,000,000`** —
+double the 2-parallel target's `--ctx-size 1,000,000`, for the same reason 500k×2 needed double
+262,144. This isn't a new mechanism, just the same formula at `--parallel 4`.
+
+Full per-quant-tier tables for all four combinations are in §11.7 above (dense×2, dense×4, Flash-Next×2,
+Flash-Next×4 are each their own table there). Headline comparison at the KV precision already proven
+in production in this repo (`q8_0`) and each model's respective current/cheapest-usable weight quant:
+
+| Scenario | `--ctx-size` | Weight quant | q8_0-KV total VRAM | Cards (32GB) |
+|---|---|---|---|---|
+| Dense × 2 (or 1M stretch) | 1,000,000 | Q4_K_XL (17.6GB, current) | ~48-54GB | **2** |
+| Dense × 4 | 2,000,000 | Q4_K_XL (17.6GB, current) | ~82-85GB | **3** |
+| Flash-Next × 2 (or 1M stretch) | 1,000,000 | UD-Q4_K_XL (111.4GB, cheapest usable) | ~126-129GB | **4**, edge |
+| Flash-Next × 4 | 2,000,000 | UD-Q4_K_XL (111.4GB, cheapest usable) | ~137-140GB | **5** |
+
+**4-parallel × 500k reachability against this repo's existing roadmap stages (§9):**
+
+- **(a) Current 1×R9700 32GB:** none of the four combinations fit — not even dense×2 at any weight/KV
+  quant (§4's own conclusion, unchanged).
+- **(b) The 2-3×R9700 roadmap already proposed in §9 (64-96GB):** covers **dense×2 fully** (stage 3, as
+  already established) and **dense×4 at `q4_0` KV with Q4_K_XL or Q6_K_XL weights** (~51-62GB, fits in
+  64-96GB) — but **not** dense×4 at higher weight quants (Q8_0/BF16 need 3-6 cards depending on KV
+  precision, per §11.7's dense×4 table) and **not any Flash-Next scenario** (minimum is 4 cards/128GB
+  even at the cheapest usable quant and tightest KV).
+- **(c) The full 4-6×R9700 stretch scenario** (`server-planing.md`'s original plan, 128-192GB pooled):
+  covers **dense×4 at every weight quant up to BF16** (worst case ~91GB at BF16+q4_0, well inside
+  128GB) and **Flash-Next×2 at `UD-Q4_K_XL`** (126-140GB, fits a 5-card/160GB build, tight on a 4-card/
+  128GB one) — but **not** Flash-Next×4 at any weight quant above `UD-Q4_K_XL`, and not Flash-Next at
+  `BF16` under any parallelism (needs 12-13 cards, an entirely different scale of build than anything
+  in this doc's roadmap).
+
+---
+
+## Sources
+
+- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) — `--ctx-size`, `--parallel`, `--cache-type-k/v`, `--kv-unified`, `--cache-reuse`, `--n-cpu-moe`, `--override-tensor` flag definitions
+- [ggml-org/llama.cpp#11681](https://github.com/ggml-org/llama.cpp/issues/11681) — real server log proving `n_ctx_per_seq = n_ctx / n_parallel`
+- [ggml-org/llama.cpp#27742](https://github.com/ggml-org/llama.cpp/pull/27742) — "model: add Qwen3.8-Flash-Next (qwen4exp)", merged 2026-08-27; architecture details, `n_stream == 1` / `-kvu` multi-slot requirement, and the conv-branch/QSA-divergence/PLE-prefill correctness caveats
+- [Qwen/Qwen3.8-27B config.json](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json)
+- [Qwen/Qwen3.8-Flash-Next config.json](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/raw/main/config.json)
+- [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
+- [unsloth/Qwen3.8-27B-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main) — 17.6GB (Q4_K_XL), 25.3GB (Q6_K_XL), 29GB (Q8_0), 54.67GB (BF16) weight sizes
+- [unsloth/Qwen3.8-Flash-Next-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main) — 111.4GB (UD-Q4_K_XL), 169GB (UD-Q6_K_XL), 188GB (Q8_0), 354GB (BF16) weight sizes, summed from each quant's per-file listing
+- [ggml-org/llama.cpp docs/docker.md](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/docs/docker.md) — `server-cuda`/`server-rocm` separate image tags
+- [AMD chipsets product page](https://www.amd.com/en/products/processors/chipsets.html)
+- [Tom's Hardware — Threadripper 3960X/3970X, sTRX4/TRX40 launch](https://www.tomshardware.com/news/amd-unveils-threadripper-3960x-and-3970x-ryzen-9-3950x-details-and-athlon-3000g/2)
+- [PCWorld — Threadripper PRO launch, 128 PCIe lanes](https://www.pcworld.com/article/393181/amd-threadripper-pro-has-64-cores-128-pcie-lanes-and-8-channel-memory-support.html)
+- [NVIDIA — GeForce RTX 3090 product page](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090/)
+- [Lowyat.net — RTX 3060 official 170W TDP](https://www.lowyat.net/2021/232659/nvidia-geforce-rtx-3060-specifications-now-official-includes-3584-cuda-cores-and-170w-tdp/)
+- [BuildMyServer — GTX 1080 180W/PCIe3.0/1×8-pin spec listing](https://buildmyserver.com/products/zotac-nvidia-geforce-gtx-1080-8gb-gddr5-180w-pcie-3-0-x16-double-wide-gpu)
+- [MSI — GT 710 1GD5 LP spec page](https://www.msi.com/Graphics-Card/GT-710-1GD5-LP/Specification)
+- [overclock3d — AMD Radeon AI PRO R9700 $1,299 MSRP](https://overclock3d.net/news/gpu-displays/amd-unveils-its-1299-radeon-ai-pro-r9700-32gb-workstation-gpu/)
+- [pricehistory.app — R9700 street price tracker](https://pricehistory.app/p/powercolor-amd-radeon-ai-pro-r9700-32gb-BFcRhGIm)
+- [bestvaluegpu.com — RTX 3090 used price tracker, Sep 2026](https://bestvaluegpu.com/history/new-and-used-rtx-3090-price-history-and-specs/)
+- [gpupoet.com — RTX 3060 12GB used listings](https://gpupoet.com/gpu/shop/nvidia-geforce-rtx-3060)
+- [Qwen Code docs — overview](https://qwenlm.github.io/qwen-code-docs/en/users/overview/)
+- [Qwen Code docs — Auto Mode (fast-model pattern)](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/)
+- This repo: [`docker-compose.yml`](../../docker-compose.yml), [`.env.example`](../../.env.example), [`docs/server-planing.md`](../server-planing.md), [`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md), [`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md), [`rocm-gpu-pin-and-render-group.md`](rocm-gpu-pin-and-render-group.md), [`fast-model-choice.md`](fast-model-choice.md)
+
+## Confidence/uncertainty summary
+
+- **High confidence:** the KV-cache-per-token formulas for both dense models and Flash-Next (computed
+  directly from each model's own `config.json`, same method this repo's prior research already used
+  and cross-checked); the `n_ctx_per_seq = n_ctx / n_parallel` division behavior (directly evidenced by
+  a real server log in a llama.cpp GitHub issue, and independently already documented in this repo's
+  own `.env.example`) — and confirmed to apply identically at `--parallel 4` since the mechanism is
+  parallel-count-agnostic; official TDP figures for GTX 1080, RTX 2080, RTX 3060, RTX 3090, GT 710
+  (each cross-checked against 2+ independent spec listings or the vendor's own product page); the
+  TR4-is-PCIe3/sTRX4-is-PCIe4 generational split (direct launch-coverage quote); the existence of
+  separate `server-cuda`/`server-rocm` llama.cpp image tags; Qwen3.8-Flash-Next's `config.json`
+  architecture params and PR #27742's merge date/status and its own stated correctness caveats and
+  `-kvu` multi-slot requirement (all directly quoted from the primary source); the weight file sizes
+  for both models at all four quant tiers (summed directly from each HF repo's real file listing, not
+  estimated).
+- **Medium confidence:** the compute-buffer/runtime-overhead estimate in §4/§11.7 (+3-6 GiB) —
+  extrapolated from one in-repo incident's before/after numbers, not a llama.cpp-documented formula,
+  and carried over to Flash-Next without re-derivation for its different architecture; the Gated
+  DeltaNet fixed recurrent-state size in §11.4 (~108 MiB/slot) — this doc's own derivation from the
+  published head-dimension config, not a value found in llama.cpp source or docs; whether
+  `--cache-type-k/v` quantization applies identically to Flash-Next's 12 full-attention layers as it
+  does to a plain transformer (reasonable extrapolation, not directly confirmed for this architecture);
+  whether `--n-cpu-moe`/`--override-tensor` actually work against Flash-Next's specific MoE tensor
+  layout (architecture-agnostic mechanism, but untested against this model by any primary source found);
+  real-world PCIe lane availability for 6 GPUs on a specific sTRX4 board (§5) — no single board's exact
+  lane map was fetched, this is a reasonable-but-unverified estimate from typical sTRX4 board behavior.
+- **Low confidence / explicitly estimated, not cited fact:** all used-GPU marketplace pricing (§8) —
+  live asking-price snapshots from a single search pass, not sold-price data; the used sTRX4
+  CPU+motherboard combo price in the roadmap's stage 4 (§9) — not researched at all in this pass,
+  flagged as a placeholder estimate; whether YaRN-scaled 500k/1M context actually holds output
+  quality for either Qwen3.8-27B or Qwen3.8-Flash-Next — no primary source (Qwen's own docs included)
+  publishes long-context quality benchmarks past the 262,144 native length for either model, so this is
+  a known-unknown carried forward from each model card's "YaRN-extensible" claim, not a verified
+  capability; whether the specific `ghcr.io/ggml-org/llama.cpp:server-rocm` image currently cached on
+  this repo's server actually postdates PR #27742's 2026-08-27 merge — not checked against the live
+  server in this pass, flagged as an action item in §11.1 rather than a confirmed fact.

docs/research/classifier-context-window-scaling.md

@@ -0,0 +1,249 @@
+# Does the qwen-classifier need to match the main model's context window, and would upgrading it to Qwen3-8B or Qwen3-Coder-30B-A3B-Instruct fit on the current R9700?
+
+**Date:** 2026-09-15
+**Question raised:** should `qwen-classifier` be resized to `LLAMA_CTX_SIZE / LLAMA_PARALLEL` (131072, matching the
+main model's per-slot context), should the classifier model itself move up to Qwen3-8B or
+Qwen3-Coder-30B-A3B-Instruct, and does that need a second GPU?
+
+**Answer: No, no, and not for this reason.** qwen-code's own docs state no context-size requirement for
+`fastModel` at all — the "match the main model" premise doesn't come from any primary source. Separately, and
+independently of context size: neither Qwen3-8B nor Qwen3-Coder-30B-A3B-Instruct fits in the ~6.1GiB of VRAM
+actually free on the card today, at *any* context length, once weights alone are counted — this is a raw-VRAM
+problem, not a context-window problem, exactly matching the "qwen8b needs to offload more to the RAM" intuition
+in the request. A second GPU would solve the VRAM problem (and incidentally remove this repo's own
+`GPU_MAX_HW_QUEUES=1` ROCm#5706 workaround from applying to this pair), but isn't deployed hardware today —
+it's a rack-build/acquisition question, not a config change.
+
+## 1. Does qwen-code require the fast/classifier model to match the main model's context window?
+
+No — checked against the user's own three linked pages, fetched directly:
+
+- **`fastModel` settings docs**: "Model used for generating prompt suggestions and speculative execution,"
+  configurable via `inherit` (main model), `fast`, a model ID, or `authType:model-id`; "Leave empty to use the
+  main model." The docs recommend "a smaller/faster model (e.g., `qwen3-coder-flash`) reduces latency and
+  cost" — **no context-window size or capacity requirement is stated anywhere on this page.**
+  Source: [qwen-code docs — Configuration / Settings, `#fastmodel`](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/settings/#fastmodel)
+
+- **Approval Mode / Auto Mode classifier docs**: describes what the classifier evaluates (shell commands,
+  network calls, out-of-workspace edits) and its allow/block behavior, but **does not name a specific model or
+  state any context-window requirement** — the only operational note is that "when the classifier API is
+  unreachable, the action is blocked rather than allowed."
+  Source: [qwen-code docs — Approval Mode, `#4-auto-mode---classifier-driven-approval`](https://qwenlm.github.io/qwen-code-docs/en/users/features/approval-mode/#4-auto-mode---classifier-driven-approval)
+
+- **Auto Mode "How it works"**: confirms the two-stage design (Stage 1: ~300ms, `{shouldBlock}` only; Stage 2:
+  chain-of-thought reconsideration, only on a Stage-1 block) and what data reaches the classifier — user text,
+  assistant tool-use calls, and tool-specific projections (truncated edit content, fetch URLs, shell command
+  text). **Tool results are explicitly never sent to the classifier.** It "uses your configured fast model
+  (`/model --fast`)," falling back to the main session model only if none is set. **No statement anywhere
+  requires or implies the fast model's context window match the main model's.**
+  Source: [qwen-code docs — Auto Mode, `#how-it-works`](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/#how-it-works)
+
+This confirms and sharpens what this repo's own `fast-model-choice.md` already found by reading qwen-code's
+source directly (`packages/core/src/permissions/classifier-transcript.ts`: `MAX_TRANSCRIPT_MESSAGES=40`,
+`MAX_HISTORICAL_ACTION_CHARS=4000`/message, worst case ~40-50K tokens, live-tested at 15,116 prompt tokens) —
+that doc already called the original `131072` in `settings.json` "copied from the main model's entry, not a
+real qwen-code requirement." The three docs pages fetched here add nothing that contradicts that: **there is no
+primary-source basis for `LLAMA_CTX_SIZE / LLAMA_PARALLEL` symmetry between the two models.** The current
+`65536` classifier ctx already carries ~1.5x margin over the real worst case.
+
+## 2. VRAM math for Qwen3-8B and Qwen3-Coder-30B-A3B-Instruct as classifier candidates
+
+Same method this repo already uses (`qwen3.8-27b-quant.md`, `fast-model-choice.md` §5): per-token KV cache =
+`layers × 2(K+V) × kv_heads × head_dim × bytes`, read directly from each model's own `config.json`.
+
+### Qwen3-8B
+
+- Architecture (`Qwen/Qwen3-8B` `config.json`): `num_hidden_layers: 36`, `num_key_value_heads: 8`,
+  `num_attention_heads: 32`, `head_dim: 128`, `hidden_size: 4096`, `max_position_embeddings: 40960`,
+  `rope_scaling: null`.
+  Source: [Qwen/Qwen3-8B `config.json`](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)
+- **Native context is 32,768 tokens**, not the 262,144 the user's brief assumed (that number belongs to
+  Qwen3-4B-Instruct-2507, a different, non-reasoning 2507-refresh model — Qwen3-8B is the earlier,
+  thinking-capable Qwen3 architecture with a materially smaller native window). Extending past 32K needs YaRN:
+  > "Qwen3 natively supports context lengths of up to 32,768 tokens. For conversations where the total length
+  > (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling
+  > techniques to handle long texts effectively."
+  and llama.cpp-specific YaRN invocation is given explicitly:
+  `./llama-cli ... -c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768`, with a documented
+  caveat that "all the notable open-source frameworks implement **static** YaRN, which means the scaling
+  factor remains constant regardless of input length, potentially impacting performance on shorter texts."
+  Source: [Qwen/Qwen3-8B-GGUF — Processing Long Texts](https://huggingface.co/Qwen/Qwen3-8B-GGUF#processing-long-texts)
+- **Weights** (official Qwen quants, fetched from the GGUF repo file list): Q5_K_M = 5.85 GB, Q8_0 = 8.71 GB.
+  Source: [Qwen/Qwen3-8B-GGUF](https://huggingface.co/Qwen/Qwen3-8B-GGUF)
+  *(Not independently verified: unsloth's equivalent `UD-Q4_K_XL` quant, which is what this repo's
+  `docker-compose.yml`/`.env.example` actually download for every model deployed so far — the unsloth file
+  size wasn't fetched, only the official Qwen quants above. Treat Q5_K_M/Q8_0 as a reasonable bound, not the
+  exact file this repo would pull.)*
+- Per-token KV cache: `36 × 2 × 8 × 128 × 2 bytes = 144 KiB/token` fp16 — identical to Qwen3-4B-Instruct-2507's
+  figure in `fast-model-choice.md` §5, since both share the same `layers/kv_heads/head_dim` triple.
+
+| Context | KV (fp16) | KV (q8_0) | KV (q4_0, current classifier setting) |
+|---|---|---|---|
+| 65,536 (current classifier ctx) | 9.0 GiB | 4.5 GiB | **2.25 GiB** |
+| 131,072 (user's proposed "match main model") | 18.0 GiB | 9.0 GiB | **4.5 GiB** |
+
+Weights + KV (q4_0, smallest realistic combo):
+
+| Context | Q5_K_M weights + q4_0 KV | Q8_0 weights + q4_0 KV |
+|---|---|---|
+| 65,536 | 5.85 + 2.25 = **8.1 GB** | 8.71 + 2.25 = **10.96 GB** |
+| 131,072 | 5.85 + 4.5 = **10.35 GB** | 8.71 + 4.5 = **13.21 GB** |
+
+### Qwen3-Coder-30B-A3B-Instruct
+
+- Architecture (fetched from the shared Qwen3-30B-A3B-family `config.json`): `num_hidden_layers: 48`,
+  `num_key_value_heads: 4`, `num_attention_heads: 32`, `head_dim: 128`, `hidden_size: 2048`, **MoE**:
+  `num_experts: 128`, `num_experts_per_tok: 8` (8 of 128 experts active per token — confirms this is a sparse
+  MoE model, not a dense one like the 27B or 8B candidates; the "active params" figure describes *compute*
+  per token, not memory footprint — **all 128 experts' weights still have to be resident** wherever the model
+  is loaded, GPU or RAM).
+  Source: [Qwen/Qwen3-30B-A3B-family `config.json`](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)
+- **UD-Q4_K_XL file size (the exact quant/quantizer this repo already standardizes on): 17.7 GB.** 30.5B total /
+  3.3B activated parameters. Native context "262,144 natively... can be extended further using Yarn to reach
+  1M tokens."
+  Source: [unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF?show_file_info=Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf)
+- Per-token KV cache: `48 × 2 × 4 × 128 × 2 bytes = 96 KiB/token` fp16 (smaller per-token than the 8B/4B
+  candidates, since `num_key_value_heads` is 4 here vs. 8 — but this saving is irrelevant given the weights
+  size below).
+
+| Context | KV (fp16) | KV (q8_0) | KV (q4_0) |
+|---|---|---|---|
+| 65,536 | 6.0 GiB | 3.0 GiB | **1.5 GiB** |
+| 131,072 | 12.0 GiB | 6.0 GiB | **3.0 GiB** |
+
+Weights + KV (q4_0):
+
+| Context | Total |
+|---|---|
+| 65,536 | 17.7 + 1.5 = **19.2 GB** |
+| 131,072 | 17.7 + 3.0 = **20.7 GB** |
+
+## 3. Does either candidate fit the ~6.1 GiB actually free on the card today?
+
+**No — neither does, at either context size, even at the smallest quant/KV-quant combination tested.**
+
+- Qwen3-8B's cheapest realistic combination (Q5_K_M weights + q4_0 KV at the *current* 65536 ctx, not even
+  the proposed 131072) is **8.1 GB — already ~2 GB over the measured 6.1 GiB free budget**, before accounting
+  for compute-buffer/batch overhead that `fast-model-choice.md` §"Implementation note" already found could add
+  meaningfully on top of the naive weights+KV estimate (that's exactly why the 4B classifier ended up needing
+  `--flash-attn on` and partial 28/36-layer offload instead of the originally-predicted comfortable full-GPU
+  fit).
+- Qwen3-Coder-30B-A3B-Instruct isn't close at any setting tested — its weights alone (17.7 GB) are triple the
+  entire free budget, and this doesn't change with context size since the weights term dominates.
+- This is a **VRAM-capacity problem, not a context-window problem** — directly confirming the "qwen8b needs to
+  offload more to the RAM" intuition in the original request. Reducing context doesn't fix it; the weights
+  don't fit regardless.
+
+**CPU/RAM offload mechanics:** llama.cpp's `--n-gpu-layers` is documented as "max. number of layers to store in
+VRAM, either an exact number, `'auto'`, or `'all'`" — the layers not selected are computed on CPU, with the
+model loaded via mmap by default (same mechanism this repo's own `.env.example` already documents for
+`LLAMA_GPU_LAYERS`: "if GPU+RAM ever can't hold the working set, the OS pages the rest in from disk
+automatically"). For the MoE Coder-30B-A3B model specifically, this repo's own `.env.example` already flags the
+more targeted alternative — `--n-cpu-moe`/`--cpu-moe`/`--override-tensor "exps"` — as the flags that "target
+Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B)," i.e. exactly this model's architecture: these offload only
+expert-tensor weights to CPU while keeping attention/shared layers and KV cache on GPU, which is the
+mechanically correct lever for an MoE model, unlike the blunt `--n-gpu-layers` used for the dense 8B/27B/4B
+models. **Neither llama.cpp's own README nor the Qwen model cards fetched here document a quantified
+performance cost for partial offload** — no primary source gives a "N layers offloaded = X% slower" figure.
+Source: [llama.cpp `tools/server/README.md`](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md)
+
+What *is* directly measured, in this repo's own deployment history: CPU-only was tried first for the current,
+much smaller 4B classifier and rejected — "too slow in practice: real classification calls blew past
+OmniRoute's 60s timeout and retry-looped (504→499→504)" (`docker-compose.yml`'s `qwen-classifier` comment
+block). An 8B dense model has roughly double the compute of the 4B model per token; a 30B-A3B model's *routing*
+overhead on CPU (choosing 8 of 128 experts per token, each a separate weight lookup) adds a different kind of
+cost that neither this repo nor the sources fetched here have measured. **Given the classifier's Stage 1 has an
+explicit ~300ms latency budget** (`fast-model-choice.md` §1, from qwen-code's own docs), and this repo already
+has one concrete data point that CPU offload breaks that budget at a smaller model size, extending either
+candidate onto significant CPU offload carries real, unquantified latency risk — the same failure mode already
+observed once, at a favorable (smaller) model size.
+
+## 4. Does a second GPU solve this, and is one actually available?
+
+**Not today.** `docs/server-planing.md` is a rack-build plan for "3-4x AMD Radeon AI PRO R9700 (32GB) GPUs" —
+a future-state document, not present inventory. Every GPU-facing comment in this repo's own
+`docker-compose.yml`/`.env.example`/`rocm-gpu-pin-and-render-group.md` consistently refers to "the single 32GB
+R9700" and measures the "~6.1GiB free" budget against one physical card holding both `llama-server` and
+`qwen-classifier`. Adding a second GPU is a hardware-acquisition and rack-build question — physically sourcing,
+installing, and power/PCIe-provisioning a card per `server-planing.md`'s own build plan — not a
+`docker-compose.yml`/`.env.example` change.
+
+**If a second GPU were added**, it would directly remove one already-documented risk for this specific pair:
+this repo's own `rocm-gpu-pin-and-render-group.md` traced the GPU-pinned-at-100%/ROCm#5706 bug to its precise
+trigger condition —
+
+> "The pin only appears with two concurrent HIP-context-holding processes **on the same GPU**... Root cause: an
+> AMD MES (Micro Engine Scheduler) firmware bug triggered by HIP hardware-queue creation."
+Source: [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706), via this repo's own
+[`rocm-gpu-pin-and-render-group.md`](rocm-gpu-pin-and-render-group.md)
+
+Since the confirmed trigger is *two HIP contexts sharing one physical card*, moving the classifier to its own,
+second GPU would put each service on a single-HIP-context card — the condition that trips the bug wouldn't
+exist for this pair anymore, and the `GPU_MAX_HW_QUEUES=1` workaround currently applied to both services
+specifically because they share one card would no longer be load-bearing for *this* pair (it would still apply
+if any future third service shared a card with either model). This wasn't independently re-verified across two
+*separate* physical cards by any source fetched in this pass — it's a direct extrapolation from the confirmed
+root cause, same category of caveat that doc's own author already flagged for its within-one-card claim.
+
+## Bottom line / recommendation
+
+1. **Don't apply `LLAMA_CTX_SIZE / LLAMA_PARALLEL` symmetry to the classifier.** No qwen-code primary source
+   states or implies the fast/classifier model needs to match the main model's context window. The real
+   requirement (§1, already established in `fast-model-choice.md`) is ~40-50K tokens worst case; the current
+   `65536` already has margin. Doubling to 131072 would only double VRAM spent on KV cache for a model that
+   won't otherwise fit anyway (§2-3).
+2. **Don't upgrade the classifier to Qwen3-8B or Qwen3-Coder-30B-A3B-Instruct on the current single-GPU setup.**
+   Neither fits the ~6.1GiB actually free, at any context size — this is a weights-size problem, not a
+   context-window problem. Forcing it would mean either (a) shrinking the main 27B model's own VRAM footprint
+   to make room (a real trade-off against the primary model, not evaluated here), or (b) CPU/partial offload,
+   which this repo has direct, measured evidence already breaks the classifier's latency budget at a *smaller*
+   model size than either candidate.
+3. **A second GPU is the clean fix for VRAM contention and would also retire the ROCm#5706 workaround's
+   relevance for this pair — but it isn't deployed hardware today.** `server-planing.md` is a future build
+   plan; this is an acquisition/rack-build decision, not something achievable via a config change right now.
+4. If the actual underlying motivation is classifier *quality* (not context capacity), that's a separate,
+   legitimate question this doc doesn't answer — worth its own research pass rather than solving it via a
+   bigger model that doesn't fit the hardware.
+
+## Sources
+
+- [qwen-code docs — Configuration / Settings, `#fastmodel`](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/settings/#fastmodel)
+- [qwen-code docs — Approval Mode, `#4-auto-mode---classifier-driven-approval`](https://qwenlm.github.io/qwen-code-docs/en/users/features/approval-mode/#4-auto-mode---classifier-driven-approval)
+- [qwen-code docs — Auto Mode, `#how-it-works`](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/#how-it-works)
+- [Qwen/Qwen3-8B `config.json`](https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json)
+- [Qwen/Qwen3-8B-GGUF](https://huggingface.co/Qwen/Qwen3-8B-GGUF)
+- [Qwen/Qwen3-8B-GGUF — Processing Long Texts](https://huggingface.co/Qwen/Qwen3-8B-GGUF#processing-long-texts)
+- [Qwen/Qwen3-30B-A3B-family `config.json`](https://huggingface.co/Qwen/Qwen3-30B-A3B/raw/main/config.json)
+- [unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF?show_file_info=Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf)
+- [llama.cpp `tools/server/README.md`](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/server/README.md)
+- [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706)
+- [docs/research/fast-model-choice.md](fast-model-choice.md) (this repo — classifier transcript sizing,
+  qwen-classifier's real deployment history)
+- [docs/research/qwen3.8-27b-quant.md](qwen3.8-27b-quant.md) (this repo — KV-cache-from-config.json method
+  reused here)
+- [docs/research/rocm-gpu-pin-and-render-group.md](rocm-gpu-pin-and-render-group.md) (this repo — ROCm#5706
+  trigger condition and `GPU_MAX_HW_QUEUES` scoping)
+- [docs/server-planing.md](../server-planing.md) (this repo — confirms only 1 of a planned 4 GPUs is deployed)
+- `docker-compose.yml`, `.env.example` (this repo — current `qwen-classifier`/`llama-server` config and the
+  measured "~6.1GiB free" VRAM figure)
+
+## Confidence / uncertainty summary
+
+- **High confidence:** qwen-code's `fastModel`/Auto-Mode docs state no context-window requirement (direct
+  quotes from all three linked pages); Qwen3-8B's native 32,768 context and YaRN caveat (direct model-card
+  quote); Qwen3-Coder-30B-A3B-Instruct's MoE architecture and 17.7GB Q4_K_XL file size (direct from the
+  quantizer's own repo page); the KV-cache-per-token math for both candidates (computed directly from each
+  model's own `config.json`, same method already validated in this repo's prior research); the ROCm#5706
+  trigger condition being scoped to two HIP contexts on the *same* GPU (direct quote from this repo's own
+  prior research, itself sourced from the upstream issue).
+- **Medium confidence:** the exact unsloth `UD-Q4_K_XL`-equivalent file size for Qwen3-8B — only the official
+  Qwen quants (Q5_K_M/Q8_0) were fetched, not unsloth's own repo, so the real number this repo would actually
+  download wasn't directly verified (bounded reasonably by the Q5_K_M figure, which is already the smallest
+  realistic option and still doesn't fit). The claim that CPU/partial-offload latency risk scales unfavorably
+  for larger/MoE models is a reasoned extrapolation from this repo's one measured data point (4B CPU-only
+  rejected) plus general MoE-routing-overhead reasoning, not a directly measured benchmark for either candidate.
+- **Low confidence / not independently verified:** whether `GPU_MAX_HW_QUEUES=1`/ROCm#5706 genuinely has zero
+  relevance across two *separate* physical GPUs (extrapolated from the confirmed same-GPU trigger condition,
+  same caveat this repo's own prior research already flagged for its own claim); no primary source found that
+  quantifies llama.cpp's actual inference-speed penalty for partial `--n-gpu-layers` or `--n-cpu-moe` offload
+  in general — this is a documented gap in the sources checked, not a guessed number.

docs/research/colibri-ai-stack-evaluation.md

@@ -0,0 +1,222 @@
+# Evaluating Colibrì (JustVugg/colibri) for this stack
+
+**Date:** 2026-09-08
+**Scope:** The user flagged https://github.com/JustVugg/colibri as something that "could revolutionize"
+this self-hosted AI stack. What is Colibrì actually, is it compatible with this stack's AMD
+ROCm/HIP-only single-GPU setup, and — even if compatible — does it fill a real gap versus what
+llama.cpp, OmniRoute, Qdrant, Neo4j, and ComfyUI already do here?
+
+## 1. What Colibrì actually is
+
+Colibrì is a pure-C, zero-runtime-dependency inference engine whose specific trick is treating
+"storage, RAM, and VRAM as a single inference hierarchy" so that huge mixture-of-experts (MoE)
+models — far bigger than any one machine's VRAM+RAM — can still run, by keeping the small dense
+layers resident and streaming the (much larger) set of routed experts from disk on demand with an
+LRU/"hot-store" cache and router-lookahead prefetching:
+
+> "Colibrì is an open-source inference engine designed to run frontier mixture-of-experts (MoE)
+> models on consumer hardware... The fundamental approach uses 'a JIT, but for weights' — parameters
+> are staged across storage tiers (VRAM/RAM/NVMe) based on measured routing patterns rather than kept
+> resident."
+— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
+
+It ships single-C-file implementations for eight specific model families — GLM-5.2/5.3,
+GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 (35B-A3B), and OLMoE
+— each requiring model weights pre-converted into Colibrì's own container format (`coli convert`),
+not arbitrary GGUF files:
+
+> "Eight model families with single C file implementations... GLM-5.2/5.3 | 744B | 372GB | 16GB+ ...
+> Kimi K3 | 2.8T | 1.6TB | 32GB+"
+— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
+
+It's meant to be run either from prebuilt binaries/releases, built from source (`./setup.sh` under
+`c/`), or via Docker (`docker/Dockerfile`, `docker/Dockerfile.slim`, `docker/docker-compose.yml` exist
+in-repo — confirmed present via the GitHub contents API, https://api.github.com/repos/JustVugg/colibri/contents/docker),
+exposing an OpenAI- and Anthropic-compatible HTTP API (`coli serve`, default `http://127.0.0.1:8000/v1`,
+plus `/v1/messages`) — the same shape OmniRoute already expects from a provider, per third-party
+summaries of `docs/api.md` and `docs/serve_protocol.md`
+([search result summary, secondary](https://github.com/JustVugg/colibri/blob/main/docs/api.md)).
+
+It launched July 10, 2026 and went viral on Hacker News the same day (453 points) on the strength of
+running the 744B-parameter GLM-5.2 model on a 25GB-RAM consumer box:
+
+> "A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744
+> billion parameters, on a regular PC with 25GB of memory."
+— https://gigazine.net/gsc_news/en/20260710-colibri-glm/ (secondary coverage)
+
+## 2. Hardware/runtime requirements — AMD ROCm or NVIDIA-only?
+
+**This is the load-bearing question given this stack runs llama.cpp on ROCm/HIP, not CUDA, on a
+single AMD Radeon AI PRO R9700.** The answer is more nuanced than a flat yes/no — verified against
+source, not just README prose:
+
+- **The engine is CPU-first; a GPU is optional at all.** `docs/quickstart.md` states plainly: "You do
+  **not** need a GPU. A GPU only helps if you have one; the engine runs CPU-only by default."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/quickstart.md
+- **AMD/ROCm support is real, shipped, and reasonably recent — not just a README claim.** It was
+  requested in [issue #69](https://github.com/JustVugg/colibri/issues/69) (opened 2026-07-11, "No
+  ROCM support"), the maintainer confirmed it was mechanically straightforward since HIP closely
+  mirrors CUDA, an initial PR (#112) added it but was **closed unmerged**, and a follow-up PR — tracked
+  as [#339](https://github.com/JustVugg/colibri/issues/69) — landed the actual mechanism that shipped:
+  a single shared CUDA kernel source (`backend_cuda.cu`) compiled either by `nvcc` or by `hipcc`
+  against a compatibility header:
+
+  > "backend_gpu_compat.h — 'one GPU backend source, two vendors.' ... maps CUDA runtime calls to HIP
+  > equivalents when compiled by hipcc with `HIP=1`... handles architecture-specific guards for rocWMMA
+  > availability and matrix core support across different GPU architectures (gfx906, gfx908, gfx11xx,
+  > etc.)."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/c/backend_gpu_compat.h (confirmed present
+  in the current `main` branch — this is not a stale/unmerged branch)
+
+  This shipped in a **tagged release**, not just an open PR — `CHANGELOG.md` lists "AMD GPU support" as
+  part of v1.1.0 (2026-07-22): "AMD GPU support, dual-SSD streaming, fmt=5/fmt=6 quantization formats."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md
+- **Notably, the community contributor who tested it used an RX 9070 XT / gfx1201** — the same RDNA4
+  architecture generation as this stack's Radeon AI PRO R9700 — per the PR #339 description: "validated
+  across CPU builds, HIP testing on gfx1201, and NVIDIA compatibility verification." That's a genuinely
+  favorable, non-generic signal for this specific card's GPU family, better than "AMD support exists
+  somewhere."
+- **But ROCm support is thinner and less documented than the CUDA/Metal paths.** `docs/` has `cuda.md`,
+  `metal.md`, `metal_implementation.md`, and `vulkan.md`, but **no `rocm.md` or `hip.md`** (confirmed via
+  the GitHub contents API listing of `docs/`, https://api.github.com/repos/JustVugg/colibri/contents/docs).
+  Model-specific tuning docs are written CUDA-only with no AMD mention at all — e.g.
+  `docs/qwen36-cuda-tier.md` references `COLI_CUDA=1`, `backend_cuda.cu`, and lists test hardware as
+  "RTX 3070 8 GB + Quadro RTX 4000" (both NVIDIA), with zero AMD/ROCm/HIP text anywhere in that
+  document. — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/qwen36-cuda-tier.md
+  So: the *general* GPU-acceleration mechanism supports ROCm/HIP and has been community-validated on
+  hardware close to the R9700, but the *per-model* tiering/tuning documentation and (presumably) most
+  of the maintainer's own benchmarking is CUDA-first. Treat AMD support as functional-but-secondary,
+  not a first-class, symmetrically-tested backend.
+- **A separate GPU-agnostic path also exists**: a Vulkan backend (`backend_vulkan.c`, confirmed present
+  in the `c/` directory listing) that the project positions as covering "AMD via Mesa/RADV" as a
+  vendor-neutral fallback, independent of the HIP path above — this would also be viable on the R9700
+  in principle, though vendor-neutral compute back ends are typically slower than a vendor SDK path
+  (HIP/ROCm) and no R9700/gfx1201-specific Vulkan numbers were found in the docs reviewed.
+
+**Bottom line on hardware fit: not a blocker.** Unlike a hard CUDA-only dependency, Colibrì's AMD/HIP
+path is real, shipped in a release, and specifically exercised on the same RDNA4 family as this
+stack's GPU — this is not a disqualifying finding the way it would be for a CUDA-only tool.
+
+## 3. License
+
+Apache License 2.0, confirmed by fetching `LICENSE` directly from the repo — a standard permissive
+license, fine for self-hosted use here (commercial or non-commercial, modification, redistribution all
+permitted, patent grant included, "AS IS" with no warranty).
+— https://raw.githubusercontent.com/JustVugg/colibri/main/LICENSE
+
+Model weights are licensed separately from the engine — e.g. the README notes "GLM-5.2 weights
+released by Z.ai under MIT"
+(https://raw.githubusercontent.com/JustVugg/colibri/main/README.md) — so each model's own license
+would need checking before use, same as with any GGUF today.
+
+## 4. Maturity signals
+
+Pulled from the GitHub API (https://api.github.com/repos/JustVugg/colibri) and the changelog:
+
+| Signal | Value |
+|---|---|
+| Repo created | 2026-07-01 |
+| First tagged release (v1.0.0) | 2026-07-19 |
+| Current version (as of today) | 1.10.2 (2026-09-06) |
+| Age at time of writing | ~10 weeks |
+| Stars / Forks | 27,047 / 2,963 |
+| Open issues | 104 |
+| Top contributor | JustVugg — 1,077 commits |
+| #2 contributor | ZacharyZcR — 163 commits |
+| Total contributors | 100+ (long tail, most in single digits) |
+| License | Apache 2.0 |
+| Archived? | No |
+
+Read honestly, this is **a viral, very-early-stage, single-maintainer-dominated project**, not a
+mature or slow-burn one:
+
+- It is ~10 weeks old today. The star count (27k) is wildly disproportionate to that age and reflects
+  a Hacker News front-page moment (453 points the day it launched,
+  https://gigazine.net/gsc_news/en/20260710-colibri-glm/), not organic multi-year adoption. Stars are
+  a popularity signal, not a quality proof, and here the ratio (huge stars, ~10 weeks old, one dominant
+  committer) is itself a maturity red flag worth naming rather than a mark in its favor.
+- Commit/release activity is genuinely fast — 14 tagged releases in under 8 weeks (v1.0.0 on 2026-07-19
+  through v1.10.2 on 2026-09-06, https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md)
+  — so it is actively maintained day-to-day, not abandoned. But that pace also means breaking changes
+  and security patches are frequent: v1.6.2 (2026-08-14) was itself "a security release: six
+  privately-reported memory-safety issues fixed... from untrusted input," and v1.10.2 (2026-09-06)
+  again touched "security fixes for image API" — both signs of a codebase still finding its footing on
+  hardening, not evidence of instability being the norm, but worth weighing given this stack would be
+  exposing any such server on an internal network via OmniRoute.
+- Contribution concentration is heavy: the maintainer (JustVugg) has ~6.6x the commits of the next
+  contributor, and the rest of the 100+ contributor list trails off into single-digit-commit
+  drive-by PRs (per the GitHub contributors API,
+  https://api.github.com/repos/JustVugg/colibri/contributors) — a classic "one person's viral project
+  plus a wave of small first-time PRs" shape, not an established multi-maintainer team.
+- The AMD/ROCm feature specifically has a short, thin history: requested 2026-07-11, shipped 2026-07-22
+  (11 days later, in v1.1.0), documented only implicitly (no dedicated `docs/hip.md`/`rocm.md`, unlike
+  every other backend) — i.e., it is the newest and least-independently-verified of the project's four
+  GPU backends (CUDA, Metal, Vulkan, HIP).
+
+## 5. What capability gap it would actually fill in this stack
+
+Concretely comparing against what's already running (`docker-compose.yml`):
+
+- **llama.cpp (ROCm) already fully GPU-resides the current model** — `--n-gpu-layers 999` on the
+  llama-server service means the whole Qwen3.8-27B-class GGUF sits in the R9700's VRAM and runs at
+  normal, fast, interactive token rates. Colibrì's entire value proposition is the *opposite* case:
+  models **too large to fit in VRAM+RAM at all**, accepted at the cost of streaming most of the model
+  from disk on every forward pass. For a model that already fits on this GPU (which is the whole point
+  of the current setup), Colibrì offers no benefit — llama.cpp is already doing the fast thing.
+- **The actual gap it could fill is running models this stack categorically cannot run today** — e.g.
+  GLM-5.2 (744B), DeepSeek V4 Flash (284B), or Kimi K3 (2.8T), none of which would ever fit on a single
+  R9700 regardless of quantization. Colibrì's own benchmarks make the cost of that explicit and, unlike
+  its capability claims, these are the project's self-reported numbers, not independently reproduced —
+  flagged as such:
+
+  > "6× RTX 5090 (full residency): 5.8-6.8 tokens/second decode... 128GB CPU-only desktop: ~1.8
+  > tokens/second (warm)... Single RTX 5070 Ti: 1.07 tokens/second... 25GB dev box: 0.05-0.1
+  > tokens/second (baseline)."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md (self-reported, unverified by
+  any third party found during this research)
+
+  At 0.05–2 tokens/second on any hardware remotely resembling a single-GPU workstation, this is not
+  usable for the interactive coding-CLI workloads this stack is built around (Claude Code CLI, Kimi
+  CLI, etc. routed through OmniRoute per the README) — those need low per-token latency for tool-calling
+  round trips, not throughput measured in seconds per token. It would only be plausible as an
+  occasional, patient, offline/batch capability (e.g. "let a huge model chew on a large doc overnight")
+  layered in *alongside*, not instead of, the current llama.cpp path.
+- **Disk footprint is a real new cost, not a marginal one**: 167GB–1.6TB per model
+  (https://raw.githubusercontent.com/JustVugg/colibri/main/README.md), which would need to be
+  provisioned in addition to the existing `models` Docker volume, GGUF downloads, Qdrant/Neo4j
+  volumes, and ComfyUI's model files already on this box.
+- **No overlap or replacement value for OmniRoute, Qdrant, Neo4j, or ComfyUI** — Colibrì is strictly an
+  inference-engine alternative to llama.cpp for a narrow, specific list of very large MoE models; it
+  does nothing related to gateway/key-management (OmniRoute's job), vector/graph storage (Qdrant/Neo4j's
+  job), or image generation (ComfyUI's job). Its own `coli serve` OpenAI/Anthropic-compatible endpoint
+  could in principle be registered as another OmniRoute provider the same way llama-server is today —
+  that part is mechanically plausible — but it would be adding a second, much slower inference backend
+  next to the existing fast one, not replacing or upgrading anything currently in the stack.
+
+## 6. Bottom line
+
+**Not a fit for this stack right now, and the "revolutionize" framing does not hold up** — but for a
+more specific reason than "wrong GPU vendor":
+
+- **ROCm/AMD support is real and not the blocker one might expect.** It shipped in a tagged release
+  (v1.1.0, 2026-07-22), lives in a compatibility header confirmed present on `main`
+  (`c/backend_gpu_compat.h`), and was community-tested on an RX 9070 XT / gfx1201 — the same RDNA4
+  family as this stack's R9700. This is genuinely worth noting as a positive, since it's the kind of
+  thing that usually *is* disqualifying for AMD-only stacks and here it isn't.
+- **The disqualifying issue is fit, not hardware**: Colibrì solves "run a model way too big for your
+  VRAM+RAM by streaming most of it from disk," at 0.05–2 tokens/second. This stack's actual situation is
+  the opposite — a model sized to fully fit and run fast on a single GPU via llama.cpp. Colibrì would add
+  no speed, capability, or reliability benefit to the model already running here, and its own numbers
+  show it isn't fast enough to serve the interactive coding-CLI use case this stack exists for.
+- It could only ever be interesting as a *bolt-on, offline-only* capability for occasionally running an
+  otherwise-impossible frontier-scale model (700B–2.8T params) for patient, non-interactive tasks — at
+  the cost of hundreds of GB to ~1.6TB of extra disk per model, on a ~10-week-old, single-maintainer,
+  still-hardening project (two security-patch releases already) with no ROCm-specific documentation and
+  the thinnest testing history of its four GPU backends.
+- **Recommendation: worth a passing watch, not worth integrating.** Revisit if/when: (a) the project
+  reaches a more established maturity point (6–12 months, broader contributor base, dedicated ROCm docs
+  bringing it to parity with the CUDA/Metal paths), and (b) there's an actual concrete need in this stack
+  to run a model in the 200B+ range that cannot fit on the R9700 — which isn't the case today (the
+  current model is deliberately sized to fit fully in VRAM). Until then, integrating it would add
+  operational surface (a new container, huge disk provisioning, a newer/less-hardened codebase) for no
+  measurable improvement over the existing llama.cpp/ROCm path.

docs/research/dashscope-websearch-selfhost-options.md

@@ -0,0 +1,412 @@
+# Research: self-hosted alternatives to DashScope for Qwen Code's built-in `web_search` tool
+
+**Question:** Qwen Code CLI's built-in `web_search` tool requires `tools.webSearch.model`
+to resolve to a "DashScope-compatible `modelProviders` entry." Is there any real,
+non-Alibaba-Cloud way to satisfy that requirement with something self-hosted —
+or is the already-working OmniRoute MCP + SearXNG path (`docs/research/omniroute-qwen-websearch.md`)
+the end of the road?
+
+**Answer, short version:** No. The client-side code that decides whether a
+`baseUrl` is "DashScope-compatible" checks the **literal hostname** against a
+hardcoded allowlist of Alibaba-owned domains, before any request is sent — it
+is not a protocol-compatibility check that a look-alike server could pass. A
+self-hosted server cannot satisfy it, full stop, unless you fork qwen-code and
+delete that check. Once you've done that, the actual wire protocol
+(OpenAI SDK `responses.create()`, SSE, specific item types) is buildable
+(a few hundred lines), but nothing you can install off the shelf implements it
+today. The already-working OmniRoute MCP + SearXNG path costs nothing further
+and does not have this problem. **Recommendation: don't build this — see
+§6.**
+
+## 1. What "DashScope Responses API" is, precisely
+
+Alibaba Cloud Model Studio (Bailian/DashScope) added an **OpenAI-compatible
+Responses API**, layered on top of its existing Chat Completions
+compatible-mode surface:
+
+- Endpoint (per Alibaba's own docs): `POST {baseUrl}/responses`, where
+  `baseUrl` is the region's compatible-mode base, e.g.
+  `https://dashscope.aliyuncs.com/compatible-mode/v1` (China/Beijing) or the
+  `-intl` / regional `*.maas.aliyuncs.com` variants.
+  Source: https://www.alibabacloud.com/help/en/model-studio/qwen-api-via-openai-responses
+  and https://www.alibabacloud.com/help/en/model-studio/compatibility-with-openai-responses-api
+- Request shape: standard Responses API (`model`, `input`, `stream`, `store`,
+  `instructions`) plus a `tools` array that can include
+  `{"type": "web_search"}`, `{"type": "web_extractor"}`, `{"type": "code_interpreter"}`
+  as **hosted, server-side tools** — the inference backend runs the search
+  itself and streams results back, the same hosted-tool pattern as OpenAI's
+  own Responses API `web_search_preview`, not a client-side function-calling
+  round trip.
+  Source: https://www.alibabacloud.com/help/en/model-studio/qwen-api-via-openai-responses
+- Response shape: an `output` array of typed items — a `web_search_call` item
+  carries `action: {type, query, sources: [{type: "url", url}]}`; narration
+  comes back as `message` items with `content: [{type, text}]`.
+  Source: same page.
+- Separately, DashScope's plain Chat Completions endpoint (not Responses)
+  has an older, unrelated `enable_search` boolean (passed via `extra_body`)
+  for models like Qwen3.8/Qwen3.6-Plus — the docs explicitly note this older
+  surface **does not return citations/sources**, which is exactly why
+  qwen-code's built-in tool uses the *Responses* API instead.
+  Source: https://docs.qwencloud.com/developer-guides/tool-calling/web-search
+
+This is the same hosted-tool pattern as OpenAI's Responses API
+`web_search_preview` (§4 confirms this directly from qwen-code's own client
+code — it literally reuses the OpenAI Node SDK's `responses.create()` call
+against a DashScope base URL).
+
+## 2. What qwen-code's own client code actually sends (ground truth)
+
+Fetched directly from `QwenLM/qwen-code`'s `main` branch,
+`packages/core/src/tools/web-search.ts` (1087 lines) and
+`packages/core/src/core/openaiContentGenerator/{constants,provider/dashscope}.ts`.
+This supersedes anything inferred from the docs pages — it's the literal
+implementation.
+
+**The request** (`web-search.ts` lines 651–699):
+
+```ts
+const client = new OpenAI({
+  apiKey,                    // from the resolved modelProviders entry's envKey
+  baseURL: backend.baseUrl,  // the modelProviders entry's baseUrl / WEB_SEARCH_BASE_URL
+  timeout: 60_000,
+  defaultHeaders: { 'User-Agent': `QwenCode/${version} (...)`, ...customHeaders },
+});
+
+const tools = [{ type: 'web_search' }];
+if (backend.webExtractor) tools.push({ type: 'web_extractor' });
+
+const requestParams = {
+  model: backend.modelId,
+  input: `Perform a web search for the query: ${query}`,
+  stream: true,
+  store: false,
+  instructions: SIDE_REQUEST_INSTRUCTIONS,   // a fixed system prompt, see source
+  tools,
+};
+
+const stream = await client.responses.create(requestParams, { signal });
+```
+
+This is the **official OpenAI Node SDK**, so `client.responses.create()`
+literally POSTs to `{baseURL}/responses` with that JSON body and reads back
+an SSE stream — there is no DashScope-specific SDK involved at all. Anything
+speaking real OpenAI Responses-API SSE syntax at that path, with these two
+extra tool types, is protocol-compatible.
+
+**What the client parses out of the SSE stream** (lines 359–509): event types
+`response.output_item.added`, `response.output_item.done`,
+`response.output_text.delta`, and terminal `response.completed` /
+`.failed` / `.incomplete` / `.cancelled`, each carrying a `response` object
+with `output: WsOutputItem[]` and `usage.x_tools.{web_search,web_extractor}.count`.
+Output items it understands: `web_search_call` (`action.query`/`action.queries`,
+`action.sources[].url`, `status`), `web_extractor_call` (`urls`, `goal`,
+`output`, `status`), and `message` (`content[].text`). It also defensively
+handles a DashScope-specific quirk: **request-level failures arrive as a bare
+SSE `event:error` with `{code, message, request_id}` and no `type`/`error`
+wrapper** — the OpenAI SDK doesn't recognize this shape, so qwen-code parses
+it itself (comment: "probe-verified"). Any replacement backend needs to emit
+exactly these item/event shapes, or qwen-code's parser silently ignores
+unrecognized item types and ultimately reports
+`WEB_SEARCH_NO_SEARCH_PERFORMED` (it treats zero `web_search_call` items as
+"no search happened," with one retry, before failing outright — see lines
+883–906).
+
+**The hard gate — this is the actual finding.** Before any request is sent,
+`evaluateWebSearchGate()` (lines 169–335) validates the resolved `baseUrl`
+through `classifyDashScopeBaseUrl()` (lines 122–157):
+
+```ts
+function classifyDashScopeBaseUrl(baseUrl: string): DashScopeBaseUrlIssue | null {
+  const url = new URL(baseUrl);
+  if (url.protocol !== 'https:') return 'insecure';
+  const hostname = url.hostname.toLowerCase();
+  const suffixes = [
+    ...DASHSCOPE_REGIONAL_HOSTS,   // dashscope.aliyuncs.com, dashscope-intl.aliyuncs.com, dashscope-us.aliyuncs.com
+    'maas.aliyuncs.com',
+    'alibaba-inc.com',
+    'aliyun-inc.com',
+  ];
+  return suffixes.some(s => hostname === s || hostname.endsWith('.' + s)) ? null : 'unknown-host';
+}
+```
+
+`DASHSCOPE_REGIONAL_HOSTS` is defined in
+`packages/core/src/core/openaiContentGenerator/provider/dashscope.ts` as
+exactly `['dashscope.aliyuncs.com', 'dashscope-intl.aliyuncs.com', 'dashscope-us.aliyuncs.com']`.
+
+**This means "DashScope-compatible" is not a protocol claim you can satisfy
+by implementing the right JSON shapes — it is a literal hostname allowlist
+checked client-side, before the request is even built.** A self-hosted server
+at `http://search.home`, `https://proxy-ai.home`, or any hostname you control
+will be rejected with *"WebSearch ... is not a DashScope-compatible
+endpoint"* regardless of what protocol it speaks, unless its hostname ends in
+one of `dashscope.aliyuncs.com`, `dashscope-intl.aliyuncs.com`,
+`dashscope-us.aliyuncs.com`, `*.maas.aliyuncs.com`, `*.alibaba-inc.com`, or
+`*.aliyun-inc.com` — domains Alibaba owns, that you cannot obtain a valid TLS
+certificate for. (There's also a separate, unrelated `DASHSCOPE_PROXY_BASE_URL`
+env var used by the *main* content generator's provider-detection code
+(`dashscope.ts` lines 244–262) for header/cache-control routing through a
+corporate proxy — it is not consulted by `classifyDashScopeBaseUrl()` at all,
+so it does not help here either.)
+
+The only way around this specific check is to **fork qwen-code and delete or
+relax `classifyDashScopeBaseUrl()`** — it's ~15 lines of open-source
+TypeScript, so this is not hard *code-wise*, but it means running a patched
+build of the CLI, not configuring the stock release.
+
+## 3. Any self-hostable server implementing this surface today? — No
+
+Checked the servers this task named:
+
+- **vLLM**: has a real `/v1/responses` implementation
+  (https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/responses/), and
+  for `gpt-oss` models specifically supports a **built-in `browser` tool**
+  with a pluggable, MCP-compliant external tool server in place of the
+  default Exa-backed reference implementation
+  (https://vllm.ai/blog/2025-08-05-gpt-oss;
+  https://github.com/vllm-project/recipes/blob/main/OpenAI/GPT-OSS.md). This
+  is the closest existing building block found — but it's gpt-oss/harmony
+  specific (not Qwen), and its tool/event shapes (`browser.search`,
+  `browser.open` harmony-channel messages) are **not** the same as DashScope's
+  `web_search_call`/`web_extractor_call` items qwen-code's parser expects, so
+  it is not drop-in — it would need a translation shim in front, at which
+  point you're building the shim anyway and don't need vLLM in the path.
+- **SGLang**: Responses API support is unclear/inconsistent per its own
+  issue tracker (https://github.com/sgl-project/sglang/issues/10038) — no
+  usable built-in web-search tool found.
+- **LiteLLM**: does expose `/v1/responses`, but has an **open bug**
+  rejecting the `web_search` tool type outright — "LiteLLM raises a
+  validation error... only `web_search_preview` is currently allowed"
+  (https://github.com/BerriAI/litellm/issues/14011). Its actual SearXNG
+  integration is the unrelated standalone `/v1/search` REST endpoint already
+  documented in `docs/research/litellm-searxng-search.md` (§1–3 there) — a
+  sibling API to chat/responses, not a Responses-API `tools:[{"type":"web_search"}]`
+  handler. It doesn't have a DashScope-mode either
+  (https://docs.litellm.ai/docs/providers/dashscope is a plain client wrapper
+  that calls the real dashscope.aliyuncs.com; nothing in it hosts a
+  DashScope-shaped server).
+- **LocalAI / Ollama**: no Responses API or DashScope-compatible mode found
+  in searches for either.
+- **A generic "OpenAI Responses API" self-hosted shim that could be relabeled**:
+  the closest match found, `teabranch/open-responses-server` (185 stars, 161
+  commits, wraps Ollama/vLLM as a Responses API with MCP support), **does not
+  implement `web_search` at all** — its own roadmap lists "Web search: crawl4ai"
+  as a *future* item, not shipped (verified live against the repo,
+  2026-09-05). No other candidate turned up in repeated GitHub searches for
+  "dashscope emulator/mock/fake server" or "responses api web_search
+  self-hosted."
+
+**Conclusion for §3: nothing installable off the shelf implements the
+DashScope Responses API's `web_search`/`web_extractor` hosted-tool surface.**
+Building it means writing your own small SSE server (see §5 sizing).
+
+## 4. Is DashScope's shape "OpenAI Responses API + web_search" reused wholesale?
+
+Yes, confirmed directly from source, not inference: qwen-code's client uses
+the **official `openai` npm package**'s `client.responses.create()` against a
+DashScope `baseURL` (§2 above) — it is not a DashScope-specific SDK or
+protocol. OpenAI's own Responses API supports a hosted `web_search_preview`
+tool with a similar `output[].type === "web_search_call"` item shape
+(OpenAI's public Responses API docs, referenced but not independently
+re-fetched here since qwen-code's source is authoritative for what it
+actually calls). DashScope's extension is the tool *name* (`web_search`
+rather than `web_search_preview` — the exact naming mismatch LiteLLM's own
+open bug in §3 stumbles on) plus the additional `web_extractor` tool and the
+`x_tools` usage-accounting field. No existing "OpenAI Responses API shim"
+project was found that already emulates `web_search_preview`/`web_search`
+server-side against a pluggable backend (see §3) — the two hosted-tool
+ecosystems (OpenAI's and DashScope's) both currently require literally
+calling out to the vendor's own cloud; nobody has open-sourced a
+self-hosted stand-in for either.
+
+## 5. LiteLLM specifically, re-examined against this exact requirement
+
+`docs/research/litellm-searxng-search.md` already established SearXNG is a
+first-class LiteLLM `search_provider` behind the **standalone** `/v1/search`
+REST endpoint (its own §1–2). That endpoint is irrelevant to qwen-code's
+`tools.webSearch.model` gate: qwen-code doesn't call an arbitrary search REST
+endpoint, it calls `POST {baseUrl}/responses` on an **OpenAI-SDK client**
+with `tools:[{type:"web_search"}]`, and gates `baseUrl` on the Alibaba
+hostname allowlist in §2. Even ignoring the hostname gate entirely (i.e.
+assuming a patched qwen-code build), LiteLLM's `/v1/responses` route
+currently **rejects** the `web_search` tool type per the open bug in §3 — so
+today, LiteLLM cannot terminate this request even as an internal component of
+a custom build. Nothing here changes the litellm-searxng-search.md
+recommendation; it remains correct and unrelated to this question.
+
+## 6. Effort assessment and recommendation
+
+**Option A — patch qwen-code + hand-roll a DashScope-Responses-shaped SSE
+server in front of SearXNG.** What it needs, concretely:
+1. Fork qwen-code, delete/relax `classifyDashScopeBaseUrl()` (§2) — trivial,
+   but means building and distributing a patched CLI, and re-patching on every
+   upstream update that touches this file or its surrounding gate logic.
+2. Write a small HTTP server exposing `POST /responses` that: accepts the
+   exact request shape in §2, calls SearXNG (`http://search.home`, already
+   reachable per `docs/research/litellm-searxng-search.md`'s `extra_hosts`
+   finding) for results, and streams back SSE events in the precise sequence
+   qwen-code's parser expects (`response.output_item.added` /
+   `.done` with a `web_search_call` item carrying `action.sources[].url`,
+   optionally a `message` item with narrated text, then
+   `response.completed`). No narration/LLM step is strictly required — an
+   empty or templated `message` still satisfies the parser as long as at
+   least one non-`failed` `web_search_call` item exists (§2's "no-search"
+   check only counts search-call items, not narration quality).
+   Realistically a few hundred lines (Node/Python + SSE), a day or so of
+   work plus debugging the exact event ordering, error-shape (`event:error`
+   quirk), and `store`/`instructions` fields the client sends but doesn't
+   strictly require echoing back.
+3. Register this server's URL as a `modelProviders` entry — except the
+   patched hostname check from step 1 is required for step 3 to pass at all,
+   so steps 1 and 2 are both mandatory, not alternatives.
+4. Maintain the fork indefinitely against upstream qwen-code releases.
+
+**Option B — do nothing further.** `docs/research/omniroute-qwen-websearch.md`
+already documents a **verified, working, fully self-hosted** path: OmniRoute's
+own `omniroute_web_search` MCP tool, backed by this stack's SearXNG instance,
+confirmed connected (`qwen mcp list` → Connected) and exercised end-to-end
+(`POST /v1/search` returned real results). This uses qwen-code's *documented,
+supported, unpatched* MCP-server extension point (`mcpServers` in
+`settings.json`) — no fork, no upstream-drift risk, no protocol shape to
+maintain.
+
+**Recommendation: do not build Option A.** The built-in `web_search` tool's
+"DashScope-compatible" requirement is, by design in qwen-code's own source, a
+hostname allowlist for Alibaba's cloud — it is not a compatibility surface
+meant to be reimplemented, and no one else has reimplemented it either (§3).
+Satisfying it self-hosted requires forking and permanently maintaining a
+patch to code whose only purpose is to *stop* you from doing that. The MCP
+path in `omniroute-qwen-websearch.md` already delivers the same end-user
+capability (web search, backed by this stack's own SearXNG, no external
+API) through qwen-code's actual supported extension point, with zero ongoing
+fork-maintenance burden. There is no functional gap Option A would close that
+Option B doesn't already close today.
+
+## Open questions / unknowns
+
+- Whether `DASHSCOPE_REGIONAL_HOSTS` or the extra suffixes
+  (`maas.aliyuncs.com`, `alibaba-inc.com`, `aliyun-inc.com`) ever change
+  across qwen-code releases — checked only against the current `main` branch
+  (fetched 2026-09-05); a future release could tighten or loosen this list.
+- Whether OpenAI's own `web_search_preview` Responses-API tool has a
+  publicly documented exact request/response JSON schema identical enough to
+  DashScope's `web_search`/`web_extractor` pair that a single shim could serve
+  both — not independently verified against OpenAI's own docs in this pass;
+  qwen-code's source (§2) is authoritative for the DashScope side only.
+- Whether `teabranch/open-responses-server`'s planned "Web search: crawl4ai"
+  roadmap item, if shipped, would end up emitting DashScope-shaped
+  `web_search_call` items or OpenAI-shaped `web_search_preview` ones — could
+  become relevant later but is speculative (unshipped) as of this research.
+
+## Sources
+
+- https://qwenlm.github.io/qwen-code-docs/en/developers/tools/web-search/ and
+  https://raw.githubusercontent.com/QwenLM/qwen-code/main/docs/developers/tools/web-search.md
+  — current built-in-tool vs. MCP options, settings keys, migration note.
+- `packages/core/src/tools/web-search.ts`,
+  `packages/core/src/core/openaiContentGenerator/constants.ts`,
+  `packages/core/src/core/openaiContentGenerator/provider/dashscope.ts` —
+  fetched directly from `QwenLM/qwen-code`'s `main` branch via
+  `raw.githubusercontent.com` on 2026-09-05; ground truth for the request
+  shape, SSE parsing, and the hostname gate (§2).
+- https://github.com/QwenLM/qwen-code/issues/3841 — prior (closed,
+  "not planned") community proposal for DashScope `enable_search` passthrough;
+  shows the feature that eventually shipped took a different path (Responses
+  API, not Chat Completions `enable_search`).
+- https://www.alibabacloud.com/help/en/model-studio/qwen-api-via-openai-responses
+  and https://www.alibabacloud.com/help/en/model-studio/compatibility-with-openai-responses-api
+  — Alibaba's own Responses API docs: endpoint, `tools` shape, `output` item
+  shape.
+- https://docs.qwencloud.com/developer-guides/tool-calling/web-search —
+  the older Chat-Completions `enable_search` mechanism and its
+  no-citations limitation.
+- https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/responses/,
+  https://vllm.ai/blog/2025-08-05-gpt-oss,
+  https://github.com/vllm-project/recipes/blob/main/OpenAI/GPT-OSS.md — vLLM's
+  Responses API and gpt-oss browser-tool/tool-server support.
+- https://github.com/sgl-project/sglang/issues/10038 — SGLang Responses API
+  support unclear.
+- https://github.com/BerriAI/litellm/issues/14011 — LiteLLM's `/v1/responses`
+  rejects the `web_search` tool type.
+- https://docs.litellm.ai/docs/providers/dashscope — LiteLLM's DashScope
+  provider is a plain client wrapper, no Responses API, no web_search.
+- https://github.com/teabranch/open-responses-server — closest
+  "self-hosted Responses API shim" found; web_search not implemented
+  (roadmap item only), checked live 2026-09-05.
+- `G:\_DEV\repos\LLM-Server\docs\research\omniroute-qwen-websearch.md` —
+  the already-working, verified self-hosted alternative this doc is weighed
+  against.
+- `G:\_DEV\repos\LLM-Server\docs\research\litellm-searxng-search.md` —
+  LiteLLM's actual (unrelated) SearXNG integration, re-confirmed as
+  orthogonal to this question in §5.
+
+## Tried it live (2026-09-05) — confirmed empirically, plus one new fact
+
+The user asked to actually run the experiment rather than stop at the analysis above.
+
+**What was done** (all local to the WSL install, reverted afterward — nothing in
+this repo or the live OmniRoute instance was left changed):
+- Patched the installed CLI file
+  `~/.local/lib/qwen-code/lib/chunks/web-search-K2FMOGS5.js` with a one-line
+  bypass in `classifyDashScopeBaseUrl()`: `if (baseUrl.includes("proxy-ai.home")) return null;`
+- Added a `tools.webSearch` block to `~/.qwen/settings.json` pointing
+  `model`/`baseUrl` at a new `qwen-experiment-websearch` `modelProviders` entry
+  using OmniRoute's existing `http://proxy-ai.home/v1` and the already-working
+  `OMNIROUTE_API_KEY`.
+- Ran `qwen` with a prompt forcing use of the built-in `web_search` tool.
+
+**Result — the client-side gate bypass worked**, confirming the research's
+read of `classifyDashScopeBaseUrl()` was accurate: qwen accepted the OmniRoute
+host as "DashScope-compatible" and attempted the tool call. It stopped at an
+interactive approval prompt first (expected — headless auto-approve wasn't
+attempted, since that flips on unrestricted auto-execution of every tool call
+at process privilege, not just this one).
+
+**New fact, not visible from static docs alone**: a direct `curl -X POST
+http://proxy-ai.home/v1/responses` (with a valid key, matching the request
+shape qwen would send) returned `{"error":{"message":"No active credentials
+for provider: codex.","type":"authentication_error","code":"invalid_api_key"}}`
+— **not** the generic "unknown route" error a nonexistent path returns (verified
+earlier in this same research thread against `/v1/search`-adjacent bogus
+paths). So `/v1/responses` **is a real, implemented OmniRoute route**, not
+merely undocumented — the earlier inference that it didn't exist was wrong;
+it exists but is hardcoded to proxy exclusively through a specific provider
+connection OmniRoute's catalog calls `codex`.
+
+**`codex` identified via `PROVIDER_REFERENCE.md`**: `id: codex`, alias `cx`,
+name "OpenAI Codex", **auth type: OAuth** — a real, personal
+ChatGPT/OpenAI-account connection, not a free/no-auth scraper provider like
+several others already connected in this instance (`felo-web`,
+`duckduckgo-web`, etc.). Checked `docs/reference/ENVIRONMENT.md` for any
+setting to redirect `/v1/responses` to a different provider — **none
+exists**; there is no `responsesProvider` or equivalent override.
+
+**Why routing isn't configurable, architecturally**: OpenAI's Responses API
+`web_search` is a *hosted* tool — the search executes inside the model
+backend's own infrastructure as part of generating the response, not as a
+client-visible round trip. Confirmed directly against llama.cpp's own
+`tools/server` docs (`github.com/ggml-org/llama.cpp/tree/master/tools/server`):
+it implements only `/v1/chat/completions` with client-side tool-calling
+(the model emits a `tool_call`; the *client* must execute it), has no
+`/v1/responses` endpoint, no hosted-tool execution, and its built-in
+`--tools` are local-only (`read_file`, `grep_search`, `exec_shell_command`,
+etc.) — none make outbound HTTP requests. So even with configurable routing,
+pointing `/v1/responses` at the local Qwen model wouldn't work: the upstream
+llama-server has nothing that could serve the hosted-tool half of the
+contract. Building that would mean OmniRoute (or a custom shim) intercepting
+the model's tool-call mid-generation and splicing in a real search — the
+same shim work priced out as not-worth-it earlier in this document, now
+confirmed to be the *only* way, not one option among several.
+
+**Conclusion holds, sharpened**: the dead end isn't just qwen-code's
+client-side hostname check anymore — even a fully self-hosted, hostname-gate-bypassed
+setup terminates at OmniRoute's `codex`-only `/v1/responses` routing, which
+itself terminates at needing a real OpenAI/ChatGPT OAuth account, which is
+exactly the kind of external paid dependency this whole line of inquiry was
+trying to avoid. `omniroute_web_search` via MCP (already working, already
+free, already self-hosted) remains the only path that actually satisfies the
+original goal.
+
+**Revert**: both the CLI patch and the `settings.json` changes were reverted
+after the test — `omniroute-search` MCP confirmed still `Connected` via
+`qwen mcp list` afterward. No lasting changes from this experiment.

docs/research/fast-model-choice.md

@@ -0,0 +1,254 @@
+# Which small model to run as the always-resident `fastModel` for qwen-code's Auto Mode classifier?
+
+**Date:** 2026-09-06
+**Budget:** ≤7GB VRAM, resident concurrently alongside the existing Qwen3.8-27B instance on the single
+32GB R9700, via the same `llama.cpp:server-rocm` image already in `docker-compose.yml`.
+**Answer: Qwen3-4B-Instruct-2507, Q8_0 GGUF (~4.3GB weights).** The prior quick pass's tentative pick
+holds up under primary-source verification, for a more specific reason than "same tokenizer family":
+it is the only strong candidate in the shortlist that is *architecturally* non-thinking (no `<think>`
+code path exists at all, vs. models that are thinking-by-default and rely on a per-call
+`enable_thinking:false` toggle that llama.cpp does not cleanly expose). It does carry one directly
+relevant, documented llama.cpp bug — but that bug is closed, has a one-flag workaround, and is
+strictly less severe than the still-open Qwen3.5/Qwen3.8-lineage bugs already documented against the
+27B model in this repo.
+
+## 1. What the classifier actually needs (grounding the requirement)
+
+Per qwen-code's own docs, Auto Mode's permission gate is a two-stage LLM classifier:
+
+- **Stage 1** — outputs only `{ shouldBlock: bool }`, ~300ms budget, thinking already disabled at the
+  request level. If `shouldBlock` is `false`, the action proceeds immediately.
+- **Stage 2** — only runs when Stage 1 blocks; uses chain-of-thought review to downgrade false
+  positives, ~3-5s budget.
+- Both stages use "your configured fast model (`/model --fast`)"; if none is configured, the full
+  session model is used instead — which is the current, too-slow state this second model is meant to
+  fix.
+
+Source: [Qwen Code docs — Auto Mode](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/),
+[QwenLM/qwen-code docs/users/features/auto-mode.md](https://github.com/QwenLM/qwen-code/blob/main/docs/users/features/auto-mode.md).
+
+A live qwen-code issue independently confirms the exact failure mode this repo already hit with the
+27B model — a model *thinking* inside the classifier path is a first-order latency problem, not a
+nice-to-have to tune later:
+
+> "for a latency-sensitive permission gate, thinking should be disabled in every stage" — enabling it
+> "makes the review path slower and more expensive, which directly worsens the timeout problem."
+
+That issue (timeouts tripping on slow inference) was closed by a PR that both loosened the stage
+timeout budgets *and* moved toward disabling thinking everywhere in the classifier.
+Source: [QwenLM/qwen-code issue #4676](https://github.com/QwenLM/qwen-code/issues/4676).
+
+Takeaway for model selection: the request-level "don't think" instruction already exists in
+qwen-code's own classifier code. What matters is whether the **model + llama.cpp combination actually
+honors it reliably** — which is precisely where the 27B model failed (see
+[`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md)) and where several shortlist candidates
+have their own version of the same problem.
+
+## 2. Candidates evaluated against primary sources
+
+| Model | Params | GGUF size (quant) | Context | License | Thinking behavior | Tool-calling | Verdict |
+|---|---|---|---|---|---|---|---|
+| **Qwen3-4B-Instruct-2507** | 4B | Q4_K_M 2.5GB / **Q8_0 4.28GB** | 262,144 native | Apache 2.0 | **Non-thinking only** — model card states it "does not generate `<think></think>` blocks in its output," full stop, no toggle needed | Yes, native `<tool_call>` format, BFCL-v3 61.9 | **Recommended** |
+| Qwen3-1.7B | 1.7B | ~1.1GB (Q4_K_M, typical) | 32,768 | Apache 2.0 | Thinking **on by default**; needs `enable_thinking:false` per call | Yes | Rejected — see §3 |
+| Qwen3-0.6B | 0.6B | ~0.4GB (Q4_K_M) | 32,768 | Apache 2.0 | Thinking on by default, same toggle issue as 1.7B | Yes, but weakest reasoning of the family | Rejected — undersized for reliability at this size, same toggle risk |
+| Llama-3.2-3B-Instruct | 3B | ~2GB (Q4_K_M, typical) | 128K | Llama 3.2 Community License — commercial use allowed, but text/EU carve-out language and an explicit >700M-MAU re-licensing clause | No thinking mode | Not natively documented on the model card fetched (no tool-call format called out) | Deprioritized — license has more fine print than Apache 2.0 for no clear benefit here |
+| Gemma-3-4b-it | 4B | Q4_K_M 2.49GB / Q8_0 4.13GB | 128K | Custom "Gemma" license (Google usage terms) | No documented thinking mode | Not documented on the model card fetched | Deprioritized — no confirmed native tool-calling story, non-Apache license |
+| Phi-4-mini-instruct | 3.8B | Q4_K_M 2.49GB / Q8_0 4.08GB | 128K | **MIT** | Not a reasoning model (that's the separate Phi-4-mini-**reasoning** model); no `<think>` tags by default | Yes — documented function-call format with dedicated tokens | Credible alternative — see §4 |
+| SmolLM3-3B | 3B | Q4_K_M ~1.9GB (typical) | 128K (64K trained + YaRN) | Apache 2.0 | **Thinking on by default** (`enable_thinking`), toggled via system-prompt flags | Yes (XML or Python-style tool calls) | Rejected — same thinking-by-default risk as Qwen3-1.7B |
+| Ministral-8B-Instruct-2410 | 8B | too large for budget at any useful quant with headroom | 128K | **Mistral Research License — commercial use requires contacting Mistral for a separate license** | Not documented as a reasoning model | Yes, documented function-calling with benchmark (31.6 vs Mistral-7B's 6.9) | Rejected — license restricts this repo's own dev-tooling use without contacting Mistral; also parameter count crowds the 7GB budget once Q8_0 + KV cache is counted |
+
+Sources (fetched directly from each model's own HF card / GGUF repo unless noted):
+[Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507),
+[unsloth/Qwen3-4B-Instruct-2507-GGUF](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF),
+[Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B),
+[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B),
+[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct),
+[google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it),
+[bartowski/google_gemma-3-4b-it-GGUF](https://huggingface.co/bartowski/google_gemma-3-4b-it-GGUF),
+[microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct),
+[bartowski/microsoft_Phi-4-mini-instruct-GGUF](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF),
+[HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B),
+[mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410).
+
+**Confidence note:** file sizes for Qwen3-1.7B/0.6B, Llama-3.2-3B, and SmolLM3-3B GGUF quants above are
+typical/approximate — those repos weren't individually re-verified against a specific GGUF file tree
+since all three families were eliminated on architectural grounds (§3) before size mattered. Sizes for
+the two models actually compared head-to-head (Qwen3-4B-Instruct-2507, Phi-4-mini-instruct) and
+Gemma-3-4b-it were pulled directly from each quantizer's own repo page.
+
+## 3. Why "thinking-by-default + per-call toggle" is disqualifying, not just a minor ding
+
+This is the deciding architectural distinction, and it's exactly the failure this second model exists
+to avoid. Qwen's own llama.cpp docs page states the toggle problem directly:
+
+> "the hard switch implemented in the chat template is not exposed in llama.cpp" for controlling
+> `enable_thinking` — the documented workaround is to supply "a custom chat template equivalent to
+> always `enable_thinking=False`" via `--chat-template-file`.
+Source: [Qwen — Run with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html).
+
+That means for Qwen3-1.7B, Qwen3-0.6B, and SmolLM3-3B — all thinking-on-by-default — reliably
+suppressing the reasoning phase in this llama.cpp/ROCm stack is not a request-body flag away; it needs
+a hand-maintained custom chat template file, which is exactly the kind of fragile, easy-to-silently-
+regress setup this task is trying to get away from (the 27B model's whole problem was reasoning_content
+being consumed before the answer). Qwen3-4B-Instruct-2507 has no such toggle to maintain in the first
+place — the model card states the non-thinking behavior as an unconditional property of the model, not
+a configurable default that has to be forced correctly on every request. This is a stronger claim than
+"same tokenizer family as the 27B" (the original quick-pass's reasoning) and is the actual basis for
+the recommendation.
+
+## 4. The one documented risk specific to Qwen3-4B-Instruct-2507 — and why it doesn't change the pick
+
+llama.cpp has its own closed, dated bug where server builds around **b8429** (March 2026)
+mis-detected Qwen3-Instruct-2507 models — the 4B included by name in the reporter's repro command — as
+thinking models, routing tool-call output into `reasoning_content` instead of `tool_calls`:
+
+> "llama.cpp b8429 incorrectly detects Qwen3-Instruct-2507 models as thinking models (`thinking = 1`).
+> This causes tool calls to be captured as `reasoning_content` instead of being parsed into the
+> `tool_calls` array."
+
+The documented, confirmed-working workaround is a single server flag:
+
+```
+llama-server -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --jinja --port 8222 --reasoning off
+```
+
+which restores `thinking = 0` and correct `finish_reason: tool_calls` output. The issue is **closed**.
+Source: [ggml-org/llama.cpp issue #20809](https://github.com/ggml-org/llama.cpp/issues/20809).
+
+This is worth flagging honestly against the recommendation, but it's materially different from the
+open, only-partially-fixed Qwen3.5/Qwen3.8-lineage parser bugs already documented in this repo's
+[`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md) (issues #21158, #20837 — both open at time
+of that research): this is a llama.cpp *server-side misdetection* bug with a one-flag fix, not an
+unresolved upstream grammar/parser defect in the Qwen3.5 architecture family itself. Concretely: add
+`--reasoning off` to this second llama-server instance's command regardless — it's a no-cost safety net
+whether or not the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build still has the bug, and it
+directly targets the exact failure mode (reasoning_content eating the completion) that ruled out the
+27B model for this role in the first place.
+
+## 5. VRAM math for the classifier role specifically
+
+Qwen3-4B-Instruct-2507 is a plain (non-hybrid) transformer — every layer is standard GQA attention, so
+unlike the 27B model's Gated-DeltaNet hybrid, KV cache scales with *all* layers, not a fraction of them.
+From the model's own `config.json`:
+
+- `num_hidden_layers`: 36, `num_key_value_heads`: 8, `head_dim`: 128
+Source: [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json).
+
+Per-token KV cache (fp16, both K and V):
+`36 layers × 2 (K+V) × 8 kv_heads × 128 head_dim × 2 bytes = 144 KiB/token`
+
+The classifier transcript is bounded by design — qwen-code's own two-stage design keeps Stage 1 to a
+`{shouldBlock}`-only judgment and Stage 2 to a chain-of-thought review of one blocked action, not an
+open-ended agent session — so a context window in the low thousands of tokens is generous headroom,
+not a tight fit:
+
+| Context | KV cache (fp16) | KV cache (q8_0, `--cache-type-k/v q8_0`) | Weights (Q8_0) | Total (q8_0 KV) | Headroom under 7GB |
+|---|---|---|---|---|---|
+| 4,096 tokens | ~0.56 GB | ~0.28 GB | 4.28 GB | **~4.56 GB** | ~2.4 GB |
+| 8,192 tokens | ~1.13 GB | ~0.56 GB | 4.28 GB | **~4.84 GB** | ~2.2 GB |
+| 32,768 tokens (generous ceiling) | ~4.5 GB | ~2.25 GB | 4.28 GB | **~6.53 GB** | ~0.5 GB (tight) |
+
+At any context length actually needed for a permission-gate classifier (thousands, not tens of
+thousands, of tokens), Q8_0 weights plus q8_0 KV cache comfortably clears the 7GB ceiling with headroom
+to spare for the compute buffer and batch overhead — matching the same `--cache-type-k q8_0
+--cache-type-v q8_0` pattern this repo already uses for the 27B instance. There's no need to drop to
+Q4_K_M (2.5GB) unless a much larger classifier context is anticipated later; Q8_0 is the better default
+here since it's a small model where quantization loss matters proportionally more, and the VRAM budget
+comfortably affords the higher-precision quant.
+
+## 6. What would change the answer
+
+- **If Phi-4-mini-instruct's MIT license matters more than matching the 27B model's tokenizer/template
+  family**, it's a legitimate second choice: confirmed non-thinking by default, confirmed native
+  function-calling format, comparable Q8_0 size (4.08GB), and a license with zero commercial-use fine
+  print (vs. Apache 2.0's still-permissive but slightly more conditional terms). It wasn't picked
+  because it has no llama.cpp-specific tool-calling track record verified in this pass (no equivalent
+  to the issue #20809 workaround search done for it), so its actual reliability on this exact
+  `llama.cpp:server-rocm` stack is less directly evidenced than Qwen3-4B-Instruct-2507's.
+- **If the classifier transcript ever needs to grow well past ~8K tokens routinely**, drop to Q4_K_M
+  (2.5GB) to keep well clear of the 7GB ceiling — the KV-cache math in §5 shows the crossover point.
+- **If llama.cpp's #20809 misdetection turns out to still reproduce** on the exact
+  `ghcr.io/ggml-org/llama.cpp:server-rocm` build this repo pulls, the fix is the one-flag
+  `--reasoning off` workaround already confirmed in that issue — not a reason to pick a different
+  model, since every thinking-capable alternative in this shortlist has an equal-or-worse version of
+  the same class of bug with less clean workarounds (custom chat-template files, per §3).
+
+## Sources
+
+- [Qwen Code docs — Auto Mode](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/)
+- [QwenLM/qwen-code — docs/users/features/auto-mode.md](https://github.com/QwenLM/qwen-code/blob/main/docs/users/features/auto-mode.md)
+- [QwenLM/qwen-code issue #4676](https://github.com/QwenLM/qwen-code/issues/4676)
+- [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
+- [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
+- [unsloth/Qwen3-4B-Instruct-2507-GGUF](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF)
+- [Qwen — Run with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html)
+- [ggml-org/llama.cpp issue #20809](https://github.com/ggml-org/llama.cpp/issues/20809)
+- [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
+- [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
+- [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
+- [google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it)
+- [bartowski/google_gemma-3-4b-it-GGUF](https://huggingface.co/bartowski/google_gemma-3-4b-it-GGUF)
+- [microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct)
+- [bartowski/microsoft_Phi-4-mini-instruct-GGUF](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF)
+- [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B)
+- [mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410)
+- [docs/research/qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) (this repo — cross-referenced
+  for the 27B model's own, still-open, tool-calling parser bugs)
+
+## Implementation note (2026-09-09) — what actually shipped, and why it differs
+
+The model pick (`Qwen3-4B-Instruct-2507`) held up and is what's deployed. Several sizing assumptions in
+this doc didn't survive contact with the real deployment, though — worth recording so the next person
+tuning this doesn't re-derive the same corrections from scratch:
+
+- **Service name is `qwen-classifier`, not `llama-server-fast`** — this doc's proposed name never got
+  used. There's no `LLAMA_FAST_CTX_SIZE`/`LLAMA_FAST_PARALLEL` in `.env.example` either; the real config
+  lives inline in `docker-compose.yml`'s `qwen-classifier` command.
+- **CPU-only was tried first and rejected** — this doc's VRAM budget analysis (§5) assumed GPU
+  residency from the start, but the actual rollout path tried CPU-only first (to sidestep VRAM
+  contention entirely) and found it too slow: real classification calls blew past OmniRoute's request
+  timeout and retry-looped. Moved to GPU after that, which is what §5's math was for all along.
+- **Q4_K_XL weights, not Q8_0** — §5's "~2.4GB headroom" case assumed Q8_0 (4.28GB). In practice, fitting
+  the classifier onto the R9700 *alongside* the 27B model (not in an assumed-empty 7GB budget) left only
+  ~6.1GB free VRAM total, and even Q4_K_XL (2.37GB) plus full-context KV cache didn't leave enough real
+  margin at full GPU offload — see the "measured live" numbers in `docker-compose.yml`'s `qwen-classifier`
+  comment block. Landed on **partial GPU offload (28/36 layers)** instead of full offload, which is not a
+  case this doc considered at all.
+- **65536 context, not 8192** — §5 sized the context "in the low thousands," reasoning from qwen-code's
+  two-stage classifier description alone. Directly reading qwen-code's actual source
+  (`packages/core/src/permissions/classifier-transcript.ts`: `MAX_TRANSCRIPT_MESSAGES=40`,
+  `MAX_HISTORICAL_ACTION_CHARS=4000`/message) puts the real worst case at ~40-50K tokens — confirmed
+  live, a real classifier call during testing hit 15,116 prompt tokens. 8192 would have been undersized
+  for real usage; 65536 gives margin without the original setting.json value (131072, copied from the
+  main model's entry, not a real qwen-code requirement) wasting VRAM for no reason.
+- **§4's `--reasoning off` recommendation was initially missed** in the first deployment pass and added
+  only once this doc was re-read while writing this note. It's now in `docker-compose.yml`'s
+  `qwen-classifier` command, per this doc's own "add it regardless, no-cost safety net" reasoning — still
+  unconfirmed whether the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build actually reproduces
+  #20809 (nothing in testing so far surfaced `reasoning_content` where `tool_calls` was expected, but
+  that wasn't specifically probed for either).
+
+## Confidence/uncertainty summary
+
+- **High confidence:** Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote);
+  qwen-code's two-stage classifier timing/design and its use of `/model --fast` (direct docs quote);
+  the existence, exact symptom, and workaround of llama.cpp issue #20809 (direct issue quote); the
+  KV-cache architecture math (computed directly from the model's own `config.json`, same method as the
+  existing `qwen3.8-27b-quant.md` research in this repo).
+- **Medium confidence:** exact GGUF file sizes for Qwen3-1.7B, Qwen3-0.6B, Llama-3.2-3B-Instruct, and
+  SmolLM3-3B — not individually re-verified against a specific quantizer's file tree since these were
+  eliminated on architectural (thinking-toggle) grounds before size became the deciding factor; treat
+  as typical/approximate, not exact.
+- **Low confidence / not independently verified:** whether the current
+  `ghcr.io/ggml-org/llama.cpp:server-rocm` image (pulled fresh) still reproduces issue #20809's
+  misdetection — the issue is closed but no changelog/PR diff was fetched to confirm the underlying
+  detection logic was actually patched vs. the reporter simply adopting the `--reasoning off`
+  workaround. Recommend a live smoke test (send one tool-calling request, confirm the response lands in
+  `tool_calls` not `reasoning_content`, and time a trivial completion) before wiring this model in as
+  the production `fastModel`, the same caveat this repo's `qwen3.8-27b-tool-calling.md` already flags
+  for the 27B model.
+- Whether Phi-4-mini-instruct has a comparably clean llama.cpp tool-calling track record was **not**
+  deep-dived (no issue-tracker search run against it) — it's flagged in §6 as a live alternative rather
+  than fully evaluated, since Qwen3-4B-Instruct-2507's architectural non-thinking guarantee and
+  same-family template consistency with the existing 27B deployment made it the clearer pick without
+  needing that extra research pass.

docs/research/image-generation-model-choice.md

@@ -0,0 +1,220 @@
+# Research: which diffusion model to target with the full ~32GB R9700
+
+**Question:** With Qwen/llama-server fully stopped (per issue #39's premise — see
+map #38), ComfyUI has the whole ~32GB R9700 (ROCm/gfx1201) to itself instead
+of the ~6GB left over during concurrent operation
+(per [`image-generation-options.md`](image-generation-options.md)). Given that
+headroom, should the build stay on FLUX.1-schnell, or move up to
+FLUX.1-dev, SD3.5-large, Qwen-Image, HunyuanImage-3.0, or Krea-2?
+
+**Answer, short version:** move up to **Qwen-Image at FP8 precision**
+(`qwen_image_fp8_e4m3fn.safetensors` diffusion weights +
+`qwen_2.5_vl_7b_fp8_scaled.safetensors` text encoder, ~25GB combined). It's
+the only one of the five upgrade candidates with **direct, hardware-specific
+evidence of running on this exact GPU architecture** (gfx1201/R9700) rather
+than a generic "ComfyUI supports ROCm" inference, it carries the cleanest
+license of the group (Apache-2.0, no revenue threshold, no non-commercial
+clause), and its 20B MMDiT is a real capability step up from schnell's
+distilled 12B (notably for text rendering and prompt adherence), while still
+fitting with real margin inside 32GB.
+
+## Why not just re-derive the schnell/dev/SDXL/SD3.5 findings
+
+`docs/research/image-generation-options.md` already covers, with primary
+sources: ComfyUI's ROCm/gfx1201 story (official AMD docs + RDNA4 blog post +
+community gfx1201 Docker images), FLUX.1-schnell vs FLUX.1-dev vs SDXL vs
+SD3.5 licenses, and FLUX GGUF VRAM figures at the ~6GB-headroom scale. None
+of that is repeated here except where the ~32GB ceiling changes the
+conclusion. This doc adds: FLUX.1-dev/SD3.5 at the *larger* headroom, plus
+three models the prior doc didn't cover at all (Qwen-Image, HunyuanImage-3.0,
+Krea-2).
+
+## Candidate comparison
+
+| Model | License (primary source) | Params | Stated/typical VRAM | ROCm/gfx1201 evidence |
+|---|---|---|---|---|
+| FLUX.1-schnell (current) | Apache-2.0 | 12B | GGUF Q4_K_S ~7GB (per prior doc) | Confirmed on gfx1201 (prior doc) |
+| **Qwen-Image** | **Apache-2.0** | 20B (20.4B DiT + 8.3B Qwen2.5-VL text encoder) | fp8 ~16GB (diffusion) + ~9.4GB (fp8 text encoder) ≈ 25GB total; bf16 needs 24GB+ and "48GB+" per some quant write-ups | **Direct**: [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) ships a pre-validated "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow specifically for the R9700 AI Pro (gfx1201), on a ROCm 7 (TheRock nightlies) toolbox |
+| FLUX.1-dev | [FLUX.1-dev Non-Commercial License](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) — non-commercial weights, outputs usable commercially | 12B | bf16 ~24GB; GGUF Q8 ~12-13GB | Same *family* evidence as schnell (gfx1201 Docker images target FLUX generally), but no R9700-specific FLUX.1-dev report found |
+| SD3.5-large | [Stability Community License](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md) — free under $1M annual revenue | 8B | bf16 ~16GB; 4-bit NF4 fits small GPUs | AMD's own ComfyUI-ROCm doc lists an "SD3.5 Simple" template workflow (per prior doc) — vendor-blessed but not R9700-specific |
+| HunyuanImage-3.0 | [tencent-hunyuan-community license](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0) | 80B total / 13B active (MoE, 64 experts) | Official repo: "≥ 3 × 80GB" VRAM for the base model, "≥ 8 × 80GB" for -Instruct; ~177GB at fp16 | **None, and actively contraindicated**: setup requires CUDA 12.8 + FlashAttention2/FlashInfer, no AMD/ROCm mention anywhere in the official repo |
+| Krea-2 (Turbo) | [Krea 2 Community License](https://www.krea.ai/krea-2-licensing) — free under $1M annual revenue, homelab/personal explicitly covered | 12-13B DiT | No official VRAM figure; community reports (RTX hardware only) cite fp8 ~16GB, GGUF ~12GB | **None found** — released [June 22, 2026 per the HF model card](https://huggingface.co/krea/Krea-2-Turbo); ComfyUI added native support per [blog.comfy.org](https://blog.comfy.org/p/krea-2-open-source-models-are-now), and GGUF quants exist ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), but no R9700/gfx1201-specific report exists yet — too new for that evidence to have accumulated |
+
+## Per-model detail
+
+### Qwen-Image — recommended
+
+- **License**: Apache-2.0, stated directly on the model card, no revenue
+  threshold, no non-commercial clause, no attribution/naming requirement.
+  The cleanest license of every model considered in this doc or the prior
+  one. Source: [Qwen/Qwen-Image on Hugging Face](https://huggingface.co/Qwen/Qwen-Image).
+- **Architecture**: 20B-parameter MMDiT (Multimodal Diffusion Transformer)
+  combined with an 8.3B Qwen2.5-VL text encoder — notably larger and more
+  capable than FLUX.1-schnell's 12B distilled model, particularly for
+  multilingual text rendering and instruction-following, per the official
+  [QwenLM/Qwen-Image GitHub repo](https://github.com/QwenLM/Qwen-Image).
+- **ComfyUI support**: native, not a wrapper/custom-node integration —
+  landed August 2025 per [ComfyUI Wiki's native-support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image),
+  with an official FP8 checkpoint (`qwen_image_fp8_e4m3fn.safetensors`) and
+  FP8-scaled text encoder (`qwen_2.5_vl_7b_fp8_scaled.safetensors`, 9.38GB)
+  published under [Comfy-Org/Qwen-Image_ComfyUI](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders).
+- **VRAM at FP8**: the official Comfy-Org FP8 diffusion checkpoint plus FP8
+  text encoder land around ~25GB combined — comfortably inside the 32GB
+  ceiling with headroom for ComfyUI runtime/VAE overhead, versus 24GB+ (some
+  sources say 48GB+) for the unquantized bf16 model. Sources: community
+  VRAM write-ups aggregated via [Comfy-Org/ComfyUI issue #10852 ("Qwen-image in 24GB VRAM and 32GB RAM")](https://github.com/Comfy-Org/ComfyUI/issues/10852)
+  and the [official ComfyUI Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/) —
+  treat exact GB figures as secondary/community-sourced, consistent with how
+  the prior doc flagged FLUX's GGUF VRAM table.
+- **Direct R9700/gfx1201 evidence (the deciding factor)**:
+  [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) is a
+  Fedora-toolbox ROCm 7 (TheRock nightlies) environment built specifically
+  for the "AMD Radeon 9700 AI PRO (32GB)" and ships a pre-validated
+  "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow
+  (plus a Qwen-Image-Edit 2511 FP8 workflow) as one of only four workflows
+  in the whole repo. This is the only model in this comparison with
+  card-architecture-specific (not just "ComfyUI supports ROCm generically")
+  validation — everything else relies on family-level or vendor-generic
+  ROCm claims. A related community discussion
+  ([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
+  does flag that "Qwen image edit doesn't always work on AMD HIP/ROCm" in
+  some configurations — worth testing the specific workflow before assuming
+  zero friction, but this is a known-quantity, actively-discussed rough edge
+  rather than a documented hard blocker.
+- **4-step Lightning LoRA**: the R9700-validated workflow pairs Qwen-Image
+  with a "Lightning" LoRA for 4-step inference — the same fast-inference
+  pattern FLUX.1-schnell uses, so switching models doesn't have to mean
+  giving up the short GPU-resident-time-per-image property the prior doc
+  called out as valuable for time-sliced use alongside llama-server (even
+  though this ticket's premise is llama-server being stopped, that pattern
+  still helps if the two services are ever run in an overlapping window).
+
+### FLUX.1-dev — solid alternative, not the pick
+
+- License permits non-commercial use of the weights; BFL's own license page
+  states generated outputs are separately usable commercially (already
+  covered in the prior doc; unchanged here). Source:
+  [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).
+- At bf16 (~24GB) or GGUF Q8 (~12-13GB), it fits the 32GB ceiling with room
+  to spare — a legitimate move up from schnell's distilled quality.
+  Source: [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
+  and community Q8 VRAM reports.
+- Loses to Qwen-Image on two counts: license (non-commercial-weights clause
+  vs. Apache-2.0 — not a hard blocker for this homelab per the prior doc's
+  own reasoning, but strictly worse) and hardware evidence (FLUX's gfx1201
+  validation is at the *family* level — the prior doc's `yurisasc/comfyui-rocm-rdna4`
+  and `charlie12345/R9700AIProComfyUIPatch` sources are about running FLUX
+  models on this card generally, not a FLUX.1-dev-specific report the way
+  kyuz0's repo is Qwen-Image-specific).
+
+### SD3.5-large — no longer the best use of the freed headroom
+
+- License unchanged from the prior doc: Stability Community License, free
+  under $1M annual revenue (irrelevant threshold for this homelab). Source:
+  [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
+- At 8B params (bf16 ~16GB), it's the smallest of the upgrade candidates —
+  which mattered when 6GB was the ceiling, but with 32GB available there's
+  no VRAM reason to pick the model the prior doc already flagged as "lower
+  fidelity than FLUX/SD3.5 by current standards" over Qwen-Image or
+  FLUX.1-dev. AMD's own ComfyUI-ROCm docs do list it as a first-party
+  example template ("SD3.5 Simple", per the prior doc), so it remains a fine
+  fallback if Qwen-Image's FP8 path hits the ROCm rough edge noted above.
+
+### HunyuanImage-3.0 — ruled out
+
+- Official repo states VRAM requirements of "≥ 3 × 80GB" for the base model
+  and "≥ 8 × 80GB" for HunyuanImage-3.0-Instruct — i.e. multi-GPU
+  datacenter-class NVIDIA clusters, not a single 32GB consumer/workstation
+  card at any precision. Source:
+  [Tencent-Hunyuan/HunyuanImage-3.0 GitHub repo](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0).
+- Setup instructions require CUDA 12.8, PyTorch 2.8.0 built for CUDA, and
+  optionally FlashAttention2/FlashInfer for MoE routing speed — no AMD or
+  ROCm path is mentioned anywhere in the official repo. Even the
+  MoE-efficient "13B active" framing doesn't help here: the tooling itself
+  assumes an NVIDIA multi-GPU cluster, and the 80B total parameter set still
+  has to be resident somewhere.
+- 32GB of headroom on one AMD card doesn't move this model into reach at
+  any precision considered here; it's excluded regardless of how much VRAM
+  frees up on this specific box.
+
+### Krea-2 (Turbo) — promising, but unverified on this hardware
+
+- Verified directly against primary sources per the ticket's instruction
+  (this is a June 2026 release, past most training cutoffs): the
+  [Hugging Face model card](https://huggingface.co/krea/Krea-2-Turbo) states
+  a release date of **June 22, 2026**, a 12-billion-parameter single-stream
+  diffusion transformer, `torch.bfloat16` as the reference precision, and
+  the **Krea 2 Community License**.
+- License, per [krea.ai/krea-2-licensing](https://www.krea.ai/krea-2-licensing):
+  non-commercial (including explicitly personal/homelab) use is free;
+  commercial use is permitted royalty-free for entities under $1M
+  trailing-12-month revenue (same shape as SD3.5's and matching this
+  homelab's use case); content-filter and AI-disclosure obligations apply if
+  deployed publicly; derivative model names must start with "Krea".
+- ComfyUI added native support for both open-weight checkpoints (Krea 2 Raw
+  and Krea 2 Turbo) per [blog.comfy.org's announcement](https://blog.comfy.org/p/krea-2-open-source-models-are-now),
+  and community GGUF quants already exist
+  ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), with
+  reports (RTX hardware only) of fp8 fitting 16GB and GGUF fitting 12GB.
+- **No AMD/ROCm mention anywhere** in the model card, and no gfx1201/R9700
+  community report was found — unsurprising given the model is roughly
+  2.5 months old at the time of this research. Its architecture (a
+  standard-shaped DiT that ComfyUI loads through its normal diffusion-model
+  nodes, per the ComfyUI blog post) gives reasonable expectation it will run
+  on the same ROCm/PyTorch backend already proven for FLUX and Qwen-Image on
+  this card, but that's an inference, not a verified fact the way
+  kyuz0's Qwen-Image workflow is.
+- **Not the pick today**, precisely because Qwen-Image already offers a
+  hardware-verified path at a comparable parameter count and VRAM budget.
+  Worth a follow-up research ticket once R9700/gfx1201-specific Krea-2
+  reports exist — the license and ComfyUI support are both already in
+  place, so the only open question is real-world ROCm behavior.
+
+## Recommendation
+
+**Qwen-Image, FP8 precision** (`qwen_image_fp8_e4m3fn.safetensors` +
+`qwen_2.5_vl_7b_fp8_scaled.safetensors`, ~25GB combined), optionally paired
+with the 4-step Lightning LoRA the R9700-specific validated workflow uses.
+It wins on all three axes the ticket asked about:
+
+1. **License**: Apache-2.0 — no restriction at all, strictly better than
+   every other candidate including the current FLUX.1-schnell pick.
+2. **ROCm/gfx1201 compatibility**: the only candidate with a workflow
+   pre-validated specifically on this GPU architecture
+   ([kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy)),
+   not just "ComfyUI supports ROCm in general."
+3. **VRAM at the ~32GB ceiling**: ~25GB at FP8 leaves real margin for
+   ComfyUI runtime/VAE overhead, without needing the multi-step,
+   non-distilled FLUX.1-dev's full 24GB bf16 footprint or accepting
+   SD3.5's lower fidelity ceiling — and it's a genuine capability upgrade
+   over schnell (20B vs. 12B, non-distilled-quality text rendering) rather
+   than just a bigger file.
+
+If the known Qwen-Image/ROCm edit-mode rough edge
+([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
+turns out to affect plain text-to-image generation too, SD3.5-large (AMD's
+own first-party "SD3.5 Simple" ComfyUI-ROCm template workflow) is the
+fallback, with FLUX.1-dev as a second option. HunyuanImage-3.0 is excluded
+outright regardless of available VRAM (CUDA-only tooling, multi-GPU
+datacenter VRAM floor). Krea-2 is worth revisiting once R9700-specific
+field reports exist.
+
+## Sources consulted
+
+- [docs/research/image-generation-options.md](image-generation-options.md) (this repo — prior findings, not re-derived)
+- [Qwen/Qwen-Image (Hugging Face)](https://huggingface.co/Qwen/Qwen-Image)
+- [QwenLM/Qwen-Image (GitHub)](https://github.com/QwenLM/Qwen-Image)
+- [ComfyUI Wiki — Qwen-Image native support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image)
+- [Comfy-Org/Qwen-Image_ComfyUI (Hugging Face, FP8 checkpoints)](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders)
+- [Comfy-Org/ComfyUI issue #10852 — Qwen-Image VRAM](https://github.com/Comfy-Org/ComfyUI/issues/10852)
+- [ComfyUI official Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/)
+- [kyuz0/amd-r9700-comfy (R9700-specific ROCm ComfyUI toolbox)](https://github.com/kyuz0/amd-r9700-comfy)
+- [pollockjj/ComfyUI-MultiGPU discussion #133 (Qwen-Image-Edit ROCm rough edge)](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133)
+- [black-forest-labs/FLUX.1-dev (Hugging Face) + LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev)
+- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
+- [stabilityai/stable-diffusion-3.5-large (Hugging Face) + LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)
+- [Tencent-Hunyuan/HunyuanImage-3.0 (GitHub)](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0)
+- [krea/Krea-2-Turbo (Hugging Face)](https://huggingface.co/krea/Krea-2-Turbo)
+- [Krea 2 Community License Agreement (krea.ai)](https://www.krea.ai/krea-2-licensing)
+- [blog.comfy.org — Krea 2 open-source models in ComfyUI](https://blog.comfy.org/p/krea-2-open-source-models-are-now)
+- [molbal/krea2-gguf (Hugging Face)](https://huggingface.co/molbal/krea2-gguf)

docs/research/image-generation-options.md

@@ -0,0 +1,249 @@
+# Research: adding local image generation to the stack
+
+**Question:** What's the best way to add local image generation alongside
+the existing Qwen3.8-27B / llama.cpp text stack, given a single AMD Radeon
+R9700 (32GB VRAM, ROCm/gfx1201 — not CUDA), routed through the OmniRoute
+gateway on the `ai-stack` Docker network?
+
+**Answer, short version:** run **ComfyUI** (official AMD-blessed ROCm
+Docker path exists, and OmniRoute already has a first-class `comfyui`
+provider — no bespoke API wrapper needed) with **FLUX.1 [schnell]**
+(Apache-2.0, 4-step, GGUF-quantizable) as the default model, falling back to
+**SDXL** for anything schnell's distilled-step license/quality tradeoffs
+don't suit. VRAM headroom against the current llama-server footprint is too
+tight for both to be resident at once at any real image quality — plan for
+**time-sliced use** (llama-server's existing lazytainer stop-on-idle pattern,
+mirrored for the image-gen service, or a manual "stop one, start the other"
+toggle), not concurrent operation.
+
+## Current VRAM baseline (from this repo)
+
+Per `docker-compose.yml` and `.env.example`, llama-server runs
+`Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights) at `--ctx-size 262144` with
+`--cache-type-k q8_0 --cache-type-v q8_0`, landing at **~25.6 GB** total
+(weights + q8_0 KV cache), leaving **~6 GB** free on the 32GB card — this
+matches the math already recorded in
+[`docs/research/qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md). Per the auto-memory
+note on this repo, real measured VRAM use has run closer to ~75% (~24 GB) in
+practice versus the theoretical estimate, which doesn't change the
+conclusion below but means the ~6 GB figure is closer to a ceiling than a
+comfortable number.
+
+**Implication:** 6 GB is not enough for any current-generation image model at
+usable quality (see VRAM table below — even the smallest practical FLUX
+quant wants ~7 GB alone, before ComfyUI's own runtime/VAE overhead). Running
+image-gen *concurrently* with llama-server resident is not realistic on this
+card. The two need to time-share the GPU, not split it.
+
+## Backend evaluation (ROCm support, checked against primary sources)
+
+### ComfyUI — recommended
+
+- **Official AMD ROCm docs exist and are current.** AMD's own ROCm docs site
+  hosts a dedicated ComfyUI install guide with a prebuilt Docker image path
+  (recommended) or build-from-source, listing ROCm 7.2.0 and 7.1.0 as
+  supported versions, explicit `--device=/dev/kfd --device=/dev/dri
+  --group-add video` flags (same device-passthrough pattern this repo
+  already uses for llama-server), and template workflows including "SD3.5
+  Simple". Officially the guide only names AMD Instinct
+  MI355X/MI325X/MI300X (datacenter cards) as supported platforms.
+  Source: [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html).
+- **The upstream ComfyUI README itself documents AMD support directly**,
+  including consumer cards: stable ROCm install via
+  `pip install torch torchvision torchaudio --index-url
+  https://download.pytorch.org/whl/rocm7.2`, plus an experimental Windows
+  build explicitly naming **RDNA 3 (RX 7000), RDNA 3.5 (Strix Halo), and
+  RDNA 4 (RX 9000 series)** — i.e. the same RDNA4 generation as the R9700 —
+  and `HSA_OVERRIDE_GFX_VERSION` workarounds for older/unlisted cards.
+  Source: [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI).
+- **AMD has published a specific RDNA4/RX 9000 ComfyUI guide** (separate
+  from the Instinct-only install page above), confirming RDNA4 consumer
+  cards are an explicitly supported, first-party-documented target, not just
+  a community workaround.
+  Source: [ROCm blog — Getting Started with ComfyUI on AMD Radeon RX 9000 Series GPUs](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html).
+- **gfx1201 (R9700's arch) specifically has active community Docker images**:
+  `yurisasc/comfyui-rocm-rdna4` targets ROCm 7.1 + PyTorch 2.9.1 with
+  `HSA_OVERRIDE_GFX_VERSION=12.0.1` / `PYTORCH_ROCM_ARCH=gfx1201` baked in,
+  and there's a published community patch specifically for R9700 AI Pro +
+  ComfyUI video-gen speedups, evidence the card is being run today, not just
+  theoretically compatible.
+  Sources: [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4),
+  [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch).
+- **Known gfx1201 caveat:** AMD's own TransformerEngine repo has an open
+  issue confirming gfx1201 is missing from the FP8 architecture table, so
+  FP8 kernels silently fall back to FP32 with ~50% throughput loss
+  (18-22 vs. 35-40 tok/s in the reporter's LLM benchmark) — not a
+  correctness blocker, but relevant if planning to use FP8-quantized image
+  models expecting native FP8 speed on this card; GGUF/Q-quants (see below)
+  avoid this path entirely since they dequantize to bf16/fp16, not fp8.
+  Source: [ROCm/TransformerEngine issue #520](https://github.com/ROCm/TransformerEngine/issues/520).
+- **Actively maintained community Docker packaging** beyond AMD's own image:
+  `YanWenKun/ComfyUI-Docker` ships parallel `rocm` (PyTorch-build-based,
+  faster releases) and `rocm7` (AMD-build-based, more comprehensive)
+  variants, both targeting ROCm 7, with ~1000 commits of ongoing history —
+  a viable alternative to the official AMD image if it lags behind ComfyUI
+  releases.
+  Source: [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker).
+
+### AUTOMATIC1111 / Forge — usable but a step down for this hardware
+
+- ROCm support for A1111/Forge is real but community-patched, not
+  first-party. The upstream `lllyasviel/stable-diffusion-webui-forge` repo's
+  own discussion thread on AMD support points users to
+  `lshqqytiger/stable-diffusion-webui-amdgpu-forge`, a community fork
+  specifically maintained for AMD, "regarded as the go-to version" for
+  running FLUX-era models on AMD — i.e. the *mainline* Forge repo does not
+  claim ROCm support itself; you're expected to run a fork.
+  Source: [lllyasviel/stable-diffusion-webui-forge discussion #67](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67).
+- No first-party AMD vendor documentation (unlike ComfyUI's AMD-authored
+  ROCm/RDNA4 blog posts above) was found for A1111/Forge specifically.
+  Given ComfyUI already has an AMD-blessed path plus a first-class OmniRoute
+  provider (below), there's no reason to take on a community fork's
+  maintenance risk instead.
+
+### InvokeAI — usable but weaker AMD story for a new-generation card
+
+- InvokeAI documents ROCm support but flags it as second-tier: "AMD GPUs
+  are only supported on Linux," and "support for newer AMD GPUs is spotty
+  ... you may experience garbled images, black images, or long startup
+  delays." Its own install docs reference ROCm 5.4.2-era wheels, notably
+  older than the ROCm 7.x this stack's llama-server image already runs on
+  gfx1201.
+  Source: [InvokeAI installation docs (mauwii mirror)](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/).
+- No OpenAI-compatible-images angle either — same drawback as A1111/Forge.
+  Not recommended as primary given ComfyUI's stronger, more current AMD
+  documentation trail.
+
+## Model choice: FLUX.1 [schnell] vs FLUX.1 [dev] vs SDXL vs SD3.5
+
+| Model | License | Params | Notes |
+|---|---|---|---|
+| **FLUX.1 [schnell]** | **Apache-2.0** — fully open, no commercial restriction | 12B | Distilled for 1-4 step inference (fast); Black Forest Labs' own model card states this license directly |
+| FLUX.1 [dev] | [FLUX.1-dev Non-Commercial License](https://github.com/black-forest-labs/flux/blob/main/model_licenses/LICENSE-FLUX1-dev) | 12B | Non-commercial for the *model/weights*; generated *outputs* are explicitly usable commercially per BFL's license page. Higher quality than schnell (more steps, non-distilled) but the weights themselves can't be redistributed/used commercially |
+| SDXL | CreativeML OpenRAIL++ (permissive, commercial-friendly) | ~3.5B | Older (2023), lower fidelity than FLUX/SD3.5 by current standards, but lowest VRAM footprint and best long-standing tooling maturity |
+| SD3.5 (Large/Medium) | Stability AI Community License — free commercial use under $1M annual revenue, else enterprise license required | 8B / 2.5B | Free for this repo's non-commercial homelab use regardless; template already listed in AMD's own ComfyUI-ROCm doc ("SD3.5 Simple") as a first-party example workflow |
+
+Sources: [black-forest-labs/flux model cards](https://github.com/black-forest-labs/flux/blob/main/model_cards/FLUX.1-dev.md),
+[black-forest-labs/FLUX.1-schnell on Hugging Face](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
+(license: apache-2.0), [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md),
+[Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5),
+[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
+
+**Recommendation: FLUX.1 [schnell].** For a private homelab, license
+enforcement isn't the deciding factor by itself, but schnell's Apache-2.0
+status removes any future ambiguity if outputs or the setup are ever shared
+or repurposed, and its whole design point — good quality in 1-4 sampling
+steps — directly addresses the VRAM/time-slicing constraint below (less
+GPU-resident time per image than a 20-50 step dev/SDXL/SD3.5 run).
+Quantized via `city96/ComfyUI-GGUF` (an actively-referenced, community-
+trusted quantization node — its GGUF Q-quants dequantize to bf16/fp16 at
+runtime, sidestepping the gfx1201 FP8 dequant bug above entirely), FLUX fits
+in a fraction of its fp16 footprint:
+
+| Precision | Approx. VRAM (model only) |
+|---|---|
+| fp16 (baseline) | ~24 GB |
+| fp8 | ~12 GB |
+| GGUF Q5_K_S | ~12-15 GB (practical quality floor) |
+| GGUF Q4_K_S | ~7 GB (quality starts degrading on hands/text below Q4) |
+
+Source: aggregated VRAM figures from GGUF-quantization write-ups referencing
+city96's FLUX GGUF conversions — treat as secondary/community sourced
+(no single BFL-published VRAM table was found), consistent across multiple
+independent sources.
+[city96/ComfyUI-GGUF README](https://github.com/city96/ComfyUI-GGUF/blob/main/README.md),
+[city96/FLUX.1-dev-gguf model card](https://huggingface.co/city96/FLUX.1-dev-gguf).
+
+**Fallback pick: SDXL.** If schnell's distilled quality ceiling proves too
+low for some use case, SDXL is the safer second choice over FLUX.1 [dev] or
+SD3.5 specifically *because* of this card's tight headroom: it's the
+smallest of the four by a wide margin, has the longest production track
+record on ROCm of any of these models, and its OpenRAIL++ license carries no
+revenue-threshold clause to track (unlike SD3.5's Community License) or
+non-commercial weight restriction (unlike FLUX.1 [dev]).
+
+## OpenAI-compatible API / OmniRoute integration
+
+This is the best news in this research: **OmniRoute already ships a
+first-class, built-in `comfyui` provider** — not a generic "point it at an
+OpenAI base URL and hope" integration. Its provider reference documents it
+explicitly: *"No API key required. Configure the local ComfyUI base URL
+(default: http://localhost:8188)."* OmniRoute's own image-routing feature
+set (`/v1/images/generations`, `/v1/images/edits`, `/v1/images/variations`,
+with automatic provider fallback) is designed for exactly this pattern:
+register ComfyUI as a backend, then any client already calling OmniRoute's
+OpenAI-compatible images endpoints reaches it with no extra shim.
+Source: [diegosouzapw/OmniRoute PROVIDER_REFERENCE.md](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md),
+[diegosouzapw/OmniRoute repo description](https://github.com/diegosouzapw/OmniRoute).
+
+This means **ComfyUI does not need an extra OpenAI-API wrapper project** —
+the wrapper projects found during this research
+([`ComfyUI-OpenAI-Compatible-API`](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API))
+turned out to be a ComfyUI *custom node* for calling *outbound* to LLM APIs
+from within a workflow (the reverse direction), not something this stack
+needs — OmniRoute's own native ComfyUI provider is the actual integration
+point, one layer up.
+
+**Confidence note:** the provider-reference detail above was fetched via an
+automated summarizer against the raw doc rather than manually re-verified
+line-by-line; re-check `PROVIDER_REFERENCE.md`'s `comfyui` entry directly
+before wiring this up, in case ComfyUI's own `/prompt` API (a
+workflow-graph-shaped API, not a simple text-prompt-in/image-out call) needs
+a specific default workflow JSON configured on the OmniRoute side to produce
+a plain text-to-image call.
+
+## Integration sketch (not a full compose — see caveats above)
+
+- New service in `docker-compose.yml`, e.g. `comfyui`, image
+  `rocm/comfyui-rocm` (or `yurisasc/comfyui-rocm-rdna4` for a gfx1201-tuned
+  build) or built from AMD's own ROCm ComfyUI Dockerfile, same
+  `/dev/kfd` + `/dev/dri` + `group_add: [video, render]` device-passthrough
+  block already used for `llama-server`, joined to the same `ai-stack`
+  network so `omniroute` can reach it as `http://comfyui:8188` — no host
+  port needed (matches the existing llama-server pattern of no published
+  port, gateway-only access).
+- Register it in OmniRoute's dashboard as a `comfyui` provider pointing at
+  that internal URL, same manual-registration pattern already used for
+  llama-server and searxng-search per `docs/proxy-key-onboarding.md`.
+- **VRAM contention is the real design problem, not networking.** Given the
+  ~6 GB headroom, the two services can't both sit GPU-resident.
+  Two workable patterns, in order of how well they fit what's already in
+  this repo:
+  1. **Mirror the existing lazytainer stop-on-idle pattern** already applied
+     to `llama-server` (`docker-compose.yml`'s `lazytainer.group.*` labels) —
+     add an equivalent idle-timeout group for `comfyui`, and rely on the two
+     services naturally not being hit at the same time for a single-user
+     homelab. This doesn't *guarantee* mutual exclusion (both could still be
+     woken concurrently and both try to fit in 6 GB free), so it's a
+     reasonable-effort fit, not a hard guarantee.
+  2. **Explicit mutual exclusion**: a small script/compose profile that
+     stops `llama-server` before starting `comfyui` (and vice versa) rather
+     than relying on lazytainer's independent idle timers — worth doing if
+     the reasonable-effort version above causes a visible OOM in practice.
+  Either way, this is a "pick one, then the other" story, not "run both."
+- Given FLUX.1 [schnell]'s 1-4 step design, a cold-start-and-generate cycle
+  (wake ComfyUI from lazytainer sleep, generate, let it idle back down) is
+  a reasonably good fit for occasional image requests through the same
+  gateway that already does this for llama-server.
+
+## Sources consulted
+
+- [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html)
+- [ROCm blog — ComfyUI on AMD Radeon RX 9000 Series (RDNA4)](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html)
+- [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI)
+- [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker)
+- [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4)
+- [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch)
+- [ROCm/TransformerEngine issue #520 (gfx1201 FP8 fallback)](https://github.com/ROCm/TransformerEngine/issues/520)
+- [lllyasviel/stable-diffusion-webui-forge discussion #67 (AMD support)](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67)
+- [InvokeAI CUDA/ROCm install docs](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/)
+- [black-forest-labs/flux GitHub repo + model cards](https://github.com/black-forest-labs/flux)
+- [black-forest-labs/FLUX.1-schnell (Hugging Face, Apache-2.0)](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
+- [black-forest-labs/FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)
+- [Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5)
+- [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)
+- [city96/ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF)
+- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) and its `PROVIDER_REFERENCE.md`
+- [yeeyou/ComfyUI-OpenAI-Compatible-API](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API) (checked and ruled out — wrong direction)
+- This repo: `docker-compose.yml`, `.env.example`, `docs/research/qwen3.8-27b-quant.md`

docs/research/langchain-pgvector-vs-litellm-pgvector.md

@@ -0,0 +1,388 @@
+# Research: LangChain+pgvector-direct RAG vs. the litellm-pgvector connector (for Gitea issue #25)
+
+**Question:** Is a LangChain + pgvector-direct retrieval architecture (LangChain's
+`PGVector` class talking to Postgres directly, LiteLLM used only for
+provider-abstracted chat/embedding calls) a better fit for this stack than the
+already-built `litellm-pgvector` connector service (`vendor/litellm-pgvector/`,
+issue #23's resolution)?
+
+**Bottom line: yes — switch.** Replace `litellm-pgvector` /
+`vendor/litellm-pgvector/` with a small hand-written FastAPI service (roughly
+80–120 lines, one new dependency set: `langchain-postgres`, `langchain-openai`,
+`fastapi`, `uvicorn`) that wraps LangChain's `PGVector` class and calls
+LiteLLM's OpenAI-compatible `/v1/embeddings` endpoint for embeddings. This
+keeps "memory served from the gateway layer" in spirit (LiteLLM still does all
+chat/embedding model calls; only the retrieval *query* now goes through a
+LangChain-based service instead of a vendored 754-line Prisma-backed
+connector), removes 754 lines of unvetted vendored source and a Prisma
+migration dependency, and — critically — removes the *guessed and unverified*
+`vector_store_registry` / `custom_llm_provider: pg_vector` config that issue
+#24 flagged as never confirmed against a live deploy. `pgvector-db` (plain
+`pgvector/pgvector:pg16`) is reusable completely as-is; only the schema
+management strategy changes (LangChain manages its own two tables instead of
+Prisma migrations).
+
+## 1. Verifying the markaicode article against LangChain's actual docs
+
+The article (<https://markaicode.com/stack/litellm-langchain-stack/>) proposes
+three services — LangServe (port 8000, wraps a LangChain chain as REST),
+LiteLLM (port 4000), and Postgres+pgvector (port 5432) — and shows:
+
+```python
+vectorstore = PGVector(
+    connection_string="postgresql://user:pass@pgvector:5432/vectordb",
+    embedding_function=your_embeddings,
+    collection_name="documents",
+)
+```
+
+**This snippet uses the deprecated class.** `connection_string` and
+`embedding_function` are constructor args of
+`langchain_community.vectorstores.pgvector.PGVector`, the older integration
+that ships inside the general-purpose `langchain-community` package. The
+current, actively-maintained integration lives in its own package,
+`langchain-postgres`, as `langchain_postgres.vectorstores.PGVector`, with a
+different constructor:
+
+```python
+from langchain_postgres import PGVector
+from langchain_openai import OpenAIEmbeddings
+
+vector_store = PGVector(
+    embeddings=embeddings,          # not embedding_function
+    collection_name="my_docs",
+    connection="postgresql+psycopg://user:pass@host:5432/db",  # not connection_string; psycopg3 DSN
+    use_jsonb=True,
+)
+```
+
+Confirmed constructor signature (`embeddings`, `connection`, `collection_name`,
+`use_jsonb`, `create_extension`, `async_mode`) via
+[reference.langchain.com/python/langchain-postgres/vectorstores/PGVector](https://reference.langchain.com/python/langchain-postgres/vectorstores/PGVector)
+and the setup walkthrough at
+[docs.langchain.com/oss/python/integrations/vectorstores/pgvector](https://docs.langchain.com/oss/python/integrations/vectorstores/pgvector).
+Two concrete corrections to the article's shape:
+
+- **Package**: `langchain-postgres` is a separate pip install
+  (`pip install -qU langchain-postgres`), not part of `langchain_community`.
+  The docs page does not say the community class is formally deprecated, but
+  it is the older/legacy path — the actively-documented integration page for
+  Postgres+pgvector points at `langchain_postgres.vectorstores.PGVector`.
+- **Driver**: `langchain-postgres` requires the `psycopg` (psycopg3) driver
+  and a `postgresql+psycopg://` DSN, not `psycopg2` / plain `postgresql://`.
+  This repo's `pgvector-db` service doesn't care (it's just Postgres), but the
+  new service's `requirements.txt` needs `psycopg[binary]`, not
+  `psycopg2-binary` (which `vendor/litellm-pgvector/requirements.txt` uses).
+- **`create_extension`** defaults to `True` — `PGVector` runs `CREATE
+  EXTENSION IF NOT EXISTS vector` itself on first connect. Source: same API
+  reference page above. This is the same requirement `litellm-pgvector`'s
+  Prisma schema has (Postgres user needs `CREATE` privilege) — **no change in
+  privilege requirements**, just who issues the DDL.
+- The article's LangServe wrapper (`add_routes()`) is a real LangChain
+  component but adds a fourth abstraction (LangServe app → LangChain
+  chain/retriever → PGVector → Postgres) for what this stack needs from only
+  two HTTP endpoints (ingest, query). A plain FastAPI app calling `PGVector`
+  methods directly is simpler and is what's specced below (§5) — LangServe is
+  overkill for a two-endpoint internal service.
+
+There is also a v2, engine-based API, `langchain_postgres.v2.vectorstores.PGVectorStore`
+(with native async via `PGEngine`/`AsyncPGVectorStore`), documented at
+[reference.langchain.com/python/integrations/langchain_postgres](https://reference.langchain.com/python/integrations/langchain_postgres/).
+It's newer and adds first-class async, but the plain `PGVector` class is the
+one shown on LangChain's main pgvector integration page, is simpler to set up
+(no separate `PGEngine` object), and this service has no need for the async
+throughput `PGVectorStore` targets (single ingest script, occasional queries).
+**Recommendation: use `langchain_postgres.vectorstores.PGVector`, not
+`PGVectorStore`** — revisit only if query volume ever demands async
+throughput.
+
+## 2. Complexity comparison: vendor/litellm-pgvector vs. a LangChain-direct service
+
+`vendor/litellm-pgvector/` as it exists in this repo today:
+
+| File | Lines |
+|---|---|
+| `main.py` | 521 |
+| `embedding_service.py` | 89 |
+| `models.py` | 85 |
+| `config.py` | 59 |
+| `prisma/schema.prisma` | 39 |
+| **Total (Python + schema)** | **793** |
+
+Plus: a `Dockerfile` that runs `prisma generate` at build time, a
+`requirements.txt` pulling in `prisma==0.11.0` (which itself vendors a Rust
+query-engine binary download at install time — a second network dependency
+inside the Docker build, on top of the git-context build that already failed
+once per the vendoring rationale in `docker-compose.yml`/`VENDORED.md`), and
+hand-rolled raw-SQL string building for every endpoint (see `main.py` lines
+86–128, 219–320, 322–402, 405–511 — table/column names are
+f-string-interpolated from `settings.table_names`/`settings.db_fields`, not
+parameterized, though the *values* are parameterized). None of this has been
+smoke-tested against a live deploy (issue #24).
+
+A LangChain-direct replacement needs to expose exactly two HTTP endpoints
+(`POST /ingest`, `POST /query` — see §5) as a thin wrapper around library
+calls that already do the embedding + SQL work:
+
+- `langchain_postgres.PGVector.add_texts(texts, metadatas)` — ingestion,
+  replaces `main.py`'s `create_embedding`/`create_embeddings_batch` (176
+  lines) and all of `models.py`'s embedding request/response schemas.
+- `langchain_postgres.PGVector.similarity_search_with_relevance_scores(query,
+  k)` — query, replaces `search_vector_store` (100 lines) and
+  `generate_query_embedding`/`embedding_service.py` (89 lines) — LangChain's
+  `OpenAIEmbeddings` (from `langchain-openai`, pointed at
+  `base_url=http://litellm:4000/v1`) does the embedding call instead of a
+  hand-written `httpx` call to LiteLLM.
+- No `create_vector_store`/`list_vector_stores` needed — one Postgres
+  "collection" (LangChain's term, one row in its `langchain_pg_collection`
+  table) is enough for this repo's single `memory-and-notes` use case;
+  `collection_name="memory-and-notes"` passed once at construction covers it.
+- No Prisma, no generated query-engine binary, no `prisma/schema.prisma` —
+  `PGVector` creates and manages its own two tables
+  (`langchain_pg_collection`, `langchain_pg_embedding`) via SQLAlchemy on
+  first use.
+
+Estimated replacement size: **~90–120 lines of Python** across the FastAPI app
++ Pydantic request/response models (a `/ingest` and `/query` endpoint, an
+`Embeddings`-backed `PGVector` instance built once at startup, an API-key
+`Depends()` check reused almost verbatim from `vendor/litellm-pgvector/main.py`
+lines 50–55) — roughly an **85% reduction** from the vendored connector's 793
+lines, with zero Prisma/migration surface and no hand-built raw SQL (parameter
+binding, vector-string formatting, and JSONB metadata handling are all
+internal to `PGVector`, not re-implemented per endpoint as in `main.py`).
+
+**This does not eliminate the need for a small custom service** — a bare
+`PGVector.similarity_search()` call is a Python library call, not an HTTP
+endpoint, and this stack's callers (the ingest script, Claude Code hooks
+described in `docs/memory-knowledgebase.md`) need HTTP. Something still has to
+be that thin wrapper; it's just ~85% smaller and has no ORM/migration layer.
+
+## 3. Does this still count as "memory served from the gateway layer"?
+
+Yes, with the same division of responsibility issue #21 established, just a
+narrower one for the connector. LiteLLM (`litellm` container) remains the
+**only** thing every client (Open WebUI, Claude Code, anything else behind the
+proxy) talks to for chat and embedding *model* calls — nothing changes there.
+What moves is which service holds the *retrieval* logic: today it's
+`litellm-pgvector` (itself calling back into `litellm:4000` for embeddings,
+per `docker-compose.yml` lines 241–246); under this alternative it's the new
+LangChain-based service, which **also** calls back into `litellm:4000/v1` for
+embeddings (via `langchain-openai`'s `OpenAIEmbeddings(base_url="http://litellm:4000/v1")`,
+since LiteLLM exposes an OpenAI-compatible API — confirmed by this repo's
+existing `local-embedding` `model_list` entry in `litellm-config.yaml` already
+being OpenAI-route-compatible). The client-facing shape is identical: no
+client (Open WebUI, a Claude Code hook, `ingest-memory.sh`) talks to Postgres
+directly, and no client bypasses LiteLLM for the actual embedding/generation
+model calls. Only the internal retrieval connector's implementation changes,
+from a vendored third-party FastAPI+Prisma app to an in-repo FastAPI+LangChain
+app. This does not contradict #21's standing decision.
+
+One nuance: today, `vector_store_registry` makes LiteLLM itself the thing a
+chat completion's `tools: [{"type": "file_search", ...}]` call reaches (per
+`docs/research/litellm-knowledgebase.md` §1) — LiteLLM intercepts the tool
+call and proxies to the registered `pg_vector` backend. **Under the
+alternative, that specific mechanism goes away**: the new service is not a
+`vector_store_registry` provider (there is no `langchain_postgres`/custom
+provider type in LiteLLM's `vector_store_registry`, and building one would
+reintroduce exactly the custom-connector complexity this ticket is trying to
+shed). Retrieval becomes a direct call to the new service's `/query` endpoint
+(or a client-side RAG step: query the service, splice results into the prompt
+before calling `litellm`) rather than an in-band `file_search` tool call
+LiteLLM resolves itself. This is a real, if modest, capability reduction:
+`{"type": "file_search", "vector_store_ids": [...]}` inside a `/chat/completions`
+call to `litellm` stops working; callers move to a two-step "search then
+send" pattern instead. Given `vector_store_registry`'s exact field shape was
+never confirmed working (issue #24) and this is the only capability lost,
+this is judged an acceptable trade for the complexity and reliability win.
+
+## 4. Ingestion, embedding-server, pgvector-db reuse
+
+**`embedding-server`** (llama.cpp, `nomic-embed-text-v1.5`, `docker-compose.yml`
+lines 38–66): fully reusable as-is. It already serves an OpenAI-compatible
+`/v1/embeddings` route (llama.cpp's `--embeddings` flag, per
+`docs/research/litellm-knowledgebase.md` §3), and `litellm-config.yaml`
+already exposes it through LiteLLM as the `local-embedding` model
+(`api_base: http://embedding-server:8080/v1`). The new service reaches it the
+same way `litellm-pgvector` does today — indirectly, by calling
+`http://litellm:4000/v1/embeddings` with `model=local-embedding` — via
+`langchain_openai.OpenAIEmbeddings(model="local-embedding",
+base_url="http://litellm:4000/v1", api_key=<virtual key>)`. `OpenAIEmbeddings`
+is LangChain's standard `Embeddings` implementation for any OpenAI-compatible
+endpoint (no custom `Embeddings` subclass needed) — confirmed via the
+constructor example in
+[docs.langchain.com/oss/python/integrations/vectorstores/pgvector](https://docs.langchain.com/oss/python/integrations/vectorstores/pgvector),
+which pairs `OpenAIEmbeddings` with `PGVector` directly.
+
+**`scripts/ingest-memory.sh`** changes, but modestly: same per-line chunking
+of `data/memory.md`/`data/claude-legacy-memory.md` (the ponytail comment in
+the current script — one chunk per non-empty, non-heading line, no real
+chunking logic needed — still holds), but it now `curl`s the new service's
+`POST /ingest` instead of `litellm-pgvector`'s two-step
+`POST /v1/vector_stores` + `POST /v1/vector_stores/{id}/embeddings/batch`.
+Concretely: build one JSON array of `{content, metadata}` per file (same
+shape the script already builds) and POST it once to `/ingest`; the new
+service calls `PGVector.add_texts(texts, metadatas)` internally, which
+embeds and inserts in one call — no separate "create the store" step, since
+`collection_name` is fixed at service startup and `PGVector` creates the
+collection row implicitly on first `add_texts`. The no-dedup caveat in the
+current script's header comment carries over unchanged (`add_texts` always
+inserts; nothing in `PGVector` deduplicates by content).
+
+**`pgvector-db`** (`pgvector/pgvector:pg16`, `docker-compose.yml` lines
+198–214): reusable completely as-is — same image, same "separate Postgres
+instance from `litellm-db`" rationale, no compose changes needed to the
+service definition itself. Only the *schema* changes: `litellm-pgvector`'s
+Prisma schema (`vendor/litellm-pgvector/prisma/schema.prisma`, 39 lines)
+defines its own `vector_stores`/embeddings tables with configurable
+table/field names (`config.py`'s `settings.table_names`/`db_fields`), applied
+via Prisma migrations at container startup (the exact behavior of which,
+per issue #24's ponytail note, was never smoke-tested). `PGVector` instead
+auto-creates two fixed tables — `langchain_pg_collection` and
+`langchain_pg_embedding` — via SQLAlchemy `Base.metadata.create_all()`-style
+setup on first connection, no separate migration step or migration tool
+needed. This is a strictly simpler schema-management story: no Prisma CLI,
+no `prisma generate` build step, no migration files to track. The existing
+`litellm_pgvector`/`PGVECTOR_DB_PASSWORD` database and role stay — the new
+service just needs its own `DATABASE_URL` pointed at the same database (or a
+fresh one; reusing `litellm_pgvector` is simplest since no data has been
+proven-loaded into it yet per issue #24).
+
+**`litellm-pgvector` (container) and `vendor/litellm-pgvector/` (source
+tree)**: removed if this verdict is adopted. Nothing else in the repo depends
+on the vendored source once the new service replaces it.
+
+## 5. Concrete replacement spec (not implemented — spec only)
+
+**New compose service**, replacing the `litellm-pgvector` block:
+
+```yaml
+  memory-retrieval:                     # name TBD; "litellm-pgvector" freed up
+    build:
+      context: ./services/memory-retrieval    # new small in-repo dir, not vendored
+    container_name: memory-retrieval
+    depends_on:
+      pgvector-db:
+        condition: service_healthy
+      litellm:
+        condition: service_healthy
+    environment:
+      - DATABASE_URL=postgresql+psycopg://litellm_pgvector:${PGVECTOR_DB_PASSWORD}@pgvector-db:5432/litellm_pgvector
+      - LITELLM_BASE_URL=http://litellm:4000/v1
+      - LITELLM_API_KEY=${MEMORY_RETRIEVAL_EMBEDDING_KEY}   # new virtual key, same pattern as LITELLM_PGVECTOR_EMBEDDING_KEY today
+      - EMBEDDING_MODEL=local-embedding
+      - SERVER_API_KEY=${MEMORY_RETRIEVAL_API_KEY}          # replaces LITELLM_PGVECTOR_API_KEY
+      - COLLECTION_NAME=memory-and-notes
+    expose:
+      - "8000"
+    restart: unless-stopped
+    networks: [ai-stack]
+```
+
+Note `DATABASE_URL` changes scheme to `postgresql+psycopg://` (psycopg3 DSN
+`langchain-postgres` requires — see §1), not the `postgresql://` scheme
+`litellm-pgvector` uses today.
+
+**Dockerfile shape** (`services/memory-retrieval/Dockerfile`): same base image
+family as today's connector — `python:3.11-slim` — `pip install` the
+requirements, copy the app, `CMD ["uvicorn", "main:app", "--host", "0.0.0.0",
+"--port", "8000"]`. No `prisma generate` step, no `build-essential`/`postgresql-client`
+needed at the OS-package level (no Prisma binary to build against; `psycopg[binary]`
+ships prebuilt wheels).
+
+**`requirements.txt`**: `fastapi`, `uvicorn[standard]`, `langchain-postgres`,
+`langchain-openai`, `psycopg[binary]`, `pydantic` — five packages vs. the
+current ten in `vendor/litellm-pgvector/requirements.txt`, and no `prisma`.
+
+**Endpoints** (2, vs. today's 5 — create store, list stores, search, create
+embedding, create embeddings batch):
+
+- `POST /ingest` — body: `{"chunks": [{"content": "...", "metadata": {...}}]}`.
+  Calls `PGVector.add_texts(texts=[...], metadatas=[...])`. Replaces both
+  `ingest-memory.sh`'s "create store" call (no longer needed — collection is
+  implicit) and its batch-embeddings call.
+- `POST /query` — body: `{"query": "...", "k": 5}`. Calls
+  `PGVector.similarity_search_with_relevance_scores(query, k=k)`, returns
+  `{"results": [{"content": ..., "metadata": ..., "score": ...}]}`.
+- `GET /health` — same trivial liveness check the current connector has
+  (`main.py` lines 514–517), kept for compose `depends_on`/`healthcheck` use
+  if desired.
+
+Bearer-token auth via `SERVER_API_KEY`, same `Depends()` pattern as
+`vendor/litellm-pgvector/main.py` lines 47–55 — that part of the existing
+connector is fine as-is and can be copied over close to verbatim.
+
+**`litellm-config.yaml` change**: remove the `vector_store_registry` block
+entirely (lines 47–61 today) — per §3, the new service isn't a
+`vector_store_registry` provider, so there's nothing to register. The
+`local-embedding` `model_list` entry (lines 25–35) is unchanged and still
+needed — it's what the new service's `OpenAIEmbeddings` client calls through
+`litellm`.
+
+## 6. Verdict
+
+**Switch to LangChain-direct.** This does invalidate part of issue #23's
+resolution and the `litellm-pgvector` container currently in
+`docker-compose.yml` — say so explicitly for scoping a follow-up
+implementation ticket:
+
+- **Torn out**: `litellm-pgvector` service block in `docker-compose.yml`,
+  `vendor/litellm-pgvector/` (entire tree), the `vector_store_registry` block
+  in `litellm-config.yaml`, `LITELLM_PGVECTOR_API_KEY`/
+  `LITELLM_PGVECTOR_EMBEDDING_KEY` env vars (replaced by new-service
+  equivalents).
+- **Survives unchanged**: `pgvector-db` (compose block as-is), `embedding-server`
+  (compose block as-is), the `local-embedding` `model_list` entry in
+  `litellm-config.yaml`, the overall "LiteLLM does all model calls" gateway
+  boundary from issue #21.
+- **New**: a small in-repo `services/memory-retrieval/` FastAPI+LangChain
+  app (~90–120 lines, spec in §5), a new compose service block, a rewritten
+  `scripts/ingest-memory.sh` pointed at `/ingest` instead of
+  `litellm-pgvector`'s two-step API, and one new virtual key
+  (`MEMORY_RETRIEVAL_EMBEDDING_KEY`) issued the same way
+  `LITELLM_PGVECTOR_EMBEDDING_KEY` was per `docs/proxy-key-onboarding.md`.
+- **Capability trade-off** (§3): the OpenAI Assistants-style `file_search`
+  tool call directly against `litellm`'s `/chat/completions` stops working
+  (no `vector_store_registry` entry to resolve it); callers move to querying
+  the new service directly and splicing results into the prompt themselves.
+  This mechanism was never confirmed working in the first place (issue #24),
+  so it's a loss of unverified capability, not working functionality.
+
+This is not implemented here — this document is research/spec only, per the
+ticket's scope. A follow-up implementation ticket should: write
+`services/memory-retrieval/`, update `docker-compose.yml` and
+`litellm-config.yaml` per §5, rewrite `scripts/ingest-memory.sh`, delete
+`vendor/litellm-pgvector/`, and update `docs/memory-knowledgebase.md`.
+
+## 7. Repo context read for this research
+
+- `docker-compose.yml` — `embedding-server`, `pgvector-db`, `litellm-pgvector`
+  service definitions (build context, env vars, network).
+- `litellm-config.yaml` — `local-embedding` model entry, `vector_store_registry`
+  block.
+- `docs/research/litellm-knowledgebase.md` — prior research establishing
+  LiteLLM's native vector-store feature has no Qdrant provider and that
+  `litellm-pgvector` was the only self-hosted path found at the time; its §1,
+  §3, §4 are cited above for the `file_search` tool-call mechanism and
+  embedding-model wiring.
+- `docs/memory-knowledgebase.md` — current user-facing setup doc for the
+  knowledgebase/memory feature.
+- `scripts/ingest-memory.sh` — current ingestion mechanism and its no-dedup
+  caveat.
+- `vendor/litellm-pgvector/{main.py,embedding_service.py,models.py,config.py,
+  prisma/schema.prisma,requirements.txt,Dockerfile}` — read in full to
+  characterize size (793 total lines Python+schema) and shape (raw-SQL
+  f-string-built queries per endpoint, Prisma-managed schema, calls back to
+  LiteLLM's `/embeddings` for query embedding).
+- Gitea issue #25 (this ticket) and its parent #21 — standing decision that
+  memory is served from the gateway layer, not a client.
+- Gitea issue #23/#24 (referenced by #25) — original `litellm-pgvector`
+  adoption and its unverified-against-live-deploy caveat.
+
+## 8. Primary sources consulted
+
+- [markaicode.com/stack/litellm-langchain-stack/](https://markaicode.com/stack/litellm-langchain-stack/) — the alternative surfaced by the user; verified and corrected above (§1).
+- [reference.langchain.com/python/langchain-postgres/vectorstores/PGVector](https://reference.langchain.com/python/langchain-postgres/vectorstores/PGVector) — `PGVector` constructor signature, `create_extension` default.
+- [docs.langchain.com/oss/python/integrations/vectorstores/pgvector](https://docs.langchain.com/oss/python/integrations/vectorstores/pgvector) — install instructions, `postgresql+psycopg://` DSN requirement, `OpenAIEmbeddings` + `PGVector` pairing example.
+- [github.com/langchain-ai/langchain-postgres](https://github.com/langchain-ai/langchain-postgres) — package README; async-function-pairing note; `PGVector` deprecated in v0.0.14+ in favor of `PGVectorStore` (v2 API).
+- [reference.langchain.com/python/integrations/langchain_postgres/](https://reference.langchain.com/python/integrations/langchain_postgres/) — v2 `PGVectorStore`/`PGEngine`/`AsyncPGVectorStore` API, compared against `PGVector` in §1.
+- `vendor/litellm-pgvector/README.md` and source files (this repo) — connector's existing endpoint/config shape, cited for the LOC comparison in §2.

docs/research/lazytainer-omniroute-idle-stop.md

@@ -0,0 +1,194 @@
+# Research: Why lazytainer's idle-stop on llama-server doesn't fire, and what switch-model.sh should do about it
+
+**Question:** lazytainer is configured on `llama-server` (`docker-compose.yml`
+`lazytainer.group.llamaserver.*` labels) but its idle-stop never triggers in
+practice — OmniRoute appears to keep the container looking "active" to
+lazytainer's packet-threshold detector. Confirm the mechanism, find root
+cause, and recommend how the future `scripts/switch-model.sh` (#43, blocked)
+should handle GPU-residency swaps between `llama-server` and a new `comfyui`
+service given this.
+
+**Answer:** Confirmed. lazytainer's detector is a dumb per-port packet
+counter with no traffic classification — it cannot tell OmniRoute's
+background provider health-check pings apart from real inference traffic,
+and there is no config knob in lazytainer or a per-provider one in OmniRoute
+that fixes this. **`switch-model.sh` should bypass lazytainer entirely** for
+the swap: drive `docker compose stop`/`up -d` directly on both services,
+rather than trying to make lazytainer's idle-stop cooperate.
+
+## Current config (`docker-compose.yml`)
+
+```yaml
+labels:
+  - "lazytainer.group.llamaserver.sleepMethod=stop"
+  - "lazytainer.group.llamaserver.ports=8080"
+  - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
+  - "lazytainer.group.llamaserver.minPacketThreshold=2"
+```
+
+`ports=8080` matches the container's real internal port (`expose: ["8080"]`,
+confirmed in the same file) — not a misconfiguration. `minPacketThreshold=2`
+is already far *below* lazytainer's own documented default of `30`, i.e. this
+deployment already tried loosening the threshold to make idle-stop easier to
+reach, not harder.
+
+## How lazytainer's detector actually works (primary source: `vmorganp/Lazytainer`)
+
+Confirmed against the project's README and Go source
+(`src/group.go`) on [github.com/vmorganp/Lazytainer](https://github.com/vmorganp/Lazytainer):
+
+- It captures packets with **gopacket/libpcap directly on the configured
+  `netInterface`** (default `eth0`), applying a BPF filter built from the
+  group's `ports` list (`"port 8080"` here, per the source's filter-string
+  construction, e.g. `"port 80 or port 81 or etc."` in the general case).
+- The filter matches **every packet to or from the port** — SYN, ACK,
+  data, FIN, everything. It is not restricted to new-connection SYNs.
+- Every `pollRate` seconds (default `30`; not overridden in this repo's
+  config) it samples a rolling packet counter (`rxHistory`) and compares the
+  delta against `minPacketThreshold`:
+  `rxHistory[0]+minPacketThreshold > rxHistory[len(rxHistory)-1]` → treated as
+  active, `inactiveSeconds` resets to 0.
+- `ignoreActiveClients` (default `false`, not set here) only changes whether
+  an ESTABLISHED-connection count is also checked; it does not add any
+  content- or source-based filtering.
+- **There is no mechanism anywhere in lazytainer to exclude specific traffic
+  (by source IP, path, header, or request type) from the packet count.** The
+  README's config table (`ports`, `inactiveTimeout`, `minPacketThreshold`,
+  `ignoreActiveClients`, `pollRate`, `sleepMethod`, `netInterface`) is
+  exhaustive — nothing else exists to tune this per-caller.
+
+Consequence: a single TCP connection to port 8080 — a bare connect + one
+small HTTP exchange + close — already produces well over `minPacketThreshold=2`
+packets purely from the handshake and teardown (SYN, SYN-ACK, ACK, ..., FIN,
+ACK), regardless of payload size or purpose. At this threshold, essentially
+*any* connection to the port counts as "active" and resets `inactiveTimeout`.
+Raising the threshold wouldn't help either — the fix would need to be
+"ignore packets from OmniRoute's health-checker," which the tool has no way
+to express; it only counts packets on a port, source-blind.
+
+## How OmniRoute actually touches registered providers (primary source: `diegosouzapw/OmniRoute`)
+
+Confirmed against
+[`docs/reference/ENVIRONMENT.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/ENVIRONMENT.md)
+in the OmniRoute repo:
+
+- OmniRoute runs a **background credential/connection health-check
+  scheduler** (`src/lib/credentialHealth/scheduler.ts`) on
+  `CREDENTIAL_HEALTH_CHECK_INTERVAL`, default `300000` ms (5 min), minimum
+  `10000` ms (10s) — this periodically re-tests each registered provider's
+  connection, which for a provider like `llama-server` (a plain HTTP base
+  URL, no API key) means an actual request/connection to
+  `llama-server:8080`.
+- Results are cached for `CREDENTIAL_HEALTH_CACHE_TTL` (default also 5 min).
+- **Only one exclusion exists, and it's hardcoded by provider category, not
+  configurable per-provider**: search providers
+  (`SEARCH_VALIDATOR_CONFIGS` in
+  `src/lib/providers/validation/searchProviders.ts`, e.g. `tavily-search`)
+  are permanently skipped because their validation call is a real billed
+  upstream query. `llama-server` is an inference provider, not a search
+  provider — it is not in this exclusion list.
+- The only toggle that actually stops the sweep is global:
+  `OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK=1`/`true`, which "disable[s]
+  background periodic testing of provider connections" for **every**
+  registered provider at once. There is no documented per-provider
+  disable/pause flag in
+  [`docs/reference/PROVIDER_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md) —
+  the dashboard's `/dashboard/providers` page is described only as where you
+  "enable, configure, and test each provider," with no documented
+  independent "pause health checks for this one provider" control.
+
+So: OmniRoute is not the sole cause, but it is a live, recurring cause. Every
+5 minutes (at most — could also be triggered ad hoc by dashboard/API use) it
+opens a connection to `llama-server:8080` purely to check the provider is
+alive, which is exactly the kind of traffic lazytainer's port-level counter
+cannot distinguish from real inference calls. With `inactiveTimeout=900`
+(15 min) and a health-check every ≤300s, the container practically always
+sees qualifying traffic before its idle timer would expire.
+
+## Root cause
+
+Two independent, both-true facts combine to defeat idle-stop:
+
+1. **lazytainer's detector is fundamentally traffic-blind** — it counts raw
+   packets on a port with no way to exclude any specific caller or traffic
+   class. This is a property of the tool, not a misconfiguration in this
+   repo (`ports=8080` is correct; `minPacketThreshold=2` is already at the
+   permissive end).
+2. **OmniRoute periodically pings every registered non-search provider**
+   (default every ≤5 min) to keep its health/availability status current,
+   and that ping is indistinguishable, at the packet level, from a real
+   inference request.
+
+Neither side offers a targeted fix: lazytainer has no allowlist/denylist by
+source, and OmniRoute's only "stop pinging" lever
+(`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK`) is all-or-nothing across every
+provider, not scoped to just `llama-server`. Tuning `minPacketThreshold`
+higher or lower doesn't change the outcome either way, since the health-check
+traffic and real traffic land on the exact same port with no distinguishing
+packet-level signature.
+
+## Recommendation for `scripts/switch-model.sh` (#43)
+
+**Bypass lazytainer entirely for the GPU-residency swap.** Drive both
+services directly:
+
+```bash
+docker compose stop llama-server
+docker compose up -d comfyui
+# ...and the reverse when swapping back
+```
+
+Justification:
+
+- The swap is a **deliberate, scripted, known-in-advance** event — the
+  script always knows exactly which service should go up and which should
+  go down. Idle-stop detection exists to handle the case where nobody knows
+  when a service last had traffic; that's not this case, so routing the
+  swap through a passive heuristic (lazytainer's idle timer) that this
+  research shows is already unreliable for `llama-server` adds a point of
+  failure for no benefit. Direct `docker compose stop`/`up -d` is
+  deterministic and immune to the packet-counting confound described above.
+- Reconfiguring lazytainer's thresholds was considered and rejected: no
+  threshold value fixes a detector that cannot distinguish OmniRoute's
+  keepalive traffic from real traffic on the same port (see Root cause).
+  This is a ceiling in the tool itself, not a tuning problem.
+- Pausing OmniRoute's polling for the swap window was also considered.
+  It's the one lever available (`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK`),
+  but it is global — it would blind OmniRoute's health status for *every*
+  provider (including `searxng-search`, if registered) for the duration of
+  the swap, and adds an extra env-toggle-and-restart step to the script for
+  a problem that direct compose control sidesteps completely. It's worth
+  flagging for #43's implementation as a *secondary* safety measure — briefly
+  disabling the sweep (or accepting that OmniRoute may show `llama-server` as
+  errored/offline for up to `CREDENTIAL_HEALTH_CHECK_INTERVAL` after it's
+  stopped) — but it should not be the primary mechanism the swap relies on.
+- This does **not** require removing the existing `lazytainer.group.llamaserver.*`
+  labels — they can stay for whatever idle-stop benefit they still provide
+  between swaps (e.g. genuinely idle periods where nothing, including
+  OmniRoute, has recently touched the container long enough to matter) while
+  `switch-model.sh` simply never depends on lazytainer to do the actual
+  stop/start for a swap.
+
+## Bottom line for #43 (blocked ticket, once unblocked)
+
+- `switch-model.sh` should call `docker compose stop <from-service>` /
+  `docker compose up -d <to-service>` directly — never rely on lazytainer's
+  idle-stop to free the GPU as part of a swap.
+- No lazytainer config change (threshold, ports, poll rate) is a viable fix;
+  the detector has no way to exclude OmniRoute's traffic by source.
+- Optionally, as a secondary hygiene step, the script may toggle
+  `OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK` around the swap (or simply
+  tolerate a stale "errored" status in OmniRoute's dashboard for up to one
+  `CREDENTIAL_HEALTH_CHECK_INTERVAL`) to avoid OmniRoute flagging the
+  just-stopped provider as failed mid-swap — but this is cosmetic/status
+  hygiene, not what makes the swap itself work.
+
+Sources: [`vmorganp/Lazytainer`](https://github.com/vmorganp/Lazytainer)
+(README config table; `src/group.go` packet-capture and threshold-comparison
+logic), [`diegosouzapw/OmniRoute` —
+`docs/reference/ENVIRONMENT.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/ENVIRONMENT.md)
+(credential health-check scheduler env vars), [`diegosouzapw/OmniRoute` —
+`docs/reference/PROVIDER_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md)
+(provider dashboard controls), this repo's `docker-compose.yml`
+(`lazytainer.group.llamaserver.*` labels, `llama-server`/`omniroute` service
+definitions).

docs/research/litellm-knowledgebase.md

@@ -0,0 +1,305 @@
+# Research: LiteLLM vector store / knowledgebase feature (for Gitea issue #23)
+
+**Question:** Can LiteLLM's built-in "knowledgebase" / vector store feature be
+used to load two Markdown fact files (`data/memory.md`,
+`data/claude-legacy-memory.md`) into a queryable knowledge base, and if so,
+what does that require from this stack (config shape, backend, embedding
+model, ingestion mechanism)?
+
+**Bottom line: no, not against this stack's existing Qdrant instance, and not
+without adding a dedicated embedding model.** LiteLLM's vector store feature
+is a *routing/registry* layer over a small set of natively-integrated
+backends (Bedrock Knowledge Bases, OpenAI Vector Stores, Azure
+Vector/AI-Search, Vertex AI RAG/Search, Gemini File Search, RAGFlow) plus one
+self-hosted option — Postgres+pgvector, via a **separate companion service**
+(`BerriAI/litellm-pgvector`), not Qdrant. There is no Qdrant provider at all.
+Ingesting the two fact files would also require running a dedicated
+embedding-capable model, which this stack doesn't currently have (the one
+llama.cpp instance serves a chat model, not started with `--embeddings`).
+
+## 1. Config shape in `config.yaml`
+
+Top-level block is **`vector_store_registry`** (this is the current key name;
+an earlier PR that introduced the feature used `vector_stores` — see
+Provenance note below), a list of entries:
+
+```yaml
+vector_store_registry:
+  - vector_store_name: "my-knowledgebase"      # optional friendly name
+    litellm_params:
+      vector_store_id: "T37J8R4WTM"             # required, provider's own ID
+      custom_llm_provider: "bedrock"            # required — selects backend
+      vector_store_description: "..."           # optional
+      vector_store_metadata: {}                 # optional
+      litellm_credential_name: "..."            # optional, reuse a named credential
+      embedding_model: "..."                    # backend-dependent — see §3
+```
+
+Source: [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase)
+(fetched directly). Earlier shape (same idea, older field names) documented
+in [BerriAI/litellm PR #10448](https://github.com/BerriAI/litellm/pull/10448)
+("[Feat] Vector Stores/KnowledgeBases - Allow defining Vector Store Configs"):
+
+```yaml
+vector_stores:
+  - vector_store_name: "bedrock-litellm-website-knowledgebase"
+    litellm_params:
+      custom_llm_provider: "bedrock"
+      id: "T37J8R4WTM"
+```
+
+This is a **registry of pointers to knowledge bases that already exist on
+the backend provider** — it is not itself a vector database. `model_list`
+entries are not where a store is attached; `vector_store_registry` is its
+own top-level sibling of `model_list`.
+
+### Referencing a vector store from a request
+
+Not via a `model_list` entry — via the **`tools`** array of a
+`/chat/completions` (or `/v1/responses`) request, OpenAI Assistants-style
+`file_search` tool:
+
+```json
+{
+  "model": "gpt-4o",
+  "messages": [...],
+  "tools": [
+    {
+      "type": "file_search",
+      "vector_store_ids": ["T37J8R4WTM"]
+    }
+  ]
+}
+```
+
+LiteLLM intercepts the tool call, looks up the referenced ID in
+`vector_store_registry`, calls that backend's native search, and injects the
+retrieved chunks into the prompt before calling the target model. Source:
+[docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
+
+There are also direct, OpenAI-compatible proxy endpoints for managing/using a
+store outside of a chat completion — see §4.
+
+## 2. Backend providers — Qdrant is NOT one of them
+
+Per the official docs page and the PR that introduced the feature, the
+natively-supported `custom_llm_provider` values are:
+
+| Provider | `custom_llm_provider` value | Notes |
+|---|---|---|
+| AWS Bedrock Knowledge Bases | `bedrock` | Original/reference implementation ([PR #10448](https://github.com/BerriAI/litellm/pull/10448)) |
+| OpenAI Vector Stores | `openai` | Wraps OpenAI's own vector store API |
+| Azure Vector Stores | `azure` | Assistants-API-only per docs |
+| Azure AI Search | `azure_ai_search` (vector search capability) | |
+| Vertex AI RAG Engine | `vertex_ai` | Added in [PR #15781](https://github.com/BerriAI/litellm/pull/15781) |
+| Vertex AI Search API | `vertex_ai/search_api` | |
+| Gemini File Search | — | |
+| RAGFlow Datasets | — | Dataset mgmt only; search unsupported per docs |
+| Postgres + pgvector | `pg_vector` (via a **separate** connector service) | See below — not built into the litellm proxy image |
+
+Source: [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase),
+cross-checked against [PR #12595](https://github.com/BerriAI/litellm/pull/12595)
+("[Feat] Vector Stores - Add Vertex RAG Engine API as a provider") and
+[PR #15781](https://github.com/BerriAI/litellm/pull/15781) ("(feat) Vector
+Stores: support Vertex AI Search API").
+
+**Qdrant is not listed anywhere** in the knowledgebase docs, the vector
+store provider PRs, or the pgvector connector's own README. LiteLLM does use
+Qdrant in one unrelated feature — **semantic response caching**
+(`docs.litellm.ai/docs/caching/all_caches`, `qdrant_api_base` /
+`qdrant_api_key` / `qdrant_collection_name` config) — but that is a cache for
+LLM *responses*, not the vector-store/knowledgebase (RAG) feature, and shares
+no config or code path with `vector_store_registry`. It would not let
+LiteLLM search Qdrant-held documents as a knowledge base.
+
+### The pgvector option is a separate microservice, not a built-in backend
+
+[`BerriAI/litellm-pgvector`](https://github.com/BerriAI/litellm-pgvector) is
+its own repo/container: a FastAPI app that exposes OpenAI-compatible vector
+store endpoints backed by Postgres with the `pgvector` extension, calling
+back out to a LiteLLM proxy's `/embeddings` endpoint to generate embeddings.
+It is registered into `vector_store_registry` like any other backend (with
+`custom_llm_provider: pg_vector` pointed at this companion service's URL),
+but it is **not Qdrant** and **not part of the main `litellm` proxy image**
+already running in this stack — it would mean deploying and operating a
+fourth service (on top of `litellm`, `litellm-db`, and `llama-server`), with
+its own Postgres database using the pgvector extension (the existing
+`litellm-db` Postgres image, `postgres:16-alpine`, does not have pgvector
+installed).
+
+### Verdict on the existing Qdrant instance
+
+**LiteLLM's knowledgebase/vector_store feature cannot point at this repo's
+existing standalone `qdrant` service.** There is no Qdrant provider type for
+`vector_store_registry`. To use LiteLLM's native feature at all, this stack
+would need to either integrate with a cloud-native backend (Bedrock/Vertex/
+Azure/OpenAI — none of which apply, this stack is local-only) or stand up
+the separate `litellm-pgvector` + pgvector-enabled Postgres stack — an
+entirely different vector store technology from the Qdrant already running
+for Open WebUI. The existing Qdrant collection Open WebUI uses for its own
+RAG/Memory feature is unrelated to and unreachable from LiteLLM's
+knowledgebase feature.
+
+## 3. Embedding model requirement
+
+LiteLLM's vector store feature **does call an embedding endpoint itself**
+when a backend needs one (pgvector explicitly; the managed cloud backends
+handle embedding server-side). The field is `embedding_model` inside a
+`vector_store_registry` entry's `litellm_params`, confirmed in
+[BerriAI/litellm issue #23980](https://github.com/BerriAI/litellm/issues/23980)
+("[Bug]: Vector store creation fails when using model mapping public model
+name for embedding_model"), which shows:
+
+```json
+"litellm_params": {
+  "vector_bucket_name": "my-embeddings",
+  "index_name": "test-index",
+  "aws_region_name": "us-east-1",
+  "embedding_model": "test-vector-store/bedrock/amazon.nova-2-multimodal-embeddings-v1:0"
+}
+```
+
+For the pgvector connector specifically, the embedding model is configured
+via its own env vars (not `vector_store_registry`, since it's a separate
+service): `EMBEDDING__MODEL`, `EMBEDDING__BASE_URL` (pointed at a LiteLLM
+proxy), `EMBEDDING__API_KEY`, `EMBEDDING__DIMENSIONS` — e.g.
+`EMBEDDING__MODEL=text-embedding-ada-002`,
+`EMBEDDING__BASE_URL=http://litellm:4000`. Source:
+[BerriAI/litellm-pgvector README](https://github.com/BerriAI/litellm-pgvector/blob/main/README.md).
+This confirms: **yes, the embedding model must be reachable as a model
+LiteLLM's proxy can call** — i.e. it needs its own `model_list` entry with
+`mode: embedding` (LiteLLM's standard way of declaring an embedding-capable
+model — see [docs.litellm.ai/docs/embedding/supported_embedding](https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/embedding/supported_embedding.md)),
+so the pgvector service (or LiteLLM itself for backends that embed
+internally) can call `POST /embeddings` against it through the proxy.
+
+### Does llama.cpp (this repo's model server) support `/embeddings`?
+
+Yes, but not enabled as currently configured, and not well-suited to the
+model already loaded. llama.cpp's server (`tools/server`, the same
+`ghcr.io/ggml-org/llama.cpp:server-rocm` image this repo uses per
+`docker-compose.yml`) exposes an OpenAI-compatible `POST /v1/embeddings`
+route (and a native `/embedding` route), **but only when started with the
+`--embeddings` flag** — source:
+[ggml-org/llama.cpp tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
+This repo's `llama-server` command in `docker-compose.yml` (lines 16–22) does
+not pass `--embeddings`, so the currently-running instance does not serve
+embeddings at all.
+
+Even if the flag were added, llama.cpp loads **one model per server
+process** — the flag would make the already-loaded Qwen3.8-27B **chat**
+model (per `litellm-config.yaml`, `qwen3.8-27b-local` /
+`Qwen3.8-27B-UD-Q4_K_XL.gguf`) emit pooled hidden-state vectors as
+"embeddings," but a generalist instruction-tuned chat model is not what
+that's trained for — embedding quality from a non-embedding-trained model is
+materially worse than a purpose-trained embedding model (e.g. BGE, Nomic
+Embed, mxbai-embed, gte). **A separate, dedicated embedding model/server
+would be needed** — either a second `llama-server` container loaded with a
+small GGUF embedding model (`--embeddings` flag on), or a different
+embedding-serving stack — and it would need its own `model_list` entry in
+`litellm-config.yaml` with `mode: embedding` for LiteLLM/litellm-pgvector to
+call it.
+
+## 4. How documents actually get ingested
+
+Two ingestion paths exist depending on backend, both are HTTP APIs — there
+is no admin-UI "add document" flow beyond store *creation*, and no CLI:
+
+**a) OpenAI-compatible vector-store file API** (used for the `openai`
+backend, and shown as the general pattern in the docs):
+
+- `POST /v1/vector_stores` — create a store. Body:
+  ```json
+  {
+    "name": "My Document Store",
+    "file_ids": ["file-abc123"],
+    "chunking_strategy": {
+      "type": "static",
+      "static": {"max_chunk_size_tokens": 800, "chunk_overlap_tokens": 400}
+    },
+    "metadata": {"key": "value"}
+  }
+  ```
+  Requires files already uploaded through a separate Files API to obtain
+  `file_id`s first — the docs page does not show a files-upload endpoint
+  under the vector-store docs directly (this is OpenAI's own two-step
+  upload-then-attach flow, proxied through). Source:
+  [docs.litellm.ai/docs/vector_stores/create](https://docs.litellm.ai/docs/vector_stores/create).
+- `POST /v1/vector_stores/{vector_store_id}/search` — query it:
+  ```json
+  {
+    "query": "What is the capital of France?",
+    "filters": {"file_ids": ["file-abc123"]},
+    "max_num_results": 5,
+    "ranking_options": {"score_threshold": 0.7},
+    "rewrite_query": true
+  }
+  ```
+  Source: [docs.litellm.ai/docs/vector_stores/search](https://docs.litellm.ai/docs/vector_stores/search).
+- `GET /vector_store/list` — list registered stores. Source:
+  [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
+- LiteLLM Admin UI: **Experimental → Vector Stores → Create Vector Store**
+  exists for registering/creating stores, and the **Logs** page shows
+  vector-store search queries/scores after use — but this is store
+  management and observability, not a bulk document-ingestion UI. Source:
+  [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
+
+**b) `litellm-pgvector` connector's own embeddings API** (only path relevant
+if this repo went the self-hosted pgvector route, since Qdrant isn't
+supported at all): direct chunk-level ingestion, no file upload step —
+
+- `POST /v1/vector_stores/{id}/embeddings` — single chunk:
+  ```json
+  {"content": "...", "embedding": [/* optional, else computed server-side */], "metadata": {}}
+  ```
+- `POST /v1/vector_stores/{id}/embeddings/batch` — array of the same shape,
+  for bulk loading.
+
+Source: [BerriAI/litellm-pgvector README](https://github.com/BerriAI/litellm-pgvector/blob/main/README.md).
+
+For a follow-up ticket that wants to programmatically load
+`data/memory.md` / `data/claude-legacy-memory.md` (dated fact lists) into a
+knowledge base, the realistic path through LiteLLM's native feature would be
+the pgvector connector's batch-embeddings endpoint (chunk the Markdown into
+facts/sections client-side, POST each as a batch) — **but that requires
+first standing up**: (1) a pgvector-enabled Postgres, (2) the
+`litellm-pgvector` service, and (3) a dedicated embedding model server
+registered in `litellm-config.yaml`. None of that reuses the Qdrant instance
+already running in this stack.
+
+## 5. Repo context read for this research
+
+- `litellm-config.yaml` — current config has one `model_list` entry
+  (`qwen3.8-27b-local`, chat-only, via llama.cpp), `router_settings`
+  (priority scheduling), `general_settings.master_key`. No
+  `vector_store_registry` block exists yet.
+- `docker-compose.yml` — confirms `qdrant` service (image `qdrant/qdrant`,
+  network `ai-stack`, no host port, only `open-webui` currently consumes it
+  via `VECTOR_DB=qdrant` / `QDRANT_URI=http://qdrant:6333`); confirms
+  `llama-server` command has no `--embeddings` flag and loads a single GGUF
+  (`Qwen3.8-27B-UD-Q4_K_XL.gguf`); confirms `litellm-db` is plain
+  `postgres:16-alpine` (no pgvector extension installed).
+- `CLAUDE.md` — points to `docs/agents/issue-tracker.md` (Gitea issues via
+  `tea`) and `docs/agents/domain.md` (domain docs convention).
+- `docs/agents/domain.md` — says to check `CONTEXT.md` and `docs/adr/` at
+  repo root before exploring, but "if any of these files don't exist,
+  proceed silently." Neither `CONTEXT.md` nor `docs/adr/` exist in this repo
+  yet, so there's no glossary/ADR conflict to flag.
+- `docs/research/` — repeated existing precedent (e.g.
+  `docs/research/proxy-shadow-pricing.md`, `docs/research/voidllm-evaluation.md`,
+  `docs/research/qwen3.8-27b-tool-calling.md`) confirms this is the
+  established location and Markdown format for this kind of investigation;
+  this file follows that convention.
+
+## Provenance note on the top-level config key name
+
+The docs page fetched live (`docs.litellm.ai/docs/completion/knowledgebase`)
+shows `vector_store_registry` as the current top-level key. The original
+feature PR ([#10448](https://github.com/BerriAI/litellm/pull/10448)) used
+`vector_stores` in its example YAML. If implementing against a specific
+pinned LiteLLM version, verify the exact key against that version's docs/
+source rather than assuming either name — this repo's `docker-compose.yml`
+pins `ghcr.io/berriai/litellm:main-stable`, a rolling tag, so the schema in
+whatever image is actually pulled should be spot-checked (e.g. `GET /openapi.json`
+against the running proxy, or grepping the image's installed
+`litellm/types/router.py` / proxy schema) before writing config against it.

docs/research/litellm-searxng-search.md

@@ -0,0 +1,260 @@
+# Research: Wiring the local SearXNG instance into LiteLLM's web-search feature
+
+**Question:** How does LiteLLM's web-search integration
+(https://docs.litellm.ai/docs/search) actually work, and what does wiring the
+local SearXNG instance (`http://search.home/`) into this repo's
+`litellm-config.yaml` require?
+
+**Answer, short version:** SearXNG **is** a natively supported provider for
+LiteLLM's `/v1/search` feature — no custom-endpoint workaround needed. But the
+feature is **not** a model-callable tool and **not** automatic
+context-injection into chat completions either — it's a **separate REST API**
+(`/v1/search/{search_tool_name}`) that a caller (Open WebUI, a script, a
+future MCP wrapper) must invoke directly, independent of any LLM call. That
+sidesteps this project's known-flaky Qwen3.8-27B tool-calling entirely, as
+long as nothing wraps the endpoint back into model-driven tool-calling.
+Reachability is the real blocker: `search.home` is a LAN mDNS/local-DNS name
+that the `litellm` container cannot resolve by default — needs an
+`extra_hosts` entry in `docker-compose.yml`.
+
+## 1. What LiteLLM's search feature actually is
+
+LiteLLM ships a **unified search API** (`/v1/search` and
+`/v1/search/{search_tool_name}`) that wraps multiple search-provider backends
+behind one Perplexity-compatible request/response shape.
+Source: https://docs.litellm.ai/docs/search
+
+Config shape in `config.yaml` (LiteLLM's own documented example, Perplexity
+shown, same shape for every provider):
+
+```yaml
+search_tools:
+  - search_tool_name: perplexity-search
+    litellm_params:
+      search_provider: perplexity
+      api_key: os.environ/PERPLEXITYAI_API_KEY
+```
+
+Call shape:
+
+```bash
+curl http://0.0.0.0:4000/v1/search/searxng-search \
+  -H "Authorization: Bearer sk-1234" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "latest AI developments", "max_results": 5}'
+```
+
+Source: https://docs.litellm.ai/docs/search
+
+**18 providers are listed as supported**, including Perplexity, Tavily, Exa
+AI, Brave, Parallel AI, Google PSE, DataForSEO, Firecrawl, **SearXNG**,
+Linkup, Serper, DuckDuckGo, SearchAPI.io, You.com, APISerpent, Bedrock
+AgentCore, Nimble, and Bing Grounding.
+Source: https://docs.litellm.ai/docs/search
+
+## 2. Is SearXNG natively supported? — Yes
+
+SearXNG is one of the 18 built-in `search_provider` values, added by
+BerriAI/litellm PR #16259 ("[Feat] add serxng search API provider").
+Source: https://github.com/BerriAI/litellm/pull/16259
+
+Config shape for SearXNG specifically:
+
+```yaml
+search_tools:
+  - search_tool_name: searxng-search
+    litellm_params:
+      search_provider: searxng
+      api_base: https://your-searxng-instance.com
+```
+
+Equivalently, the base URL can be supplied via the `SEARXNG_API_BASE`
+environment variable instead of an inline `api_base` key — SearXNG has no API
+key of its own (it's an unauthenticated local meta-search engine), so this is
+the one provider in the list where `litellm_params` doesn't need a secret.
+Sources: https://github.com/BerriAI/litellm/pull/16259,
+https://docs.litellm.ai/docs/search
+
+**For this repo**, the addition to `litellm-config.yaml` (research only — not
+applied here) would look like:
+
+```yaml
+search_tools:
+  - search_tool_name: searxng-search
+    litellm_params:
+      search_provider: searxng
+      api_base: http://search.home/
+```
+
+No custom-endpoint or "generic OpenAI-compatible /v1/web_search" fallback is
+needed — the concern in the ticket that LiteLLM's docs "may assume a hosted
+provider like Tavily/Serper" turned out not to apply; SearXNG is a first-class
+`search_provider` value, same shape as every hosted one.
+
+## 3. Tool-call vs. automatic injection vs. a third thing
+
+The ticket asked to determine whether this is (a) a tool the model must
+explicitly call, or (b) automatic pre-retrieval/context-injection like
+Perplexity's own search-augmented answers. **It's neither** — it's a
+**standalone REST endpoint** that sits alongside `/v1/chat/completions`, not
+wired into it:
+
+> The documentation indicates this is a separate REST endpoint the
+> application calls directly. The page presents `/search` as a standalone API
+> endpoint alongside chat completions, not as an automatic injection feature.
+> Users explicitly invoke the search endpoint; LiteLLM does not automatically
+> inject search results into completions.
+
+Source: https://docs.litellm.ai/docs/search (fetched content, describing the
+`/v1/search` and `/v1/search/{search_tool_name}` endpoints as siblings of
+`/v1/chat/completions`, not a chat-completion parameter or automatic
+retrieval step)
+
+Practical effect: whatever calls this endpoint — Open WebUI's own web-search
+feature, a shell script, a future MCP server — does so with a plain HTTP call.
+**LiteLLM's model routing and Qwen3.8-27B's tool-calling reliability are not
+in that path at all**, unless something downstream chooses to expose this
+endpoint back to the model *as* a function-calling tool (e.g. an MCP wrapper
+that hands the model a `web_search` tool definition backed by this endpoint —
+that would reintroduce the model-must-emit-a-correct-tool-call problem, but
+that's a choice made one layer up, not something LiteLLM's `/v1/search`
+feature forces).
+
+## 4. Network reachability: `search.home` from inside the `litellm` container
+
+`docker-compose.yml`'s `litellm` service joins only the `ai-stack` bridge
+network (`networks: [ai-stack]`, line 116) and gets DNS resolution from
+Docker's embedded DNS server for that network — which resolves other
+containers by service/container name (`llama-server`, `qdrant`, etc., as
+already used at `api_base: http://llama-server:8080/v1` in
+`litellm-config.yaml` line 8) but has **no visibility into the LAN's mDNS/
+local-DNS namespace** that resolves `search.home` on the host machine or on
+LAN clients. So `http://search.home/` will not resolve from inside the
+`litellm` container as configured today — this matches the ticket's
+suspicion, and is standard Docker bridge-networking behavior, not specific to
+this repo.
+Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` (litellm service, lines
+94–124; `networks:` block, lines 154–156)
+
+No `extra_hosts`, `host.docker.internal`, or `network_mode: host` pattern
+exists yet anywhere in this compose file to crib from — this would be the
+first. (One service, `lazytainer`, already uses `network_mode: host`, but for
+an unrelated reason — Docker-socket/host-port introspection — and switching
+`litellm` to host networking would be a much bigger blast-radius change than
+this ticket needs, dropping the `ai-stack` network isolation for every other
+port `litellm` exposes.)
+Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` lines 144–152
+
+**Recommendation: `extra_hosts` on the `litellm` service**, mapping
+`search.home` to its LAN IP, e.g.:
+
+```yaml
+litellm:
+  ...
+  extra_hosts:
+    - "search.home:192.0.2.10"   # replace with SearXNG's actual LAN IP
+```
+
+This is the smallest, most local fix: one line, scoped to the one service
+that needs it, no change to network topology or isolation, and it keeps
+`litellm-config.yaml`'s `api_base: http://search.home/` value human-readable
+(matching this repo's existing preference for symbolic hostnames like
+`ai.home` / `proxy.ai.home` documented in `docs/network-access.md`) rather
+than hardcoding the LAN IP directly into the YAML config. The IP needs to stay
+in sync if SearXNG's host ever gets a new DHCP lease — same caveat that would
+apply to any hardcoded-IP alternative, just isolated to one `extra_hosts`
+line instead of buried in the search config.
+
+`host.docker.internal` is not applicable here: that special hostname
+resolves to the Docker **host's** own IP (useful for reaching a service
+running directly on the host machine's loopback), not to arbitrary LAN mDNS
+names — it wouldn't help resolve `search.home` unless SearXNG happens to run
+on the same physical host as this compose stack.
+
+## 5. Interaction with this project's known-flaky Qwen3.8-27B tool-calling
+
+`docs/research/qwen3.8-27b-tool-calling.md` (2026-08-24) found, with medium-
+to-high confidence, that Qwen3.8-27B's tool-calling through llama.cpp
+inherits open/partially-fixed upstream parser bugs from the Qwen3.5 lineage
+(issues #21158, #20837 in `ggml-org/llama.cpp`) — tool calls can be emitted
+but not recognized, or land as inert XML inside a reasoning block, especially
+with thinking enabled.
+Source: `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md`
+(section 3, "Bottom line" section)
+
+Given section 3 above (`/v1/search` is a standalone endpoint, not a
+model-tool), **that flakiness has no bearing on the recommended integration
+path**: nothing about calling `POST /v1/search/searxng-search` from Open
+WebUI or a script asks Qwen3.8-27B to emit a tool call at all. The search
+happens (or doesn't) independent of the model's tool-calling grammar/parser
+entirely.
+
+The risk **would** resurface only if a *different* design choice is made
+later — e.g. wrapping this same SearXNG-backed endpoint as an MCP tool or a
+`tools=[...]` function definition handed to Qwen3.8-27B in a chat-completion
+request, so the model itself decides when to search. That path would inherit
+every bug documented in `qwen3.8-27b-tool-calling.md` (calls silently dropped,
+calls trapped inside `<think>` blocks, etc.) and would need the live smoke
+test that doc recommends before being trusted unattended. **That is not what
+LiteLLM's `/v1/search` feature itself requires** — it's a choice a caller
+could additionally make on top of it.
+
+## Recommendation
+
+1. Add a `search_tools` block to `litellm-config.yaml` using
+   `search_provider: searxng` and `api_base: http://search.home/` (see
+   section 2) — no custom/generic-endpoint workaround needed, this is a
+   first-class supported provider.
+2. Add `extra_hosts: ["search.home:<LAN IP>"]` to the `litellm` service in
+   `docker-compose.yml` (see section 4) so the container can resolve the
+   hostname; confirm the IP is stable (static DHCP reservation) since
+   `extra_hosts` is a static mapping baked in at container start.
+3. Treat `/v1/search` as a plain HTTP integration point, not a model tool —
+   whatever calls it (Open WebUI, a script) should call the REST endpoint
+   directly rather than exposing it to Qwen3.8-27B as a function-calling
+   tool, to avoid inheriting this project's documented tool-calling
+   flakiness (section 5). If model-driven search-tool-calling is wanted
+   later, that's a separate decision that should be smoke-tested against the
+   caveats in `qwen3.8-27b-tool-calling.md` first.
+
+This is research only — `litellm-config.yaml` and `docker-compose.yml` are
+not modified by this doc.
+
+## Sources
+
+- https://docs.litellm.ai/docs/search — LiteLLM search feature docs:
+  endpoints, `search_tools` config shape, provider list, standalone-endpoint
+  behavior.
+- https://github.com/BerriAI/litellm/pull/16259 — SearXNG provider
+  implementation: `search_provider: searxng`, `api_base` /
+  `SEARXNG_API_BASE` config.
+- `g:\_DEV\repos\LLM-Server\docker-compose.yml` — `litellm` service
+  definition (lines 94–124), `ai-stack` network block (lines 154–156),
+  `lazytainer`'s `network_mode: host` precedent (lines 144–152).
+- `g:\_DEV\repos\LLM-Server\litellm-config.yaml` — current proxy config
+  conventions (`model_list`, `litellm_params`, `api_base` usage at line 8).
+- `g:\_DEV\repos\LLM-Server\docs\network-access.md` — this repo's existing
+  `*.home` / NPM hostname conventions.
+- `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md` —
+  prior findings on Qwen3.8-27B tool-calling reliability via llama.cpp.
+
+## Confidence/uncertainty summary
+
+- **High confidence:** SearXNG is a native, first-class `search_provider` in
+  LiteLLM's `/v1/search` feature (directly documented and confirmed via the
+  implementing PR); the feature is a standalone REST endpoint separate from
+  chat completions, not automatic context-injection and not itself a
+  model-callable tool.
+- **Medium confidence:** the exact field name (`api_base` vs. relying solely
+  on `SEARXNG_API_BASE`) — two independent fetch passes against
+  docs.litellm.ai returned slightly different renderings of the same example
+  (one showed `api_key: os.environ/SEARXNG_API_BASE`, the other and the PR
+  fetch showed `api_base: <url>`); the PR-sourced `api_base` form is treated
+  as authoritative here since it comes from the implementing code change, but
+  this should be smoke-tested against the actual deployed LiteLLM image
+  (`ghcr.io/berriai/litellm:main-stable`) before being relied on verbatim.
+- **Not independently verified:** SearXNG's actual LAN IP/hostname stability
+  on this network, and whether the deployed LiteLLM version
+  (`main-stable`, per `docker-compose.yml` line 95) already includes PR
+  #16259 — worth a quick `docker exec litellm pip show litellm` / changelog
+  check before wiring this in for real.

docs/research/omniroute-account-semaphore-timeout.md

@@ -0,0 +1,84 @@
+# OmniRoute's per-connection semaphore timeout — hardcoded, not a setting
+
+**Date:** 2026-09-09
+
+Any OmniRoute connection whose upstream can only handle a small, fixed number of concurrent requests
+(this repo's `llama-server`/`qwen-classifier`, both effectively single-GPU-slot-limited) can hit a hard
+30-second reject once more requests are in flight than the connection's `maxConcurrent` allows — even
+though the request would have succeeded fine if it had just waited its turn. This surfaced first as the
+`pr-agent`/`CodersPlacePI` 429/504 investigation (see the issue tracker), then again while sizing
+`qwen-classifier`. Recorded here so it doesn't have to be re-diagnosed from scratch next time.
+
+## The error
+
+```
+{"error":{"message":"Semaphore timeout after 30000ms for <provider>:<connectionId>","type":"rate_limit_error","code":"rate_limit_exceeded"}}
+```
+
+## Root cause (confirmed against OmniRoute's own source, [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute))
+
+`open-sse/services/accountSemaphore.ts`:
+
+```ts
+const DEFAULT_TIMEOUT_MS = 30_000;
+...
+function createSemaphoreTimeoutError(semaphoreKey, timeoutMs) {
+  const error = new Error(`Semaphore timeout after ${timeoutMs}ms for ${semaphoreKey}`);
+  error.code = "SEMAPHORE_TIMEOUT"; // classified upstream as HTTP 429 rate_limit_exceeded
+  return error;
+}
+```
+
+Called from `open-sse/handlers/chatCore.ts`:
+
+```ts
+await acquireAccountSemaphore(accountSemaphoreKey, {
+  maxConcurrency: accountSemaphoreMaxConcurrency, // = the connection's maxConcurrent
+  signal: streamController.signal,
+  // no timeoutMs passed → always falls back to the hardcoded 30_000 default
+})
+```
+
+This is **not** the same thing as OmniRoute's documented quota-share concurrency gate
+(`open-sse/services/combo/quotaShareConcurrency.ts`, key prefix `qsconn:`), which is deliberately
+fail-open per its own doc comment ("a saturated queue or timeout proceeds without a slot rather than
+ever rejecting a dispatchable request") — that one only matters for quota-share combos. The account
+semaphore above is a *different*, always-on gate keyed `provider:connectionId`, has no fail-open path,
+and its 30-second timeout is a bare `await` with nothing passed to override it — not exposed via
+`/api/resilience`, not an env var, not a dashboard toggle, not documented anywhere in
+`docs/reference/ENVIRONMENT.md`. It's a hardcoded constant in vendored code.
+
+Also **not** the same as `requestQueue.maxWaitMs` (visible via `GET /api/resilience`, this deployment
+already has it at `86400000`) — that one bounds a Bottleneck-managed *execution* timer that starts only
+after dispatch, surfaces as HTTP 504 `RATE_LIMIT_EXECUTION_TIMEOUT`, and is unrelated to the 429 above.
+
+## What actually fixes it
+
+The 30s ceiling itself cannot be raised — no config surface reaches it in the current OmniRoute build.
+Two real options:
+
+1. **Bypass the semaphore, let the upstream's own queue absorb concurrency instead.**
+   `maxConcurrency == null || maxConcurrency <= 0` fully bypasses `accountSemaphore.ts` (see
+   `isBypassed()`) — no gate, no 30s timer, requests pass straight through to the upstream. This only
+   works if the upstream itself queues gracefully with no reject-timeout of its own — confirmed true for
+   llama.cpp's server (`tools/server/server-queue.cpp` has no queue-wait timeout; excess requests just
+   wait for a free slot). If you do this, also raise the connection's own
+   `providerSpecificData.timeoutMs` (bounded 1ms–24h, `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS`) generously —
+   that's the timer that now matters: "did the upstream return response headers in time," which on
+   llama.cpp means the full queue-wait-then-generate time, since llama.cpp sends **zero bytes, not even
+   headers**, while a request sits queued (confirmed in `server-context.cpp`: `res->status = 200` is only
+   set after the first generated token exists).
+2. **Reduce how often more than `maxConcurrent` requests actually stack up** — e.g. the
+   `pr-agent`/Gitea webhook fix (narrowing the subscribed event list so one PR action doesn't fire 3+
+   near-simultaneous AI calls). Doesn't remove the ceiling, just makes it less likely to be hit.
+
+Applied in this repo: `llama-server`'s OmniRoute connection has `maxConcurrent: null` and
+`providerSpecificData.timeoutMs: 1200000` (20 min — matches worst-case 2-slots-busy + queued + own
+generation time). `qwen-classifier` uses a much shorter `timeoutMs: 120000` since it isn't
+GPU-contended the same way — see `docs/coding-cli-setup/qwen-code.md`.
+
+## Sources
+
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `open-sse/services/accountSemaphore.ts`, `open-sse/handlers/chatCore.ts`, `open-sse/services/combo/quotaShareConcurrency.ts`, `open-sse/services/rateLimitManager.ts`, `docs/architecture/RESILIENCE_GUIDE.md`, `docs/reference/ENVIRONMENT.md`
+- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `tools/server/server-queue.cpp`, `tools/server/server-context.cpp`
+- `src/shared/validation/providerSpecificData.ts` (OmniRoute) — `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS` bound

docs/research/omniroute-direct-response-timeout-outage-2026-09-15.md

@@ -0,0 +1,397 @@
+# A 48-minute total outage on `qwen3.8-27b-local`, and the third OmniRoute timeout mechanism this repo hadn't documented yet
+
+**Date:** 2026-09-15
+
+**Verdict:** The error — `"[504]: Direct response did not start within 30000ms — retrying on a fresh socket"` —
+comes from a **third, previously-undocumented OmniRoute timeout mechanism** (`OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`,
+default 30s), distinct from both timeouts already recorded in
+[`omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) and
+[`omniroute-non-ping-sse-stream-timeout.md`](./omniroute-non-ping-sse-stream-timeout.md). It exists specifically to
+recover from a *stale pooled TCP socket* by retrying once on a brand-new connection — but in the incident analyzed
+here, **both** the original attempt and the fresh-socket retry timed out, repeatedly, for 46 requests over 48
+straight minutes with zero successes. That pattern rules out a stale-socket explanation (a fresh socket bypasses
+the pool entirely) and points instead at the upstream itself — `llama-server`, or the R9700 GPU underneath it —
+being genuinely unresponsive for the whole window. The best primary-source match for that symptom is an **open,
+still-unresolved AMD ROCm bug specific to this exact GPU** ([ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630)):
+an MES-firmware hang during generation on `gfx1201`/R9700 that leaves the process alive but stuck, sometimes for
+no logged reason at all. No config change fixes this — raising the 30s timeout only makes each failed attempt
+take longer to give up, it doesn't un-wedge a hung GPU.
+
+## The evidence
+
+**Source:** `omniroute-request-logs-6h-2026-09-15.json`, a 339-entry OmniRoute request-log export the user pulled
+from the dashboard, covering `2026-09-15T12:49:52Z`–`18:44:54Z`. Every entry with a non-200 status (66 of them)
+is a `POST /v1/chat/completions` against `qwen3.8-27b-local` (`/models/Qwen3.8-27B-UD-Q4_K_XL.gguf`), all on the
+same `connectionId` (`649a2d3e-7527-488e-9b8a-dc4ac2624176`) and the same `provider`
+(`openai-compatible-chat-a7bda643-6687-41f4-b75d-fd2cab746874`) — a single upstream connection, not a fan-out
+artifact.
+
+Sorting every error by timestamp shows **one continuous outage**, not scattered slow requests:
+
+- Last successful `/v1/chat/completions` before the outage: `15:18:12.285Z`
+- **First failure:** `15:25:06.415Z` — status 504, `"[504]: Direct response did not start within 30000ms —
+  retrying on a fresh socket"`, duration `60186ms`
+- **Every single `/v1/chat/completions` attempt** from `15:25:06.415Z` through `16:13:29.502Z` failed — 46× 504
+  (all clustered `60021`-`60324ms`, i.e. two back-to-back 30s attempts, both failing) interleaved with 20× 499
+  (`"Request aborted"` / `"Client disconnected: request_signal_aborted"`, durations `1.4s`-`99.97s` — these are
+  qwen-code giving up client-side while OmniRoute was still mid-retry)
+- **First successful recovery:** `16:13:50.388Z`, `20873ms` — 21 minutes after the last failure attempt cluster,
+  i.e. the very next attempt after the outage window succeeded normally
+- No successful `/v1/chat/completions` call appears anywhere inside the `15:25:06Z`-`16:13:29Z` window — confirmed
+  by filtering all 154 `/v1/chat/completions` log entries in that range: every one is 504 or 499.
+
+`git log --since=2026-09-14 --until=2026-09-16` shows **zero commits** in this repo on 2026-09-15 — the outage
+correlates with no deploy, `scripts/update.sh` run, or config change on this end.
+
+**A second, independent data point** (see caveat below): the user also pasted a large raw text table, copied
+directly from OmniRoute's dashboard UI rather than the JSON export, showing the **`qwen-classifier`** connection
+(`qwen3-4b`, both the `-UD-Q4_K_XL.gguf` file this repo's `docker-compose.yml` currently defaults to, and a
+`-UD-Q8_K_XL.gguf` variant that appears **nowhere** in this repo's checked-in `docker-compose.yml`/`.env.example`
+— either tested by hand against `LLAMA_CLASSIFIER_MODEL_FILE` outside version control, or evidence of drift worth
+checking directly on the server) failing repeatedly across two accounts (`Haylan`, `qwen-cli-main`), with the
+same signature: `TI: 0|TO: 0` (zero tokens either direction — failed before generating anything) and durations
+clustering at exactly `60.0`-`60.4s` for the 504s. That's the identical two-attempts-at-30s-each shape as the 27B
+outage above, strongly suggesting the same underlying mechanism, though — important caveat — **this table is not
+present in the 6-hour JSON export and covers a different, longer time range** (timestamps back to `01:21` and
+`23:54` on unspecified dates), so it cannot be directly time-correlated against the 27B outage above. Treat it as
+corroborating evidence that this failure mode recurs on both local model connections, not as proof they failed at
+the same moment.
+
+## Root cause: `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`, a mechanism built for a different problem
+
+Confirmed directly against OmniRoute's own source, [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
+(same repo the two existing timeout docs already cite) — `open-sse/utils/directResponseStartTimeout.ts`:
+
+```ts
+const DEFAULT_DIRECT_HEADERS_TIMEOUT_MS = 30_000;
+const DIRECT_RESPONSE_START_TIMEOUT_CODE = "DIRECT_RESPONSE_START_TIMEOUT";
+
+export function resolveDirectHeadersTimeoutMs(
+  env: Record<string, string | undefined> = process.env
+): number {
+  const raw = env.OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS;
+  if (raw == null || raw.trim() === "") return DEFAULT_DIRECT_HEADERS_TIMEOUT_MS;
+  ...
+}
+
+function createDirectResponseStartTimeout(timeoutMs: number): Error & { code: string } {
+  const err = new Error(
+    `Direct response did not start within ${timeoutMs}ms — retrying on a fresh socket`
+  ) as Error & { code: string };
+  ...
+}
+```
+
+"Direct" here means **direct (no-proxy) egress** — confirmed in `open-sse/utils/proxyFetch.ts`, which routes any
+connection with no configured upstream HTTP proxy through this path (as opposed to OmniRoute's separate
+proxy/relay egress paths). Every local connection in this repo (`llama-server`, `qwen-classifier`, both reached
+over the `ai-stack` Docker network with no proxy) is "direct" — so this timeout mechanism governs **every**
+request to either local model, streaming or non-streaming alike, not just non-streaming JSON responses as the
+name might suggest.
+
+### Why it exists (and why it didn't help here)
+
+Two OmniRoute issues, both with dedicated regression tests in the repo, explain the actual design intent:
+
+- **#4252** (`tests/unit/proxyfetch-retry-fresh-socket-4252.test.ts`): "Undici dispatcher fails on direct provider
+  requests in 502 bursts" — the default direct dispatcher pools keep-alive sockets; some upstreams silently close
+  idle pooled sockets, so the next request reusing one fails with `UND_ERR_SOCKET`. Fix: retry once on a **fresh,
+  no-keep-alive dispatcher** (`getRetryDispatcher()`, a different instance from `getDefaultDispatcher()`) so the
+  retry can't grab another already-dead pooled socket.
+- **#10214** (`tests/unit/proxyfetch-direct-response-start-timeout-10214.test.ts`): "Direct (no-proxy) requests
+  stall on a silently-dropped pooled keep-alive socket until the caller's deadline or a service restart" — the
+  harder case: a pooled socket that dies **without even an error**, just silence. Undici's `headersTimeout`
+  default (600s) is far too slow to catch this in practice, and the existing #4252 retry never fires because no
+  error is thrown to trigger it. The fix bounds each direct attempt's response-start wait to
+  `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` (30s default) via `directFetchWithBoundedResponseStart`, and retries
+  **once** on the same fresh no-keep-alive dispatcher from #4252 when that bound is hit.
+
+Both fixes assume the *socket* is the problem, not the upstream. `directFetchWithBoundedResponseStart`'s own
+implementation (`open-sse/utils/directResponseStartTimeout.ts`) is exactly two attempts: pooled, then fresh. When
+attempt 2 — a brand-new socket that cannot possibly be a zombie pooled connection — **also** times out at 30s,
+the retry logic has nothing left to try and the request fails with `DIRECT_RESPONSE_START_TIMEOUT_CODE`
+(surfaced as the 504 seen in the logs). A fresh socket succeeding to *connect* but the *server* never sending a
+response is exactly what "the upstream process is alive but stuck" looks like from OmniRoute's side — it can't
+distinguish "GPU is wedged mid-generation" from "stale pooled socket," because both present as "nothing came
+back in 30s." 46 consecutive both-attempts-failed cycles over 48 minutes is far outside what a transient stale-socket
+burst (the scenario #4252/#10214 were built for) would produce; it's consistent with a sustained upstream
+outage instead.
+
+**Not currently configured in this repo**: `grep`-ing `docker-compose.yml` and `.env.example` for
+`OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` finds nothing — this deployment runs on the unmodified 30s default. (This
+is also, notably, a *fourth* data point alongside the two already-documented mechanisms and the `requestQueue.maxWaitMs`/
+`RATE_LIMIT_EXECUTION_TIMEOUT` timer mentioned in passing in `omniroute-account-semaphore-timeout.md` — OmniRoute
+has at least four independent timeout knobs guarding different stages of a request's life, three of them
+30-second-flavored by default, which is worth keeping in mind the next time an unfamiliar timeout string shows up.)
+
+## Why the upstream itself was likely unresponsive: ROCm/legacy-rocm-build#6630
+
+This session has **no SSH/shell access to the actual R9700 server** — there's no SSH config, and nothing in
+`scripts/` does remote exec, confirmed by inspecting `scripts/update.sh` and the absence of any `~/.ssh/config`
+entry for the box. So none of the following is confirmed against this specific incident's `dmesg`/`rocm-smi`/
+`docker logs` output — it's the closest primary-source match to the *symptom*, not a diagnosis of *this* outage.
+
+[ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630) ("gfx1201 R9700 ROCm 7.14
+llama.cpp generation hang, MES queue failure and PSP reset -62; Vulkan passes") is an **open**, actively
+investigated issue (created 2026-08-19, most recent update 2026-08-28, no fix landed) that reproduces on **the
+exact same GPU this repo runs on** — Radeon AI PRO R9700, `gfx1201` — running **llama.cpp with `-fa on`** (this
+repo's `llama-server` also runs `--flash-attn on`), with a controlled Vulkan-vs-ROCm A/B: Vulkan completes
+normally, ROCm hangs during token generation. Direct quotes:
+
+> "During the ROCm generation stall: GPU busy reached 100%, memory busy remained 0%, VRAM use was only about 1.1
+> GB, **the container remained alive but made no output progress**."
+
+> "when MES stops responding, **the driver can stay unaware of it indefinitely** — the failure only surfaces when
+> something happens to send the next MES message... if a run is left alone after it stops making progress, the
+> kernel prints nothing at all, so a hang can look like a slow workload rather than a fault."
+
+> "the failure is probabilistic, not deterministic... a single passing run on this host does not indicate a
+> healthy configuration."
+
+The thread (12 comments as of this research pass, an AMD engineer `harkgill-amd` participating) has ruled out,
+one at a time, `uni_mes=0`, `mes_log_enable=1`, ROCm 6.4.4, ROCm 7.14, ROCm 10.0.0 stable, the latest TheRock
+nightly, and GFXOFF-disable — **no confirmed fix or workaround exists in the thread as of this research pass**.
+A related comment on the same issue (`chrisfranson`) reports the identical MES `REMOVE_QUEUE`/MODE1-reset
+signature from a **completely unrelated workload** (headless LibreOffice with OpenCL) on the same `gfx1201`
+silicon, reinforcing that this is a driver/firmware-level fault under general GPU load, not something specific
+to llama.cpp's request pattern.
+
+**This is a different bug from the one already mitigated in this repo.** `docs/research/rocm-gpu-pin-and-render-group.md`
+already documents and works around `ROCm/ROCm#5706` (clock/power pinned at boost whenever two concurrent HIP
+contexts share the GPU — fixed via `GPU_MAX_HW_QUEUES=1`, already set on both `llama-server` and `qwen-classifier`
+in `docker-compose.yml`). #5706's symptom is elevated power draw with the GPU still working; #6630's symptom is
+generation fully halting with `gpu_busy=100%`/`mem_busy=0%` and MES no longer responding at all — a real hang, not
+a clock-pin inefficiency. `GPU_MAX_HW_QUEUES=1` targets #5706's specific trigger (hardware-queue oversubscription
+across concurrent HIP processes) and has no evidence in #6630's thread of affecting that bug — #6630 reproduces
+in single-GPU, single-process benchmarks with no second HIP context involved at all, so the already-applied fix
+should not be assumed to help here.
+
+## What would actually resolve this vs. what wouldn't
+
+- **Raising `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`** — not recommended as a fix. It would make each failed attempt
+  take longer before giving up (worse latency during a real hang), without addressing why the GPU stopped
+  responding. It's the right lever only if future evidence shows genuinely-slow-but-working responses being
+  mistaken for hangs (the same shape as the already-fixed `REQUEST_TIMEOUT_MS` issue in
+  `omniroute-non-ping-sse-stream-timeout.md`) — this incident's 46-for-46 both-attempts-failed pattern over 48
+  minutes doesn't fit that shape.
+- **Concrete next step for whoever has server access when this recurs**: check `dmesg | grep -i amdgpu` and
+  `journalctl -k` on the R9700 host for `MES(...) failed to respond`, `GPU reset begin`, or `PSP resume failed`
+  lines matching #6630's signature, and `docker logs llama-server`/`docker logs qwen-classifier` to see whether
+  the process was alive-but-stuck (consistent with #6630) versus crashed/restarted (which would point elsewhere).
+  Capturing this during a live incident is the only way to move this from "best primary-source match" to
+  "confirmed root cause."
+- **Monitor [ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630)** for a fix —
+  it's open and active (AMD engineer engaged as of 2026-08-26); no released ROCm version as of this research pass
+  is confirmed clean.
+- **The `-UD-Q8_K_XL.gguf` classifier variant in the pasted table but absent from version control** is worth a
+  direct look on the server (`cat .env` / `docker inspect qwen-classifier` for the actual `LLAMA_CLASSIFIER_MODEL_FILE`
+  in effect) — outside this research pass's reach without server access, flagged here so it isn't lost.
+
+## Recovery: how the hang actually clears (or doesn't) — addendum, 2026-09-15
+
+Follow-up question: what actually recovers the socket once this hits, given it's been observed to stay wedged
+for days at a time? Pulled the full comment thread on
+[ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630) directly via the GitHub API
+(12 comments, `angelhalo` as primary reporter, `harkgill-amd` as the responding AMD engineer, plus one
+corroborating report from `chrisfranson` on unrelated hardware/workload) — the earlier research pass's source list
+cited this issue but hadn't read the full thread. Three things fall directly out of it:
+
+**There is no reliable in-band recovery.** The driver doesn't notice the hang on its own — direct quote:
+"when MES stops responding, the driver can stay unaware of it indefinitely — the failure only surfaces when
+something happens to send the next MES message." In a captured live hang, "every driver-managed ring is
+completely idle while the GPU reports 100% busy," `dmesg` has zero amdgpu lines, and no task is in D-state —
+so from the OS's perspective nothing is wrong; only sending the GPU another command (which killing/restarting
+the stuck process does) triggers the driver to discover the wedge and attempt its own MODE1 reset.
+
+**Once triggered, that reset itself is a coin flip across three documented outcomes**, not a guaranteed fix:
+1. **Clean recovery** — `GPU reset succeeded, trying to resume`, PSP resumes, the card comes back (VRAM is
+   wiped — "VRAM is lost due to GPU reset!" — so the container needs a real restart to reload the model, a plain
+   process respawn isn't enough even when the reset itself works).
+2. **Failed resume** — `PSP resume failed`, `GPU reset end with ret = -62` (the original report's own outcome) —
+   the reset attempt itself fails, leaving the GPU in a worse state than before.
+3. **Full kernel soft-lockup** — `chrisfranson`'s independent report (different workload — headless LibreOffice
+   OpenCL, different card — RX 9070 XT, same `gfx1201` silicon) hit outcome 2 or 3 twice out of three times:
+   "the whole system hard-locked (kernel soft lockup pegging a CPU at ~90-100% softirq, requiring a physical
+   power cycle)."
+
+`angelhalo` deliberately left one hang untouched rather than killing the process, to observe it without
+contaminating the state with a reset: "I recovered only with a subsequent cold power cycle" — no reset was ever
+triggered because nothing sent the GPU another message. Their standard test procedure between every single run
+in this thread is "a cold power cycle (AC removed, ≥30 s)," specifically **not** a warm/soft reboot — stated
+reason: "on this card a MODE1 reset takes the host down with it," meaning even the OS's own reboot path can't
+be trusted to come back cleanly once this GPU is in a bad state. This is the practical answer to "why does it
+stay stuck for days": nothing about the hang self-clears, `docker`'s `restart: unless-stopped` policy never
+fires because the container process is alive and never exits (confirmed: this repo's `llama-server` and
+`qwen-classifier` services have no `healthcheck` block at all — only `omniroute` itself does, a plain TCP
+connect check on its own dashboard port, which says nothing about whether `llama-server`/`qwen-classifier` are
+responding) — so a hang persists until a human notices the symptom (requests failing) and manually intervenes,
+and "days" is just however long that takes to notice on a homelab box, not a property of the hang itself.
+
+**No fix or reliable mitigation exists as of this reading (2026-08-28, the thread's latest comment).**
+`harkgill-amd` (AMD) could not reproduce locally and asked for a nightly-driver retest; `angelhalo` retested and
+it still failed. Every other variable tested still hangs: ROCm 6.4.4 through 10.0.0 stable, TheRock nightlies,
+`amdgpu.uni_mes=0`, `cwsr_enable=0`, `mes_log_enable=1`, GFXOFF disabled, two different physical R9700 cards, both
+llama.cpp and vLLM. The thread's own conclusion, as of the last comment: "a probabilistic lost-completion event"
+with no known trigger to avoid and no known driver/firmware combination that's clean.
+
+**Practical takeaway for this repo, given no upstream fix exists:**
+- A restart *might* recover it, *might* make it worse (failed PSP resume), and *might* take the whole host down
+  requiring a physical power cycle — there's no way to know in advance which outcome a given hang will produce.
+- Nothing currently watches for this automatically. Docker's `restart: unless-stopped` is the wrong tool (process
+  doesn't exit) — recovering automatically would need a `healthcheck` against `llama-server`'s own `/health`
+  endpoint (llama.cpp's built-in liveness endpoint) paired with something that acts on an `unhealthy` status,
+  since Docker itself doesn't restart on failed healthchecks without an external watcher (e.g. `willfarrell/autoheal`
+  or equivalent) — not evaluated here, flagged as a real gap, not a recommendation to implement blind: an
+  automated restart during a hang that's about to fail its PSP resume and lock the host could turn a
+  "requests are failing" incident into "the box needs a physical power cycle" automatically and unattended,
+  which is a real downside worth weighing against faster detection.
+- Given the reset outcome is unpredictable, the safest manual recovery when this is caught live is: restart the
+  affected container, then immediately check `dmesg | grep -i amdgpu` for `PSP resume failed` or a soft-lockup
+  signature before assuming it's fixed — if either appears, a full reboot (and per this thread's own testing
+  practice, possibly a genuine AC power cycle rather than a warm reboot) is the next step, not a second restart
+  attempt.
+
+## Caveats and open questions
+
+- **JSON export vs. pasted table are two different, non-overlapping captures.** The JSON file is a precise 6-hour
+  window with full per-request detail; the pasted table is a longer, dashboard-UI-copied range with less
+  structure and no verifiable overlap with the JSON file's timestamps. A fresh multi-day JSON export (same
+  `request-logs` endpoint used to produce the file analyzed here) would let a future pass check whether the
+  classifier's failures and the 27B model's outage are literally simultaneous (strong evidence for a shared
+  GPU-level cause) or independent recurrences of the same mechanism on separate schedules.
+- **Why the outage self-recovered after ~48 minutes with no observed restart is unexplained.** #6630's thread
+  describes hangs resolving via an explicit GPU reset (sometimes failing, requiring reboot) — not a case of a
+  hang clearing on its own after a fixed interval. Nothing in the available data (no server access) confirms
+  whether a restart happened that isn't visible from OmniRoute's logs, or whether this specific hang genuinely
+  self-cleared, which would be a data point *against* the #6630 hypothesis worth capturing next time.
+- **Live reproduction was not attempted.** The user suggested testing tool-calls against the classifier via the
+  Windows-side qwen-code CLI (`C:\Users\aerli\AppData\Local\qwen-code\bin\qwen.cmd`) to try to reproduce a
+  "Direct response did not start" failure live. Skipped for this pass: qwen-code requires an interactive/
+  already-authenticated session to drive meaningfully, and deliberately trying to reproduce a GPU hang against
+  the shared production classifier risked a genuine 60s+ stall on infrastructure other work depends on, for
+  uncertain diagnostic payoff given the strength of the log-based and source-based evidence already gathered.
+  Worth doing deliberately, with server access on hand to capture `rocm-smi`/`dmesg` simultaneously, rather than
+  as a quick check from this pass.
+
+## Sources
+
+- `omniroute-request-logs-6h-2026-09-15.json` — OmniRoute dashboard request-log export provided by the user
+  (2026-09-15, 339 entries, `12:49:52Z`-`18:44:54Z`)
+- User-pasted OmniRoute dashboard table (`qwen-classifier`/`qwen3-4b` failures, separate capture window)
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) —
+  `open-sse/utils/directResponseStartTimeout.ts`, `open-sse/utils/proxyFetch.ts`, `open-sse/utils/proxyDispatcher.ts`,
+  `tests/unit/proxyfetch-direct-response-start-timeout-10214.test.ts`, `tests/unit/proxyfetch-retry-fresh-socket-4252.test.ts`,
+  `open-sse/handlers/chatCore/upstreamTimeouts.ts`
+- [ROCm/legacy-rocm-build#6630](https://github.com/ROCm/legacy-rocm-build/issues/6630) — open R9700/gfx1201
+  llama.cpp generation-hang issue, full comment thread (2026-08-19 through 2026-08-28)
+- [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the
+  first already-documented 30s OmniRoute timeout (account semaphore, 429, hardcoded)
+- [`docs/research/omniroute-non-ping-sse-stream-timeout.md`](./omniroute-non-ping-sse-stream-timeout.md) — the
+  second already-documented timeout (first-SSE-event deadline, `REQUEST_TIMEOUT_MS`-derived)
+- [`docs/research/rocm-gpu-pin-and-render-group.md`](./rocm-gpu-pin-and-render-group.md) — the already-mitigated,
+  *different* R9700/gfx1201 MES bug ([ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706), clock-pin/power,
+  not a hang)
+- Local `docker-compose.yml`, `.env.example` (grepped directly, confirming `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`
+  is unset / on the 30s default, and that `GPU_MAX_HW_QUEUES=1` is already applied to both GPU services)
+- `git log --since=2026-09-14 --until=2026-09-16` (this repo, confirming zero commits during the outage window)
+
+## Confidence / uncertainty summary
+
+- **High confidence**: the exact mechanism and semantics of `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS` /
+  `directFetchWithBoundedResponseStart` (read directly from OmniRoute's own source and its two regression test
+  files, which spell out the intent in comments referencing the originating issues); the JSON log's chronology,
+  single-connection scope, and the "two ~30s attempts, both failing" shape of every 504 (computed directly from
+  the log file); that this timeout is unconfigured in this repo (direct grep); that no commits landed in this
+  repo during the outage window (direct `git log`).
+- **Medium confidence**: that ROCm/legacy-rocm-build#6630 is the actual root cause of *this specific* outage.
+  The GPU model, driver-family symptom shape (`gpu_busy=100%`/`mem_busy=0%`, alive-but-stuck, sometimes
+  logged/sometimes silent), and `-fa on` usage all match closely, and it's an open/unresolved/actively-discussed
+  issue as of this research pass — but nothing from this incident's own `dmesg`/`rocm-smi` output was available
+  to confirm it directly (no SSH access), so this is the best primary-source match to the symptom, not a
+  confirmed diagnosis.
+- **Low confidence / open**: why the outage recovered on its own after ~48 minutes with no observed restart;
+  whether the classifier's pasted-table failures share the exact same triggering event as the 27B outage
+  analyzed here (same failure signature, but no verified time-overlap between the two data sources); the
+  provenance of the `-UD-Q8_K_XL.gguf` classifier variant seen in the pasted table but absent from version
+  control.
+
+## Live test, 2026-09-15: the classifier is measurably too slow at its own documented worst case — independent of any hang
+
+Before scoping a fix, tested the live `qwen-classifier` backend directly against realistic worst-case load,
+per the user's request to gather fresh evidence rather than design blind. Two attempts to reproduce this through
+qwen-code itself first surfaced an unrelated, separately-useful finding; the direct backend test below is what
+actually answered the question.
+
+### qwen-code's own headless mode never reaches the classifier
+
+Ran `qwen --approval-mode auto <prompt>` (positional/one-shot, non-interactive) from the Windows-side install
+(`C:\Users\aerli\AppData\Local\qwen-code\bin\qwen.cmd`, which has both `fastModel` and the `omniroute-search` MCP
+server already configured), asking it to run a shell `dir` and use the web-search MCP tool. Both attempts hit a
+wall before any classifier request was even sent:
+
+- The MCP tool call was refused outright: `Warning: Tool "mcp__omniroute-search__search" requires user approval
+  but cannot execute in non-interactive mode. ... use the -y flag (YOLO mode)`.
+- The shell tool: the model itself reported `run_shell_command` as "not registered" in this session and silently
+  substituted a read-only `glob` call instead — no approval prompt, no classifier call, no system warning printed
+  (unlike the MCP case), across two separate clean runs.
+
+**Conclusion: one-shot headless `qwen <prompt>` invocations don't exercise Auto Mode's classifier at all for
+approval-requiring tools** — they're declined or silently rerouted before the classifier ever gets a request.
+The classifier only fires in a genuinely interactive session, where it substitutes for the human's live approval
+decision. This wasn't previously documented anywhere in this repo and is worth keeping in mind: headless qwen-code
+testing is not a valid way to probe classifier behavior, live or otherwise. (Not investigated further: whether
+`qwen serve`/`--input-format stream-json` headless-agent modes behave differently — plausible, since they're
+built for exactly this kind of automation, but out of scope for this pass.)
+
+### Direct backend test: real classifier latency at realistic token counts
+
+Given headless qwen-code couldn't drive this, sent shaped classifier requests straight to
+`http://proxy-ai.home/v1/chat/completions` (model `qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`,
+confirmed present in `GET /v1/models`) using a temporary scoped API key, mimicking the `{shouldBlock}`
+JSON-verdict shape and the `MAX_TRANSCRIPT_MESSAGES=40` / `MAX_HISTORICAL_ACTION_CHARS=4000` structure this
+repo's own `fast-model-choice.md` already read out of qwen-code's `classifier-transcript.ts` source. Five calls,
+in order:
+
+| # | Prompt tokens | `cached_tokens` | Wall time | Notes |
+|---|---|---|---|---|
+| 1 | 1,000 | 3 | 2.24s | Small prompt, genuinely fresh — healthy baseline. |
+| 2 | 51,234 | 51,233 | **64.21s** | First send of a large synthetic worst-case transcript (~40 highly self-similar 4K-char "historical action" blocks). Near-total cache hit reported, yet still the slowest call — see caveat below. |
+| 3 | 51,234 | 51,233 | 0.07s | **Identical repeat of #2.** Same `chatcmpl-...` id as #2 came back — this is OmniRoute short-circuiting an exact-duplicate request via a proxy-level response cache, not fresh inference. Confirms #2/#3's `cached_tokens` field is not a reliable proxy for wall-clock latency on its own. |
+| 4 | 1,000 | 3 | 0.03s | Identical repeat of #1 — same id, same response-cache short-circuit. |
+| 5 | 30,958 | 919 | **28.66s** | Fresh, non-repeated worst-case-shaped transcript (different random content, no internal self-similarity to trigger cache effects). Mostly-uncached (919/30,958) — this is the clean data point. |
+
+Call #5 is the one to trust: **~31K genuinely-fresh prompt tokens took 28.7 seconds** on the current
+`--n-gpu-layers 28` (of 36) / `--cache-type-k/v q4_0` / `--parallel 1` configuration, with the backend otherwise
+idle and healthy (no hang in progress). Extrapolating that rate to the repo's own documented worst case (a real
+15,116-token call observed live per `fast-model-choice.md`, and a theoretical ceiling around 40-50K tokens per
+`classifier-transcript.ts`'s limits) puts a genuine worst-case classifier call at **roughly 30-65 seconds of
+normal, non-hung processing time** — consistent with call #2's 64.21s, even though that call's own cache
+metadata is too muddied by internal prompt self-similarity to use as a second clean sample.
+
+**This directly overlaps both binding timeouts**: qwen-code's own client-side classifier stage timeout
+(`stage1Ms`/`stage2Ms`, `60000` each in the Windows-side `settings.json` observed this session, `30000`/`60000`
+in the WSL-side one) and `OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`'s 30-second-per-attempt window documented above.
+A worst-case classifier call landing in the 30-65s range will **routinely** trip one or both of these timeouts
+on its own, with the backend never having hung at all — the exact same 499/504/"Request aborted" shape as the
+GPU-hang hypothesis produces, but from an entirely mundane, deterministic cause: **the classifier's current
+configuration is simply too slow for the request sizes qwen-code's own classifier-transcript design allows.**
+
+### What this changes
+
+This doesn't rule out ROCm/legacy-rocm-build#6630 — the 27B model's 48-minute total outage (zero successes, not
+just slow ones) doesn't fit a "just slow" explanation, and remains best matched by a real GPU hang. But it does
+mean **the classifier's own recurring failures (the pasted-table evidence) very plausibly have a second,
+independent, non-probabilistic cause that a healthcheck/restart-on-hang design wouldn't fix at all** — restarting
+a classifier that's merely slow-but-working at worst-case load just interrupts a call that would have succeeded,
+and would fire repeatedly under normal peak usage, not just during a rare hang. Any fix that only targets "detect
+and recover from an unresponsive GPU" leaves this second failure mode untouched. Two independent levers worth
+weighing before finalizing a scope: raising the classifier's own timeouts to match its real worst-case latency
+(cheap, immediate, but does nothing for actual hangs), and/or speeding up the classifier itself (full GPU offload
+if VRAM allows, a faster quant, or capping the transcript size client-side) to bring worst-case latency back
+under the existing timeouts.
+
+**Not investigated in this pass**: whether call #2's 64.21s (vs. call #5's extrapolated ~45-48s at a similar
+token count) reflects genuine non-linear slowdown at the very largest context sizes, real concurrent contention
+from other production traffic sharing the same `--parallel 1` slot during the test, or is just noise from a
+single sample each — worth a few more clean, uniquely-content, worst-case-sized calls at different times of day
+before treating either number as precise.

docs/research/omniroute-memory-tool-injection-breaks-classifier.md

@@ -0,0 +1,216 @@
+# OmniRoute's builtin memory tools silently hijack qwen-code's classifier tool-call, not a model or GPU problem
+
+**Date:** 2026-09-15
+
+**Verdict:** The `"Classifier stage 1 unavailable"` / `"Auto Mode couldn't classify this action"` failures are **not**
+a GPU hang, not a timeout, and not a Qwen3-4B quality problem. Confirmed directly from a live debug log: the fast
+model (`qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`) *is* being used (`stage=fast` in every classifier
+log line), and it responds well within its timeout (18.7s against a 30-60s budget). The failure is
+`"Error: Invalid side query response: params must have required property 'shouldBlock'"` — a **schema-validation
+failure on an in-time response**. Root cause, confirmed directly against OmniRoute's own source
+(`open-sse/handlers/chatCore/memorySkillsInjection.ts`): **OmniRoute silently appends its own builtin memory tools
+(`memory_save`/`update`/`search`/`delete`) to every non-streaming chat completion's `tools` array**, whenever
+memory is enabled for the calling key — regardless of what tools the caller declared. qwen-code's classifier forces
+`tool_choice: ANY` (call *some* tool, not a specific one) so it can get a structured `respond_in_schema` JSON
+response. With OmniRoute's extra tools spliced in, the small model sometimes picks the injected `memory_save` tool
+instead — and since qwen-code's client only extracts the classifier's answer from a `respond_in_schema` function
+call (not from a stray `memory_save` call, even if the answer also happens to be present as plain text), the
+result validated against `STAGE1_SCHEMA` is empty, producing exactly the observed error.
+
+## The debug-log evidence
+
+Captured directly from a `-d` (debug) qwen-code run, `C:\Users\aerli\.qwen\debug\886d00eb-...txt`:
+
+```
+21:09:56 [DEBUG] [CLASSIFIER] ALLOW stage=fast tool=mcp__omniroute-search__search durationMs=15412
+21:10:28 [WARN]  [CLASSIFIER] failUnavailable stage=fast durationMs=18727 reason="Classifier stage 1 unavailable" cause="Error: Invalid side query response: params must have required property 'shouldBlock'"
+```
+
+Both lines are tagged `stage=fast` — qwen-code's own internal label confirming the classifier used the configured
+fast model both times, settling a live question this session raised about whether the classifier was silently
+falling back to the main 27B model. It wasn't. The second call's 18.7s duration is comfortably inside this
+deployment's configured `stage1Ms` (30000-60000ms depending on install) — this was not a timeout.
+
+## Reproduced directly
+
+Sent the exact same request shape qwen-code's `classifyAction()`/`generateJson()` builds — a forced tool call to a
+synthetic `respond_in_schema` function with `STAGE1_SCHEMA` (`{required: ["shouldBlock"], properties: {shouldBlock:
+{type: "boolean"}}}`) as its only declared tool, `tool_choice: "required"` — directly against
+`POST /v1/chat/completions` on this deployment (model `qwen3-4b//models/Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`).
+The very first response:
+
+```json
+{
+  "message": {
+    "content": "{\"shouldBlock\": true}",
+    "tool_calls": [{
+      "type": "function",
+      "function": {
+        "name": "memory_save",
+        "arguments": "{\"key\": \"tool_call_review\", \"content\": \"web_fetch call to https://www.swfte.com/ai/leaderboard was blocked due to potential data exfiltration or unauthorized external access.\", \"type\": \"factual\"}"
+      }
+    }]
+  },
+  "tool_results": [{ "output": "{\"success\":true,\"memory\":{\"id\":\"84389721-...\", ...}, \"message\":\"Memory saved successfully\"}" }]
+}
+```
+
+`memory_save` was never declared in the request's `tools` array — only `respond_in_schema` was. OmniRoute added it
+server-side, the model called it instead of (or alongside) the intended tool, and OmniRoute **auto-executed it**
+(`tool_results` shows a real memory record was created, with a UUID and timestamp — this isn't inert). The correct
+`{"shouldBlock": true}` answer is sitting right there in `content`, but qwen-code's `generateJson()` only ever
+looks for it inside a `respond_in_schema` function call (`functionCalls.find(call => call.name ===
+"respond_in_schema")`); when that's absent it falls through to a raw-text JSON parse of `getResponseText(result)`
+— and OpenAI-client conventions generally don't populate a response's plain-text field when `finish_reason` is
+`tool_calls` rather than `stop`, so that fallback likely returns nothing, yielding `{}` against a schema that
+requires `shouldBlock`. This is a single confirmed repro, not a guaranteed-every-time reproduction — the injected
+tool doesn't win every time (an immediate identical follow-up request returned an OmniRoute-cached copy of the same
+response, not a fresh sample — see the cache caveat in
+[`omniroute-direct-response-timeout-outage-2026-09-15.md`](./omniroute-direct-response-timeout-outage-2026-09-15.md)),
+but it reproduces the *exact* failure shape from the live debug log on the first genuine attempt.
+
+## Root cause, confirmed in OmniRoute's own source
+
+[diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) —
+`open-sse/handlers/chatCore/memorySkillsInjection.ts`:
+
+```ts
+if (memoryOwnerId && memorySettings?.enabled && body.stream !== true) {
+  // Server-side builtin memory tools (memory_save/update/search/delete) are
+  // executed by the gateway's tool-call interception, which runs only on the
+  // non-stream path. Stream clients (opencode etc.) execute tools client-side,
+  // so for them these tools would be announced but never executed; they should
+  // use the MCP memory tools (omniroute_memory_*) instead.
+  const existingTools = Array.isArray(body.tools) ? body.tools : [];
+  ...
+  const memoryTools = buildMemoryToolsForProvider(...).filter(tool => !existingToolNames.has(name));
+  if (memoryTools.length > 0) {
+    body = { ...body, tools: [...existingTools, ...memoryTools] };
+  }
+}
+```
+
+This runs unconditionally for any non-streaming request from a key with memory enabled — there is no exemption
+for a caller that already set `tool_choice` to force a *specific* tool. qwen-code's classifier is exactly this
+case: a single-purpose, forced-`ANY`, non-streaming tool call, which is precisely the shape this injection logic
+was not written to avoid interfering with.
+
+`src/lib/memory/settings.ts` confirms `enabled: false` is the *default* — memory is off by default in a fresh
+OmniRoute install specifically because of injected-context cost, per its own comment:
+
+> "Off by default: enabling memory injects up to `maxTokens` (~2k) of retrieved context into every chat request,
+> which is billed — a surprising cost for new installs... Opt in explicitly via Settings → Memory... Per-request
+> opt-out is also available via the `x-omniroute-no-memory` header."
+
+This deployment has memory enabled (confirmed live by the reproduction above), which is presumably a deliberate
+choice for other workflows (chat memory across sessions) — but it has an undocumented-to-this-repo side effect on
+any caller using forced-tool-call classification.
+
+## What would fix this
+
+Two per-target exclusions were checked live against this deployment and confirmed **not to exist**:
+
+- **Per-API-key memory override**: `GET /api/keys` was fetched directly (the temporary key handed to this session
+  turned out to carry admin access, well beyond the plain `/v1` workload scope its name implied). Every key's full
+  field list was inspected — `noLog`, `scopes`, `allowedModels`, `rateLimits`, `disableNonPublicModels`, etc. — with
+  no memory-related field anywhere.
+- **Per-model/connection override**: `GET /api/providers/<id>` for the classifier's own connection
+  (`qwen3-4b`, id `b78ceb4c-52f8-47ae-b245-483baa6e3fc2`) was fetched directly. `providerSpecificData` (`prefix`,
+  `apiType`, `baseUrl`, `nodeName`, `timeoutMs`, `apiKeyHealth`) has no memory field either — consistent with the
+  source: `memoryOwnerId` is resolved purely from the *calling key* (`resolveMemoryOwnerId(apiKeyInfo)`), before
+  OmniRoute has even picked a provider, so it can't know or care that this particular request targets the
+  classifier model specifically.
+
+**The fix that was actually available and is now applied**: `x-omniroute-no-memory`, OmniRoute's own per-*request*
+opt-out (not per-key or per-model), confirmed end-to-end and traced through both sides:
+
+- OmniRoute's handling, confirmed directly in `open-sse/handlers/chatCore.ts` and its own test suite
+  (`tests/unit/no-memory-header.test.ts`): `memoryOwnerId = isNoMemoryRequested(headers) ? null : resolveMemoryOwnerId(...)`
+  — a null owner id short-circuits *both* branches in `injectMemoryAndSkills` (context injection and tool
+  injection). The test suite gives the exact accepted values: `"true"`, `"1"`, `"yes"` (case-insensitive on both
+  the header name and value); `"false"`/`"0"`/`"no"`/empty do not trigger it.
+- qwen-code's support for sending it, confirmed against the installed bundle, *not* just the docs: `modelProviders.
+  openai[].generationConfig.customHeaders` (documented at
+  [model-providers](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/model-providers/)) flows into
+  `DefaultOpenAICompatibleProvider.buildClient()` (`chunk-CXTPVBFA.js`), which passes it straight into the
+  underlying `OpenAI` SDK client as `defaultHeaders` — applied to every request made through that one model entry,
+  and *only* that entry (confirmed this is the generic OpenAI-compatible-chat client, the same one the classifier's
+  `apiType: "chat"` connection uses — not Anthropic- or Responses-API-specific plumbing).
+
+**Applied**, 2026-09-15: added to the `qwen3-4b-classifier` entry in the Windows-side `~/.qwen/settings.json`
+(`C:\Users\aerli\.qwen\settings.json`), inside its `generationConfig`, alongside the existing `contextWindowSize`
+and `extra_body`:
+
+```json
+"customHeaders": { "x-omniroute-no-memory": "true" }
+```
+
+Scoped to this one model entry only — the main `qwen3.8-27b-local` connection's `generationConfig` is untouched,
+so its own memory-context behavior (if any is relied on elsewhere) is unaffected. qwen-code's own
+`[MODEL_PROVIDERS_HOT_RELOAD]` settings watcher (confirmed present in this session's debug log) should pick this
+up on the already-running session without a restart. **Not yet verified live** — the next classifier failure (or a
+deliberate repro, per the "Reproduced directly" section above) should confirm no `memory_save`-shaped tool call
+appears in the response once this is in effect.
+
+- **Remaining fallback, if the header approach doesn't hold up**: disable memory globally for this deployment
+  (`PATCH /api/settings/memory`, `enabled: false`, or Settings → Memory in the dashboard) — blunt, but confirmed to
+  work by definition since `enabled: false` is every fresh install's default.
+- **Also worth doing regardless**: file this upstream with OmniRoute. Their own code already special-cases one
+  caller type (streaming clients) right next to this injection logic; a similar exemption for a caller that already
+  set `tool_choice` to force one specific tool would be a clean fix on their end that doesn't depend on every
+  client remembering to send an opt-out header.
+- **Not a fix, and not the problem**: nothing on the classifier-model or llama.cpp side. Qwen3-4B-Instruct-2507
+  correctly produced the right answer (`{"shouldBlock": true}`) in the one reproduction captured here — the model
+  was never at fault.
+
+## Scope note
+
+This session's earlier hypothesis that the 27B model's 48-minute total outage
+([`omniroute-direct-response-timeout-outage-2026-09-15.md`](./omniroute-direct-response-timeout-outage-2026-09-15.md))
+was caused by a ROCm/gfx1201 GPU hang is set aside here per explicit direction, not retracted — that was a
+different incident (zero successes for 48 straight minutes, a shape this memory-injection bug doesn't produce) and
+this finding doesn't bear on it either way.
+
+## Sources
+
+- Live debug log, `C:\Users\aerli\.qwen\debug\886d00eb-5b2b-4d84-b1ef-60909f75eec2.txt` (this session, 2026-09-15)
+- Direct reproduction against this deployment's `POST /v1/chat/completions` (this session, 2026-09-15)
+- Live `GET /api/keys`, `GET /api/providers`, `GET /api/providers/b78ceb4c-52f8-47ae-b245-483baa6e3fc2`,
+  `GET /api/settings/memory` against this deployment's OmniRoute instance (this session, 2026-09-15) — confirmed no
+  per-key or per-connection memory field exists in either schema
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) —
+  `open-sse/handlers/chatCore/memorySkillsInjection.ts`, `open-sse/handlers/chatCore.ts` (the
+  `isNoMemoryRequested`/`resolveMemoryOwnerId` branch), `src/lib/memory/settings.ts`, `src/lib/memory/injection.ts`,
+  `open-sse/mcp-server/tools/memoryTools.ts`, `tests/unit/no-memory-header.test.ts` (exact accepted header
+  name/value set)
+- [Qwen Code docs — Model Providers](https://qwenlm.github.io/qwen-code-docs/en/users/configuration/model-providers/)
+  (`customHeaders` field, documented under `generationConfig`)
+- Installed qwen-code bundle — `chunk-N7VWZDWW.js`, `chunk-HBU7EKY4.js` (`classifyAction`, `runSideQuery`,
+  `resolveDefaultModel`, `generateJson`, `resolveFastModelSelector`, `getFastModel`) and `chunk-CXTPVBFA.js`
+  (`DefaultOpenAICompatibleProvider.buildHeaders()`/`buildClient()`, confirming `customHeaders` reaches the actual
+  OpenAI SDK client as `defaultHeaders` for the plain chat-completions path the classifier uses) — all read
+  directly from the bundled (unminified variable names) source, not inferred from docs alone
+- Applied fix: `C:\Users\aerli\.qwen\settings.json`, `qwen3-4b-classifier` entry's `generationConfig.customHeaders`
+  (this session, 2026-09-15)
+
+## Confidence / uncertainty summary
+
+- **High confidence**: the fast model is genuinely used for classification (`stage=fast` in qwen-code's own debug
+  log, both on success and failure); the failure is a schema-validation error on an in-time response, not a
+  timeout (18.7s duration, explicit error text); OmniRoute's `memorySkillsInjection.ts` unconditionally injects
+  builtin memory tools into non-streaming completions for any memory-enabled key, with no exemption for
+  forced-single-tool callers (read directly from source); no per-key or per-model/connection memory override
+  exists in this OmniRoute version (confirmed by reading the complete live schema of both, not by absence of
+  documentation); `x-omniroute-no-memory: true` is a real, working per-request opt-out on OmniRoute's side (its
+  own test suite) and is reachable from qwen-code via `modelProviders.openai[].generationConfig.customHeaders`,
+  traced to the exact HTTP client the classifier's connection type uses (not inferred from docs alone — confirmed
+  against the bundled source's actual header-merging code).
+- **Medium confidence**: that this exact tool-injection mechanism explains the *specific* production failures seen
+  earlier in this session's testing (the reproduction matches the failure shape and the source confirms the
+  mechanism exists and applies to this call pattern, but the live debug-log failure itself wasn't captured
+  mid-flight with response inspection — only its aftermath, the error message).
+- **Low confidence / not verified**: the exact conditions under which the model picks the injected tool over the
+  intended one (one clean reproduction on the first attempt, not a characterized hit rate — the failure may not be
+  deterministic, so the `customHeaders` fix should still be watched rather than assumed to have fully resolved it
+  on the strength of this write-up alone); whether the applied `customHeaders` fix has been confirmed live yet
+  (not as of this writing — see "Applied" above).

docs/research/omniroute-non-ping-sse-stream-timeout.md

@@ -0,0 +1,60 @@
+# OmniRoute's "non-ping SSE" first-token deadline — a different timer than `STREAM_IDLE_TIMEOUT_MS`
+
+**Date:** 2026-09-10
+
+`STREAM_IDLE_TIMEOUT_MS` was raised to 180000 on 2026-09-09 (see docker-compose.yml's `omniroute`
+service) specifically to give contended `llama-server` prefill room to produce a first token. It didn't
+work: the very next morning, qwen-code sessions against `qwen3.8-27b-local` still hit repeated
+
+```
+Stream produced no non-ping SSE event within 95000ms
+```
+
+(and once at 115000ms) — both well under the 180s the compose fix set, and well under the connection's
+own `providerSpecificData.timeoutMs: 1200000` (confirmed live via `GET /api/providers/<id>`). Neither of
+those settings bounds this failure.
+
+## Root cause
+
+Per OmniRoute's own docs (`docs/reference/ENVIRONMENT.md`, "Timeout Settings" section) and a maintainer
+reply in [diegosouzapw/OmniRoute#10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602):
+
+| Variable | Default | Governs |
+|---|---|---|
+| `REQUEST_TIMEOUT_MS` | 600000 (10 min) | Overall upstream request budget. **The first non-ping SSE event's deadline inherits this one.** |
+| `STREAM_IDLE_TIMEOUT_MS` | 120000 (2 min) | Max gap between *successive* SSE chunks once streaming has already started — does not govern the wait for the first chunk. |
+| `STREAM_PING_INTERVAL_MS` | 30000 (30s) | How often OmniRoute emits its own keepalive pings on the stream — these explicitly do not count as "non-ping" events, so they can't rescue a request against the first deadline. |
+
+So the 2026-09-09 fix tuned the wrong timer for this failure mode: `STREAM_IDLE_TIMEOUT_MS` only matters
+once `llama-server` has already emitted something. The "no token at all yet" case — exactly what a large
+compact-prompt prefill on a contended local model produces — is bounded by `REQUEST_TIMEOUT_MS` instead.
+
+The observed 95000ms/115000ms figures are also *not* `REQUEST_TIMEOUT_MS`'s raw 600000ms default: OmniRoute
+computes the first-event deadline as **remaining budget**, not a flat timer — `REQUEST_TIMEOUT_MS` minus
+time already spent in OmniRoute's own request-queue/retry/cooldown cycle (`requestRetry: 3`,
+`connectionCooldown.apikey.baseCooldownMs`, provider breaker) before the request was actually dispatched
+to `llama-server`. Confirmed live via `GET /api/settings` → `resilienceSettings` on this deployment. Most
+of the 10-minute default budget was being burned by retries before the final attempt even started.
+
+## Fix
+
+Set `REQUEST_TIMEOUT_MS` explicitly, generously — applied in docker-compose.yml as
+`OMNIROUTE_REQUEST_TIMEOUT_MS` (default 1800000 / 30 min), same pattern as
+`OMNIROUTE_STREAM_IDLE_TIMEOUT_MS`. This doesn't replace the 2026-09-09 `STREAM_IDLE_TIMEOUT_MS` fix —
+that one still matters for mid-stream stalls after generation has started — it addresses the separate
+"nothing has arrived yet" case that fix didn't cover.
+
+Raising `REQUEST_TIMEOUT_MS` buys headroom; it doesn't address *why* prefill on a 50K+ token compact
+prompt can take that long in the first place. `llama-server` had no `--cache-reuse` flag set — every
+request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged
+from the previous turn. Added `--cache-reuse 256` (docker-compose.yml) so llama.cpp reuses cached KV for
+any matching ≥256-token chunk via KV-shift instead of reprocessing it, which is the actual fix for
+compact-prompt prefill time — the timeout bump above is a safety margin around it, not a substitute.
+
+## Sources
+
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `docs/reference/ENVIRONMENT.md`
+  ("Timeout Settings"), [Discussion #10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602)
+- Live `GET /api/providers/<connectionId>`, `GET /api/settings`, `GET /api/resilience` against this
+  deployment's OmniRoute instance (2026-09-10)
+- [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the related-but-distinct 30s semaphore/429 investigation

docs/research/omniroute-qwen-websearch.md

@@ -0,0 +1,330 @@
+# OmniRoute + Qwen Code CLI web search — setup research
+
+Investigates how to (a) confirm/complete OmniRoute's routing to this stack's local
+Qwen model, and (b) enable Qwen Code CLI's web-search tool, for a user running
+`qwen` from WSL against this repo's docker-compose stack.
+
+## What's already configured (verified live in WSL, 2026-09-05)
+
+Checked via `wsl.exe -- bash -lc '...'` against `~/.qwen/`:
+
+- **qwen-code CLI is installed**: `which qwen` → `/home/haylan/.local/bin/qwen`, `qwen --version` → `0.23.0`.
+- **`~/.qwen/settings.json` already points at this stack's OmniRoute gateway**, in the exact shape OmniRoute's own `setup-qwen` command produces (see below):
+  ```json
+  "modelProviders": {
+    "openai": [
+      {
+        "id": "qwen3.8-27b-local//models/Qwen3.8-27B-UD-Q4_K_XL.gguf",
+        "name": "qwen3.8-27b-local",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://proxy-ai.home/v1",
+        "generationConfig": { "contextWindowSize": 131072 }
+      }
+    ]
+  },
+  "security": { "auth": { "selectedType": "openai" } },
+  "model": {
+    "name": "qwen3.8-27b-local//models/Qwen3.8-27B-UD-Q4_K_XL.gguf",
+    "baseUrl": "http://proxy-ai.home/v1"
+  }
+  ```
+  This targets `http://proxy-ai.home/v1` (this repo's OmniRoute gateway hostname per `docs/network-access.md`), reads the API key from the `OMNIROUTE_API_KEY` env var, and matches `docs/coding-cli-setup.md`'s convention of naming the registered provider `qwen3.8-27b-local`. Two backup files (`settings.json.bak-cbm-*`, `settings.json.save`) show earlier iterations of the same config — this was set up deliberately, not a stray default.
+  - **Not a gap — verified correct**: `contextWindowSize: 131072` matches `LLAMA_CTX_SIZE / LLAMA_PARALLEL` (`262144 / 2`), not half of it. `docker-compose.yml` (lines 21–22) runs llama-server with `--ctx-size ${LLAMA_CTX_SIZE:-262144} --parallel ${LLAMA_PARALLEL:-2}`, and `.env.example` (line 39) spells out that each of the two concurrent slots gets `LLAMA_CTX_SIZE / LLAMA_PARALLEL` tokens — i.e. 131072 per slot, matching commit `23e90fe` ("cap concurrent slots at 2"). So `~/.qwen/settings.json`'s value is correctly sized to what one slot actually offers; no fix needed here.
+  - **Unverified**: whether `OMNIROUTE_API_KEY` is actually set in the WSL environment or in a `~/.qwen/.env` file — `env | grep -i qwen` in the same session showed no `OMNIROUTE_API_KEY` in the *current* shell (only `PATH` entries matched `qwen`), and `~/.qwen/.env` wasn't checked (missed in the executed probe — see Open questions). If it's unset, `qwen` calls will fail auth against OmniRoute regardless of the `web_search` setup below.
+- **No web-search config exists yet**: `env | grep -i tavily` and `env | grep -i search` both returned nothing; `settings.json` has no `tools.webSearch` key and no `mcpServers` entry for Tavily/Bailian/GLM search or for OmniRoute's own MCP server (it does have an unrelated `mcpServers.codebase-memory-mcp` stdio entry for this repo's own codebase-memory tool). `grep -ril "tavily\|websearch\|web_search\|web-search" ~/.qwen` matched only unrelated project chat-log files (from an unrelated `shopware-420-seeds` project), not any config.
+- **Conclusion**: model routing (a) is already done. Web search (b) is not configured at all — no API key, no MCP server, no built-in-tool setting.
+
+## (a) OmniRoute → local Qwen model routing
+
+Sources: this repo's `docker-compose.yml` (lines 63–133) and `.env.example`
+(lines 54–90); `README.md` §"AI gateway (OmniRoute)"; `docs/coding-cli-setup.md`;
+OmniRoute's own docs at `github.com/mckazzy/OmniRoute-run-qwen`, ref
+`release/v3.8.50`.
+
+**Current repo state**: `docker-compose.yml`'s `omniroute` service comment (lines 63–67)
+states routing is registered "once through the dashboard or `POST /api/providers`
+after first boot, not checked into this repo." `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true`
+and `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true` are already set (lines 84–88) so the
+dashboard/API will accept `llama-server`'s container-internal address instead of
+rejecting it as a private URL.
+
+**Confirmed against OmniRoute's own reference docs**
+(`docs/reference/ENVIRONMENT.md` at the pinned ref):
+- `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` — default `false`; the doc says it is
+  **"REQUIRED for self-hosted providers"** (it names LM Studio, Ollama, vLLM,
+  Llamafile, Triton, SearXNG). Confirms the repo's own comment is correct and necessary.
+- `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS` — default `true` ("local-first"); `false`
+  would block localhost/LAN/private ranges outright (cloud-metadata IPs stay
+  blocked either way).
+- `OMNIROUTE_WS_BRIDGE_SECRET` — "REQUIRED in production — when unset, all WS
+  bridge requests are rejected," generated via `openssl rand -base64 32` — matches
+  this repo's comment (lines 89–93) and `scripts/update.sh` autofill.
+
+`docs/reference/PROVIDER_REFERENCE.md` (same ref) lists **`llama-cpp`** as a
+built-in "Local, self-hosted" provider ID:
+
+> "Configure the OpenAI-compatible base URL (default: `http://127.0.0.1:8080/v1`)"
+
+This is a good match for this stack's `llama-server` container, which exposes
+port 8080 only on the internal `ai-stack` Docker network (`docker-compose.yml`
+lines 30–33, "No published host port"). Inside that network the service is
+reachable by its Compose service name, so the base URL to register should be
+`http://llama-server:8080/v1`, not `127.0.0.1` (127.0.0.1 inside the OmniRoute
+container would mean OmniRoute itself, not llama-server — they're different
+containers on the same bridge network).
+
+**Concrete steps** (dashboard, matching `docs/proxy-key-onboarding.md`'s
+existing pattern for reaching the dashboard):
+
+1. Reach the dashboard: from the R9700 box, `docker inspect -f
+   '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`, then browse
+   `http://<that-ip>:20128`; from elsewhere, SSH-tunnel
+   `ssh -L 20128:<container-ip>:20128 <host>` then browse `localhost:20128`.
+2. Providers → Add provider → **llama.cpp** (`llama-cpp` provider ID per
+   `PROVIDER_REFERENCE.md`).
+3. Set base URL to `http://llama-server:8080/v1` (the Compose service name — both
+   containers share the `ai-stack` network per `docker-compose.yml`'s `networks:
+   [ai-stack]` on both services). No API key needed (llama-server's endpoint is
+   unauthenticated internally, per `docs/network-access.md`).
+4. Register the model under that provider using the naming this repo already
+   assumes downstream (`qwen3.8-27b-local`, per `docs/coding-cli-setup.md` line 8)
+   — pick a model ID/name here and keep it consistent everywhere a CLI config
+   references it (`~/.qwen/settings.json`'s existing entry already assumes this name).
+5. Mint or reuse a virtual API key for the `qwen-code-cli` workload per
+   `docs/proxy-key-onboarding.md` (label `qwen-code-cli`), and confirm it's the
+   value behind `OMNIROUTE_API_KEY` in the WSL environment (or `~/.qwen/.env` —
+   see Open questions) that `~/.qwen/settings.json`'s `envKey` references.
+
+**OmniRoute's own automation for this exact CLI** — `docs/guides/CLI-INTEGRATIONS.md`
+at the pinned ref documents a dedicated `omniroute setup-qwen` command:
+
+> `omniroute setup-qwen --model qwen/qwen3.8-max-preview` — writes
+> `~/.qwen/settings.json` (V4 `modelProviders.openai` array) and stores
+> `OMNIROUTE_API_KEY` in `~/.qwen/.env`; supports `--yes` (non-interactive),
+> `--config-path` / `--env-path` (custom locations), and works in local or remote mode.
+
+The `~/.qwen/settings.json` found on this machine has exactly the V4
+`modelProviders.openai` shape this command produces, and the two `.bak`/`.save`
+files back that up — this was very likely already run once, pointed at whichever
+model ID was registered in the dashboard at the time (the `id` field embeds the
+GGUF filename, `qwen3.8-27b-local//models/Qwen3.8-27B-UD-Q4_K_XL.gguf`, matching
+`.env.example`'s `LLAMA_MODEL_FILE`). Re-running it after registering/renaming
+the provider in step 2–4 above is the fastest way to refresh this file if the
+registered model ID ever changes (`contextWindowSize: 131072` itself is already
+correct — see note above on `--parallel`).
+
+## (b) Qwen Code CLI web search
+
+Sources: `qwenlm.github.io/qwen-code-docs/en/developers/tools/web-search/`,
+`.../en/developers/tools/mcp-server/`, `.../en/users/configuration/settings/`;
+OmniRoute's `docs/frameworks/MCP-SERVER.md` and `docs/reference/PROVIDER_REFERENCE.md`
+at `release/v3.8.50`.
+
+**Qwen Code's web-search docs page states plainly**: the *original* built-in
+`web_search` tool ("Tavily/Google/GLM/DashScope multi-provider") **"and its
+configuration were removed."** Current options, per that same page:
+
+1. **New built-in `web_search` tool** — DashScope-only now, not multi-provider.
+   Needs `tools.webSearch.enabled: true` and `tools.webSearch.model` (e.g.
+   `"qwen3.6-plus"`) in `settings.json`, or equivalent env vars if `settings.json`
+   can't be edited; requires a `DASHSCOPE_API_KEY` (Alibaba Cloud). It "issues a
+   self-contained search request to a small auxiliary model with DashScope's
+   server-side `web_search` (and `web_extractor`) tools, and returns the
+   narrated findings plus source URLs" — i.e. it calls out to Alibaba's cloud,
+   not this stack's local model or SearXNG.
+   - **Caveat**: `users/configuration/settings/` (the canonical settings-schema
+     page) does **not** list `tools.webSearch` anywhere among its documented
+     `tools.*` keys — only `tools.sandbox`, `tools.shell`, `tools.core`,
+     `tools.exclude`, `tools.disabled`. This key may be genuinely undocumented
+     there, or newer than that page's last update. Treat `tools.webSearch` as
+     unconfirmed against the settings schema itself — verify with `qwen --help`
+     or by testing once a `DASHSCOPE_API_KEY` is available (see Open questions).
+2. **MCP-based search** — three named services: Alibaba Cloud Bailian WebSearch,
+   Tavily WebSearch, GLM WebSearch Prime — each added as an `mcpServers` entry
+   in `settings.json`. Confirmed schema from `developers/tools/mcp-server/`:
+   HTTP/SSE servers use `httpUrl` (or `url` for SSE) plus an optional `headers`
+   object, e.g.:
+   ```json
+   { "mcpServers": { "tavily": {
+       "httpUrl": "https://mcp.tavily.com/mcp/?tavilyApiKey=${TAVILY_API_KEY}"
+   } } }
+   ```
+   (stdio servers instead use `command`/`args`/`env`/`cwd`, as the existing
+   `codebase-memory-mcp` entry in this machine's `~/.qwen/settings.json` does.)
+
+**Neither of Qwen Code's own two paths uses this stack's existing SearXNG
+integration.** But OmniRoute — already in front of this stack's model — has its
+own MCP server with a **built-in multi-provider web-search tool**, and this
+repo already wires SearXNG through OmniRoute (`README.md` §"Web search":
+"The gateway also fronts SearXNG-backed web search"; `.env.example`'s
+`SEARXNG_LAN_IP` / `search.home` extra_hosts entry in `docker-compose.yml`
+lines 106–109). OmniRoute's `docs/frameworks/MCP-SERVER.md` (pinned ref):
+
+> "Web search through OmniRoute search gateway
+> (Serper/Brave/Perplexity/Exa/Tavily/Google PSE/Linkup/SearchAPI/SearXNG) with
+> failover" — exposed as an `omniroute_web_search` tool requiring the
+> `execute:search` scope.
+
+And `docs/reference/PROVIDER_REFERENCE.md` lists `searxng-search` as one of its
+12 built-in search-provider IDs: **"API key is optional. Set your SearXNG base
+URL. Some instances may require a bearer token for access."** — meaning
+SearXNG can be registered as a search provider in the OmniRoute dashboard the
+same way `llama-cpp` is registered as a model provider, no separate API key
+needed for a self-hosted SearXNG instance.
+
+**This means the path that reuses what's already deployed in this stack (SearXNG,
+already reachable from OmniRoute via `search.home`) is: connect qwen-code to
+OmniRoute's MCP server, not to Tavily/DashScope/GLM directly.** Concrete steps:
+
+1. In the OmniRoute dashboard, register SearXNG as a search provider
+   (`searxng-search`), pointing at `http://search.home` (already resolvable
+   inside the OmniRoute container via the `extra_hosts` entry in
+   `docker-compose.yml`). This may already be done — `README.md` implies the
+   gateway already fronts SearXNG-backed search, but confirm live in the
+   dashboard since, per the same `docker-compose.yml` comment (lines 63–67),
+   provider registration isn't checked into this repo.
+2. Mint an API key scoped for MCP search use — OmniRoute's `MCP-SERVER.md`
+   names `execute:search` (to actually call the search tool) and `mcp:connect`
+   (narrow, MCP-connect-only) as the relevant scopes; `manage`/`admin` also work
+   but are broader than needed.
+3. Add an `mcpServers` entry to `~/.qwen/settings.json` pointing at OmniRoute's
+   MCP endpoint, using the same `httpUrl`/`headers` shape Qwen Code already
+   supports for Tavily:
+   ```json
+   {
+     "mcpServers": {
+       "omniroute-search": {
+         "httpUrl": "http://proxy-ai.home/api/mcp/stream",
+         "headers": { "Authorization": "Bearer ${OMNIROUTE_SEARCH_KEY}" }
+       }
+     }
+   }
+   ```
+   (`proxy-ai.home` matches the hostname the model-provider entry already uses
+   in this same file; swap in whatever host:port actually fronts OmniRoute's API
+   port from WSL — `docs/network-access.md` says `proxy-ai.home` points at
+   `${OMNIROUTE_PORT:-4000}`, the *API* port, and `docker-compose.yml`/`.env.example`
+   separately track `OMNIROUTE_API_PORT` (default `20129`) as the
+   container-internal port — confirm which one NPM actually proxies to before
+   trusting the `/api/mcp/stream` path resolves through `proxy-ai.home` unchanged;
+   this wasn't independently verified against a live instance, see Open questions.)
+4. Set `OMNIROUTE_SEARCH_KEY` in the WSL shell profile (or in `~/.qwen/.env`,
+   consistent with how `setup-qwen` already stores `OMNIROUTE_API_KEY` there).
+5. Restart `qwen`; the model should now see an MCP tool for web search backed by
+   this stack's own SearXNG, routed and rate-limited the same way its LLM calls
+   already are.
+
+If instead the goal is simply "get *any* web search working fastest, reuse
+nothing," the plain Tavily-MCP or DashScope built-in-tool paths above are
+simpler (one API key, no dashboard provider registration) — but they bypass
+this stack's OmniRoute/SearXNG setup entirely and send queries to an external
+paid API instead.
+
+## Follow-up verification (2026-09-05, live checks)
+
+- **`OMNIROUTE_API_KEY` — confirmed set and working.** It's exported from
+  `~/.bashrc` (line 133), *not* `~/.qwen/.env` — invisible to a non-interactive
+  `bash -lc` probe because `.bashrc`'s standard top-of-file guard
+  (`case $- in *i*) ;; *) return;; esac`) skips the rest of the file for
+  non-interactive shells; a real interactive shell (`bash -ic`, or `wsl` +
+  `qwen` as actually run) sources it fine. Verified: `curl -H "Authorization:
+  Bearer $OMNIROUTE_API_KEY" http://proxy-ai.home/v1/models` → `200`. Routing (a)
+  is confirmed end-to-end, no action needed.
+- **MCP endpoint located from primary source** — fetched OmniRoute's
+  `docs/frameworks/MCP-SERVER.md` at `release/v3.8.50` directly. Resolves the
+  port ambiguity above: the MCP server runs on **port 20128** (dashboard/API
+  port), paths `/api/mcp/stream` (streamable HTTP), `/api/mcp/sse`, and
+  `/api/mcp/status`. It states: `/api/mcp/*` is in OmniRoute's `LOCAL_ONLY` authz
+  tier (`src/server/authz/routeGuard.ts`) — loopback-only by default; a
+  non-loopback client needs a key carrying the `manage` scope or the narrower
+  `mcp:connect` scope (added v3.8.0), and the server's Settings must have
+  `mcpEnabled` on with the matching `mcpTransport` selected. `omniroute_web_search`
+  itself additionally needs `execute:search`. No separate "MCP key type" exists —
+  same provider keys, different scopes.
+- **Live probe result**: `curl http://proxy-ai.home:20128/api/mcp/status` returns
+  `{"error":{"code":"AUTH_001","message":"Authentication required"}}` **identically
+  with or without** the `Authorization: Bearer $OMNIROUTE_API_KEY` header — the
+  existing model-routing key isn't recognized on this route at all, consistent
+  with it lacking `mcp:connect`/`manage`/`execute:search` scope and/or
+  `mcpEnabled` not yet being turned on in the dashboard. This is dashboard-side
+  state (not in git, no session credentials available from this environment) —
+  genuinely needs a human with dashboard access, not another probe.
+- **Config prepared** to unblock as soon as that's done: added an
+  `omniroute-search` entry to `~/.qwen/settings.json`'s `mcpServers` (backed up
+  the prior file first as `settings.json.bak-wayfinder-<timestamp>`):
+  ```json
+  "omniroute-search": {
+    "httpUrl": "http://proxy-ai.home:20128/api/mcp/stream",
+    "headers": { "Authorization": "Bearer ${OMNIROUTE_SEARCH_KEY}" }
+  }
+  ```
+  Deliberately a separate env var (`OMNIROUTE_SEARCH_KEY`), not reusing
+  `OMNIROUTE_API_KEY`, so the search-scoped key stays distinct from the
+  model-routing key — matches `docs/proxy-key-onboarding.md`'s per-workload
+  key pattern.
+
+## Resolution (2026-09-05, completed)
+
+The dashboard steps above turned out to need a different diagnosis than
+originally guessed — walked through live with a `oma_live_...` management
+token and a rotating set of `sk-...` provider keys the user supplied:
+
+- **`/api/providers` (management API) showed zero search providers at all**
+  — not a misconfigured `searxng-search` entry, it simply didn't exist as a
+  connection anymore (9 connections total, all LLM/chat providers). The
+  CHANGELOG at the pinned ref was checked and shows `/v1/search` under active
+  feature development (a `feat(search)` entry adding Firecrawl support), so
+  this wasn't an OmniRoute-side removal of the search system — the row was
+  just gone from this instance's own database (reason unconfirmed: update
+  migration vs. prior manual removal).
+- **Created it via the API**, not the dashboard UI — `POST /api/providers`
+  turned out to accept the same generic connection schema used for LLM
+  providers: `{"provider":"searxng-search","name":"searxng"}` (Zod-validated;
+  an empty-body POST surfaced the required fields). Then
+  `PATCH /api/providers/<id>` with `{"providerSpecificData":{"baseUrl":"http://search.home/search"}}`
+  set the real URL, replacing the catalog default.
+- **Verified end-to-end**: `POST /v1/search` with `provider: "searxng-search"`
+  returned real results (5 hits, `search_cost_usd: 0`, `cached: false`,
+  `response_time_ms: 4495`) — confirms `search.home`'s `extra_hosts` mapping
+  in `docker-compose.yml` resolves correctly from inside the OmniRoute
+  container and the whole chain (OmniRoute → SearXNG → results) works.
+- **`/api/mcp/status` confirmed `scopesEnforced: false`** on this instance —
+  the `mcp:connect`/`execute:search` scope requirement documented upstream
+  isn't actually being enforced here, so any valid provider key connects.
+  `mcpEnabled: true` already, transport `streamable-http`.
+- **Key rotation caveat hit live**: the first `sk-...` key the user shared
+  went from working to a flat 401 on *every* route (including `/v1/models`)
+  partway through testing — consistent with it having been revoked/rotated
+  server-side. A second key worked immediately. If this setup stops working
+  later, check for exactly this before re-diagnosing the whole chain.
+- **Final `~/.qwen/settings.json` `mcpServers` entry** (confirmed connected
+  via `qwen mcp list` → `✓ omniroute-search: ... - Connected`):
+  ```json
+  "omniroute-search": {
+    "httpUrl": "http://proxy-ai.home/api/mcp/stream",
+    "headers": { "Authorization": "Bearer ${OMNIROUTE_SEARCH_KEY}" }
+  }
+  ```
+  `OMNIROUTE_SEARCH_KEY` is exported in `~/.bashrc` alongside the existing
+  `OMNIROUTE_API_KEY`, holding the second (working) `sk-...` key.
+
+**Status: done.** `qwen` in WSL has a connected `omniroute-search` MCP server
+backed by this stack's own SearXNG instance — no external search API, no
+Alibaba DashScope dependency. Not yet exercised: an actual `qwen` chat turn
+that triggers the `omniroute_web_search` tool call (only the MCP handshake
+and the raw `/v1/search` call were verified directly).
+
+## Open questions / unverified
+
+- **`tools.webSearch.*` settings.json schema** — described on Qwen Code's
+  web-search doc page but absent from the canonical settings-schema page; not
+  independently confirmed (e.g. via `qwen --help` or source) — moot for this
+  setup since the MCP path (above) is what's being wired in, not the
+  DashScope-only built-in tool.
+- **DashScope vs SearXNG data-residency/cost tradeoffs** — out of scope here,
+  but worth noting the built-in `web_search` tool sends queries to Alibaba
+  Cloud regardless of this stack being otherwise fully self-hosted.
+- OmniRoute's own docs (already flagged in this repo's `README.md`) describe
+  stealth/anti-detection and TLS-interception features elsewhere in its repo;
+  none of that is exercised by anything in this note, but it's the same caveat
+  `README.md` already carries forward from issue #31.

docs/research/ponytail-audit-2026-09-09.md

@@ -0,0 +1,64 @@
+# Ponytail audit — repo-wide over-engineering scan, 2026-09-09
+
+Whole-tree audit (ponytail-audit skill), not a diff review. The only real
+code in this repo is `scripts/update.sh` (253L) and `scripts/switch-model.sh`
+(50L), plus `docker-compose.yml` and `.env.example`; the rest is docs.
+`switch-model.sh` and the compose comments (ROCm GID workarounds,
+`GPU_MAX_HW_QUEUES` rationale, lazytainer label placement) are load-bearing
+and lean — left alone. Scope: over-engineering and complexity only;
+correctness/security/performance out of scope. Findings ranked biggest cut
+first. One-shot report — nothing was applied.
+
+## Findings
+
+1. **`delete:` the `downloader-fast` line — it references a service removed
+   in `5d6a17f` ("feat: remove llama-server-fast") and no longer exists in
+   `docker-compose.yml`.** Under `set -euo pipefail`,
+   `docker compose run downloader-fast` errors on the unknown service and
+   **aborts every `update.sh` run** right after config sync, before omniroute
+   comes up. Dead code that also breaks the mandatory deploy flow.
+   *Remove the line.* [scripts/update.sh:235]
+
+2. **`delete:` the entire gum path — `ensure_gum()` (~27L, L59–85), the
+   `GUM_VERSION`/`GUM_DIR`/`GUM_BIN` vars (L56–58), the vendored
+   `scripts/vendor/gum_0.14.5_Linux_x86_64.tar.gz` (4.4MB checked into git),
+   the download fallback, and the `if [ -n "$gum_bin" ]` branch (L123,
+   L127–130).** The plain-bash fallback (L131–148) already makes the
+   *identical* decision (which keys take the new value) whenever gum is
+   absent; the gum TUI is a speculative nicer prompt on top of a working
+   path. ~4.4MB in git + arch detection + a `.cache/gum` layer, all to
+   prettify a rare interactive conflict. *Replacement: nothing — always use
+   the plain-bash one-screen prompt.* [scripts/update.sh, scripts/vendor/]
+
+3. **`delete:` stale `llama-server-fast` / `fastModel` references — the fast
+   model was removed but the docs still describe a two-model Qwen Code
+   setup.** `.env.example:98` comment still lists it;
+   `docs/coding-cli-setup/index.md:32` says "2 models: chat + `fastModel`";
+   and `docs/coding-cli-setup/qwen-code.md` carries a whole fast-model
+   section (11 refs: the `fastModel` config block, `LLAMA_FAST_CTX_SIZE`
+   notes, Qwen3-4B). *Rewrite to single-model.* [docs/coding-cli-setup/
+   qwen-code.md, index.md, .env.example:98]
+
+4. **`shrink:` the 3× repeated `test -f … || curl …` blocks in
+   `downloader-comfyui` (YAML L85–99, ~15 lines) → a `for` loop over the 3
+   model files (~5 lines).** *Low confidence:* the env var names are
+   non-uniform (`COMFYUI_DIFFUSION_MODEL_FILE` / `TEXT_ENCODER_FILE` /
+   `VAE_FILE`), so the loop needs a small `case` — marginal win, and it
+   matches the house "one-off downloader" style. [docker-compose.yml]
+
+5. **`yagni:` (verify-first) qdrant + neo4j run with no consumer in the
+   stack yet** — added ahead of the RAG app via the `feat-rag-databases`
+   merge; nothing writes to them. Two always-on DBs for a feature that
+   isn't wired. *Confirm the RAG consumer is still on the roadmap before
+   keeping both; the compose comment already concedes neo4j "can absorb
+   qdrant's job later."* Low confidence — deliberate tracked decision, and
+   cheap to leave running. [docker-compose.yml]
+
+## Net
+
+`net: -45 lines script/compose (+~15 stale doc lines), -1 dep (gum, 4.4MB
+vendored binary) possible.`
+
+No out-of-scope (correctness/security/performance) findings. #1 is the one
+to fix first — it's not just bloat, it's the deploy script halting on every
+run.

docs/research/rocm-gpu-pin-and-render-group.md

@@ -0,0 +1,95 @@
+# Research: GPU pinned at 100% with two concurrent llama.cpp containers, and the intermittent "render" group startup error
+
+**Question:** After adding `llama-server-fast` (#44), real-hardware testing on the R9700
+showed `rocm-smi` pinned at 100% GPU / ~73-101W whenever both `llama-server` and
+`llama-server-fast` run concurrently, dropping to 3% / ~25-60W the moment either one
+alone is stopped. Separately, `docker compose up` intermittently failed with
+`Error response from daemon: unable to find group render: no matching entries in group file`
+— confirmed new since the second GPU service was added. See issue #5's comment thread
+for the raw `rocm-smi`/`free -h` output this doc is diagnosing.
+
+## GPU pin: root cause and fix
+
+**Confirmed via #5's own data**: either container alone is fine (3% GPU, low power).
+The pin only appears with two concurrent HIP-context-holding processes on the same
+GPU. This matches `ROCm/ROCm#5706` (already flagged as a risk in map #1) — full
+comment thread confirms:
+
+- Root cause: an AMD MES (Micro Engine Scheduler) firmware bug triggered by HIP
+  hardware-queue creation, pinning the GPU at boost clock the moment ROCm
+  initializes a queue. Not llama.cpp-specific — reproduced with vLLM and bare
+  PyTorch ROCm too. Source: [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706)
+  (`tcgu-amd`, AMD engineer, confirms MES firmware root cause; closed as
+  "fixed" in March, but a report as recent as May 24 shows it recurring even on
+  patched firmware/kernel).
+- **Validated workaround**: `GPU_MAX_HW_QUEUES=1` as a container env var. One
+  report ran a controlled before/after on the exact image this stack uses
+  (`ghcr.io/ggml-org/llama.cpp:server-rocm`, R9700/gfx1201):
+  baseline 100% GPU / 95W → with the var set, 3% GPU / 22W, VRAM unchanged.
+  Source: same thread, `interconnectedMe`'s comment.
+- **Semantics** (why this should apply to our two-container case, not just the
+  single-process case tested above): `GPU_MAX_HW_QUEUES` is a **per-process**
+  HIP runtime setting — it caps how many HSA/hardware queues *that process's*
+  HIP runtime allocates, default higher (over-subscription is what causes the
+  penalty). Source: [AMD ROCm workload-optimization docs](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html).
+  Since it's per-process, setting it on *each* container independently is the
+  correct scope — it should reduce total concurrent hardware-queue creation
+  across both processes, which is the trigger condition MES chokes on.
+  **Caveat**: no primary source explicitly tested two concurrent containers
+  both set to `GPU_MAX_HW_QUEUES=1` — this is a well-grounded extrapolation
+  from confirmed per-process semantics and the same root-cause mechanism, not
+  a directly-reproduced fix for our exact topology. Verify with `rocm-smi`
+  after applying, both containers up.
+
+## "unable to find group render" — a real Docker bug, not flaky hardware
+
+This is a known, documented Docker limitation, not something specific to this
+stack: `group_add` by **name** requires Docker to resolve that name against
+the **container's own** `/etc/group` file — if the image doesn't define a
+`render` entry there (common for minimal/slim base images), resolution fails.
+Source: [docker/cli#4714](https://github.com/docker/cli/issues/4714)
+("`docker run --group-add` by name doesn't add group from host as
+documented") and [docker/compose#7277](https://github.com/docker/compose/issues/7277)
+(same "no matching entries in group file" error).
+
+Confirms why it's now intermittent rather than always-broken: this repo's
+`docker-compose.yml` uses `group_add: [video, render]` (plain names) on
+**three** GPU services now (`llama-server`, `llama-server-fast`, `comfyui`).
+Docker Compose starts containers concurrently, and each does its own
+name-resolution lookup independently — with only one GPU service before #44,
+the resolution almost always won its race; with two (soon three, once
+`comfyui`'s downloader/model land per #46) the odds of losing that race and
+hitting the unresolved-name path go up. This is consistent with the user's
+own observation that it's new since the second GPU service.
+
+**Fix, already precedented in this repo**: `scripts/update.sh` already
+resolves the host's real `video`/`render` **numeric GIDs** for the `comfyui`
+service (`COMFYUI_VIDEO_GID`/`COMFYUI_RENDER_GID`, passed as app-level env
+vars) — but `comfyui`'s own `group_add:` still uses plain names too, so it
+isn't actually protected by that either. The correct fix per the Docker
+issues above: use the resolved **numeric GIDs** in `group_add:` itself
+(Compose accepts numeric strings directly), not names, on all three GPU
+services. Numeric GIDs skip the name-resolution step entirely, eliminating
+both the flakiness and the race.
+
+## Recommendation
+
+1. Add `GPU_MAX_HW_QUEUES=1` to both `llama-server` and `llama-server-fast`'s
+   `environment:` blocks. Verify with `rocm-smi` after redeploy, both
+   containers up — this is the one part of this doc that's extrapolated
+   rather than directly reproduced, so real confirmation matters here.
+2. Resolve host `video`/`render` GIDs once (generalize the existing
+   `COMFYUI_VIDEO_GID`/`COMFYUI_RENDER_GID` pattern in `scripts/update.sh`
+   to shared `HOST_VIDEO_GID`/`HOST_RENDER_GID` vars), and switch
+   `group_add:` on all three GPU services (`llama-server`,
+   `llama-server-fast`, `comfyui`) from `[video, render]` (names) to the
+   resolved numeric GIDs. Removes the race entirely rather than reducing its
+   odds.
+
+## Sources
+
+- [ROCm/ROCm#5706 — full comment thread](https://github.com/ROCm/ROCm/issues/5706)
+- [AMD ROCm — MI300/MI350 workload optimization docs (GPU_MAX_HW_QUEUES)](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
+- [docker/cli#4714 — group_add by name doesn't work as documented](https://github.com/docker/cli/issues/4714)
+- [docker/compose#7277 — "no matching entries in group file"](https://github.com/docker/compose/issues/7277)
+- This repo's issue #5 (real-hardware `rocm-smi`/`free -h` evidence this doc diagnoses)

litellm-config.yaml

@@ -1,34 +0,0 @@
-model_list:
-  - model_name: qwen3.8-27b-local
-    litellm_params:
-      # Static name — llama.cpp serves whatever model it loaded regardless of
-      # what's requested here; this string isn't shell-expanded (this file
-      # isn't docker-compose.yml, .env vars don't reach it).
-      model: openai/qwen3.8-27b-local
-      api_base: http://llama-server:8080/v1
-      api_key: local
-      # Qwen3 is a reasoning model — it spends output tokens on
-      # reasoning_content before ever writing content. Callers that don't
-      # set their own max_tokens (Open WebUI's default request didn't) hit
-      # llama.cpp's low default, so the model runs out mid-thought and
-      # content comes back empty. This is a floor, not a cap — any caller
-      # that passes its own max_tokens still overrides it.
-      max_tokens: 4096
-    model_info:
-      # Shadow cloud-cost estimate — priced against Claude Sonnet 5's published
-      # rate, not real spend (this proxy only ever routes to the local model).
-      # Source: https://platform.claude.com/docs/en/about-claude/pricing,
-      # checked 2026-08-25. Update these two numbers if that page changes.
-      input_cost_per_token: 0.000002   # $2 / MTok
-      output_cost_per_token: 0.00001   # $10 / MTok
-
-router_settings:
-  # ponytail: LiteLLM's request-prioritization scheduler is beta (see
-  # docs/proxy-request-priority.md) — exact settings key/shape must be
-  # confirmed against LiteLLM's current docs and smoke-tested against
-  # llama.cpp before workloads depend on it. Single-instance deployment,
-  # no Redis configured — add one only if the scheduler turns out to need it.
-  enable_priority_scheduling: true
-
-general_settings:
-  master_key: os.environ/LITELLM_MASTER_KEY

scripts/download-model.sh

@@ -1,12 +0,0 @@
-#!/usr/bin/env bash
-# Downloads the model GGUF straight into the `models` named volume via a
-# one-off container — no huggingface-cli or host bind-mount needed.
-#
-# ponytail: hardcodes the one model this stack is built for (see the
-# `downloader` service in docker-compose.yml for the actual URL/filename).
-# Set LLAMA_MODEL_FILE in .env first if you're using a different quant.
-set -euo pipefail
-cd "$(dirname "$0")/.."
-
-docker compose --profile tools run --rm downloader
-echo "Model downloaded into the 'models' volume."

scripts/generate-secrets.sh

@@ -1,30 +0,0 @@
-#!/usr/bin/env bash
-# Generates random values for the secrets docker-compose.yml requires
-# (LITELLM_MASTER_KEY, LITELLM_SALT_KEY, LITELLM_DB_PASSWORD, UI_PASSWORD)
-# and writes them into .env — creating it from .env.example first if it
-# doesn't exist.
-#
-# ponytail: only fills in blank values, never overwrites ones you've already
-# set — safe to re-run. Re-running won't touch LITELLM_SALT_KEY once it's
-# set; changing it after first run makes existing encrypted data unreadable.
-set -euo pipefail
-cd "$(dirname "$0")/.."
-
-[ -f .env ] || cp .env.example .env
-
-set_if_blank() {
-  local key="$1" value="$2"
-  if grep -qE "^${key}=.*[^[:space:]]" .env; then
-    echo "${key}: already set, skipping."
-  else
-    sed -i "s|^${key}=.*|${key}=${value}|" .env
-    echo "${key}: generated."
-  fi
-}
-
-set_if_blank LITELLM_MASTER_KEY "$(openssl rand -hex 32)"
-set_if_blank LITELLM_SALT_KEY "$(openssl rand -hex 32)"
-set_if_blank LITELLM_DB_PASSWORD "$(openssl rand -hex 32)"
-set_if_blank UI_PASSWORD "$(openssl rand -hex 16)"
-
-echo "Done. Review .env, then set OPENWEBUI_LITELLM_KEY per docs/proxy-key-onboarding.md."

scripts/switch-model.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Swap GPU residency between llama-server (Qwen) and comfyui — they never
+# run concurrently, VRAM doesn't fit both (see issue #38's map). Manual
+# invocation only, no auto-switching.
+#
+# Bypasses lazytainer entirely and drives docker compose directly — its
+# idle-stop can't be used for this. Root cause (see
+# docs/research/lazytainer-omniroute-idle-stop.md, issue #40): lazytainer's
+# packet-threshold detector is source-blind and can't tell OmniRoute's
+# periodic health-check pings apart from real traffic on the same port, so
+# it never reliably sleeps a service on its own. A scripted swap always
+# knows which service should go up/down, so it doesn't need that heuristic.
+#
+# OmniRoute may show the just-stopped provider as errored/offline in its
+# dashboard for up to CREDENTIAL_HEALTH_CHECK_INTERVAL (default 5 min) after
+# a swap — cosmetic, not a functional problem (see the research doc above).
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+usage() {
+  echo "Usage: $0 {qwen|comfyui}" >&2
+  echo "  qwen    - stop comfyui, start llama-server" >&2
+  echo "  comfyui - stop llama-server, start comfyui" >&2
+  exit 1
+}
+
+[ $# -eq 1 ] || usage
+
+case "$1" in
+  qwen)
+    from=comfyui
+    to=llama-server
+    ;;
+  comfyui)
+    from=llama-server
+    to=comfyui
+    ;;
+  *)
+    usage
+    ;;
+esac
+
+echo "==> stopping $from"
+docker compose stop "$from"
+
+echo "==> starting $to"
+docker compose up -d "$to"
+
+echo "==> status"
+docker compose ps

scripts/update.sh

@@ -0,0 +1,254 @@
+#!/usr/bin/env bash
+# The one command to run after any change to this repo (compose file, .env,
+# or a git pull) to bring the running stack in sync. Ensures secrets/keys
+# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
+# changed — safe to run any time, including with nothing to do.
+#
+# Tunable config values (LLAMA_*, ports, timeouts — anything with a real
+# default in .env.example) are synced from .env.example every run. A value
+# already matching is left alone silently. A value that DIFFERS from the
+# server's current .env is a conflict: interactively, you're shown every
+# conflict on one screen (via gum) and choose which to accept — unpicked
+# keys keep the server's current value. Non-interactively (no TTY — cron,
+# CI, piped), any conflict is a hard error unless --force is passed, which
+# accepts every new value automatically. Secrets and host-resolved values
+# (blank in .env.example — OMNIROUTE_*_SECRET/_KEY/_SALT/_PASSWORD,
+# SEARXNG_LAN_IP, COMFYUI_PUID/PGID, HOST_VIDEO_GID/RENDER_GID) are never
+# touched by this — they keep going through set_if_blank as before.
+#
+# omniroute's own routing/provider config (llama-server, search) lives in
+# its dashboard, not a checked-in file like the old litellm-config.yaml —
+# see issue #31 and docs/proxy-key-onboarding.md.
+#
+# ponytail: no rollback/backup logic — this is a single-user homelab box,
+# not a fleet. If a bad config lands, `git revert` + re-run is the recovery
+# path, not this script.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+FORCE=false
+for arg in "$@"; do
+  case "$arg" in
+    --force) FORCE=true ;;
+    *) echo "Usage: $0 [--force]" >&2; exit 1 ;;
+  esac
+done
+
+# Must run before anything else touches a file this script itself reads
+# (docker-compose.yml, .env.example, this script's own remaining lines) —
+# a self-updating script isn't guaranteed atomic against its own file
+# changing mid-run, so pulling later can execute a mix of old and new
+# script/compose content in one pass. Bit us for real: old GID-resolution
+# code ran, then this pulled in new var names docker-compose.yml now
+# requires, and nothing re-ran the (now-current) resolution step for
+# them — see issue #5's thread.
+echo "==> git pull"
+git pull --ff-only
+
+[ -f .env ] || cp .env.example .env
+
+echo "==> syncing tracked config values from .env.example"
+# ponytail: gum (charmbracelet/gum) is a single static binary. Vendored as
+# a release tarball under scripts/vendor/ (checked into git) for the R9700
+# box, which has no outbound internet access — the download fallback below
+# is only for other archs / when the vendored copy is missing or stale.
+# Cached under .cache/gum/ (gitignored) so repeat runs don't re-extract.
+GUM_VERSION="0.14.5"
+GUM_DIR="$(pwd)/.cache/gum"
+GUM_BIN="$GUM_DIR/gum"
+ensure_gum() {
+  command -v gum >/dev/null 2>&1 && { echo "gum"; return; }
+  [ -x "$GUM_BIN" ] && { echo "$GUM_BIN"; return; }
+  mkdir -p "$GUM_DIR"
+  local arch tmpdir vendored
+  case "$(uname -m)" in
+    x86_64) arch="x86_64" ;;
+    aarch64|arm64) arch="arm64" ;;
+    *) echo "no gum build for $(uname -m), falling back to plain prompts" >&2; echo ""; return ;;
+  esac
+  tmpdir="$(mktemp -d)"
+  vendored="$(pwd)/scripts/vendor/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
+  if [ -f "$vendored" ]; then
+    tar -xz -C "$tmpdir" -f "$vendored"
+  else
+    local url="https://github.com/charmbracelet/gum/releases/download/v${GUM_VERSION}/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
+    if ! curl -fsSL "$url" | tar -xz -C "$tmpdir" 2>/dev/null; then
+      echo "no vendored gum for $arch and couldn't download from $url (no internet egress? falling back to plain prompts)" >&2
+    fi
+  fi
+  if [ -n "$(find "$tmpdir" -name gum -type f 2>/dev/null)" ]; then
+    find "$tmpdir" -name gum -type f -exec cp {} "$GUM_BIN" \;
+    chmod +x "$GUM_BIN" 2>/dev/null || true
+  fi
+  rm -rf "$tmpdir"
+  [ -x "$GUM_BIN" ] && echo "$GUM_BIN" || echo ""
+}
+
+# Collect every key where .env.example has a real (non-blank) default:
+# missing from .env -> just add it (no conflict, nothing to decide);
+# present and identical -> leave alone silently; present and different ->
+# a conflict to resolve below.
+conflict_keys=()
+conflict_old=()
+conflict_new=()
+while IFS='=' read -r key value; do
+  [ -n "$value" ] || continue
+  if ! grep -qE "^${key}=" .env; then
+    echo "${key}=${value}" >> .env
+    continue
+  fi
+  current="$(grep -E "^${key}=" .env | head -1 | cut -d= -f2-)"
+  if [ "$current" != "$value" ]; then
+    conflict_keys+=("$key")
+    conflict_old+=("$current")
+    conflict_new+=("$value")
+  fi
+done < <(grep -E '^[A-Za-z_][A-Za-z0-9_]*=.+' .env.example)
+
+if [ "${#conflict_keys[@]}" -gt 0 ]; then
+  if [ "$FORCE" = true ]; then
+    for i in "${!conflict_keys[@]}"; do
+      key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
+      sed -i "s|^${key}=.*|${key}=${new}|" .env
+      echo "${key}: ${conflict_old[$i]} -> ${new} (--force)"
+    done
+  elif [ ! -t 0 ] || [ ! -t 1 ]; then
+    echo "ERROR: ${#conflict_keys[@]} config value(s) in .env differ from .env.example, and this isn't an interactive terminal:" >&2
+    for i in "${!conflict_keys[@]}"; do
+      echo "  ${conflict_keys[$i]}: ${conflict_old[$i]} (current) vs ${conflict_new[$i]} (.env.example)" >&2
+    done
+    echo "Re-run interactively to choose per-key, or pass --force to accept every new value." >&2
+    exit 1
+  else
+    gum_bin="$(ensure_gum)"
+    labels=()
+    for i in "${!conflict_keys[@]}"; do
+      labels+=("${conflict_keys[$i]}: ${conflict_old[$i]} -> ${conflict_new[$i]}")
+    done
+    if [ -n "$gum_bin" ]; then
+      selected="$(printf '%s\n' "${labels[@]}" | "$gum_bin" choose --no-limit --selected "$(printf '%s\n' "${labels[@]}" | paste -sd,)" --header "Config differs from .env.example — selected keys take the new value, unselected keep the server's current value:")"
+    else
+      # ponytail: plain-bash fallback if gum couldn't be fetched (offline,
+      # unsupported arch) — same one-screen-of-conflicts idea, cruder UI.
+      echo "Config differs from .env.example. Enter space-separated numbers to KEEP the server's current value (all others take the new value), or press enter to take every new value:"
+      for i in "${!conflict_keys[@]}"; do
+        echo "  $((i+1))) ${labels[$i]}"
+      done
+      read -r -p "> " keep_nums
+      selected=""
+      for i in "${!conflict_keys[@]}"; do
+        case " $keep_nums " in
+          *" $((i+1)) "*) ;;
+          *) selected="${selected}${labels[$i]}"$'\n' ;;
+        esac
+      done
+    fi
+    for i in "${!conflict_keys[@]}"; do
+      key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
+      if printf '%s\n' "$selected" | grep -qxF "${labels[$i]}"; then
+        sed -i "s|^${key}=.*|${key}=${new}|" .env
+        echo "${key}: ${conflict_old[$i]} -> ${new}"
+      else
+        echo "${key}: kept ${conflict_old[$i]} (server value)"
+      fi
+    done
+  fi
+fi
+
+# Handles all three cases: the KEY=value line is missing entirely (.env
+# predates that var being added to .env.example — sed can't fix what isn't
+# there, so this appends it), present but blank, or already set.
+set_if_blank() {
+  local key="$1" value="$2"
+  if grep -qE "^${key}=.*[^[:space:]]" .env; then
+    echo "${key}: already set, skipping."
+  elif grep -qE "^${key}=" .env; then
+    sed -i "s|^${key}=.*|${key}=${value}|" .env
+    echo "${key}: set."
+  else
+    echo "${key}=${value}" >> .env
+    echo "${key}: added (was missing from .env)."
+  fi
+}
+
+echo "==> filling in missing secrets"
+# Random values — safe to re-run, never overwrites what's already set.
+# OMNIROUTE_STORAGE_ENCRYPTION_KEY especially: never change it after first
+# run, existing encrypted data becomes unreadable if you do (same caveat as
+# LiteLLM's old LITELLM_SALT_KEY).
+set_if_blank OMNIROUTE_INITIAL_PASSWORD "$(openssl rand -hex 16)"
+set_if_blank OMNIROUTE_JWT_SECRET "$(openssl rand -base64 48)"
+set_if_blank OMNIROUTE_API_KEY_SECRET "$(openssl rand -hex 32)"
+set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
+set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
+set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
+set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
+set_if_blank NEO4J_PASSWORD "$(openssl rand -hex 16)"
+
+echo "==> resolving SEARXNG_LAN_IP"
+# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
+# not from inside the omniroute container (see docs/research/litellm-searxng-search.md,
+# still the relevant background even though omniroute replaced litellm — see issue #31).
+searxng_ip="$(getent hosts search.home 2>/dev/null | awk '{print $1}' | head -1)"
+if [ -n "$searxng_ip" ]; then
+  set_if_blank SEARXNG_LAN_IP "$searxng_ip"
+else
+  echo "SEARXNG_LAN_IP: couldn't resolve search.home from this host, set it manually if still blank."
+fi
+
+echo "==> resolving ComfyUI host UID"
+# yurisasc/comfyui-rocm7.1 wants these as env vars, not just group_add in
+# compose — resolve from this host, same pattern as SEARXNG_LAN_IP.
+set_if_blank COMFYUI_PUID "$(id -u)"
+set_if_blank COMFYUI_PGID "$(id -g)"
+
+echo "==> resolving host video/render GIDs (shared by every GPU service)"
+# Numeric GIDs, not names, in docker-compose.yml's group_add: — Docker
+# resolves a *named* group_add entry against the container's own /etc/group,
+# not the host's, and fails unpredictably (worse with multiple GPU services
+# starting concurrently and racing on the same lookup) — see
+# docs/research/rocm-gpu-pin-and-render-group.md and issue #5.
+video_gid="$(getent group video 2>/dev/null | cut -d: -f3)"
+render_gid="$(getent group render 2>/dev/null | cut -d: -f3)"
+if [ -n "$video_gid" ]; then
+  set_if_blank HOST_VIDEO_GID "$video_gid"
+else
+  echo "HOST_VIDEO_GID: no 'video' group on this host, set it manually if still blank."
+fi
+if [ -n "$render_gid" ]; then
+  set_if_blank HOST_RENDER_GID "$render_gid"
+else
+  echo "HOST_RENDER_GID: no 'render' group on this host, set it manually if still blank."
+fi
+
+echo "==> validating compose config"
+docker compose config -q
+
+echo "==> pulling images"
+docker compose pull --ignore-buildable
+
+echo "==> rebuilding local-build services"
+docker compose build --pull
+
+echo "==> ensuring models are downloaded (skips already-present files)"
+docker compose --profile tools run --rm downloader
+docker compose --profile tools run --rm downloader-fast
+docker compose --profile tools run --rm downloader-classifier
+docker compose --profile tools run --rm downloader-comfyui
+
+echo "==> bringing up omniroute"
+docker compose up -d --wait omniroute
+
+# ponytail: no scripted key-minting yet, unlike the old LiteLLM /key/generate
+# flow — omniroute's POST /api/keys needs a dashboard login session
+# (ManagementSessionAuth), not a static bearer key, and that flow hasn't
+# been verified against a live instance (see issue #37). No in-stack
+# workload needs a key right now (nothing left calls the gateway besides
+# coding CLIs, which mint their own by hand per docs/proxy-key-onboarding.md)
+# — revisit this script once that flow is automatable.
+
+echo "==> recreating changed services"
+docker compose up -d --remove-orphans
+
+echo "==> status"
+docker compose ps