Merge pull request 'Fix/omniroute pr agent timeout' (#54) from fix/omniroute-pr-agent-timeout into main

Reviewed-on: #54

.env.example

@@ -29,18 +29,27 @@ LLAMA_GPU_LAYERS=999
 # this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
 # 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
 LLAMA_CTX_SIZE=262144
-# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
-# default) with no flag set — under concurrent subagent fan-out, 4 requests
-# split the same GPU compute, so a large-context prefill can queue behind
-# others long enough to blow past OmniRoute's stream-idle timeout, which then
-# cancels the request (see issue-tracker notes on the timeout/cancel loop).
-# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
-# raise back toward 4 if throughput (not latency) becomes the bottleneck
-# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
-# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
-# checking that per-slot number stays comfortably above observed usage.
+# Concurrent request slots — the real hardware ceiling for this GPU, not a
+# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
+# default; dropped to 2 because more contended prefill was blowing requests
+# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
+# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
+# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
+# the timeout that matters moved to OmniRoute's per-connection
+# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
+# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
+# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
+# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
+# comfortably above observed usage.
 LLAMA_PARALLEL=2
 
+# Dedicated CPU-only backend for qwen-code's tool-call harmfulness classifier
+# (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
+# qwen-classifier service comment for the why. 131072 ctx / q8_0 KV / Q4_K_M-
+# class weights ≈ 12GiB, fits gameserver's ~17GiB free system RAM with room
+# to spare.
+LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
+
 # --- Lazytainer ---
 # Seconds of inactivity before llama-server is stopped. 900 = 15 min.
 LAZYTAINER_INACTIVE_TIMEOUT=900

AGENTS.md

@@ -1,2 +1 @@
-# AGENTS.md
-Agent instructions for this repo live in [CLAUDE.md](./CLAUDE.md) — read it before working here.
+@CLAUDE.md

QWEN.md

@@ -1,57 +1 @@
-# QWEN.md
-
-Agent instructions for working in this repo (issue tracker, domain docs, mandatory deploy flow) live in [CLAUDE.md](./CLAUDE.md) — read it first.
-
-## Project Overview
-
-Local AI inference stack for a single AMD Radeon AI PRO R9700 (32GB VRAM, gfx1201/ROCm) homelab box. Not a code project — it's a **Docker Compose deployment** plus operational docs and scripts. The stack:
-
-- **llama.cpp (ROCm)** serves Qwen3.8-27B (`Qwen3.8-27B-UD-Q4_K_XL.gguf`, fully GPU-resident, 262K context with q8_0 KV cache). Internal-only: no published host port, no auth of its own.
-- **OmniRoute** (AI gateway, replaced LiteLLM in issue #31) fronts everything: per-workload API keys, usage tracking, SearXNG-backed web search. Split ports: API `${OMNIROUTE_API_PORT:-20129}`, dashboard `${OMNIROUTE_DASHBOARD_PORT:-20128}` (dashboard is host/LAN-only, never published externally).
-- **ComfyUI** (yurisasc's ROCm image, gfx1201-tuned) for local image generation (Qwen-Image FP8). Shares the GPU with llama-server — **never runs concurrently** with it; use `./scripts/switch-model.sh`.
-- **Lazytainer** auto-suspends llama-server after idle (15 min default). Note: its packet-threshold detector can't reliably distinguish OmniRoute's health pings from real traffic (issue #40) — the scripted swap in `switch-model.sh` exists because of this.
-- **RAG stores**: Qdrant (vector, 6333) + Neo4j (graph, 7474/7687).
-- All services live on the `ai-stack` docker network. External clients reach the gateway via `proxy-ai.home` / `proxy-ai.haylan.ch` (Nginx Proxy Manager); see `docs/network-access.md`.
-
-Coding CLIs (Claude Code, Kimi, OpenCode, Qwen Code) point at the gateway, never at llama-server directly — see `docs/coding-cli-setup/index.md`.
-
-**Known risk**: Qwen3.8-27B tool-calling against llama.cpp's Anthropic shim has open upstream parser bugs — see `docs/research/qwen3.8-27b-tool-calling.md`. Don't trust it for unattended agentic work until smoke-tested (issues #5, #17).
-
-## Key Files
-
-| File | Purpose |
-|---|---|
-| `docker-compose.yml` | The whole stack. Comments in it are load-bearing (ROCm GID workarounds, GPU_MAX_HW_QUEUES, timeout rationale) — read before editing. |
-| `.env.example` | Defaults for every tunable. Secrets/host-resolved values are blank and auto-filled by `update.sh`. |
-| `scripts/update.sh` | **The one command to run after any repo change** on the server. Creates `.env`, fills blank secrets (openssl), resolves `SEARXNG_LAN_IP`/GIDs, syncs tunables from `.env.example` (conflicts are interactive or hard errors non-interactively), downloads missing model files, pulls/builds, recreates only what changed. Idempotent. |
-| `scripts/switch-model.sh {qwen\|comfyui}` | Manual GPU-residency swap between llama-server and comfyui. |
-| `docs/proxy-key-onboarding.md` | Minting per-workload API keys (manual, dashboard-only — no scripted flow yet, issue #37). |
-| `docs/coding-cli-setup/` | Per-CLI endpoint/wire-format/config recipes. |
-| `docs/research/` | Research trail behind every major decision (model choice, quant, ROCm quirks, gateway selection). Read the relevant doc before re-litigating a decision. |
-| `docs/agents/issue-tracker.md` | Gitea/`tea` CLI conventions for this repo's issue tracking. |
-| `docs/agents/domain.md` | How to consume `CONTEXT.md` + `docs/adr/` (both may not exist yet — proceed silently if absent). |
-
-## Working in This Repo
-
-### Deploying changes — mandatory flow
-
-The running stack lives on a **separate box** (the R9700 server), not wherever this repo is edited. After any change to `docker-compose.yml`, `.env.example`, or a `scripts/` file:
-
-1. Commit and push.
-2. Run `./scripts/update.sh` **on the server** to apply it.
-3. If this session has no shell access to the server, say so explicitly and tell the user to run it — never describe a change as done without step 2.
-
-### Issue tracker
-
-Issues live as Gitea issues on `git.arthurerlich.de` (repo `haylan/LLM-Server`). Use the **`tea` CLI** (already authenticated) — conventions in `docs/agents/issue-tracker.md`. Large efforts are tracked via wayfinder map issues (`wayfinder:map` label) with child tickets and native dependency blocking.
-
-### Conventions
-
-- **`ponytail:` comments** mark deliberate simplifications with their known ceiling and upgrade path (e.g. lazytainer timeout tuning lives in compose labels, not a separate config file; no rollback logic in `update.sh` — `git revert` + re-run is recovery). Keep them when editing nearby code; they encode "why this looks like a shortcut".
-- **Secrets stay blank in `.env.example`** and are filled by `update.sh` via `set_if_blank` — never hardcode or commit real secrets. `OMNIROUTE_STORAGE_ENCRYPTION_KEY` and the like must never change after first run (encrypted data becomes unreadable).
-- **GPU group access is numeric GIDs** (`HOST_VIDEO_GID`/`HOST_RENDER_GID`), resolved by `update.sh` — don't switch `group_add` to named groups (Docker resolves names against the container's `/etc/group`, not the host's; see `docs/research/rocm-gpu-pin-and-render-group.md`).
-- **One-off downloaders** (`downloader*` services) use `test -f` guards so re-runs skip existing files; they run as root because the named volume is root-owned.
-- Comments in this repo are unusually dense and explanatory — that's the house style. When changing behavior, update the comment explaining *why*, not just the *what*.
-- Tunables with real defaults live in `.env.example`; `update.sh` syncs them into the server's `.env` every run. A divergent server value is a conflict, not a silent overwrite.
-
 @CLAUDE.md

docker-compose.yml

@@ -46,6 +46,39 @@ services:
       - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
       - "lazytainer.group.llamaserver.minPacketThreshold=2"
 
+  # Dedicated backend for qwen-code's tool-call harmfulness classifier
+  # (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
+  # 27B connection — every classification call then queued behind whatever
+  # heavy generation was already running on that model's 2 GPU slots (issue
+  # tracker: OmniRoute semaphore/pr-agent investigation). CPU-only, own
+  # process, own queue: structurally can't contend with llama-server for a
+  # GPU slot. Needs >=131072 ctx (qwen-code requirement); Qwen3-4B-Instruct-2507
+  # is the smallest Qwen3 that supports that natively (262144) without
+  # RoPE-scaling — the smaller 0.6B/1.7B/4B (non-2507) models only go to
+  # 40960. Sized for gameserver's ~17GiB free RAM: q8_0/q8_0 KV at full
+  # 131072 ctx is ~9.8GiB + ~2.3GiB Q4_K_M-class weights ≈ 12GiB, comfortable
+  # headroom, and better KV quality than the q4_0 that would've been needed
+  # to squeeze this onto the GPU's ~6GiB free VRAM instead.
+  qwen-classifier:
+    image: ghcr.io/ggml-org/llama.cpp:server
+    container_name: qwen-classifier
+    volumes:
+      - models:/models
+    command: >
+      -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+      --host 0.0.0.0
+      --port 8080
+      --n-gpu-layers 0
+      --ctx-size 131072
+      --parallel 1
+      --cache-type-k q8_0
+      --cache-type-v q8_0
+      --jinja
+    expose:
+      - "8080"
+    restart: unless-stopped
+    networks: [ai-stack]
+
   # ponytail: one-off downloader, not a standing service — run via
   # `docker compose --profile tools run --rm downloader`. Folded into
   # scripts/update.sh, which runs this every time; the `test -f` guard is
@@ -67,6 +100,22 @@ services:
         curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
         https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
 
+  # Same pattern as downloader above, separate service so this one small
+  # file doesn't get re-checked/re-pulled by the big model's job.
+  downloader-classifier:
+    image: curlimages/curl:latest
+    profiles: ["tools"]
+    user: root
+    volumes:
+      - models:/models
+    entrypoint: ["sh", "-c"]
+    command:
+      - >
+        test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
+        echo "already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+
   # Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
   # text encoder, VAE) — same test -f guard pattern as downloader above.
   # See docs/research/image-generation-model-choice.md and issue #42.

scripts/update.sh

@@ -233,6 +233,7 @@ docker compose build --pull
 echo "==> ensuring models are downloaded (skips already-present files)"
 docker compose --profile tools run --rm downloader
 docker compose --profile tools run --rm downloader-fast
+docker compose --profile tools run --rm downloader-classifier
 docker compose --profile tools run --rm downloader-comfyui
 
 echo "==> bringing up omniroute"