Merge pull request 'Fix/omniroute pr agent timeout' (#54) from fix/omniroute-pr-agent-timeout into main

Reviewed-on: #54

.env.example

@@ -29,18 +29,27 @@ LLAMA_GPU_LAYERS=999
 # this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
 # 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
 LLAMA_CTX_SIZE=262144
-# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
-# default) with no flag set — under concurrent subagent fan-out, 4 requests
-# split the same GPU compute, so a large-context prefill can queue behind
-# others long enough to blow past OmniRoute's stream-idle timeout, which then
-# cancels the request (see issue-tracker notes on the timeout/cancel loop).
-# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
-# raise back toward 4 if throughput (not latency) becomes the bottleneck
-# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
-# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
-# checking that per-slot number stays comfortably above observed usage.
+# Concurrent request slots — the real hardware ceiling for this GPU, not a
+# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
+# default; dropped to 2 because more contended prefill was blowing requests
+# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
+# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
+# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
+# the timeout that matters moved to OmniRoute's per-connection
+# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
+# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
+# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
+# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
+# comfortably above observed usage.
 LLAMA_PARALLEL=2
 
+# Dedicated CPU-only backend for qwen-code's tool-call harmfulness classifier
+# (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
+# qwen-classifier service comment for the why. 131072 ctx / q8_0 KV / Q4_K_M-
+# class weights ≈ 12GiB, fits gameserver's ~17GiB free system RAM with room
+# to spare.
+LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
+
 # --- Lazytainer ---
 # Seconds of inactivity before llama-server is stopped. 900 = 15 min.
 LAZYTAINER_INACTIVE_TIMEOUT=900
@@ -105,34 +114,6 @@ COMFYUI_PGID=
 HOST_VIDEO_GID=
 HOST_RENDER_GID=
 
-# --- llama.cpp / fast model (second, always-resident instance — see
-# docs/research/fast-model-choice.md and issue #44) ---
-# Qwen3-4B-Instruct-2507: architecturally non-thinking (never emits
-# <think> blocks, unlike Qwen3-1.7B/0.6B which need a per-call toggle) —
-# picked specifically so it stays fast enough for qwen-code's Auto Mode
-# classifier (Stage 1 wants ~300ms). Same publisher (unsloth) as the main
-# model for consistency.
-LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
-# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
-# is dense too.
-LLAMA_FAST_GPU_LAYERS=999
-# --ctx-size is the TOTAL across every LLAMA_FAST_PARALLEL slot, not per
-# request — same halving already called out for the main model above.
-# Was PARALLEL=2, silently halving this to 4096/slot — too small: a real
-# classifier call (hints + environment + recent tool-call history) hit
-# "exceeds the available context size (4096 tokens)" in practice, which
-# qwen-code surfaces as "Auto Mode couldn't classify this action
-# (Classifier stage 1 unavailable)" — see issue #5. Fixed by dropping to
-# a single slot instead of raising ctx-size (no extra VRAM, and this
-# service doesn't need concurrent classifier calls the way the main
-# model needs concurrent chat sessions) — the full 8192 now goes to the
-# one slot. If hints.allow/softDeny/hardDeny ever approach their
-# 50-entries-each ceiling, raise LLAMA_FAST_CTX_SIZE instead — qwen-code
-# caps those at 200 chars x 150 entries plus 40,000 chars of
-# historical-action context, which can exceed 8192 tokens worst-case.
-LLAMA_FAST_CTX_SIZE=8192
-LLAMA_FAST_PARALLEL=1
-
 # --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
 # image-generation-model-choice.md and issue #42) ---
 # Three files: diffusion weights, text encoder, VAE — all from the official

.gitignore

@@ -5,4 +5,4 @@
 data/
 .leankg/
 .cache/
-.qwen/temp
+.qwen/tmp

AGENTS.md

@@ -0,0 +1 @@
+@CLAUDE.md

QWEN.md

@@ -0,0 +1 @@
+@CLAUDE.md

docker-compose.yml

@@ -15,8 +15,7 @@ services:
     ipc: host
     # Caps this process's HIP hardware-queue allocation — works around
     # ROCm/ROCm#5706 (GPU pinned at 100%/boost-clock whenever two
-    # concurrent HIP contexts touch this card, confirmed on real hardware
-    # against llama-server-fast below). See the research doc above.
+    # concurrent HIP contexts touch this card). See the research doc above.
     environment:
       - GPU_MAX_HW_QUEUES=1
     volumes:
@@ -47,48 +46,38 @@ services:
       - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
       - "lazytainer.group.llamaserver.minPacketThreshold=2"
 
-  llama-server-fast:
-    image: ghcr.io/ggml-org/llama.cpp:server-rocm
-    container_name: llama-server-fast
-    devices:
-      - /dev/kfd
-      - /dev/dri
-    group_add:
-      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
-      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
-    security_opt:
-      - seccomp=unconfined
-    ipc: host
-    # See llama-server's identical setting above — same fix, same bug.
-    environment:
-      - GPU_MAX_HW_QUEUES=1
+  # Dedicated backend for qwen-code's tool-call harmfulness classifier
+  # (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
+  # 27B connection — every classification call then queued behind whatever
+  # heavy generation was already running on that model's 2 GPU slots (issue
+  # tracker: OmniRoute semaphore/pr-agent investigation). CPU-only, own
+  # process, own queue: structurally can't contend with llama-server for a
+  # GPU slot. Needs >=131072 ctx (qwen-code requirement); Qwen3-4B-Instruct-2507
+  # is the smallest Qwen3 that supports that natively (262144) without
+  # RoPE-scaling — the smaller 0.6B/1.7B/4B (non-2507) models only go to
+  # 40960. Sized for gameserver's ~17GiB free RAM: q8_0/q8_0 KV at full
+  # 131072 ctx is ~9.8GiB + ~2.3GiB Q4_K_M-class weights ≈ 12GiB, comfortable
+  # headroom, and better KV quality than the q4_0 that would've been needed
+  # to squeeze this onto the GPU's ~6GiB free VRAM instead.
+  qwen-classifier:
+    image: ghcr.io/ggml-org/llama.cpp:server
+    container_name: qwen-classifier
     volumes:
       - models:/models
     command: >
-      -m /models/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf}
+      -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
       --host 0.0.0.0
       --port 8080
-      --n-gpu-layers ${LLAMA_FAST_GPU_LAYERS:-999}
-      --ctx-size ${LLAMA_FAST_CTX_SIZE:-8192}
-      --parallel ${LLAMA_FAST_PARALLEL:-1}
-      --flash-attn on
+      --n-gpu-layers 0
+      --ctx-size 131072
+      --parallel 1
       --cache-type-k q8_0
       --cache-type-v q8_0
-      --reasoning off
       --jinja
-    # Second, always-resident llama.cpp instance — small non-thinking model
-    # used as qwen-code's Auto Mode classifier fastModel, alongside the main
-    # 27B instance above. See docs/research/fast-model-choice.md and #44.
-    # Same ai-stack-only pattern as llama-server: no published host port.
     expose:
       - "8080"
     restart: unless-stopped
     networks: [ai-stack]
-    labels:
-      - "lazytainer.group.llamaserverfast.sleepMethod=stop"
-      - "lazytainer.group.llamaserverfast.ports=8080"
-      - "lazytainer.group.llamaserverfast.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
-      - "lazytainer.group.llamaserverfast.minPacketThreshold=2"
 
   # ponytail: one-off downloader, not a standing service — run via
   # `docker compose --profile tools run --rm downloader`. Folded into
@@ -111,9 +100,9 @@ services:
         curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
         https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
 
-  # Same test -f guard pattern as downloader above — fetches the second,
-  # smaller model for llama-server-fast. See issue #44.
-  downloader-fast:
+  # Same pattern as downloader above, separate service so this one small
+  # file doesn't get re-checked/re-pulled by the big model's job.
+  downloader-classifier:
     image: curlimages/curl:latest
     profiles: ["tools"]
     user: root
@@ -122,15 +111,14 @@ services:
     entrypoint: ["sh", "-c"]
     command:
       - >
-        test -f /models/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf} &&
+        test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
         echo "already downloaded, skipping" ||
-        curl -L --fail --create-dirs -o /models/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf}
-        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf}
+        curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
 
   # Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
-  # text encoder, VAE) — same test -f guard pattern as downloader/
-  # downloader-fast above. See docs/research/image-generation-model-choice.md
-  # and issue #42.
+  # text encoder, VAE) — same test -f guard pattern as downloader above.
+  # See docs/research/image-generation-model-choice.md and issue #42.
   #
   # ponytail: target paths assume ComfyUI's standard models/ layout under
   # BASE_STORAGE_PATH (/storage) — same "not independently confirmed against
@@ -184,7 +172,7 @@ services:
       # This image also wants GID env vars directly (its own README asks
       # for both these and group_add above) — same HOST_VIDEO_GID/
       # HOST_RENDER_GID resolved by scripts/update.sh, shared with
-      # llama-server/llama-server-fast now instead of comfyui-only vars.
+      # llama-server now instead of comfyui-only vars.
       - PUID=${COMFYUI_PUID}
       - PGID=${COMFYUI_PGID}
       - VIDEO_GID=${HOST_VIDEO_GID}
@@ -220,8 +208,6 @@ services:
     depends_on:
       llama-server:
         condition: service_started
-      llama-server-fast:
-        condition: service_started
     volumes:
       - omniroute-data:/app/data
     env_file: .env

docs/research/colibri-ai-stack-evaluation.md

@@ -0,0 +1,222 @@
+# Evaluating Colibrì (JustVugg/colibri) for this stack
+
+**Date:** 2026-09-08
+**Scope:** The user flagged https://github.com/JustVugg/colibri as something that "could revolutionize"
+this self-hosted AI stack. What is Colibrì actually, is it compatible with this stack's AMD
+ROCm/HIP-only single-GPU setup, and — even if compatible — does it fill a real gap versus what
+llama.cpp, OmniRoute, Qdrant, Neo4j, and ComfyUI already do here?
+
+## 1. What Colibrì actually is
+
+Colibrì is a pure-C, zero-runtime-dependency inference engine whose specific trick is treating
+"storage, RAM, and VRAM as a single inference hierarchy" so that huge mixture-of-experts (MoE)
+models — far bigger than any one machine's VRAM+RAM — can still run, by keeping the small dense
+layers resident and streaming the (much larger) set of routed experts from disk on demand with an
+LRU/"hot-store" cache and router-lookahead prefetching:
+
+> "Colibrì is an open-source inference engine designed to run frontier mixture-of-experts (MoE)
+> models on consumer hardware... The fundamental approach uses 'a JIT, but for weights' — parameters
+> are staged across storage tiers (VRAM/RAM/NVMe) based on measured routing patterns rather than kept
+> resident."
+— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
+
+It ships single-C-file implementations for eight specific model families — GLM-5.2/5.3,
+GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 (35B-A3B), and OLMoE
+— each requiring model weights pre-converted into Colibrì's own container format (`coli convert`),
+not arbitrary GGUF files:
+
+> "Eight model families with single C file implementations... GLM-5.2/5.3 | 744B | 372GB | 16GB+ ...
+> Kimi K3 | 2.8T | 1.6TB | 32GB+"
+— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
+
+It's meant to be run either from prebuilt binaries/releases, built from source (`./setup.sh` under
+`c/`), or via Docker (`docker/Dockerfile`, `docker/Dockerfile.slim`, `docker/docker-compose.yml` exist
+in-repo — confirmed present via the GitHub contents API, https://api.github.com/repos/JustVugg/colibri/contents/docker),
+exposing an OpenAI- and Anthropic-compatible HTTP API (`coli serve`, default `http://127.0.0.1:8000/v1`,
+plus `/v1/messages`) — the same shape OmniRoute already expects from a provider, per third-party
+summaries of `docs/api.md` and `docs/serve_protocol.md`
+([search result summary, secondary](https://github.com/JustVugg/colibri/blob/main/docs/api.md)).
+
+It launched July 10, 2026 and went viral on Hacker News the same day (453 points) on the strength of
+running the 744B-parameter GLM-5.2 model on a 25GB-RAM consumer box:
+
+> "A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744
+> billion parameters, on a regular PC with 25GB of memory."
+— https://gigazine.net/gsc_news/en/20260710-colibri-glm/ (secondary coverage)
+
+## 2. Hardware/runtime requirements — AMD ROCm or NVIDIA-only?
+
+**This is the load-bearing question given this stack runs llama.cpp on ROCm/HIP, not CUDA, on a
+single AMD Radeon AI PRO R9700.** The answer is more nuanced than a flat yes/no — verified against
+source, not just README prose:
+
+- **The engine is CPU-first; a GPU is optional at all.** `docs/quickstart.md` states plainly: "You do
+  **not** need a GPU. A GPU only helps if you have one; the engine runs CPU-only by default."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/quickstart.md
+- **AMD/ROCm support is real, shipped, and reasonably recent — not just a README claim.** It was
+  requested in [issue #69](https://github.com/JustVugg/colibri/issues/69) (opened 2026-07-11, "No
+  ROCM support"), the maintainer confirmed it was mechanically straightforward since HIP closely
+  mirrors CUDA, an initial PR (#112) added it but was **closed unmerged**, and a follow-up PR — tracked
+  as [#339](https://github.com/JustVugg/colibri/issues/69) — landed the actual mechanism that shipped:
+  a single shared CUDA kernel source (`backend_cuda.cu`) compiled either by `nvcc` or by `hipcc`
+  against a compatibility header:
+
+  > "backend_gpu_compat.h — 'one GPU backend source, two vendors.' ... maps CUDA runtime calls to HIP
+  > equivalents when compiled by hipcc with `HIP=1`... handles architecture-specific guards for rocWMMA
+  > availability and matrix core support across different GPU architectures (gfx906, gfx908, gfx11xx,
+  > etc.)."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/c/backend_gpu_compat.h (confirmed present
+  in the current `main` branch — this is not a stale/unmerged branch)
+
+  This shipped in a **tagged release**, not just an open PR — `CHANGELOG.md` lists "AMD GPU support" as
+  part of v1.1.0 (2026-07-22): "AMD GPU support, dual-SSD streaming, fmt=5/fmt=6 quantization formats."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md
+- **Notably, the community contributor who tested it used an RX 9070 XT / gfx1201** — the same RDNA4
+  architecture generation as this stack's Radeon AI PRO R9700 — per the PR #339 description: "validated
+  across CPU builds, HIP testing on gfx1201, and NVIDIA compatibility verification." That's a genuinely
+  favorable, non-generic signal for this specific card's GPU family, better than "AMD support exists
+  somewhere."
+- **But ROCm support is thinner and less documented than the CUDA/Metal paths.** `docs/` has `cuda.md`,
+  `metal.md`, `metal_implementation.md`, and `vulkan.md`, but **no `rocm.md` or `hip.md`** (confirmed via
+  the GitHub contents API listing of `docs/`, https://api.github.com/repos/JustVugg/colibri/contents/docs).
+  Model-specific tuning docs are written CUDA-only with no AMD mention at all — e.g.
+  `docs/qwen36-cuda-tier.md` references `COLI_CUDA=1`, `backend_cuda.cu`, and lists test hardware as
+  "RTX 3070 8 GB + Quadro RTX 4000" (both NVIDIA), with zero AMD/ROCm/HIP text anywhere in that
+  document. — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/qwen36-cuda-tier.md
+  So: the *general* GPU-acceleration mechanism supports ROCm/HIP and has been community-validated on
+  hardware close to the R9700, but the *per-model* tiering/tuning documentation and (presumably) most
+  of the maintainer's own benchmarking is CUDA-first. Treat AMD support as functional-but-secondary,
+  not a first-class, symmetrically-tested backend.
+- **A separate GPU-agnostic path also exists**: a Vulkan backend (`backend_vulkan.c`, confirmed present
+  in the `c/` directory listing) that the project positions as covering "AMD via Mesa/RADV" as a
+  vendor-neutral fallback, independent of the HIP path above — this would also be viable on the R9700
+  in principle, though vendor-neutral compute back ends are typically slower than a vendor SDK path
+  (HIP/ROCm) and no R9700/gfx1201-specific Vulkan numbers were found in the docs reviewed.
+
+**Bottom line on hardware fit: not a blocker.** Unlike a hard CUDA-only dependency, Colibrì's AMD/HIP
+path is real, shipped in a release, and specifically exercised on the same RDNA4 family as this
+stack's GPU — this is not a disqualifying finding the way it would be for a CUDA-only tool.
+
+## 3. License
+
+Apache License 2.0, confirmed by fetching `LICENSE` directly from the repo — a standard permissive
+license, fine for self-hosted use here (commercial or non-commercial, modification, redistribution all
+permitted, patent grant included, "AS IS" with no warranty).
+— https://raw.githubusercontent.com/JustVugg/colibri/main/LICENSE
+
+Model weights are licensed separately from the engine — e.g. the README notes "GLM-5.2 weights
+released by Z.ai under MIT"
+(https://raw.githubusercontent.com/JustVugg/colibri/main/README.md) — so each model's own license
+would need checking before use, same as with any GGUF today.
+
+## 4. Maturity signals
+
+Pulled from the GitHub API (https://api.github.com/repos/JustVugg/colibri) and the changelog:
+
+| Signal | Value |
+|---|---|
+| Repo created | 2026-07-01 |
+| First tagged release (v1.0.0) | 2026-07-19 |
+| Current version (as of today) | 1.10.2 (2026-09-06) |
+| Age at time of writing | ~10 weeks |
+| Stars / Forks | 27,047 / 2,963 |
+| Open issues | 104 |
+| Top contributor | JustVugg — 1,077 commits |
+| #2 contributor | ZacharyZcR — 163 commits |
+| Total contributors | 100+ (long tail, most in single digits) |
+| License | Apache 2.0 |
+| Archived? | No |
+
+Read honestly, this is **a viral, very-early-stage, single-maintainer-dominated project**, not a
+mature or slow-burn one:
+
+- It is ~10 weeks old today. The star count (27k) is wildly disproportionate to that age and reflects
+  a Hacker News front-page moment (453 points the day it launched,
+  https://gigazine.net/gsc_news/en/20260710-colibri-glm/), not organic multi-year adoption. Stars are
+  a popularity signal, not a quality proof, and here the ratio (huge stars, ~10 weeks old, one dominant
+  committer) is itself a maturity red flag worth naming rather than a mark in its favor.
+- Commit/release activity is genuinely fast — 14 tagged releases in under 8 weeks (v1.0.0 on 2026-07-19
+  through v1.10.2 on 2026-09-06, https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md)
+  — so it is actively maintained day-to-day, not abandoned. But that pace also means breaking changes
+  and security patches are frequent: v1.6.2 (2026-08-14) was itself "a security release: six
+  privately-reported memory-safety issues fixed... from untrusted input," and v1.10.2 (2026-09-06)
+  again touched "security fixes for image API" — both signs of a codebase still finding its footing on
+  hardening, not evidence of instability being the norm, but worth weighing given this stack would be
+  exposing any such server on an internal network via OmniRoute.
+- Contribution concentration is heavy: the maintainer (JustVugg) has ~6.6x the commits of the next
+  contributor, and the rest of the 100+ contributor list trails off into single-digit-commit
+  drive-by PRs (per the GitHub contributors API,
+  https://api.github.com/repos/JustVugg/colibri/contributors) — a classic "one person's viral project
+  plus a wave of small first-time PRs" shape, not an established multi-maintainer team.
+- The AMD/ROCm feature specifically has a short, thin history: requested 2026-07-11, shipped 2026-07-22
+  (11 days later, in v1.1.0), documented only implicitly (no dedicated `docs/hip.md`/`rocm.md`, unlike
+  every other backend) — i.e., it is the newest and least-independently-verified of the project's four
+  GPU backends (CUDA, Metal, Vulkan, HIP).
+
+## 5. What capability gap it would actually fill in this stack
+
+Concretely comparing against what's already running (`docker-compose.yml`):
+
+- **llama.cpp (ROCm) already fully GPU-resides the current model** — `--n-gpu-layers 999` on the
+  llama-server service means the whole Qwen3.8-27B-class GGUF sits in the R9700's VRAM and runs at
+  normal, fast, interactive token rates. Colibrì's entire value proposition is the *opposite* case:
+  models **too large to fit in VRAM+RAM at all**, accepted at the cost of streaming most of the model
+  from disk on every forward pass. For a model that already fits on this GPU (which is the whole point
+  of the current setup), Colibrì offers no benefit — llama.cpp is already doing the fast thing.
+- **The actual gap it could fill is running models this stack categorically cannot run today** — e.g.
+  GLM-5.2 (744B), DeepSeek V4 Flash (284B), or Kimi K3 (2.8T), none of which would ever fit on a single
+  R9700 regardless of quantization. Colibrì's own benchmarks make the cost of that explicit and, unlike
+  its capability claims, these are the project's self-reported numbers, not independently reproduced —
+  flagged as such:
+
+  > "6× RTX 5090 (full residency): 5.8-6.8 tokens/second decode... 128GB CPU-only desktop: ~1.8
+  > tokens/second (warm)... Single RTX 5070 Ti: 1.07 tokens/second... 25GB dev box: 0.05-0.1
+  > tokens/second (baseline)."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md (self-reported, unverified by
+  any third party found during this research)
+
+  At 0.05–2 tokens/second on any hardware remotely resembling a single-GPU workstation, this is not
+  usable for the interactive coding-CLI workloads this stack is built around (Claude Code CLI, Kimi
+  CLI, etc. routed through OmniRoute per the README) — those need low per-token latency for tool-calling
+  round trips, not throughput measured in seconds per token. It would only be plausible as an
+  occasional, patient, offline/batch capability (e.g. "let a huge model chew on a large doc overnight")
+  layered in *alongside*, not instead of, the current llama.cpp path.
+- **Disk footprint is a real new cost, not a marginal one**: 167GB–1.6TB per model
+  (https://raw.githubusercontent.com/JustVugg/colibri/main/README.md), which would need to be
+  provisioned in addition to the existing `models` Docker volume, GGUF downloads, Qdrant/Neo4j
+  volumes, and ComfyUI's model files already on this box.
+- **No overlap or replacement value for OmniRoute, Qdrant, Neo4j, or ComfyUI** — Colibrì is strictly an
+  inference-engine alternative to llama.cpp for a narrow, specific list of very large MoE models; it
+  does nothing related to gateway/key-management (OmniRoute's job), vector/graph storage (Qdrant/Neo4j's
+  job), or image generation (ComfyUI's job). Its own `coli serve` OpenAI/Anthropic-compatible endpoint
+  could in principle be registered as another OmniRoute provider the same way llama-server is today —
+  that part is mechanically plausible — but it would be adding a second, much slower inference backend
+  next to the existing fast one, not replacing or upgrading anything currently in the stack.
+
+## 6. Bottom line
+
+**Not a fit for this stack right now, and the "revolutionize" framing does not hold up** — but for a
+more specific reason than "wrong GPU vendor":
+
+- **ROCm/AMD support is real and not the blocker one might expect.** It shipped in a tagged release
+  (v1.1.0, 2026-07-22), lives in a compatibility header confirmed present on `main`
+  (`c/backend_gpu_compat.h`), and was community-tested on an RX 9070 XT / gfx1201 — the same RDNA4
+  family as this stack's R9700. This is genuinely worth noting as a positive, since it's the kind of
+  thing that usually *is* disqualifying for AMD-only stacks and here it isn't.
+- **The disqualifying issue is fit, not hardware**: Colibrì solves "run a model way too big for your
+  VRAM+RAM by streaming most of it from disk," at 0.05–2 tokens/second. This stack's actual situation is
+  the opposite — a model sized to fully fit and run fast on a single GPU via llama.cpp. Colibrì would add
+  no speed, capability, or reliability benefit to the model already running here, and its own numbers
+  show it isn't fast enough to serve the interactive coding-CLI use case this stack exists for.
+- It could only ever be interesting as a *bolt-on, offline-only* capability for occasionally running an
+  otherwise-impossible frontier-scale model (700B–2.8T params) for patient, non-interactive tasks — at
+  the cost of hundreds of GB to ~1.6TB of extra disk per model, on a ~10-week-old, single-maintainer,
+  still-hardening project (two security-patch releases already) with no ROCm-specific documentation and
+  the thinnest testing history of its four GPU backends.
+- **Recommendation: worth a passing watch, not worth integrating.** Revisit if/when: (a) the project
+  reaches a more established maturity point (6–12 months, broader contributor base, dedicated ROCm docs
+  bringing it to parity with the CUDA/Metal paths), and (b) there's an actual concrete need in this stack
+  to run a model in the 200B+ range that cannot fit on the R9700 — which isn't the case today (the
+  current model is deliberately sized to fit fully in VRAM). Until then, integrating it would add
+  operational surface (a new container, huge disk provisioning, a newer/less-hardened codebase) for no
+  measurable improvement over the existing llama.cpp/ROCm path.

docs/research/ponytail-audit-2026-09-09.md

@@ -0,0 +1,64 @@
+# Ponytail audit — repo-wide over-engineering scan, 2026-09-09
+
+Whole-tree audit (ponytail-audit skill), not a diff review. The only real
+code in this repo is `scripts/update.sh` (253L) and `scripts/switch-model.sh`
+(50L), plus `docker-compose.yml` and `.env.example`; the rest is docs.
+`switch-model.sh` and the compose comments (ROCm GID workarounds,
+`GPU_MAX_HW_QUEUES` rationale, lazytainer label placement) are load-bearing
+and lean — left alone. Scope: over-engineering and complexity only;
+correctness/security/performance out of scope. Findings ranked biggest cut
+first. One-shot report — nothing was applied.
+
+## Findings
+
+1. **`delete:` the `downloader-fast` line — it references a service removed
+   in `5d6a17f` ("feat: remove llama-server-fast") and no longer exists in
+   `docker-compose.yml`.** Under `set -euo pipefail`,
+   `docker compose run downloader-fast` errors on the unknown service and
+   **aborts every `update.sh` run** right after config sync, before omniroute
+   comes up. Dead code that also breaks the mandatory deploy flow.
+   *Remove the line.* [scripts/update.sh:235]
+
+2. **`delete:` the entire gum path — `ensure_gum()` (~27L, L59–85), the
+   `GUM_VERSION`/`GUM_DIR`/`GUM_BIN` vars (L56–58), the vendored
+   `scripts/vendor/gum_0.14.5_Linux_x86_64.tar.gz` (4.4MB checked into git),
+   the download fallback, and the `if [ -n "$gum_bin" ]` branch (L123,
+   L127–130).** The plain-bash fallback (L131–148) already makes the
+   *identical* decision (which keys take the new value) whenever gum is
+   absent; the gum TUI is a speculative nicer prompt on top of a working
+   path. ~4.4MB in git + arch detection + a `.cache/gum` layer, all to
+   prettify a rare interactive conflict. *Replacement: nothing — always use
+   the plain-bash one-screen prompt.* [scripts/update.sh, scripts/vendor/]
+
+3. **`delete:` stale `llama-server-fast` / `fastModel` references — the fast
+   model was removed but the docs still describe a two-model Qwen Code
+   setup.** `.env.example:98` comment still lists it;
+   `docs/coding-cli-setup/index.md:32` says "2 models: chat + `fastModel`";
+   and `docs/coding-cli-setup/qwen-code.md` carries a whole fast-model
+   section (11 refs: the `fastModel` config block, `LLAMA_FAST_CTX_SIZE`
+   notes, Qwen3-4B). *Rewrite to single-model.* [docs/coding-cli-setup/
+   qwen-code.md, index.md, .env.example:98]
+
+4. **`shrink:` the 3× repeated `test -f … || curl …` blocks in
+   `downloader-comfyui` (YAML L85–99, ~15 lines) → a `for` loop over the 3
+   model files (~5 lines).** *Low confidence:* the env var names are
+   non-uniform (`COMFYUI_DIFFUSION_MODEL_FILE` / `TEXT_ENCODER_FILE` /
+   `VAE_FILE`), so the loop needs a small `case` — marginal win, and it
+   matches the house "one-off downloader" style. [docker-compose.yml]
+
+5. **`yagni:` (verify-first) qdrant + neo4j run with no consumer in the
+   stack yet** — added ahead of the RAG app via the `feat-rag-databases`
+   merge; nothing writes to them. Two always-on DBs for a feature that
+   isn't wired. *Confirm the RAG consumer is still on the roadmap before
+   keeping both; the compose comment already concedes neo4j "can absorb
+   qdrant's job later."* Low confidence — deliberate tracked decision, and
+   cheap to leave running. [docker-compose.yml]
+
+## Net
+
+`net: -45 lines script/compose (+~15 stale doc lines), -1 dep (gum, 4.4MB
+vendored binary) possible.`
+
+No out-of-scope (correctness/security/performance) findings. #1 is the one
+to fix first — it's not just bloat, it's the deploy script halting on every
+run.

scripts/switch-model.sh

@@ -11,12 +11,6 @@
 # it never reliably sleeps a service on its own. A scripted swap always
 # knows which service should go up/down, so it doesn't need that heuristic.
 #
-# llama-server-fast (the small classifier model, issue #44) is NOT part of
-# this swap — it's meant to stay always-resident. Worst case with comfyui up
-# is comfyui (~25GB, Qwen-Image FP8) + llama-server-fast (~5GB) ≈ 30GB,
-# still under the 32GB card but tight — unverified on real hardware, check
-# `docker compose ps` / VRAM usage after the first real swap.
-#
 # OmniRoute may show the just-stopped provider as errored/offline in its
 # dashboard for up to CREDENTIAL_HEALTH_CHECK_INTERVAL (default 5 min) after
 # a swap — cosmetic, not a functional problem (see the research doc above).

scripts/update.sh

@@ -233,6 +233,7 @@ docker compose build --pull
 echo "==> ensuring models are downloaded (skips already-present files)"
 docker compose --profile tools run --rm downloader
 docker compose --profile tools run --rm downloader-fast
+docker compose --profile tools run --rm downloader-classifier
 docker compose --profile tools run --rm downloader-comfyui
 
 echo "==> bringing up omniroute"