Merge branch 'research/opencode-auto-compact'

Implement code changes to enhance functionality and improve performance
fix(omniroute): raise REQUEST_TIMEOUT_MS and enable --cache-reuse to stop non-ping SSE stream aborts
2026-09-14 07:44:32 +02:00 · 2026-09-11 13:15:30 +02:00 · 2026-09-10 10:12:02 +02:00 · 2026-09-09 19:10:32 +02:00 · 2026-09-09 18:10:39 +02:00 · 2026-09-09 17:04:05 +02:00
21 changed files with 1938 additions and 112 deletions
@@ -29,18 +29,25 @@ LLAMA_GPU_LAYERS=999
 # this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
 # 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
 LLAMA_CTX_SIZE=262144
-# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
-# default) with no flag set — under concurrent subagent fan-out, 4 requests
-# split the same GPU compute, so a large-context prefill can queue behind
-# others long enough to blow past OmniRoute's stream-idle timeout, which then
-# cancels the request (see issue-tracker notes on the timeout/cancel loop).
-# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
-# raise back toward 4 if throughput (not latency) becomes the bottleneck
-# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
-# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
-# checking that per-slot number stays comfortably above observed usage.
+# Concurrent request slots — the real hardware ceiling for this GPU, not a
+# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
+# default; dropped to 2 because more contended prefill was blowing requests
+# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
+# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
+# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
+# the timeout that matters moved to OmniRoute's per-connection
+# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
+# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
+# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
+# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
+# comfortably above observed usage.
 LLAMA_PARALLEL=2

+# Dedicated GPU-resident backend for qwen-code's tool-call harmfulness
+# classifier (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
+# qwen-classifier service comment for the why and the VRAM/context math.
+LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
+
 # --- Lazytainer ---
 # Seconds of inactivity before llama-server is stopped. 900 = 15 min.
 LAZYTAINER_INACTIVE_TIMEOUT=900
@@ -64,6 +71,11 @@ OMNIROUTE_DASHBOARD_PORT=20128
 # cancels it (which cancels the matching llama-server task too). 180s gives
 # contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
 OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
+# Wait budget for the *first* SSE token specifically (distinct from the
+# inter-chunk timeout above) — see
+# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a
+# contended, large-context prefill even after retries eat into the budget.
+OMNIROUTE_REQUEST_TIMEOUT_MS=1800000
 # Random values, filled in automatically by ./scripts/update.sh — leave
 # blank. Bootstrap dashboard admin password (log in at the dashboard port,
 # change it there afterwards — this is only the first-boot value):
@@ -5,4 +5,4 @@
 data/
 .leankg/
 .cache/
-.qwen/temp
+.qwen/tmp
@@ -0,0 +1 @@
+@CLAUDE.md
@@ -0,0 +1 @@
+@CLAUDE.md
@@ -14,7 +14,7 @@ See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/

 llama.cpp's own API is internal-only — everything routes through the AI gateway below.

-Pointing Claude Code CLI, Kimi CLI, or OpenCode CLI at the local endpoint: see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md).
+Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).

 **Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.

@@ -26,15 +26,10 @@ An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fr
 - Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
 - Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).

-Coding CLIs (see [`docs/coding-cli-setup.md`](docs/coding-cli-setup.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
+Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).

 **Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.

 ### Web search

 The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
-
-## What's not here
-
- **Open WebUI** — this stack has no chat UI; every client is a coding CLI. Removed rather than kept idle.
- **Gateway-level knowledgebase/memory** (`litellm-pgvector`, `pgvector-db`, a dedicated embedding model) — removed as unwanted, unrelated to OmniRoute's own lack of parity with it (see issue #31's #34). Superseded by OmniRoute's own built-in memory feature (opt-in via the dashboard, Settings → Memory): vector store is its bundled `sqlite-vec`, embeddings are a local ONNX model (Transformers.js, ~400MB, downloaded into the `omniroute-data` volume on first use) — no external services, no static config here.
@@ -30,7 +30,16 @@ services:
      --flash-attn on
      --cache-type-k q8_0
      --cache-type-v q8_0
+      --cache-reuse 256
      --jinja
+    # --cache-reuse 256: reuse cached KV for any matching prompt chunk of at
+    # least 256 tokens (KV-shift, no reprocessing) instead of reprefilling
+    # from scratch every request. Directly targets the actual root cause
+    # behind the OmniRoute non-ping SSE timeout, not just the symptom — see
+    # docs/research/omniroute-non-ping-sse-stream-timeout.md. Pairs with
+    # OmniRoute's promptCacheAffinityEnabled (dashboard default), which keeps
+    # a conversation's requests pinned to the same slot so there's a matching
+    # prefix to reuse.
    # No published host port: llama-server is reached only via the omniroute
    # gateway on the ai-stack docker network now — see issue #15. Its
    # unauthenticated API no longer needs to be LAN-reachable directly.
@@ -46,6 +55,82 @@ services:
      - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
      - "lazytainer.group.llamaserver.minPacketThreshold=2"

+  # Dedicated backend for qwen-code's tool-call harmfulness classifier
+  # (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
+  # 27B connection — every classification call then queued behind whatever
+  # heavy generation was already running on that model's 2 GPU slots (issue
+  # tracker: OmniRoute semaphore/pr-agent investigation).
+  #
+  # Tried CPU-only first (own process avoids the GPU queue entirely) — too
+  # slow in practice: real classification calls blew past OmniRoute's 60s
+  # timeout and retry-looped (504→499→504...). Moved to GPU instead.
+  #
+  # Context sizing: qwen-code's classifier transcript is hard-capped in its
+  # own source (MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000
+  # per message, packages/core/src/permissions/classifier-transcript.ts) —
+  # worst case is ~40-50K tokens, nowhere near the 131072 originally set in
+  # settings.json (that number was copied from the main model's entry, not
+  # a real qwen-code requirement). 65536 ctx gives ~1.5x margin over that
+  # worst case. Qwen3-4B-Instruct-2507 is still the model choice — smallest
+  # Qwen3 with long native context (262144) without RoPE-scaling, in case
+  # that margin ever needs to grow.
+  #
+  # VRAM: weights+KV math (~4.9GiB) predicted comfortable headroom in the
+  # ~6.1GiB free on the R9700, but measured live it actually used ~5.85GiB —
+  # left only ~700MB free, too tight. Dropping --batch-size/--ubatch-size
+  # barely moved it (~768MB free) — wrong lever. Actual cause: llama-server
+  # runs with --flash-attn on but this service was missing it — without
+  # flash attention the unfused attention compute buffer at 65536 ctx is
+  # much larger (roughly O(n^2) intermediate buffers vs flash-attn's fused,
+  # near-linear workspace), dwarfing the naive weights+KV estimate. Added
+  # --flash-attn on to match llama-server; re-verify with rocm-smi after
+  # deploy before trusting any of these numbers again. GPU_MAX_HW_QUEUES=1
+  # carried over from llama-server's comment above — same ROCm/ROCm#5706
+  # clock-pinning bug applies now that two HIP contexts (this +
+  # llama-server) share the card.
+  #
+  # --reasoning off is a no-cost safety net, not a confirmed-needed fix:
+  # ggml-org/llama.cpp#20809 (closed) documents some server builds
+  # misdetecting Qwen3-Instruct-2507 models as thinking models, routing
+  # tool-call output into reasoning_content instead of tool_calls — exactly
+  # the failure mode that ruled out the 27B model for this role in the
+  # first place. Whether the current image build still has it was never
+  # independently confirmed (see docs/research/fast-model-choice.md §4/§6).
+  qwen-classifier:
+    image: ghcr.io/ggml-org/llama.cpp:server-rocm
+    container_name: qwen-classifier
+    devices:
+      - /dev/kfd
+      - /dev/dri
+    group_add:
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
+    security_opt:
+      - seccomp=unconfined
+    ipc: host
+    environment:
+      - GPU_MAX_HW_QUEUES=1
+    volumes:
+      - models:/models
+    command: >
+      -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+      --host 0.0.0.0
+      --port 8080
+      --n-gpu-layers 28
+      --ctx-size 65536
+      --parallel 1
+      --batch-size 512
+      --ubatch-size 128
+      --flash-attn on
+      --reasoning off
+      --cache-type-k q4_0
+      --cache-type-v q4_0
+      --jinja
+    expose:
+      - "8080"
+    restart: unless-stopped
+    networks: [ai-stack]
+
  # ponytail: one-off downloader, not a standing service — run via
  # `docker compose --profile tools run --rm downloader`. Folded into
  # scripts/update.sh, which runs this every time; the `test -f` guard is
@@ -67,6 +152,22 @@ services:
        curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
        https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}

+  # Same pattern as downloader above, separate service so this one small
+  # file doesn't get re-checked/re-pulled by the big model's job.
+  downloader-classifier:
+    image: curlimages/curl:latest
+    profiles: ["tools"]
+    user: root
+    volumes:
+      - models:/models
+    entrypoint: ["sh", "-c"]
+    command:
+      - >
+        test -f /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} &&
+        echo "already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
+
  # Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
  # text encoder, VAE) — same test -f guard pattern as downloader above.
  # See docs/research/image-generation-model-choice.md and issue #42.
@@ -191,6 +292,16 @@ services:
      # LLAMA_PARALLEL above for the other half of this fix). Raised here so
      # it's tracked in git instead of a dashboard-only setting.
      - STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
+      # Different timer than STREAM_IDLE_TIMEOUT_MS above — that one only
+      # bounds gaps *between* SSE chunks once streaming has started.
+      # REQUEST_TIMEOUT_MS bounds the wait for the *first* non-ping SSE
+      # event, and it's what was still firing ("Stream produced no non-ping
+      # SSE event within 95000ms") the morning after the timeout above was
+      # raised — see docs/research/omniroute-non-ping-sse-stream-timeout.md.
+      # Default 600000 (10 min) per OmniRoute's own docs, but the effective
+      # deadline is remaining budget after retries/cooldowns eat into it, not
+      # a flat timer, so raised well past the default for headroom.
+      - REQUEST_TIMEOUT_MS=${OMNIROUTE_REQUEST_TIMEOUT_MS:-1800000}
    # Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
    # network can't resolve search.home on its own.
    extra_hosts:
@@ -1,91 +0,0 @@
-# Pointing a coding-agent CLI at this stack
-
-This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
-
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
-
-Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — this doc assumes `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
-
-`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
-
-**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
-
-> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every setup below inherits this risk identically, regardless of which CLI or wire format you use. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
-
-## Claude Code CLI
-
-Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
-
-```bash
-export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
-export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
-claude
-```
-
-Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
-
-## Kimi CLI
-
-Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
-
-```toml
-[providers.openai]
-type = "openai"
-base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
-api_key = "<kimi-cli virtual key>"
-```
-
-If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
-
-## OpenCode CLI
-
-Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
-
-**Install**:
-```bash
-curl -fsSL https://opencode.ai/install | bash
-```
-
-**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
-
-```json
-{
-  "$schema": "https://opencode.ai/config.json",
-  "provider": {
-    "aiproxy": {
-      "npm": "@ai-sdk/openai-compatible",
-      "name": "AI proxy (local)",
-      "options": {
-        "baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
-        "apiKey": "<opencode-cli virtual key>"
-      },
-      "models": {
-        "qwen3.8-27b-local": {
-          "name": "Qwen3.8-27B",
-          "limit": { "context": 65536, "output": 8192 }
-        }
-      }
-    }
-  }
-}
-```
-
-Set `limit.context` to match whatever `LLAMA_CTX_SIZE` this stack is actually running with (`.env`), not a value assumed from the model card — OpenCode uses it for its own context-management bookkeeping, not the server.
-
-Select the model with `aiproxy/qwen3.8-27b-local`.
-
-**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk above):
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
-
-## Summary
-
-| CLI | Wire format | Endpoint | Config |
-|---|---|---|---|
-| Claude Code | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
-| Kimi CLI | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
-| OpenCode | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
-
-Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/research/opencode-cli-setup.md`, `docs/proxy-key-onboarding.md`.
@@ -0,0 +1,13 @@
+# Claude Code CLI
+
+[← back to overview](index.md)
+
+Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
+
+```bash
+export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
+export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
+claude
+```
+
+Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
@@ -0,0 +1,34 @@
+# Pointing a coding-agent CLI at this stack
+
+[← back to README](../../README.md)
+
+This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
+
+- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
+- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
+
+Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
+
+`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
+
+**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
+
+> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
+
+## Per-CLI setup
+
+- [Claude Code CLI](claude-code.md)
+- [Kimi CLI](kimi-cli.md)
+- [OpenCode CLI](opencode.md)
+- [Qwen Code CLI](qwen-code.md)
+
+## Summary
+
+| CLI | Wire format | Endpoint | Config |
+|---|---|---|---|
+| [Claude Code](claude-code.md) | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
+| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
+| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
+| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` |
+
+Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`, `docs/research/omniroute-account-semaphore-timeout.md` (a connection that can only handle a few concurrent requests — like `llama-server` or `qwen-classifier` — hits a hardcoded 30s reject once more requests queue up than its `maxConcurrent`, unless configured around it).
@@ -0,0 +1,14 @@
+# Kimi CLI
+
+[← back to overview](index.md)
+
+Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
+
+```toml
+[providers.openai]
+type = "openai"
+base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
+api_key = "<kimi-cli virtual key>"
+```
+
+If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
@@ -0,0 +1,45 @@
+# OpenCode CLI
+
+[← back to overview](index.md)
+
+Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
+
+**Install**:
+```bash
+curl -fsSL https://opencode.ai/install | bash
+```
+
+**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "aiproxy": {
+      "npm": "@ai-sdk/openai-compatible",
+      "name": "AI proxy (local)",
+      "options": {
+        "baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "apiKey": "<opencode-cli virtual key>"
+      },
+      "models": {
+        "qwen3.8-27b-local": {
+          "name": "Qwen3.8-27B",
+          "limit": { "context": 131072, "output": 8192 }
+        }
+      }
+    }
+  }
+}
+```
+
+Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server.
+
+Select the model with `aiproxy/qwen3.8-27b-local`.
+
+**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)):
+- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
+- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
+- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
+
+Further reading: `docs/research/opencode-cli-setup.md`.
@@ -0,0 +1,130 @@
+# Qwen Code CLI
+
+[← back to overview](index.md)
+
+Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier. Both are registered as separate providers in OmniRoute but reachable through the same gateway URL.
+
+## Why a second model exists
+
+Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's per-tool-call safety gate — it decides whether to auto-approve or block a shell command / tool call before it runs. It was originally aliased onto the main 27B model's own OmniRoute connection. That broke two ways in practice (see `docs/research/fast-model-choice.md` for the model research, and the issue-tracker history for the full incident):
+
+- **Queued behind heavy work.** Every classification call competed for the main model's 2 GPU slots with whatever real generation was already running, so a classifier check could sit blocked for minutes.
+- **CPU-only was tried first and was too slow.** Isolating the classifier onto its own CPU-only llama.cpp instance avoided the GPU queue entirely, but real classification calls (which can carry a non-trivial conversation transcript, not just the bare tool call) blew past OmniRoute's request timeout and retry-looped.
+
+The fix: a dedicated, GPU-resident `qwen-classifier` service (`docker-compose.yml`) running a small model (`Qwen3-4B-Instruct-2507`) on its own **partial** GPU offload — enough layers on the R9700 to be fast, sized to leave real VRAM headroom next to the 27B model rather than trusting a naive weights+KV estimate (see that service's comment block in `docker-compose.yml` for the actual measured numbers and the two wrong turns — batch-size tuning, then flash-attn — before partial offload turned out to be the real lever).
+
+## `~/.qwen/settings.json`
+
+```json
+{
+  "modelProviders": {
+    "openai": [
+      {
+        "id": "<main-model-provider-id-in-omniroute>",
+        "name": "qwen3.8-27b-local",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "generationConfig": { "contextWindowSize": 131072 }
+      },
+      {
+        "id": "<classifier-provider-id-in-omniroute>",
+        "name": "qwen3-4b-classifier",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "generationConfig": {
+          "contextWindowSize": 65536,
+          "extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
+        }
+      }
+    ]
+  },
+  "security": { "auth": { "selectedType": "openai" } },
+  "model": {
+    "name": "<main-model-provider-id-in-omniroute>",
+    "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
+  },
+  "fastModel": "<classifier-provider-id-in-omniroute>"
+}
+```
+
+- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=<qwen-code-cli virtual key>` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
+- **`contextWindowSize` for the main model is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it from `.env`: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**.
+- **The classifier's `contextWindowSize` (65536) is not per-slot math** — `qwen-classifier` runs `--parallel 1`, so its whole `--ctx-size` belongs to the one slot. 65536 isn't a guess either: qwen-code's own source hard-caps the classifier transcript (`MAX_TRANSCRIPT_MESSAGES=40`, `MAX_HISTORICAL_ACTION_CHARS=4000`/message in `packages/core/src/permissions/classifier-transcript.ts`) — worst case is ~40-50K tokens, so 65536 gives real margin without wasting VRAM the way the original 131072 (copied from the main model's entry, not an actual qwen-code requirement) would have.
+- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see `fast-model-choice.md` §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
+- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
+
+## Web search via OmniRoute
+
+Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
+
+```js
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+
+const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home";
+const API_KEY = process.env.OMNIROUTE_API_KEY;
+
+if (!API_KEY) {
+  console.error("OMNIROUTE_API_KEY is not set in the environment.");
+  process.exit(1);
+}
+
+const server = new McpServer({ name: "omniroute-search", version: "1.0.0" });
+
+server.registerTool(
+  "search",
+  {
+    description: "Web/news search via OmniRoute's /v1/search endpoint.",
+    inputSchema: { query: z.string().describe("Search query") },
+  },
+  async ({ query }) => {
+    const res = await fetch(`${BASE_URL}/v1/search`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` },
+      body: JSON.stringify({ query }),
+    });
+    const text = await res.text();
+    if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true };
+    return { content: [{ type: "text", text }] };
+  }
+);
+
+await server.connect(new StdioServerTransport());
+```
+
+Register it in `~/.qwen/settings.json`:
+
+```json
+{
+  "mcpServers": {
+    "omniroute-search": { "command": "node", "args": ["<path-to>/index.mjs"] }
+  },
+  "tools": { "webSearch": { "enabled": false } }
+}
+```
+
+It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.
+
+**Non-interactive mode (`qwen -p ...`) needs this tool explicitly allow-listed.** MCP tools require interactive confirmation by default; `--approval-mode auto` alone doesn't bypass that for a non-interactive run — pass `--allowed-tools mcp__omniroute-search__search` (or `-y` for full YOLO) alongside `-p`, or the search call never reaches the classifier at all and silently no-ops. Confirmed live: without the allow-list, only the tool calls the CLI's non-interactive gate lets through end up as classifier requests.
+
+## Auto Mode tuning
+
+Auto Mode's action classifier calls the fast model above. Even on the dedicated GPU-resident instance, give it real timeout headroom rather than trusting OmniRoute's default — and since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:
+
+```json
+{
+  "permissions": {
+    "autoMode": {
+      "classifier": { "timeouts": { "stage1Ms": 600000 } },
+      "hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] }
+    }
+  }
+}
+```
+
+`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each.
+
+Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see `docs/network-access.md` for reaching the dashboard API). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.
+
+Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.
@@ -4,7 +4,7 @@ This stack has no chat UI — every client is a coding CLI reaching the AI gatew

 ## llama.cpp's raw API stays LAN-only — deliberately

-The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode — see `docs/coding-cli-setup.md`) don't reach it directly at all now; they go through the gateway below, same as everything else.
+The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else.

 If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.

@@ -0,0 +1,737 @@
+# Research: hardware roadmap to 500k-token context × 2 parallel agents (1M stretch)
+
+**Date:** 2026-09-11
+
+**Question:** What VRAM does 500k-token context × 2 parallel llama-server slots (and a 1M-token
+stretch goal) actually cost for the Qwen3 family, and what hardware roadmap gets there from the
+current single-R9700 setup — given the user's stated plan to add an older (PCIe 4.0) Threadripper
+for lane count, reuse existing RAM/PSU (~200W headroom / one spare 8-pin), mix in already-owned
+NVIDIA cards (GTX 1080 8GB, RTX 2080 8GB, GT 710 1GB) for the classifier role, and price used GPUs
+at roughly $20-30/GB VRAM?
+
+**Answer, short version:** The two goals ("500k × 2 parallel" and "1M stretch") turn out to need
+**the same total VRAM budget** — because of how llama-server's `--ctx-size` and `--parallel` interact
+(§2), 500k × 2 slots and a single 1M-token slot both require setting `--ctx-size 1000000`. At
+`q4_0`-quantized KV cache that's **~33 GB** (weights + KV) for Qwen3.8-27B, at `q8_0` it's **~48 GB**,
+at fp16 it's **~79 GB** — before compute-buffer overhead. That does not fit on the current single
+32GB R9700 at any KV precision, and comfortably fits on two 32GB-class cards only at `q8_0`/`q4_0`.
+The user's $20-30/GB pricing intuition holds for last-gen used consumer cards (RTX 3060 12GB) but
+**not** for RTX 3090 24GB (~$44/GB currently) or a second R9700 (~$41/GB, new — no used market yet
+for a card released mid-2026). The stated Threadripper plan needs to specifically target the
+**non-PRO Threadripper 3000 series on sTRX4** (64 lanes, PCIe 4.0) — older Threadripper on the
+original TR4 socket (1000/2000 series) is PCIe 3.0 only, which doesn't match the user's own PCIe 4.0
+requirement. The power budget (~200W / one spare 8-pin) is exhausted by a *single* mid-tier used GPU
+addition — a PSU upgrade is not optional past the very first stage. See §7 for the roadmap.
+
+---
+
+## 1. Current state (from this repo)
+
+From `docker-compose.yml` and `.env.example` at the repo root:
+
+- **Main model:** `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights), `--ctx-size 262144`,
+  `--parallel 2`, `--flash-attn on`, `--cache-type-k q8_0 --cache-type-v q8_0`, `--n-gpu-layers 999`,
+  on one AMD Radeon AI PRO R9700 (32GB, ROCm/HIP, `gfx1201`).
+- **Classifier ("fast") model:** `Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf`, `--ctx-size 65536`,
+  `--parallel 1`, `--n-gpu-layers 28` (partial offload), `--cache-type-k/v q4_0`, its own container on
+  the *same* R9700, sharing VRAM with the main model — see
+  [`docker-compose.yml`](../../docker-compose.yml) lines ~58-99 and
+  [`fast-model-choice.md`](fast-model-choice.md).
+- `.env.example` already documents the exact fact this research turns on: *"Each slot gets
+  `LLAMA_CTX_SIZE / LLAMA_PARALLEL` tokens of context"* — i.e. today's 262144 ctx-size ÷ 2 parallel
+  slots means each real request only gets **~131K tokens**, not the full 262144, confirmed in-repo
+  before any external source was checked.
+- Prior research already worked out the KV-cache formula for this exact model
+  ([`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md)) — this doc reuses and extends that math for the
+  500k/1M targets rather than re-deriving it.
+
+`docs/server-planing.md` describes a **different, earlier plan**: a 4× AMD Radeon AI PRO R9700 rig
+on a Gigabyte MZ32-AR0 (single-socket SP3/EPYC, 128 PCIe 4.0 lanes), fully AMD/ROCm. The user's plan
+in this ticket is not that — it pivots toward an older **Threadripper** (SP3's sibling desktop-HEDT
+socket family, not SP3 itself) and explicitly wants to mix in already-owned **NVIDIA** cards. These
+two plans are **not the same build** and, per §6, ROCm and CUDA cards cannot share one llama.cpp
+process — they can only coexist as separate containers on separate cards. Treat `server-planing.md`
+as superseded context, not the active plan, unless the user says otherwise.
+
+---
+
+## 2. llama-server parallelism: does each slot get its own full `--ctx-size`, or is it divided?
+
+**Divided.** This is the single fact that changes the whole budget by 2×, confirmed from three
+independent primary sources:
+
+1. **This repo's own `.env.example`** (quoted above) already documents it for the current deployment.
+2. **llama.cpp's own server README**, fetched directly
+   ([`tools/server/README.md`](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)):
+   `--ctx-size (-c)`: *"size of the prompt context (default: 0, 0 = loaded from model)"*;
+   `--parallel (-np)`: *"number of server slots (default: -1, -1 = auto)"* — the docs list these as
+   independent flags, but don't spell out the division themselves.
+3. **A real user's server log**, quoted verbatim in
+   [ggml-org/llama.cpp#11681](https://github.com/ggml-org/llama.cpp/issues/11681), is the actual proof:
+   running with `--ctx-size 327680 --parallel 6` produces `n_ctx = 327680`,
+   `n_ctx_per_seq = 54613` — i.e. `327680 / 6 ≈ 54613`. The reporter explicitly asked for a
+   `--ctx-size-per-seq`-style flag to *avoid* this division; no such flag exists as of the fetch date.
+
+Practical consequence: **to get 500,000 usable tokens on each of 2 parallel slots, `--ctx-size` must
+be set to 1,000,000, not 500,000.** The KV cache is sized off the *total* `--ctx-size`
+(`--kv-unified`, on by default when slots are auto per the README's `-kvu` entry, uses one shared
+pool sized to the full `n_ctx`) — so the VRAM cost of "500k × 2 parallel" and "one 1M-token slot"
+is **identical**: both require `--ctx-size 1000000`. This is a genuinely useful finding for the
+roadmap — reaching the 500k×2 target and the 1M stretch goal cost the same VRAM; the only difference
+is `--parallel 1` vs `--parallel 2` at deploy time, a config change with zero extra hardware cost.
+
+`--cache-type-k` / `--cache-type-v` accept `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`
+(default `f16`), per the same README fetch. The repo already uses `q8_0` on the main model and `q4_0`
+on the classifier, so both quantization tiers used in the math below are already-proven-working
+configurations in this stack, not hypothetical flags.
+
+---
+
+## 3. KV-cache math per model
+
+### Qwen3.8-27B (hybrid Gated-DeltaNet / attention)
+
+Reusing the architecture params already pulled from
+[`Qwen/Qwen3.8-27B/config.json`](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json) in
+[`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md), re-verified directly for this doc: `num_hidden_layers=64`,
+`full_attention_interval=4` → **16 of 64 layers are standard KV-caching attention**, the other 48 are
+Gated DeltaNet linear-attention layers with a small, context-length-*independent* recurrent state
+(tens of MB total, negligible next to the attention KV cache — ignored below).
+`num_key_value_heads=4` (GQA), `head_dim=256`. Native context `max_position_embeddings=262144`
+(YaRN-extensible to 1M per the model card — **both the 500k and 1M targets exceed native context and
+require RoPE/YaRN scaling**, which is a real quality caveat, not just a memory one — Qwen has not
+published independent long-context quality benchmarks past native length that this research found).
+
+Per-token KV cache, fp16, both K and V, across the 16 full-attention layers:
+
+```
+16 layers × 2 (K+V) × 4 kv_heads × 256 head_dim × 2 bytes = 64 KiB/token
+```
+
+| Total ctx-size | KV cache, fp16 | KV cache, q8_0 | KV cache, q4_0 |
+|---|---|---|---|
+| 262,144 (current) | ~16.0 GiB | ~8.0 GiB | ~4.0 GiB |
+| 500,000 | ~30.5 GiB | ~15.3 GiB | ~7.6 GiB |
+| **1,000,000 (500k×2, or 1M stretch)** | **~61.0 GiB** | **~30.5 GiB** | **~15.3 GiB** |
+
+(`q8_0` is 8-bit vs. fp16's 16-bit → exactly half; `q4_0` is 4-bit → exactly quarter, per llama.cpp's
+own cache-type byte widths.)
+
+### Qwen3-4B-Instruct-2507 (plain GQA transformer, classifier role)
+
+From [`Qwen/Qwen3-4B-Instruct-2507/config.json`](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
+(already pulled in [`fast-model-choice.md`](fast-model-choice.md)): `num_hidden_layers=36` — every
+layer is standard attention here (no hybrid split), `num_key_value_heads=8`, `head_dim=128`.
+
+```
+36 layers × 2 (K+V) × 8 kv_heads × 128 head_dim × 2 bytes = 144 KiB/token
+```
+
+The classifier's real transcript ceiling is ~40-50K tokens (qwen-code's own
+`MAX_TRANSCRIPT_MESSAGES=40` × `MAX_HISTORICAL_ACTION_CHARS=4000`, per `fast-model-choice.md` §"what
+actually shipped") — nowhere near 500k/1M, so the classifier does **not** need to grow for this
+roadmap; it stays exactly as deployed today, on its own small allocation. Per-token cost is included
+here only because it feeds the "does the classifier's dedicated GPU need to change" question in §6.
+
+---
+
+## 4. Total VRAM budget: 500k × 2 parallel, and the 1M stretch
+
+Weights: `Qwen3.8-27B-UD-Q4_K_XL.gguf` is **17.6 GB**, confirmed directly from the
+[unsloth/Qwen3.8-27B-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main)
+(already verified in `qwen3.8-27b-quant.md`).
+
+Per §2, both "500k × 2 parallel" and "1M stretch" require `--ctx-size 1000000` — same KV budget:
+
+| KV precision | KV cache | + weights (17.6 GB) | + est. compute-buffer/runtime overhead* | **Realistic total** |
+|---|---|---|---|---|
+| fp16 (default) | 61.0 GiB | 78.6 GiB | +3-6 GiB | **~82-85 GB** |
+| q8_0 (proven in this stack today) | 30.5 GiB | 48.1 GiB | +3-6 GiB | **~51-54 GB** |
+| q4_0 (proven in this stack today, on the classifier) | 15.3 GiB | 32.9 GiB | +3-6 GiB | **~36-39 GB** |
+
+\* *Estimate, not a cited figure* — llama.cpp's flash-attention compute buffer scales closer to
+linear than the unfused-attention path, per this repo's own measured note in `docker-compose.yml`'s
+`qwen-classifier` comment (unfused attention buffers ballooned unexpectedly at 65536 ctx; flash-attn
+fixed it). `--flash-attn on` is already the deployed default for the main model, so the linear-ish
+regime applies, but no primary source gives an exact formula for this buffer size at 1M context — the
+3-6 GiB band is this doc's estimate based on the ratio observed in that in-repo incident, not a
+llama.cpp-documented number. Budget for the high end of that range when sizing hardware.
+
+**Bottom line:** at `q4_0` KV (the most aggressive, already-proven-in-this-repo tier), 500k×2 /
+1M needs **~36-39 GB** total VRAM for the 27B model alone. That does not fit one 32GB card at any
+precision — it needs at least two 32GB-class cards, or one ≥40GB card. At `q8_0` (the precision this
+repo already runs in production for quality reasons), budget **~51-54 GB** — two 32GB cards (64GB
+pooled) clears this with room to spare; a single 48GB-class card would not.
+
+(§11 below extends this table to higher weight-quant tiers — Q6_K_XL, Q8_0, BF16 — for users who want
+better output quality than `Q4_K_XL`, and to a Flash-Next alternative architecture; see §11.6-§11.7.)
+
+---
+
+## 5. CPU/motherboard: which Threadripper generations give PCIe 4.0, and how many lanes for GPUs
+
+AMD's own product/chipset pages, cross-checked against the launch reviews that quote them directly:
+
+| Platform | Socket | PCIe generation | Total CPU-provided lanes |
+|---|---|---|---|
+| Threadripper 1000/2000 series ("1920X", "2950X", etc.) | **TR4** | **PCIe 3.0 only** | 60-64 |
+| Threadripper 3000 series (3960X/3970X/3990X) | **sTRX4** | **PCIe 4.0** | 64 |
+| Threadripper 7000 series (non-PRO) | sTR5 | PCIe 5.0 (48 lanes) + PCIe 4.0 (24-32 lanes) | ~72-80 |
+| Threadripper PRO 3000WX/5000WX | sWRX8 | PCIe 4.0 | **128** |
+| Threadripper PRO 7000WX | sTR5 (WRX90) | PCIe 5.0 (128 lanes) + a few PCIe 3.0 | **128** |
+
+Sources: [AMD chipsets product page](https://www.amd.com/en/products/processors/chipsets.html)
+(sWRX8/socket listing), corroborated by
+[Tom's Hardware — Threadripper 3960X/3970X, sTRX4/TRX40 launch coverage](https://www.tomshardware.com/news/amd-unveils-threadripper-3960x-and-3970x-ryzen-9-3950x-details-and-athlon-3000g/2)
+(*"the 3rd Gen TR CPUs carry the same 64 PCIe lanes but double bandwidth by moving from Gen 3.0 to
+Gen 4.0"* — explicit confirmation TR4/1000-2000-series is PCIe 3.0 while sTRX4/3000-series is PCIe
+4.0), [PCWorld — Threadripper PRO launch](https://www.pcworld.com/article/393181/amd-threadripper-pro-has-64-cores-128-pcie-lanes-and-8-channel-memory-support.html)
+(*"128 PCIe lanes"* for PRO).
+
+**This directly matters for the user's plan.** "An older Threadripper... for more PCIe lanes" is
+ambiguous between two real, very different chips:
+
+- **TR4 (1000/2000 series)** — cheapest used option, but **PCIe 3.0** — does not meet the user's own
+  stated PCIe 4.0 requirement, and PCIe 3.0 x8 per GPU roughly halves inter-GPU/host transfer
+  bandwidth (matters more for training/tensor-parallel than for llama.cpp's inference-time layer
+  splitting, but still a real downgrade vs. the R9700's native PCIe 5.0).
+- **sTRX4 (3000 series, non-PRO)** — the correct "older Threadripper with PCIe 4.0" target: 64 lanes,
+  4-5 years old, real used-market availability, no PRO price premium.
+- **Threadripper PRO (3000WX/5000WX)** doubles the lane count to 128 but at meaningfully higher used
+  cost (workstation-tier, lower volume, sWRX8 boards are pricier than sTRX4/TRX40 boards) — worth it
+  only if 6 full-bandwidth (x16) GPU slots are actually needed; at x8-per-card (adequate for inference)
+  64 lanes already covers 6 GPUs with lanes to spare for NVMe/chipset.
+
+**Lane budget for 6 GPUs on sTRX4 (64 lanes), estimated (no vendor spec gives a topology this
+specific — treat this bullet as an estimate):** typical sTRX4 boards reserve ~4 lanes for the
+chipset uplink and commonly wire 1-2 M.2 slots directly to the CPU (4 lanes each) — so realistic
+GPU-available lanes land around 44-52 of the 64, i.e. **6 GPUs at x8 electrical each (48 lanes) is
+plausible but board-model-dependent**; x16-each for 6 cards is not possible on 64 lanes regardless of
+board. x8 electrical is not a meaningful inference-speed penalty for llama.cpp (weights are loaded
+once; the ongoing per-token traffic across PCIe is small compared to compute), so this is an
+acceptable tradeoff, not a real bottleneck for this workload.
+
+---
+
+## 6. Power budget vs. the ~200W / one spare 8-pin headroom
+
+Official/vendor TDPs:
+
+| Card | TDP | Source |
+|---|---|---|
+| GTX 1080 (owned) | 180W, one 8-pin | [confirmed 180W, PCIe 3.0 x16, 1× 8-pin](https://buildmyserver.com/products/zotac-nvidia-geforce-gtx-1080-8gb-gddr5-180w-pcie-3-0-x16-double-wide-gpu) — spec matches NVIDIA's own launch figures reported across multiple outlets incl. Tom's Hardware |
+| RTX 2080 (owned) | 215W | Cross-checked across gpuzoo/cputronic/notebookcheck spec pages, consistent at 215W |
+| GT 710 (owned) | ~19W, **no external power connector** (slot power only) | [MSI/EVGA/Zotac GT 710 spec pages](https://www.msi.com/Graphics-Card/GT-710-1GD5-LP/Specification) |
+| RTX 3060 12GB (candidate purchase) | 170W, one 8-pin | [NVIDIA-confirmed 170W TDP, one 8-pin connector](https://www.lowyat.net/2021/232659/nvidia-geforce-rtx-3060-specifications-now-official-includes-3584-cuda-cores-and-170w-tdp/) |
+| RTX 3090 24GB (candidate purchase) | 350W, two 8-pin, [NVIDIA's own RTX 3090 product page](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090/) lists 350W and a 750W PSU minimum | NVIDIA official |
+| R9700 32GB (already deployed / "more of the same") | 300W (per this repo's `server-planing.md`, consistent with AMD's own R9700 product page framing it as a 300W-class card) | in-repo prior research |
+
+**Against the stated ~200W / one spare 8-pin budget:**
+
+- Adding **one RTX 3060 12GB** (170W, one 8-pin) is the *only* candidate in this list that fits the
+  stated headroom as-is — it uses the one spare connector and stays under 200W.
+- Adding the already-owned **GTX 1080** (180W) as the classifier's dedicated card also just barely
+  fits (180W ≤ 200W, one 8-pin) — this is a genuinely free option since the card is already owned and
+  its power draw is within budget, unlike every purchase candidate below.
+- Adding the already-owned **RTX 2080** (215W) **exceeds** the stated 200W headroom by 15W — technically
+  over budget on paper, though real-world draw is usually a bit under rated TDP; flag it as marginal,
+  not safely fitting.
+- Adding a **second R9700** (300W) or an **RTX 3090** (350W, needs two 8-pin — the user has only one
+  spare) both blow well past the current power budget on both watts and connector count.
+- **The GT 710 draws no meaningful power (~19W, no PCIe power connector at all)** — it is free from a
+  power-budget standpoint regardless of what else is added.
+
+**PSU upgrade trigger:** the very first stage that adds *any* GPU beyond a GTX 1080-class card (180W,
+one 8-pin) or an RTX 3060 12GB (170W, one 8-pin) exhausts the stated headroom. Any stage that reaches
+for a second 32GB-class card (R9700 or equivalent) or any 300W+ card **requires a PSU upgrade before
+that stage**, not after — see the roadmap table in §7 for exactly which stage that is.
+
+---
+
+## 7. Mixed-GPU feasibility: ROCm + CUDA, and is the GT 710 usable at all
+
+**ROCm and CUDA are different llama.cpp builds, but that's exactly the pattern already in this
+repo.** `ghcr.io/ggml-org/llama.cpp` publishes both `server-rocm` and `server-cuda` as separate,
+independently-built image tags (confirmed present on the [ggml-org container registry](https://github.com/orgs/ggml-org/packages/container/llama.cpp)
+and documented in [`docs/docker.md`](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/docs/docker.md)
+— *"server-cuda: Same as `server` but compiled with CUDA support"*, *"server-rocm: Same as `server`
+but compiled with ROCm support"*). You cannot mix backends inside one process/container, but you
+**can** run one `server-rocm` container pinned to the R9700 and a separate `server-cuda` container
+pinned to an NVIDIA card, simultaneously, on the same host — this is architecturally identical to
+today's `llama-server` + `qwen-classifier` two-container split in `docker-compose.yml`, just with a
+different image tag for the NVIDIA-backed service and NVIDIA's container runtime (`nvidia-container-toolkit`
+ `--gpus` / device reservation, the CUDA-world equivalent of this repo's `/dev/kfd`+`/dev/dri`+
+numeric-GID ROCm pattern documented in
+[`rocm-gpu-pin-and-render-group.md`](rocm-gpu-pin-and-render-group.md)). None of that doc's ROCm-specific
+findings (the `GPU_MAX_HW_QUEUES=1` MES firmware workaround, the numeric-GID `group_add` fix) apply to
+an NVIDIA/CUDA container — those are ROCm-stack-specific bugs, not general multi-GPU-container issues.
+
+**Is this an implicit AMD→NVIDIA rebuild, or additive?** Worth surfacing explicitly since the two
+source plans conflict on this: `server-planing.md` is an AMD-only, ROCm-only 4×R9700 plan. This
+ticket's plan is **additive/mixed** — keep the R9700 running the main model under ROCm, and bolt on
+NVIDIA cards under CUDA for secondary roles (classifier, or a second inference GPU for the big model
+if going the "more of the same type" route means buying NVIDIA instead of more R9700s). Both are
+internally consistent, but they are different end-states — flag this choice back to the user rather
+than assuming one.
+
+**Splitting the *main* 27B model itself across mixed AMD+NVIDIA silicon in one process is not
+possible** — llama.cpp's multi-GPU tensor-split only works within a single backend build. To use
+both an R9700 and an NVIDIA card for the *same* model's layers, all the compute-hosting cards need to
+be the same backend (all-ROCm or all-CUDA) in that one process. This is why §5's roadmap treats "add
+GPU capacity to the main model" and "add a GPU for the classifier" as separable purchases with
+different backend constraints, not a single mixed pool.
+
+**Is the GT 710 usable for anything in this pipeline? No.** Reasoning:
+
+- 1GB VRAM cannot hold any meaningful fraction of either model's weights (17.6 GB / 2.4-4.3 GB) —
+  even a handful of transformer layers at Q4 quantization exceeds 1GB.
+- It's Kepler-generation silicon (192 CUDA cores, no tensor cores) — llama.cpp's CUDA backend
+  technically supports pre-Turing cards, but at this VRAM size there's nothing to usefully offload.
+- It draws power from the PCIe slot only, no external connector — genuinely free to keep installed.
+- **Plausible actual use: dedicate it as the box's display-output card**, so every compute-capable
+  GPU (R9700, and whichever NVIDIA cards get added) can be fully headless/compute-only with none of
+  their VRAM or a display output tied up driving a monitor — a real, if minor, use for it. This is
+  this doc's own inference from the spec facts above, not a claim found in any primary source.
+
+---
+
+## 8. GPU market pricing vs. the $20-30/GB assumption
+
+| Card | VRAM | Backend | Current used-market price (estimate — see caveat) | $/GB |
+|---|---|---|---|---|
+| RTX 3060 12GB | 12GB | CUDA | ~$240-300 used (eBay listings, [gpupoet.com tracker](https://gpupoet.com/gpu/shop/nvidia-geforce-rtx-3060): *"from $239"*, [eBay live listings](https://www.ebay.com/shop/rtx-3060-12gb) averaging ~$488 asking but with a $239 floor) | **~$20-25/GB** — matches the stated assumption |
+| RTX 3090 24GB | 24GB | CUDA | ~$1,010-1,050 used ([bestvaluegpu.com Sep 2026 tracker](https://bestvaluegpu.com/history/new-and-used-rtx-3090-price-history-and-specs/), [xda-developers coverage](https://www.xda-developers.com/used-rtx-3090-still-best-for-local-ai-in-value/)) | **~$42-44/GB** — well above the stated assumption |
+| R9700 32GB ("more of the same type") | 32GB | ROCm | **New only — $1,299 MSRP**, street price $1,400-1,585 as of this research ([overclock3d](https://overclock3d.net/news/gpu-displays/amd-unveils-its-1299-radeon-ai-pro-r9700-32gb-workstation-gpu/), [pricehistory.app tracker](https://pricehistory.app/p/powercolor-amd-radeon-ai-pro-r9700-32gb-BFcRhGIm)) — too recent a release (2026) for a used market to exist yet | **~$41-50/GB, and not a used-market price at all** |
+
+**Caveat on all three price figures:** these are live marketplace asking-price snapshots pulled via
+web search on 2026-09-11, not sold-price data or a vendor spec sheet — treat as directional, not
+exact. eBay asking prices in particular run above realized sale prices.
+
+**Correction to the user's stated assumption:** $20-30/GB is a good estimate specifically for
+**last-generation mainstream used cards** (RTX 3060 12GB fits it almost exactly) but **not** for
+high-VRAM flagship cards like the RTX 3090 (~1.5-2× that rate) or for "more of the same type" R9700
+units, which aren't used-market at all yet and sit even higher per GB than the 3090. If the plan is
+"cheapest path to more VRAM," multiple RTX 3060 12GB cards (or similar mid-tier used cards) beat one
+RTX 3090 on $/GB, at the cost of needing more PCIe slots and more total wattage/connectors to reach
+the same aggregate VRAM — which is exactly the tradeoff the Threadripper lane-count plan in §5 is
+for.
+
+---
+
+## 9. Step-by-step roadmap
+
+All "resulting max context" figures assume `--parallel 2` and the KV precision stated; per §2, the
+`--ctx-size` value shown is the *total* (pre-division) value to pass to llama-server.
+
+| Stage | Hardware change | Est. cost | Backend | Usable VRAM (main-model pool) | Max context @ parallel=2 (`q4_0` KV) | PSU upgrade triggered? |
+|---|---|---|---|---|---|---|
+| **0 (current)** | 1× R9700 32GB, in production | $0 | ROCm | 32GB (shared with classifier) | ~131K/slot today at `q8_0` KV (262144 total ÷ 2) | No |
+| **1 — classifier isolation** | Move classifier onto the already-owned **GTX 1080** (180W, own container, `server-cuda`), freeing the R9700 entirely for the main model. Matches the existing dual-model pattern qwen-code's own docs describe (§10) and this repo's `qwen-classifier` service already implements, just on separate silicon instead of a shared card. | $0 (already owned) | ROCm (main) + CUDA (classifier) | R9700's full 32GB now available to the main model alone | ~262K/slot @ `q8_0` (unchanged ctx-size, no more classifier contention) | **No** — 180W GTX 1080 fits the stated ~200W/one-8-pin headroom |
+| **2 — second big-model GPU** | Add **one more 32GB-class card** for the main model. Cheapest correct-backend option: a second R9700 (~$1,300-1,585 new, ROCm, same backend as the first — required if tensor-splitting one model across two cards) | ~$1,300-1,585 | ROCm | 64GB pooled | `--ctx-size 500000 --parallel 1` fits at `q4_0` (~33GB) or `q8_0` (~48GB, tight but fits in 64GB) — **not yet 500k×2** | **Yes** — 300W card, no spare 8-pin left after stage 1 |
+| **3 — reach 500k × 2 / 1M stretch** | No further hardware if stage 2's 64GB pool is used with `--cache-type-k/v q4_0`: `--ctx-size 1000000 --parallel 2` needs ~33-39GB (§4), fits inside 64GB with real headroom for the compute buffer. If `q8_0` KV is required instead (this repo's current quality bar for the main model), the ~51-54GB need is tight-to-marginal on 64GB — a **third** 32GB card (~96GB pool) removes the risk. | $0 (reuses stage 2) or +$1,300-1,585 for a 3rd card if `q8_0` KV is required | ROCm | 64GB (q4_0 case) or 96GB (q8_0 case) | **500k×2 parallel achieved**, and the 1M stretch goal is the *same config* with `--parallel 1` instead of 2 (§2) | Already upgraded at stage 2 |
+| **4 — optional CPU/lane platform swap** | Only needed if the plan is to keep scaling past 2-3 big cards, or to add several small used cards (RTX 3060 12GB) for extra headroom/throughput rather than raw ctx-size. Swap to **non-PRO Threadripper 3000-series (sTRX4)** — 64 PCIe 4.0 lanes, ~x8-per-slot for up to 6 GPUs (§5). Threadripper PRO 3000WX/5000WX (128 lanes) only if x16-per-card matters or 6+ full-bandwidth slots are wanted. | Used sTRX4 CPU+board: roughly $400-800 combined on the used market (not independently priced in this pass — **estimate**, not cited) | n/a (platform only) | n/a | n/a | Independent of GPU wattage — driven by whatever GPU count/wattage stage 5+ adds |
+| **5+ — scale-out via small used cards** | Add RTX 3060 12GB units (~$20-25/GB, the assumption that actually holds, §8) instead of more 32GB flagship cards, once lane count (stage 4) supports it — useful for extra parallel slots / throughput beyond the 500k×2 target rather than for raising ctx-size further (500k×2/1M is already met at stage 3). | ~$240-300/card | CUDA (separate container per §6) | +12GB pooled per card, but on a *different backend* from the ROCm main model — usable for extra classifier/small-model capacity or a separate CUDA-backend llama-server instance, not as additional tensor-split VRAM for the ROCm main model | Unchanged for the main model; adds parallel capacity elsewhere | Yes, cumulative — each additional 170W card needs PSU headroom stage 2 already consumed |
+
+**Where the existing dual-model pattern sits in this roadmap:** it's stage 1, and it's free. The
+qwen-code docs pattern (main model + a small, always-resident, non-thinking fast/classifier model —
+see §10) is already implemented in this repo; the only roadmap-relevant change is *which GPU* the
+classifier sits on, moving it off the R9700 entirely onto an already-owned NVIDIA card frees the
+R9700's full 32GB for the 500k×2/1M push instead of splitting it with the classifier as happens
+today.
+
+### 9.1 Upgrade path, as diagrams
+
+Diagram form of the same §9 table and §11.9's dense-vs-Flash-Next call — nothing new is claimed here,
+this is a visual index back into the cited sections above.
+
+**Stage-by-stage hardware path** (PSU-upgrade triggers and target reached called out inline):
+
+```mermaid
+flowchart TD
+    S0["Stage 0 — today<br/>1x R9700 32GB, ROCm<br/>classifier shares the card<br/>$0"]
+    S1["Stage 1 — classifier isolation<br/>+ GTX 1080 (owned, 180W, CUDA)<br/>R9700 freed for main model<br/>$0 · PSU OK (180W fits ~200W headroom)"]
+    S2["Stage 2 — 2nd big-model GPU<br/>+1x R9700 32GB (ROCm)<br/>64GB pooled<br/>~$1,300-1,585 · PSU UPGRADE REQUIRED (300W, no 8-pin left)"]
+    S3q4["Stage 3a — q4_0 KV<br/>--ctx-size 1,000,000 --parallel 2<br/>~33-39GB, fits in 64GB<br/>$0 (reuses stage 2)"]
+    S3q8["Stage 3b — q8_0 KV (current prod quality)<br/>~51-54GB, tight on 64GB<br/>+1x R9700 -> 96GB removes risk<br/>+~$1,300-1,585"]
+    TARGET(["500k x2 parallel reached<br/>= 1M stretch goal, same VRAM<br/>(--parallel 1 vs 2 is a config flag, §2)"])
+    S4["Stage 4 — platform swap (optional)<br/>sTRX4 Threadripper 3000, 64 PCIe4 lanes<br/>only needed past 2-3 big cards<br/>~$400-800 (estimate, §9)"]
+    S5["Stage 5+ — scale out<br/>+RTX 3060 12GB cards (CUDA, separate backend)<br/>extra parallel/throughput, not more ctx-size<br/>~$240-300/card · PSU upgrade each card"]
+
+    S0 --> S1 --> S2
+    S2 --> S3q4 --> TARGET
+    S2 --> S3q8 --> TARGET
+    TARGET -.->|"only if scaling past this"| S4 --> S5
+
+    style TARGET fill:#2e7d32,color:#fff,stroke:#1b5e20
+    style S2 fill:#8a5a00,color:#fff,stroke:#5c3d00
+    style S5 fill:#8a5a00,color:#fff,stroke:#5c3d00
+```
+
+**Model choice, and the one open question that could change it** (§11.9):
+
+```mermaid
+flowchart TD
+    Q{"Goal: 500k-1M ctx<br/>within a $20-30/GB VRAM budget?"}
+    D["Dense Qwen3.8-27B<br/>17.6-54.7GB weights (Q4_K_XL-BF16)<br/>reaches 500k x2 on 2-3 cards,<br/>500k x4 on 2-6 cards depending on quant<br/>(§11.7 tables)"]
+    F{"Try --n-cpu-moe:<br/>offload MoE experts to system RAM?<br/>(untested for this model, §11.8)"}
+    FBAD["Flash-Next, all-GPU weights<br/>111-354GB just for weights<br/>needs 4-13 cards before any KV cost<br/>NOT recommended at this budget (§11.9)"]
+    FGOOD["Flash-Next, experts in system RAM<br/>GPU VRAM could shrink a lot<br/>2.67x cheaper KV/token becomes relevant<br/>UNVERIFIED — prototype on real server first"]
+    CAVEAT["+ real caveat either way:<br/>PR #27742 flags unverified conv branch,<br/>3% QSA divergence, prefill-pos-0-only PLE<br/>(§11.1) — dense model carries no such flag"]
+
+    Q --> D
+    Q -->|"considering Flash-Next instead"| F
+    F -->|"works well"| FGOOD
+    F -->|"doesn't help / untested"| FBAD
+    FGOOD --> CAVEAT
+    FBAD --> CAVEAT
+
+    style D fill:#2e7d32,color:#fff,stroke:#1b5e20
+    style FBAD fill:#8a1c1c,color:#fff,stroke:#5c1212
+    style FGOOD fill:#8a5a00,color:#fff,stroke:#5c3d00
+```
+
+---
+
+## 10. Qwen-code's own docs on the fast-model/classifier pattern
+
+Fetched directly per the user's link:
+[qwenlm.github.io/qwen-code-docs/en/users/overview/](https://qwenlm.github.io/qwen-code-docs/en/users/overview/)
+— **the overview page itself does not describe the dual-model/classifier pattern**; it only covers
+single-model-provider setup (Alibaba ModelStudio / third-party / custom provider), one model at a
+time. The actual fast-model/classifier documentation lives on the **Auto Mode** page instead, which
+this repo's own `fast-model-choice.md` already fetched and cited in detail:
+[qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/) —
+summary (see `fast-model-choice.md` §1 for the full quote): a two-stage classifier gate, both stages
+using "your configured fast model (`/model --fast`)", Stage 1 a ~300ms `{shouldBlock}`-only check,
+Stage 2 a ~3-5s chain-of-thought review that only runs on a Stage-1 block. Nothing in either page
+gives a recommended *context size* or *model size* for the fast model beyond what's implied by that
+latency budget — this repo's own prior research (`fast-model-choice.md`) derived the actual context
+requirement from qwen-code's source code instead (`packages/core/src/permissions/classifier-transcript.ts`),
+since the docs pages don't state one. No new information changes that prior doc's conclusion; this
+section exists to confirm the overview page was checked directly as instructed and doesn't contradict
+or add to it.
+
+---
+
+## 11. Alternative: Qwen3.8-Flash-Next (MoE, hybrid attention)
+
+The user also wants to weigh switching (or adding) **Qwen3.8-Flash-Next** — a 125B-total/6B-active MoE
+with a hybrid recurrent-attention architecture — against staying on dense Qwen3.8-27B, and separately
+wants this section to cover **going up in weight quant** (Q4_K_XL → Q6_K_XL → Q8_0 → BF16/fp16) for
+*both* models, not just Q4. Feasibility first, since it gates everything else.
+
+### 11.1 Feasibility verdict: supported, but immature — read before trusting any number below
+
+Checked directly against the primary sources the task named:
+
+- **llama.cpp mainline support exists.** [PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742)
+  ("model: add Qwen3.8-Flash-Next (qwen4exp)") was **merged into `master` on 2026-08-27** by ngxson.
+  It adds the full architecture: Gated DeltaNet layers (sigmoid-gated linear attention), QSA
+  ("Qwen Sparse Attention", operating at micro-block granularity), hyper-connections, and the PLE
+  n-gram embedding table. `llama.cpp`'s own docs list CPU/CUDA/Metal/ROCm as supported backends for
+  it — this is not a CUDA-only feature.
+- **This repo's pinned image is a floating tag, not a version pin.** `docker-compose.yml` runs
+  `ghcr.io/ggml-org/llama.cpp:server-rocm` with no date/digest suffix — a rolling "latest ROCm server
+  build" tag, not a release version. The merge is from 2026-08-27, and today is 2026-09-11 (~2 weeks
+  later), so a **fresh pull** of `server-rocm` should include it — but whatever image is already
+  cached/running on the R9700 box may predate the merge. **Action before touching this model on the
+  server: `docker compose pull llama-server` and check the startup log's build/commit banner is dated
+  on/after 2026-08-27**, not just "the tag says server-rocm."
+- **Real, primary-source-flagged immaturity — this is the part that should temper enthusiasm.** The
+  PR's own description/review discussion states: *"The conv branch itself is still numerically
+  unverified because the fixture zeroes its weights"*; QSA sparse attention *"diverges on 3 percent of
+  positions"* above its budget threshold; the PLE depthwise convolution *"is exact only for a prefill
+  that starts at position 0"* (i.e. correctness is not guaranteed once `--cache-reuse`/prompt-caching
+  is in play — a flag this repo already turns on for the dense model per the latest commit). None of
+  that is disqualifying, but it is a primary-source admission that this is a fresh, not-fully-verified
+  implementation, not a mature, widely-battle-tested one like the dense Qwen3.8-27B path.
+- **Multi-slot serving needs an explicit new flag.** The same PR states: *"`set_input_qsa` asserted
+  `n_stream == 1`, so llama-server could not serve this model with more than one slot unless `-kvu`
+  was passed."* Per the server README (§2), `--kv-unified`/`-kvu` defaults to enabled **only when slot
+  count is auto** (`-1`). This repo's compose file sets `--parallel ${LLAMA_PARALLEL:-2}` **explicitly**
+  (not auto) — so adopting Flash-Next with `--parallel` > 1 requires **adding `--kv-unified` (or
+  `-kvu`) to the launch flags**, a real deploy-time change, not something that "just works" by copying
+  today's flag set onto a new model file.
+
+**Verdict: yes, runnable** on this repo's backend (ROCm, mainline, no dev branch needed) as long as the
+image is pulled after 2026-08-27 and `-kvu` is added for multi-slot use — but treat it as
+**usable-with-caution**, not a drop-in swap, given the PR author's own unresolved-correctness notes.
+
+### 11.2 Architecture, verified against `config.json` directly
+
+Fetched from `Qwen/Qwen3.8-Flash-Next`'s `config.json` (unsloth's GGUF repo repackages the same base
+model): `num_hidden_layers=48`, `hidden_size=2560`, `num_attention_heads=24`, `num_key_value_heads=2`,
+`head_dim=256`, `max_position_embeddings=262144` (same native/extensible-to-1M framing as the dense
+model — same YaRN quality caveat from §3 applies here too, unverified past native length), `num_experts=512`,
+`num_experts_per_tok=10`, and the linear-attention head config: `linear_num_key_heads=16`,
+`linear_num_value_heads=48`, `linear_key_head_dim=128`, `linear_value_head_dim=128`.
+
+Layer pattern (confirmed both from the model card's own description and `config.json`'s
+`full_attention_interval=4`): every 4th layer is full/QSA attention, the other 3 are Gated DeltaNet —
+**12 of 48 layers grow a real KV cache; the other 36 have a fixed-size recurrent state that does not
+grow with context length.** (24% full-attention layers vs. the dense model's 16-of-64 = 25% — similar
+ratio, but the *absolute* per-layer KV cost differs because `num_key_value_heads` is 2 here vs. 4 on
+the dense model — see below.)
+
+### 11.3 Per-token growing-KV-cache cost
+
+```
+12 full-attention layers × 2 (K+V) × 2 kv_heads × 256 head_dim × 2 bytes (fp16) = 24 KiB/token
+```
+
+| Total ctx-size | KV cache, fp16 | KV cache, q8_0 | KV cache, q4_0 |
+|---|---|---|---|
+| 262,144 (native) | ~6.0 GiB | ~3.0 GiB | ~1.5 GiB |
+| 500,000 | ~11.4 GiB | ~5.7 GiB | ~2.9 GiB |
+| **1,000,000 (500k×2, or 1M stretch)** | **~22.9 GiB** | **~11.4 GiB** | **~5.7 GiB** |
+| **2,000,000 (500k×4)** | **~45.8 GiB** | **~22.9 GiB** | **~11.4 GiB** |
+
+`--cache-type-k/v` are the same generic llama.cpp KV-cache-quantization flags used elsewhere in this
+doc; nothing in the PR or the server README suggests they're handled differently for the 12
+full-attention layers of a hybrid model — they quantize the same growing K/V buffers as on a plain
+transformer. (No primary source explicitly confirms this for *this* architecture specifically — flagged
+as a reasonable extrapolation, not a directly-cited fact, same caveat class as this doc's other
+estimates.)
+
+### 11.4 Fixed (non-growing) recurrent state — Gated DeltaNet layers
+
+The 36 Gated DeltaNet layers each keep a fixed-size recurrent state (an outer-product-style
+key×value matrix per head) that does **not** scale with context length — only with slot/sequence
+count. Sized from `config.json`'s linear-attention head params:
+
+```
+36 layers × linear_num_value_heads(48) × linear_key_head_dim(128) × linear_value_head_dim(128) × 4 bytes (fp32 state)
+≈ 36 × 48 × 128 × 128 × 4 bytes ≈ 108 MiB per slot
+```
+
+This is **this doc's own derivation from the published head-dimension params, not a value pulled
+directly from llama.cpp source or docs** — the PR text confirms the state exists per-stream/per-slot
+but doesn't publish an exact byte formula, so treat the ~108 MiB/slot figure as an estimate, medium
+confidence. Even at 4 parallel slots that's under half a gigabyte — **negligible** next to both the
+growing KV cache (GBs) and the weights (tens to hundreds of GB) computed below. The headline
+implication holds regardless of the exact multiplier: Flash-Next's "big memory line item" is the MoE
+weights, not the attention state of any kind.
+
+### 11.5 Magnitude vs. the dense model — how much cheaper is KV, really
+
+At the same total ctx-size, Flash-Next's growing KV cache is **24 KiB/token vs. the dense model's
+64 KiB/token — 2.67× smaller**, i.e. Flash-Next's KV budget is **37.5%** of the dense model's at
+identical context length. This is a real, significant win *for the KV-cache line item specifically* —
+but see §11.9: it's a much smaller slice of a much bigger total, because the weights move the other
+way by a far larger factor.
+
+### 11.6 Weight sizes — verified from each unsloth GGUF repo's actual file listing
+
+Fetched directly from the HF file trees (not estimated from ratios), current as of this research pass:
+
+| Quant tier | Qwen3.8-27B (dense) | Qwen3.8-Flash-Next (MoE) |
+|---|---|---|
+| Q4_K_XL (`UD-Q4_K_XL`) | **17.6 GB** (existing baseline) | **111.4 GB** (4 parts: 10.9MB + 49.9GB + 49.4GB + 12.1GB) |
+| Q6_K_XL (`UD-Q6_K_XL`) | **25.3 GB** | **169 GB** (6 parts) |
+| Q8_0 | **29 GB** | **188 GB** (6 parts) |
+| BF16/fp16 | **54.67 GB** (50GB + 4.67GB, 2 parts) | **354 GB** (8 parts) |
+
+Sources: [unsloth/Qwen3.8-27B-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main)
+and its `BF16/` subfolder; [unsloth/Qwen3.8-Flash-Next-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main)
+and its `UD-Q4_K_XL/`, `UD-Q6_K_XL/`, `Q8_0/`, `BF16/` subfolders (per-file sizes summed). The
+preliminary Q8_0 figure floated before this research pass (~192GB) was slightly high — the real
+listing sums to **188 GB**; everything else in the preliminary list was accurate to within rounding.
+
+**The weight-quant axis and the KV-cache-quant axis are independent knobs.** Raising weight quality
+(Q4_K_XL → BF16) does not require raising `--cache-type-k/v` — the two flags are unrelated, and this
+repo already proves that pattern works (`q8_0` KV cache is deployed today against `Q4_K_XL` weights).
+A user chasing **maximum output quality** can run e.g. **BF16 weights + `q4_0` KV cache** — full-precision
+weights for quality, still-compressed KV for context budget — or any other combination in the tables
+below; nothing about picking a higher weight quant forces a matching KV precision.
+
+### 11.7 Total VRAM: does it fit, across quant tiers and both parallelism targets
+
+All totals = weights + growing KV cache + an estimated **+3-6 GB** compute-buffer/runtime overhead
+(same estimate band as §4, carried over — not re-derived for this architecture; flagged medium
+confidence there too). "Cards" = ceil(total ÷ 32GB), i.e. how many R9700-class 32GB cards it takes.
+
+#### Dense Qwen3.8-27B — 500k × 2 parallel / 1M stretch (`--ctx-size 1,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| Q4_K_XL (17.6GB) | ~79-85GB (**3**) | ~48-54GB (**2**) | ~33-39GB (**2**) |
+| Q6_K_XL (25.3GB) | ~89-92GB (**3**) | ~59-62GB (**2**, tight) | ~44-47GB (**2**) |
+| Q8_0 (29GB) | ~93-96GB (**3**, edge) | ~63-66GB (**2-3**, edge) | ~47-50GB (**2**) |
+| BF16 (54.67GB) | ~119-122GB (**4**) | ~88-91GB (**3**) | ~73-76GB (**3**) |
+
+#### Dense Qwen3.8-27B — 500k × 4 parallel (`--ctx-size 2,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| Q4_K_XL (17.6GB) | ~143-146GB (**5**) | ~82-85GB (**3**) | ~51-54GB (**2**) |
+| Q6_K_XL (25.3GB) | ~150-153GB (**5**) | ~89-92GB (**3**) | ~59-62GB (**2**, tight) |
+| Q8_0 (29GB) | ~154-157GB (**5**) | ~93-96GB (**3**, edge) | ~63-66GB (**2-3**, edge) |
+| BF16 (54.67GB) | ~180-183GB (**6**) | ~119-122GB (**4**) | ~88-91GB (**3**) |
+
+#### Qwen3.8-Flash-Next — 500k × 2 parallel / 1M stretch (`--ctx-size 1,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| UD-Q4_K_XL (111.4GB) | ~137-140GB (**5**) | ~126-129GB (**4**, edge) | ~120-123GB (**4**) |
+| UD-Q6_K_XL (169GB) | ~195-198GB (**7**) | ~183-186GB (**6**) | ~178-181GB (**6**) |
+| Q8_0 (188GB) | ~214-217GB (**7**) | ~202-205GB (**7**) | ~197-200GB (**7**) |
+| BF16 (354GB) | ~357-360GB (**12**) | ~357-360GB (**12**) | ~357-360GB (**12**) |
+
+#### Qwen3.8-Flash-Next — 500k × 4 parallel (`--ctx-size 2,000,000`)
+
+| Weight quant | fp16 KV → total (cards) | q8_0 KV → total (cards) | q4_0 KV → total (cards) |
+|---|---|---|---|
+| UD-Q4_K_XL (111.4GB) | ~160-163GB (**6**) | ~137-140GB (**5**) | ~126-129GB (**4**, edge) |
+| UD-Q6_K_XL (169GB) | ~218-221GB (**7**) | ~195-198GB (**7**) | ~183-186GB (**6**) |
+| Q8_0 (188GB) | ~234-237GB (**8**) | ~211-214GB (**7**) | ~199-202GB (**7**) |
+| BF16 (354GB) | ~397-400GB (**13**) | ~377-380GB (**12**) | ~366-369GB (**12**) |
+
+(Flash-Next's KV precision barely moves the total at any weight quant above `UD-Q6_K_XL` — the weights
+so dominate the budget that KV quantization stops mattering for the "how many cards" question. This
+is the clearest signal in this whole section: for Flash-Next, the weight-quant choice is the entire
+hardware-sizing decision; for the dense model, KV precision still matters a lot.)
+
+### 11.8 CPU MoE-expert offload — the one lever that could change this calculus
+
+Flash-Next is a 512-expert/10-active-per-token MoE, and llama.cpp has a purpose-built flag for exactly
+this shape of model, confirmed directly from the server README: **`--n-cpu-moe`** — *"keep the Mixture
+of Experts (MoE) weights of the first N layers in the CPU"* — plus the more general
+**`--override-tensor`** (*"override tensor buffer type"*, pattern-matched by tensor name) that the same
+flag is built on top of. Both are generic, architecture-agnostic llama.cpp mechanisms (they match on
+tensor name patterns, not model type), so there's no reason to expect them not to apply to Flash-Next's
+MoE tensors specifically — but this pass found **no primary source that has actually tested
+`--n-cpu-moe` against this specific qwen4exp architecture**, so treat "it works here" as plausible,
+not confirmed.
+
+If it does work as expected, this changes the whole weight-VRAM picture in §11.7: the ~90-95% of
+Flash-Next's weight footprint that's MoE expert tensors could live in system RAM while attention
+projections, the shared/non-expert tensors, and the full KV cache stay on GPU — meaning a much smaller
+GPU-VRAM number than the "all weights on GPU" tables above, at the cost of PCIe/RAM-bandwidth-bound
+inference speed for whichever experts get selected per token (this repo has no benchmark of that
+tradeoff, and it's highly system-RAM-bandwidth-dependent, so no number is given here — flagged as an
+escape hatch worth prototyping directly on the server, not something this research values responsibly
+without a real test run).
+
+### 11.9 Net recommendation: dense Qwen3.8-27B vs. Flash-Next, for this user's stated goal
+
+**Net loss for this user's goal, as things stand — stay on dense Qwen3.8-27B.** Reasoning:
+
+- The user's target (500k×2 or 500k×4, on a $20-30/GB-VRAM budget, GPUs in 32GB increments) is a
+  **VRAM-budget-constrained** goal, and §11.7 shows Flash-Next's *weights alone* (111-354GB depending
+  on quant) dwarf the entire dense-model total-VRAM figure from §4/§11.7 (33-183GB depending on quant)
+  at every parallelism target. Flash-Next's much cheaper per-token KV cache (§11.5, real and verified)
+  is a rounding error next to that weight-size gap — the "2.67× cheaper KV" win doesn't come close to
+  offsetting a "6-20× larger weight footprint," so at $20-30/GB-VRAM the *dense* model reaches 500k×2
+  or 500k×4 for a fraction of the card count and dollar cost that Flash-Next needs even at its lowest
+  usable quant (`UD-Q4_K_XL`, 4-5 cards minimum) — before even factoring in §11.1's immaturity flags.
+- The one scenario that could flip this verdict is `--n-cpu-moe` actually working well for this
+  architecture (§11.8) — if most of those 111-354GB of expert weights can sit in system RAM at
+  acceptable throughput, Flash-Next's GPU-VRAM number could shrink dramatically and its real KV-cache
+  advantage would start to matter. That is untested here and shouldn't be assumed; it's the one
+  concrete next step worth trying on the actual server before ruling Flash-Next out permanently.
+- Independent of VRAM: §11.1's primary-source-flagged correctness caveats (unverified conv branch,
+  3%-divergence QSA, prefill-position-0-only PLE exactness) are a real quality/stability risk on a
+  production coding-agent stack that dense Qwen3.8-27B simply doesn't carry, since it's been running
+  in this repo already.
+
+---
+
+## 12. 4-parallel × 500k scenario — all four combinations side by side
+
+Per §2's already-established, cited rule (`n_ctx_per_seq = n_ctx / n_parallel`,
+[ggml-org/llama.cpp#11681](https://github.com/ggml-org/llama.cpp/issues/11681)), the same division
+applies at 4 slots: **500k tokens on each of 4 parallel slots requires `--ctx-size 2,000,000`** —
+double the 2-parallel target's `--ctx-size 1,000,000`, for the same reason 500k×2 needed double
+262,144. This isn't a new mechanism, just the same formula at `--parallel 4`.
+
+Full per-quant-tier tables for all four combinations are in §11.7 above (dense×2, dense×4, Flash-Next×2,
+Flash-Next×4 are each their own table there). Headline comparison at the KV precision already proven
+in production in this repo (`q8_0`) and each model's respective current/cheapest-usable weight quant:
+
+| Scenario | `--ctx-size` | Weight quant | q8_0-KV total VRAM | Cards (32GB) |
+|---|---|---|---|---|
+| Dense × 2 (or 1M stretch) | 1,000,000 | Q4_K_XL (17.6GB, current) | ~48-54GB | **2** |
+| Dense × 4 | 2,000,000 | Q4_K_XL (17.6GB, current) | ~82-85GB | **3** |
+| Flash-Next × 2 (or 1M stretch) | 1,000,000 | UD-Q4_K_XL (111.4GB, cheapest usable) | ~126-129GB | **4**, edge |
+| Flash-Next × 4 | 2,000,000 | UD-Q4_K_XL (111.4GB, cheapest usable) | ~137-140GB | **5** |
+
+**4-parallel × 500k reachability against this repo's existing roadmap stages (§9):**
+
+- **(a) Current 1×R9700 32GB:** none of the four combinations fit — not even dense×2 at any weight/KV
+  quant (§4's own conclusion, unchanged).
+- **(b) The 2-3×R9700 roadmap already proposed in §9 (64-96GB):** covers **dense×2 fully** (stage 3, as
+  already established) and **dense×4 at `q4_0` KV with Q4_K_XL or Q6_K_XL weights** (~51-62GB, fits in
+  64-96GB) — but **not** dense×4 at higher weight quants (Q8_0/BF16 need 3-6 cards depending on KV
+  precision, per §11.7's dense×4 table) and **not any Flash-Next scenario** (minimum is 4 cards/128GB
+  even at the cheapest usable quant and tightest KV).
+- **(c) The full 4-6×R9700 stretch scenario** (`server-planing.md`'s original plan, 128-192GB pooled):
+  covers **dense×4 at every weight quant up to BF16** (worst case ~91GB at BF16+q4_0, well inside
+  128GB) and **Flash-Next×2 at `UD-Q4_K_XL`** (126-140GB, fits a 5-card/160GB build, tight on a 4-card/
+  128GB one) — but **not** Flash-Next×4 at any weight quant above `UD-Q4_K_XL`, and not Flash-Next at
+  `BF16` under any parallelism (needs 12-13 cards, an entirely different scale of build than anything
+  in this doc's roadmap).
+
+---
+
+## Sources
+
+- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) — `--ctx-size`, `--parallel`, `--cache-type-k/v`, `--kv-unified`, `--cache-reuse`, `--n-cpu-moe`, `--override-tensor` flag definitions
+- [ggml-org/llama.cpp#11681](https://github.com/ggml-org/llama.cpp/issues/11681) — real server log proving `n_ctx_per_seq = n_ctx / n_parallel`
+- [ggml-org/llama.cpp#27742](https://github.com/ggml-org/llama.cpp/pull/27742) — "model: add Qwen3.8-Flash-Next (qwen4exp)", merged 2026-08-27; architecture details, `n_stream == 1` / `-kvu` multi-slot requirement, and the conv-branch/QSA-divergence/PLE-prefill correctness caveats
+- [Qwen/Qwen3.8-27B config.json](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json)
+- [Qwen/Qwen3.8-Flash-Next config.json](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/raw/main/config.json)
+- [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
+- [unsloth/Qwen3.8-27B-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main) — 17.6GB (Q4_K_XL), 25.3GB (Q6_K_XL), 29GB (Q8_0), 54.67GB (BF16) weight sizes
+- [unsloth/Qwen3.8-Flash-Next-GGUF file tree](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main) — 111.4GB (UD-Q4_K_XL), 169GB (UD-Q6_K_XL), 188GB (Q8_0), 354GB (BF16) weight sizes, summed from each quant's per-file listing
+- [ggml-org/llama.cpp docs/docker.md](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/docs/docker.md) — `server-cuda`/`server-rocm` separate image tags
+- [AMD chipsets product page](https://www.amd.com/en/products/processors/chipsets.html)
+- [Tom's Hardware — Threadripper 3960X/3970X, sTRX4/TRX40 launch](https://www.tomshardware.com/news/amd-unveils-threadripper-3960x-and-3970x-ryzen-9-3950x-details-and-athlon-3000g/2)
+- [PCWorld — Threadripper PRO launch, 128 PCIe lanes](https://www.pcworld.com/article/393181/amd-threadripper-pro-has-64-cores-128-pcie-lanes-and-8-channel-memory-support.html)
+- [NVIDIA — GeForce RTX 3090 product page](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090/)
+- [Lowyat.net — RTX 3060 official 170W TDP](https://www.lowyat.net/2021/232659/nvidia-geforce-rtx-3060-specifications-now-official-includes-3584-cuda-cores-and-170w-tdp/)
+- [BuildMyServer — GTX 1080 180W/PCIe3.0/1×8-pin spec listing](https://buildmyserver.com/products/zotac-nvidia-geforce-gtx-1080-8gb-gddr5-180w-pcie-3-0-x16-double-wide-gpu)
+- [MSI — GT 710 1GD5 LP spec page](https://www.msi.com/Graphics-Card/GT-710-1GD5-LP/Specification)
+- [overclock3d — AMD Radeon AI PRO R9700 $1,299 MSRP](https://overclock3d.net/news/gpu-displays/amd-unveils-its-1299-radeon-ai-pro-r9700-32gb-workstation-gpu/)
+- [pricehistory.app — R9700 street price tracker](https://pricehistory.app/p/powercolor-amd-radeon-ai-pro-r9700-32gb-BFcRhGIm)
+- [bestvaluegpu.com — RTX 3090 used price tracker, Sep 2026](https://bestvaluegpu.com/history/new-and-used-rtx-3090-price-history-and-specs/)
+- [gpupoet.com — RTX 3060 12GB used listings](https://gpupoet.com/gpu/shop/nvidia-geforce-rtx-3060)
+- [Qwen Code docs — overview](https://qwenlm.github.io/qwen-code-docs/en/users/overview/)
+- [Qwen Code docs — Auto Mode (fast-model pattern)](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/)
+- This repo: [`docker-compose.yml`](../../docker-compose.yml), [`.env.example`](../../.env.example), [`docs/server-planing.md`](../server-planing.md), [`qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md), [`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md), [`rocm-gpu-pin-and-render-group.md`](rocm-gpu-pin-and-render-group.md), [`fast-model-choice.md`](fast-model-choice.md)
+
+## Confidence/uncertainty summary
+
+- **High confidence:** the KV-cache-per-token formulas for both dense models and Flash-Next (computed
+  directly from each model's own `config.json`, same method this repo's prior research already used
+  and cross-checked); the `n_ctx_per_seq = n_ctx / n_parallel` division behavior (directly evidenced by
+  a real server log in a llama.cpp GitHub issue, and independently already documented in this repo's
+  own `.env.example`) — and confirmed to apply identically at `--parallel 4` since the mechanism is
+  parallel-count-agnostic; official TDP figures for GTX 1080, RTX 2080, RTX 3060, RTX 3090, GT 710
+  (each cross-checked against 2+ independent spec listings or the vendor's own product page); the
+  TR4-is-PCIe3/sTRX4-is-PCIe4 generational split (direct launch-coverage quote); the existence of
+  separate `server-cuda`/`server-rocm` llama.cpp image tags; Qwen3.8-Flash-Next's `config.json`
+  architecture params and PR #27742's merge date/status and its own stated correctness caveats and
+  `-kvu` multi-slot requirement (all directly quoted from the primary source); the weight file sizes
+  for both models at all four quant tiers (summed directly from each HF repo's real file listing, not
+  estimated).
+- **Medium confidence:** the compute-buffer/runtime-overhead estimate in §4/§11.7 (+3-6 GiB) —
+  extrapolated from one in-repo incident's before/after numbers, not a llama.cpp-documented formula,
+  and carried over to Flash-Next without re-derivation for its different architecture; the Gated
+  DeltaNet fixed recurrent-state size in §11.4 (~108 MiB/slot) — this doc's own derivation from the
+  published head-dimension config, not a value found in llama.cpp source or docs; whether
+  `--cache-type-k/v` quantization applies identically to Flash-Next's 12 full-attention layers as it
+  does to a plain transformer (reasonable extrapolation, not directly confirmed for this architecture);
+  whether `--n-cpu-moe`/`--override-tensor` actually work against Flash-Next's specific MoE tensor
+  layout (architecture-agnostic mechanism, but untested against this model by any primary source found);
+  real-world PCIe lane availability for 6 GPUs on a specific sTRX4 board (§5) — no single board's exact
+  lane map was fetched, this is a reasonable-but-unverified estimate from typical sTRX4 board behavior.
+- **Low confidence / explicitly estimated, not cited fact:** all used-GPU marketplace pricing (§8) —
+  live asking-price snapshots from a single search pass, not sold-price data; the used sTRX4
+  CPU+motherboard combo price in the roadmap's stage 4 (§9) — not researched at all in this pass,
+  flagged as a placeholder estimate; whether YaRN-scaled 500k/1M context actually holds output
+  quality for either Qwen3.8-27B or Qwen3.8-Flash-Next — no primary source (Qwen's own docs included)
+  publishes long-context quality benchmarks past the 262,144 native length for either model, so this is
+  a known-unknown carried forward from each model card's "YaRN-extensible" claim, not a verified
+  capability; whether the specific `ghcr.io/ggml-org/llama.cpp:server-rocm` image currently cached on
+  this repo's server actually postdates PR #27742's 2026-08-27 merge — not checked against the live
+  server in this pass, flagged as an action item in §11.1 rather than a confirmed fact.
@@ -0,0 +1,222 @@
+# Evaluating Colibrì (JustVugg/colibri) for this stack
+
+**Date:** 2026-09-08
+**Scope:** The user flagged https://github.com/JustVugg/colibri as something that "could revolutionize"
+this self-hosted AI stack. What is Colibrì actually, is it compatible with this stack's AMD
+ROCm/HIP-only single-GPU setup, and — even if compatible — does it fill a real gap versus what
+llama.cpp, OmniRoute, Qdrant, Neo4j, and ComfyUI already do here?
+
+## 1. What Colibrì actually is
+
+Colibrì is a pure-C, zero-runtime-dependency inference engine whose specific trick is treating
+"storage, RAM, and VRAM as a single inference hierarchy" so that huge mixture-of-experts (MoE)
+models — far bigger than any one machine's VRAM+RAM — can still run, by keeping the small dense
+layers resident and streaming the (much larger) set of routed experts from disk on demand with an
+LRU/"hot-store" cache and router-lookahead prefetching:
+
+> "Colibrì is an open-source inference engine designed to run frontier mixture-of-experts (MoE)
+> models on consumer hardware... The fundamental approach uses 'a JIT, but for weights' — parameters
+> are staged across storage tiers (VRAM/RAM/NVMe) based on measured routing patterns rather than kept
+> resident."
+— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
+
+It ships single-C-file implementations for eight specific model families — GLM-5.2/5.3,
+GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 (35B-A3B), and OLMoE
+— each requiring model weights pre-converted into Colibrì's own container format (`coli convert`),
+not arbitrary GGUF files:
+
+> "Eight model families with single C file implementations... GLM-5.2/5.3 | 744B | 372GB | 16GB+ ...
+> Kimi K3 | 2.8T | 1.6TB | 32GB+"
+— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
+
+It's meant to be run either from prebuilt binaries/releases, built from source (`./setup.sh` under
+`c/`), or via Docker (`docker/Dockerfile`, `docker/Dockerfile.slim`, `docker/docker-compose.yml` exist
+in-repo — confirmed present via the GitHub contents API, https://api.github.com/repos/JustVugg/colibri/contents/docker),
+exposing an OpenAI- and Anthropic-compatible HTTP API (`coli serve`, default `http://127.0.0.1:8000/v1`,
+plus `/v1/messages`) — the same shape OmniRoute already expects from a provider, per third-party
+summaries of `docs/api.md` and `docs/serve_protocol.md`
+([search result summary, secondary](https://github.com/JustVugg/colibri/blob/main/docs/api.md)).
+
+It launched July 10, 2026 and went viral on Hacker News the same day (453 points) on the strength of
+running the 744B-parameter GLM-5.2 model on a 25GB-RAM consumer box:
+
+> "A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744
+> billion parameters, on a regular PC with 25GB of memory."
+— https://gigazine.net/gsc_news/en/20260710-colibri-glm/ (secondary coverage)
+
+## 2. Hardware/runtime requirements — AMD ROCm or NVIDIA-only?
+
+**This is the load-bearing question given this stack runs llama.cpp on ROCm/HIP, not CUDA, on a
+single AMD Radeon AI PRO R9700.** The answer is more nuanced than a flat yes/no — verified against
+source, not just README prose:
+
+- **The engine is CPU-first; a GPU is optional at all.** `docs/quickstart.md` states plainly: "You do
+  **not** need a GPU. A GPU only helps if you have one; the engine runs CPU-only by default."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/quickstart.md
+- **AMD/ROCm support is real, shipped, and reasonably recent — not just a README claim.** It was
+  requested in [issue #69](https://github.com/JustVugg/colibri/issues/69) (opened 2026-07-11, "No
+  ROCM support"), the maintainer confirmed it was mechanically straightforward since HIP closely
+  mirrors CUDA, an initial PR (#112) added it but was **closed unmerged**, and a follow-up PR — tracked
+  as [#339](https://github.com/JustVugg/colibri/issues/69) — landed the actual mechanism that shipped:
+  a single shared CUDA kernel source (`backend_cuda.cu`) compiled either by `nvcc` or by `hipcc`
+  against a compatibility header:
+
+  > "backend_gpu_compat.h — 'one GPU backend source, two vendors.' ... maps CUDA runtime calls to HIP
+  > equivalents when compiled by hipcc with `HIP=1`... handles architecture-specific guards for rocWMMA
+  > availability and matrix core support across different GPU architectures (gfx906, gfx908, gfx11xx,
+  > etc.)."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/c/backend_gpu_compat.h (confirmed present
+  in the current `main` branch — this is not a stale/unmerged branch)
+
+  This shipped in a **tagged release**, not just an open PR — `CHANGELOG.md` lists "AMD GPU support" as
+  part of v1.1.0 (2026-07-22): "AMD GPU support, dual-SSD streaming, fmt=5/fmt=6 quantization formats."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md
+- **Notably, the community contributor who tested it used an RX 9070 XT / gfx1201** — the same RDNA4
+  architecture generation as this stack's Radeon AI PRO R9700 — per the PR #339 description: "validated
+  across CPU builds, HIP testing on gfx1201, and NVIDIA compatibility verification." That's a genuinely
+  favorable, non-generic signal for this specific card's GPU family, better than "AMD support exists
+  somewhere."
+- **But ROCm support is thinner and less documented than the CUDA/Metal paths.** `docs/` has `cuda.md`,
+  `metal.md`, `metal_implementation.md`, and `vulkan.md`, but **no `rocm.md` or `hip.md`** (confirmed via
+  the GitHub contents API listing of `docs/`, https://api.github.com/repos/JustVugg/colibri/contents/docs).
+  Model-specific tuning docs are written CUDA-only with no AMD mention at all — e.g.
+  `docs/qwen36-cuda-tier.md` references `COLI_CUDA=1`, `backend_cuda.cu`, and lists test hardware as
+  "RTX 3070 8 GB + Quadro RTX 4000" (both NVIDIA), with zero AMD/ROCm/HIP text anywhere in that
+  document. — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/qwen36-cuda-tier.md
+  So: the *general* GPU-acceleration mechanism supports ROCm/HIP and has been community-validated on
+  hardware close to the R9700, but the *per-model* tiering/tuning documentation and (presumably) most
+  of the maintainer's own benchmarking is CUDA-first. Treat AMD support as functional-but-secondary,
+  not a first-class, symmetrically-tested backend.
+- **A separate GPU-agnostic path also exists**: a Vulkan backend (`backend_vulkan.c`, confirmed present
+  in the `c/` directory listing) that the project positions as covering "AMD via Mesa/RADV" as a
+  vendor-neutral fallback, independent of the HIP path above — this would also be viable on the R9700
+  in principle, though vendor-neutral compute back ends are typically slower than a vendor SDK path
+  (HIP/ROCm) and no R9700/gfx1201-specific Vulkan numbers were found in the docs reviewed.
+
+**Bottom line on hardware fit: not a blocker.** Unlike a hard CUDA-only dependency, Colibrì's AMD/HIP
+path is real, shipped in a release, and specifically exercised on the same RDNA4 family as this
+stack's GPU — this is not a disqualifying finding the way it would be for a CUDA-only tool.
+
+## 3. License
+
+Apache License 2.0, confirmed by fetching `LICENSE` directly from the repo — a standard permissive
+license, fine for self-hosted use here (commercial or non-commercial, modification, redistribution all
+permitted, patent grant included, "AS IS" with no warranty).
+— https://raw.githubusercontent.com/JustVugg/colibri/main/LICENSE
+
+Model weights are licensed separately from the engine — e.g. the README notes "GLM-5.2 weights
+released by Z.ai under MIT"
+(https://raw.githubusercontent.com/JustVugg/colibri/main/README.md) — so each model's own license
+would need checking before use, same as with any GGUF today.
+
+## 4. Maturity signals
+
+Pulled from the GitHub API (https://api.github.com/repos/JustVugg/colibri) and the changelog:
+
+| Signal | Value |
+|---|---|
+| Repo created | 2026-07-01 |
+| First tagged release (v1.0.0) | 2026-07-19 |
+| Current version (as of today) | 1.10.2 (2026-09-06) |
+| Age at time of writing | ~10 weeks |
+| Stars / Forks | 27,047 / 2,963 |
+| Open issues | 104 |
+| Top contributor | JustVugg — 1,077 commits |
+| #2 contributor | ZacharyZcR — 163 commits |
+| Total contributors | 100+ (long tail, most in single digits) |
+| License | Apache 2.0 |
+| Archived? | No |
+
+Read honestly, this is **a viral, very-early-stage, single-maintainer-dominated project**, not a
+mature or slow-burn one:
+
+- It is ~10 weeks old today. The star count (27k) is wildly disproportionate to that age and reflects
+  a Hacker News front-page moment (453 points the day it launched,
+  https://gigazine.net/gsc_news/en/20260710-colibri-glm/), not organic multi-year adoption. Stars are
+  a popularity signal, not a quality proof, and here the ratio (huge stars, ~10 weeks old, one dominant
+  committer) is itself a maturity red flag worth naming rather than a mark in its favor.
+- Commit/release activity is genuinely fast — 14 tagged releases in under 8 weeks (v1.0.0 on 2026-07-19
+  through v1.10.2 on 2026-09-06, https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md)
+  — so it is actively maintained day-to-day, not abandoned. But that pace also means breaking changes
+  and security patches are frequent: v1.6.2 (2026-08-14) was itself "a security release: six
+  privately-reported memory-safety issues fixed... from untrusted input," and v1.10.2 (2026-09-06)
+  again touched "security fixes for image API" — both signs of a codebase still finding its footing on
+  hardening, not evidence of instability being the norm, but worth weighing given this stack would be
+  exposing any such server on an internal network via OmniRoute.
+- Contribution concentration is heavy: the maintainer (JustVugg) has ~6.6x the commits of the next
+  contributor, and the rest of the 100+ contributor list trails off into single-digit-commit
+  drive-by PRs (per the GitHub contributors API,
+  https://api.github.com/repos/JustVugg/colibri/contributors) — a classic "one person's viral project
+  plus a wave of small first-time PRs" shape, not an established multi-maintainer team.
+- The AMD/ROCm feature specifically has a short, thin history: requested 2026-07-11, shipped 2026-07-22
+  (11 days later, in v1.1.0), documented only implicitly (no dedicated `docs/hip.md`/`rocm.md`, unlike
+  every other backend) — i.e., it is the newest and least-independently-verified of the project's four
+  GPU backends (CUDA, Metal, Vulkan, HIP).
+
+## 5. What capability gap it would actually fill in this stack
+
+Concretely comparing against what's already running (`docker-compose.yml`):
+
+- **llama.cpp (ROCm) already fully GPU-resides the current model** — `--n-gpu-layers 999` on the
+  llama-server service means the whole Qwen3.8-27B-class GGUF sits in the R9700's VRAM and runs at
+  normal, fast, interactive token rates. Colibrì's entire value proposition is the *opposite* case:
+  models **too large to fit in VRAM+RAM at all**, accepted at the cost of streaming most of the model
+  from disk on every forward pass. For a model that already fits on this GPU (which is the whole point
+  of the current setup), Colibrì offers no benefit — llama.cpp is already doing the fast thing.
+- **The actual gap it could fill is running models this stack categorically cannot run today** — e.g.
+  GLM-5.2 (744B), DeepSeek V4 Flash (284B), or Kimi K3 (2.8T), none of which would ever fit on a single
+  R9700 regardless of quantization. Colibrì's own benchmarks make the cost of that explicit and, unlike
+  its capability claims, these are the project's self-reported numbers, not independently reproduced —
+  flagged as such:
+
+  > "6× RTX 5090 (full residency): 5.8-6.8 tokens/second decode... 128GB CPU-only desktop: ~1.8
+  > tokens/second (warm)... Single RTX 5070 Ti: 1.07 tokens/second... 25GB dev box: 0.05-0.1
+  > tokens/second (baseline)."
+  — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md (self-reported, unverified by
+  any third party found during this research)
+
+  At 0.05–2 tokens/second on any hardware remotely resembling a single-GPU workstation, this is not
+  usable for the interactive coding-CLI workloads this stack is built around (Claude Code CLI, Kimi
+  CLI, etc. routed through OmniRoute per the README) — those need low per-token latency for tool-calling
+  round trips, not throughput measured in seconds per token. It would only be plausible as an
+  occasional, patient, offline/batch capability (e.g. "let a huge model chew on a large doc overnight")
+  layered in *alongside*, not instead of, the current llama.cpp path.
+- **Disk footprint is a real new cost, not a marginal one**: 167GB–1.6TB per model
+  (https://raw.githubusercontent.com/JustVugg/colibri/main/README.md), which would need to be
+  provisioned in addition to the existing `models` Docker volume, GGUF downloads, Qdrant/Neo4j
+  volumes, and ComfyUI's model files already on this box.
+- **No overlap or replacement value for OmniRoute, Qdrant, Neo4j, or ComfyUI** — Colibrì is strictly an
+  inference-engine alternative to llama.cpp for a narrow, specific list of very large MoE models; it
+  does nothing related to gateway/key-management (OmniRoute's job), vector/graph storage (Qdrant/Neo4j's
+  job), or image generation (ComfyUI's job). Its own `coli serve` OpenAI/Anthropic-compatible endpoint
+  could in principle be registered as another OmniRoute provider the same way llama-server is today —
+  that part is mechanically plausible — but it would be adding a second, much slower inference backend
+  next to the existing fast one, not replacing or upgrading anything currently in the stack.
+
+## 6. Bottom line
+
+**Not a fit for this stack right now, and the "revolutionize" framing does not hold up** — but for a
+more specific reason than "wrong GPU vendor":
+
+- **ROCm/AMD support is real and not the blocker one might expect.** It shipped in a tagged release
+  (v1.1.0, 2026-07-22), lives in a compatibility header confirmed present on `main`
+  (`c/backend_gpu_compat.h`), and was community-tested on an RX 9070 XT / gfx1201 — the same RDNA4
+  family as this stack's R9700. This is genuinely worth noting as a positive, since it's the kind of
+  thing that usually *is* disqualifying for AMD-only stacks and here it isn't.
+- **The disqualifying issue is fit, not hardware**: Colibrì solves "run a model way too big for your
+  VRAM+RAM by streaming most of it from disk," at 0.05–2 tokens/second. This stack's actual situation is
+  the opposite — a model sized to fully fit and run fast on a single GPU via llama.cpp. Colibrì would add
+  no speed, capability, or reliability benefit to the model already running here, and its own numbers
+  show it isn't fast enough to serve the interactive coding-CLI use case this stack exists for.
+- It could only ever be interesting as a *bolt-on, offline-only* capability for occasionally running an
+  otherwise-impossible frontier-scale model (700B–2.8T params) for patient, non-interactive tasks — at
+  the cost of hundreds of GB to ~1.6TB of extra disk per model, on a ~10-week-old, single-maintainer,
+  still-hardening project (two security-patch releases already) with no ROCm-specific documentation and
+  the thinnest testing history of its four GPU backends.
+- **Recommendation: worth a passing watch, not worth integrating.** Revisit if/when: (a) the project
+  reaches a more established maturity point (6–12 months, broader contributor base, dedicated ROCm docs
+  bringing it to parity with the CUDA/Metal paths), and (b) there's an actual concrete need in this stack
+  to run a model in the 200B+ range that cannot fit on the R9700 — which isn't the case today (the
+  current model is deliberately sized to fit fully in VRAM). Until then, integrating it would add
+  operational surface (a new container, huge disk provisioning, a newer/less-hardened codebase) for no
+  measurable improvement over the existing llama.cpp/ROCm path.
@@ -195,6 +195,39 @@ comfortably affords the higher-precision quant.
 - [docs/research/qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) (this repo — cross-referenced
  for the 27B model's own, still-open, tool-calling parser bugs)

+## Implementation note (2026-09-09) — what actually shipped, and why it differs
+
+The model pick (`Qwen3-4B-Instruct-2507`) held up and is what's deployed. Several sizing assumptions in
+this doc didn't survive contact with the real deployment, though — worth recording so the next person
+tuning this doesn't re-derive the same corrections from scratch:
+
+- **Service name is `qwen-classifier`, not `llama-server-fast`** — this doc's proposed name never got
+  used. There's no `LLAMA_FAST_CTX_SIZE`/`LLAMA_FAST_PARALLEL` in `.env.example` either; the real config
+  lives inline in `docker-compose.yml`'s `qwen-classifier` command.
+- **CPU-only was tried first and rejected** — this doc's VRAM budget analysis (§5) assumed GPU
+  residency from the start, but the actual rollout path tried CPU-only first (to sidestep VRAM
+  contention entirely) and found it too slow: real classification calls blew past OmniRoute's request
+  timeout and retry-looped. Moved to GPU after that, which is what §5's math was for all along.
+- **Q4_K_XL weights, not Q8_0** — §5's "~2.4GB headroom" case assumed Q8_0 (4.28GB). In practice, fitting
+  the classifier onto the R9700 *alongside* the 27B model (not in an assumed-empty 7GB budget) left only
+  ~6.1GB free VRAM total, and even Q4_K_XL (2.37GB) plus full-context KV cache didn't leave enough real
+  margin at full GPU offload — see the "measured live" numbers in `docker-compose.yml`'s `qwen-classifier`
+  comment block. Landed on **partial GPU offload (28/36 layers)** instead of full offload, which is not a
+  case this doc considered at all.
+- **65536 context, not 8192** — §5 sized the context "in the low thousands," reasoning from qwen-code's
+  two-stage classifier description alone. Directly reading qwen-code's actual source
+  (`packages/core/src/permissions/classifier-transcript.ts`: `MAX_TRANSCRIPT_MESSAGES=40`,
+  `MAX_HISTORICAL_ACTION_CHARS=4000`/message) puts the real worst case at ~40-50K tokens — confirmed
+  live, a real classifier call during testing hit 15,116 prompt tokens. 8192 would have been undersized
+  for real usage; 65536 gives margin without the original setting.json value (131072, copied from the
+  main model's entry, not a real qwen-code requirement) wasting VRAM for no reason.
+- **§4's `--reasoning off` recommendation was initially missed** in the first deployment pass and added
+  only once this doc was re-read while writing this note. It's now in `docker-compose.yml`'s
+  `qwen-classifier` command, per this doc's own "add it regardless, no-cost safety net" reasoning — still
+  unconfirmed whether the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build actually reproduces
+  #20809 (nothing in testing so far surfaced `reasoning_content` where `tool_calls` was expected, but
+  that wasn't specifically probed for either).
+
 ## Confidence/uncertainty summary

 - **High confidence:** Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote);
@@ -0,0 +1,84 @@
+# OmniRoute's per-connection semaphore timeout — hardcoded, not a setting
+
+**Date:** 2026-09-09
+
+Any OmniRoute connection whose upstream can only handle a small, fixed number of concurrent requests
+(this repo's `llama-server`/`qwen-classifier`, both effectively single-GPU-slot-limited) can hit a hard
+30-second reject once more requests are in flight than the connection's `maxConcurrent` allows — even
+though the request would have succeeded fine if it had just waited its turn. This surfaced first as the
+`pr-agent`/`CodersPlacePI` 429/504 investigation (see the issue tracker), then again while sizing
+`qwen-classifier`. Recorded here so it doesn't have to be re-diagnosed from scratch next time.
+
+## The error
+
+```
+{"error":{"message":"Semaphore timeout after 30000ms for <provider>:<connectionId>","type":"rate_limit_error","code":"rate_limit_exceeded"}}
+```
+
+## Root cause (confirmed against OmniRoute's own source, [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute))
+
+`open-sse/services/accountSemaphore.ts`:
+
+```ts
+const DEFAULT_TIMEOUT_MS = 30_000;
+...
+function createSemaphoreTimeoutError(semaphoreKey, timeoutMs) {
+  const error = new Error(`Semaphore timeout after ${timeoutMs}ms for ${semaphoreKey}`);
+  error.code = "SEMAPHORE_TIMEOUT"; // classified upstream as HTTP 429 rate_limit_exceeded
+  return error;
+}
+```
+
+Called from `open-sse/handlers/chatCore.ts`:
+
+```ts
+await acquireAccountSemaphore(accountSemaphoreKey, {
+  maxConcurrency: accountSemaphoreMaxConcurrency, // = the connection's maxConcurrent
+  signal: streamController.signal,
+  // no timeoutMs passed → always falls back to the hardcoded 30_000 default
+})
+```
+
+This is **not** the same thing as OmniRoute's documented quota-share concurrency gate
+(`open-sse/services/combo/quotaShareConcurrency.ts`, key prefix `qsconn:`), which is deliberately
+fail-open per its own doc comment ("a saturated queue or timeout proceeds without a slot rather than
+ever rejecting a dispatchable request") — that one only matters for quota-share combos. The account
+semaphore above is a *different*, always-on gate keyed `provider:connectionId`, has no fail-open path,
+and its 30-second timeout is a bare `await` with nothing passed to override it — not exposed via
+`/api/resilience`, not an env var, not a dashboard toggle, not documented anywhere in
+`docs/reference/ENVIRONMENT.md`. It's a hardcoded constant in vendored code.
+
+Also **not** the same as `requestQueue.maxWaitMs` (visible via `GET /api/resilience`, this deployment
+already has it at `86400000`) — that one bounds a Bottleneck-managed *execution* timer that starts only
+after dispatch, surfaces as HTTP 504 `RATE_LIMIT_EXECUTION_TIMEOUT`, and is unrelated to the 429 above.
+
+## What actually fixes it
+
+The 30s ceiling itself cannot be raised — no config surface reaches it in the current OmniRoute build.
+Two real options:
+
+1. **Bypass the semaphore, let the upstream's own queue absorb concurrency instead.**
+   `maxConcurrency == null || maxConcurrency <= 0` fully bypasses `accountSemaphore.ts` (see
+   `isBypassed()`) — no gate, no 30s timer, requests pass straight through to the upstream. This only
+   works if the upstream itself queues gracefully with no reject-timeout of its own — confirmed true for
+   llama.cpp's server (`tools/server/server-queue.cpp` has no queue-wait timeout; excess requests just
+   wait for a free slot). If you do this, also raise the connection's own
+   `providerSpecificData.timeoutMs` (bounded 1ms–24h, `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS`) generously —
+   that's the timer that now matters: "did the upstream return response headers in time," which on
+   llama.cpp means the full queue-wait-then-generate time, since llama.cpp sends **zero bytes, not even
+   headers**, while a request sits queued (confirmed in `server-context.cpp`: `res->status = 200` is only
+   set after the first generated token exists).
+2. **Reduce how often more than `maxConcurrent` requests actually stack up** — e.g. the
+   `pr-agent`/Gitea webhook fix (narrowing the subscribed event list so one PR action doesn't fire 3+
+   near-simultaneous AI calls). Doesn't remove the ceiling, just makes it less likely to be hit.
+
+Applied in this repo: `llama-server`'s OmniRoute connection has `maxConcurrent: null` and
+`providerSpecificData.timeoutMs: 1200000` (20 min — matches worst-case 2-slots-busy + queued + own
+generation time). `qwen-classifier` uses a much shorter `timeoutMs: 120000` since it isn't
+GPU-contended the same way — see `docs/coding-cli-setup/qwen-code.md`.
+
+## Sources
+
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `open-sse/services/accountSemaphore.ts`, `open-sse/handlers/chatCore.ts`, `open-sse/services/combo/quotaShareConcurrency.ts`, `open-sse/services/rateLimitManager.ts`, `docs/architecture/RESILIENCE_GUIDE.md`, `docs/reference/ENVIRONMENT.md`
+- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `tools/server/server-queue.cpp`, `tools/server/server-context.cpp`
+- `src/shared/validation/providerSpecificData.ts` (OmniRoute) — `MAX_PROVIDER_SPECIFIC_TIMEOUT_MS` bound
@@ -0,0 +1,60 @@
+# OmniRoute's "non-ping SSE" first-token deadline — a different timer than `STREAM_IDLE_TIMEOUT_MS`
+
+**Date:** 2026-09-10
+
+`STREAM_IDLE_TIMEOUT_MS` was raised to 180000 on 2026-09-09 (see docker-compose.yml's `omniroute`
+service) specifically to give contended `llama-server` prefill room to produce a first token. It didn't
+work: the very next morning, qwen-code sessions against `qwen3.8-27b-local` still hit repeated
+
+```
+Stream produced no non-ping SSE event within 95000ms
+```
+
+(and once at 115000ms) — both well under the 180s the compose fix set, and well under the connection's
+own `providerSpecificData.timeoutMs: 1200000` (confirmed live via `GET /api/providers/<id>`). Neither of
+those settings bounds this failure.
+
+## Root cause
+
+Per OmniRoute's own docs (`docs/reference/ENVIRONMENT.md`, "Timeout Settings" section) and a maintainer
+reply in [diegosouzapw/OmniRoute#10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602):
+
+| Variable | Default | Governs |
+|---|---|---|
+| `REQUEST_TIMEOUT_MS` | 600000 (10 min) | Overall upstream request budget. **The first non-ping SSE event's deadline inherits this one.** |
+| `STREAM_IDLE_TIMEOUT_MS` | 120000 (2 min) | Max gap between *successive* SSE chunks once streaming has already started — does not govern the wait for the first chunk. |
+| `STREAM_PING_INTERVAL_MS` | 30000 (30s) | How often OmniRoute emits its own keepalive pings on the stream — these explicitly do not count as "non-ping" events, so they can't rescue a request against the first deadline. |
+
+So the 2026-09-09 fix tuned the wrong timer for this failure mode: `STREAM_IDLE_TIMEOUT_MS` only matters
+once `llama-server` has already emitted something. The "no token at all yet" case — exactly what a large
+compact-prompt prefill on a contended local model produces — is bounded by `REQUEST_TIMEOUT_MS` instead.
+
+The observed 95000ms/115000ms figures are also *not* `REQUEST_TIMEOUT_MS`'s raw 600000ms default: OmniRoute
+computes the first-event deadline as **remaining budget**, not a flat timer — `REQUEST_TIMEOUT_MS` minus
+time already spent in OmniRoute's own request-queue/retry/cooldown cycle (`requestRetry: 3`,
+`connectionCooldown.apikey.baseCooldownMs`, provider breaker) before the request was actually dispatched
+to `llama-server`. Confirmed live via `GET /api/settings` → `resilienceSettings` on this deployment. Most
+of the 10-minute default budget was being burned by retries before the final attempt even started.
+
+## Fix
+
+Set `REQUEST_TIMEOUT_MS` explicitly, generously — applied in docker-compose.yml as
+`OMNIROUTE_REQUEST_TIMEOUT_MS` (default 1800000 / 30 min), same pattern as
+`OMNIROUTE_STREAM_IDLE_TIMEOUT_MS`. This doesn't replace the 2026-09-09 `STREAM_IDLE_TIMEOUT_MS` fix —
+that one still matters for mid-stream stalls after generation has started — it addresses the separate
+"nothing has arrived yet" case that fix didn't cover.
+
+Raising `REQUEST_TIMEOUT_MS` buys headroom; it doesn't address *why* prefill on a 50K+ token compact
+prompt can take that long in the first place. `llama-server` had no `--cache-reuse` flag set — every
+request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged
+from the previous turn. Added `--cache-reuse 256` (docker-compose.yml) so llama.cpp reuses cached KV for
+any matching ≥256-token chunk via KV-shift instead of reprocessing it, which is the actual fix for
+compact-prompt prefill time — the timeout bump above is a safety margin around it, not a substitute.
+
+## Sources
+
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) — `docs/reference/ENVIRONMENT.md`
+  ("Timeout Settings"), [Discussion #10602](https://github.com/diegosouzapw/OmniRoute/discussions/10602)
+- Live `GET /api/providers/<connectionId>`, `GET /api/settings`, `GET /api/resilience` against this
+  deployment's OmniRoute instance (2026-09-10)
+- [`docs/research/omniroute-account-semaphore-timeout.md`](./omniroute-account-semaphore-timeout.md) — the related-but-distinct 30s semaphore/429 investigation
@@ -0,0 +1,358 @@
+# OpenCode CLI: how auto-compact actually decides to trigger
+
+**Date:** 2026-09-03
+**Scope:** Resolves haylan/LLM-Server issue #27 (part of #26) — the config key(s), trigger
+threshold/formula, reasoning-token accounting, and per-model tunability of OpenCode's
+context-compaction behavior, and whether any of it is specific to hosted providers vs. the
+`@ai-sdk/openai-compatible` path this repo's `docs/research/opencode-cli-setup.md` documents for
+llama.cpp/litellm-style backends.
+
+**Freshness note:** `opencode-cli-setup.md` (dated 2026-08-24) does not cover compaction at all
+beyond one sentence ("§5: OpenCode's `limit.context`/`limit.output` fields ... don't change what the
+server actually accepts ... may miscalculate when to compact/summarize"). This document supersedes
+that gap. It is based on:
+1. Live docs fetches from https://opencode.ai/docs/ (2026-09-03), and
+2. A fresh `git clone` of https://github.com/anomalyco/opencode at commit
+   `b578b7261fc9ec4917fe272df5cc4bd8a056cd5d` (2026-09-03T09:47:21+08:00 — same day as this
+   research), `package.json` version `1.18.27`. Source-code claims below are cited by file path in
+   that clone and are the **highest-confidence source in this doc** — they're what actually ships,
+   not a doc description or a third-party claim.
+
+## Confidence scheme
+
+- **High** — read directly from the current source code in the repo, or a docs page quoted
+  verbatim.
+- **Medium** — inferred from source code behavior that isn't spelled out in a single line/comment
+  (i.e., I traced call sites to confirm it, rather than reading one authoritative line).
+- **Low** — plausible but not directly confirmed in the sources checked; flagged as an open
+  question.
+
+---
+
+## Verdict summary (answers to the four questions asked in #27)
+
+| Question | Answer | Confidence |
+|---|---|---|
+| Config key(s) controlling threshold/behavior | Single global top-level `compaction` object in `opencode.json`: `auto`, `prune`, `reserved`, `tail_turns`, `preserve_recent_tokens`. **No `threshold` percentage key exists.** | High |
+| Is it a hardcoded percentage of `limit.context`? | **No.** It's `usedTokens >= (contextLimit − reservedBuffer)`, where `reservedBuffer` defaults to `min(20_000, min(model.limit.output, 32_768) or default)`, i.e. compaction reserves room for one more max-size reply, not a flat 75%/95% cutoff. Some closed GitHub feature requests describe it as "hardcoded 75%" — that claim is **not what the current source does** (see §2). | High (source), contradicts a stale community claim (see §2) |
+| Does compaction count reasoning/`reasoning_content` tokens? | **Yes, in practice**, via the provider's `usage.total_tokens` (which for llama.cpp/litellm includes every generated token, reasoning or not) — but the token bookkeeping OpenCode itself derives (`tokens.output`, `tokens.reasoning`) explicitly **splits reasoning out of `output`**, and the compaction trigger's own fallback arithmetic (used only if the provider omits `total_tokens`) **omits `tokens.reasoning` entirely**. See §3 for the exact mechanism and the one edge case where reasoning tokens could be undercounted. | High (source), Medium (edge-case behavior when `total_tokens` is absent) |
+| Per-model or single global behavior? | **Global only.** The `compaction` block is a top-level config key, not nested under `provider.<id>.models.<id>` or any per-model schema. It cannot be disabled or tuned for one model while enabled for another. The *indirect* lever is each model's own `limit.context`/`limit.output` (which you already set per-model for custom providers), since those numbers feed the same global formula per-model. Multiple GitHub feature requests (#11314, #11930, #8140, #16375) ask for per-model/per-agent configurability; all are open or closed-not-planned as of this check. | High |
+
+---
+
+## 1. The config surface (verbatim from source + docs)
+
+Global (or project) `opencode.json`:
+
+```json
+{
+  "compaction": {
+    "auto": true,
+    "prune": false,
+    "reserved": 20000,
+    "tail_turns": null,
+    "preserve_recent_tokens": null
+  }
+}
+```
+
+Field descriptions, quoted verbatim from the config schema
+(`packages/core/src/v1/config/config.ts`, lines ~149–166 in the cloned repo):
+
+- `auto` — *"Enable automatic compaction when context is full (default: true)"*
+- `prune` — *"Enable pruning of old tool outputs (default: false)"*
+- `tail_turns` — *"Maximum number of recent user turns, including their following
+  assistant/tool responses, to keep verbatim during compaction. By default retention is limited
+  only by the preserved token budget."*
+- `preserve_recent_tokens` — *"Maximum number of tokens from recent turns to preserve verbatim
+  after compaction"*
+- `reserved` — *"Token buffer for compaction. Leaves enough window to avoid overflow during
+  compaction."*
+
+Sources:
+- https://opencode.ai/docs/config/ (fetched 2026-09-03) confirms `auto`/`prune`/`reserved` with the
+  same defaults and descriptions; the docs page does **not** mention `tail_turns` or
+  `preserve_recent_tokens` — those two are documented only in the source schema, not (yet) on the
+  public docs page. **Confidence: High** on all five keys existing and their defaults; the
+  docs-vs-source gap on the last two is itself notable (docs page is behind the schema).
+- `packages/core/src/v1/config/config.ts` (cloned repo, ~line 149) — schema + descriptions.
+
+There is no `threshold`, `percent`, or similarly-named key anywhere in the config schema. I grepped
+`packages/core/src/v1/config/` and `packages/core/src/config.ts` for `compaction` and found only the
+struct above — no percentage field exists in the current schema (High confidence; direct grep of
+current source).
+
+**Not nested under provider/model.** The `compaction` key sits at the top level of `opencode.json`,
+a sibling of `provider`, `model`, `agent`, etc. — not inside
+`provider.<id>.models.<model-id>` (the block this repo's `opencode-cli-setup.md` §3 documents for
+declaring the llama.cpp provider). Confirmed by reading the full top-level config struct in
+`packages/core/src/v1/config/config.ts` (~lines 95–170): `compaction` is a direct sibling of
+`provider`, not a child of it.
+
+There *is* a separate, easily-confused concept: `agent.compaction` (also in that same file, ~line
+105) — this only lets you assign a **different agent/model to perform the summarization step
+itself** (e.g. run the compaction LLM call on a cheaper model), not a per-model *threshold*
+override. It does not change when compaction triggers.
+
+## 2. The actual trigger formula (not a flat percentage)
+
+Live code, `packages/opencode/src/session/overflow.ts` (the shipped/default compaction-trigger
+path — see "which code path ships" note at the end of this section):
+
+```ts
+const COMPACTION_BUFFER = 20_000
+
+export function usable(input: { cfg: ConfigV1.Info; model: Provider.Model; outputTokenMax?: number }) {
+  const context = input.model.limit.context
+  if (context === 0) return 0
+
+  const reserved =
+    input.cfg.compaction?.reserved ??
+    Math.min(COMPACTION_BUFFER, ProviderTransform.maxOutputTokens(input.model, input.outputTokenMax))
+  return input.model.limit.input
+    ? Math.max(0, input.model.limit.input - reserved)
+    : Math.max(0, context - ProviderTransform.maxOutputTokens(input.model, input.outputTokenMax))
+}
+
+export function isOverflow(input: {
+  cfg: ConfigV1.Info
+  tokens: SessionV1.Assistant["tokens"]
+  model: Provider.Model
+  outputTokenMax?: number
+}) {
+  if (input.cfg.compaction?.auto === false) return false
+  if (input.model.limit.context === 0) return false
+
+  const count =
+    input.tokens.total || input.tokens.input + input.tokens.output + input.tokens.cache.read + input.tokens.cache.write
+  return count >= usable(input)
+}
+```
+
+In plain terms:
+- Compaction triggers when the running token count (`count`) reaches or exceeds
+  `usable = contextLimit − reservedBuffer` (or, if the model declares a separate
+  `limit.input`, `usable = limit.input − reservedBuffer` instead).
+- `reservedBuffer` defaults to `min(20_000, maxOutputTokens)`, where `maxOutputTokens = min(model.limit.output, 32_768) || 32_768`
+  (`OUTPUT_TOKEN_MAX` constant, `packages/opencode/src/provider/transform.ts` — `maxOutputTokens()`
+  at ~line 1468). It can be overridden with `compaction.reserved`.
+- If `model.limit.context` is `0` or unset, compaction is **silently disabled entirely** — this
+  matters for custom `@ai-sdk/openai-compatible` providers where `limit.context` is a value the
+  user must supply by hand (per `opencode-cli-setup.md` §3); omit it, and auto-compact never fires
+  for that model.
+- Compaction is disabled outright if `compaction.auto === false`.
+
+This is **not** a fixed 75%/90%/95%-of-context cutoff. It's `context − reservedOutputBuffer`, which
+in practice usually lands somewhere in the 80–95%+ range depending on the model's own
+`limit.output` relative to `limit.context` — but it's a token-count subtraction, not a percentage
+multiplication, and there is no config key to set a percentage.
+
+**On the "hardcoded 75%" claim**: a closed GitHub feature request
+(https://github.com/anomalyco/opencode/issues/11314, "Feature Request: Configurable Context
+Compaction Threshold") asserts *"Currently, OpenCode triggers context compaction at a hardcoded 75%
+threshold of a model's context window."* That is a **user's claim in a feature-request issue, not a
+maintainer statement or a source citation**, and it does not match the formula actually in the
+current source (which is a token-count subtraction tied to output-buffer size, not a flat
+percentage). Treat that issue as evidence that *users perceive/want configurability*, not as an
+accurate description of the mechanism. A second, similarly-shaped request
+(https://github.com/anomalyco/opencode/issues/11930) instead describes the current default as "100%
+threshold" — the two community reports disagree with each other, which is itself a signal neither
+is a reliable description of internals; the source code is what should be trusted here.
+**Confidence: High** on the source-code formula; **High** that the 75%/100% claims in those two
+issues are user speculation, not verified facts — both are closed as "not planned" with no
+maintainer confirmation of the underlying mechanism in the content fetched.
+
+**Which code path ships**: the repo contains two parallel implementations — the one quoted above
+(`packages/opencode/src/session/overflow.ts` + `packages/opencode/src/session/compaction.ts`,
+using `ConfigV1`/`SessionV1` types) and a newer, structurally different one
+(`packages/core/src/session/compaction.ts` + `packages/core/src/session/runner/llm.ts`, using
+`Config.Entry`/`SessionMessage` types, with its own `compactIfNeeded`/`compactAfterOverflow`
+functions and a `buffer` config field instead of `reserved`). Tracing the call path: the newer
+runner is only reached when `flags.experimentalNativeLlm` is true, gated by the
+`OPENCODE_EXPERIMENTAL_NATIVE_LLM` env var (`packages/opencode/src/effect/runtime-flags.ts`,
+`packages/opencode/src/session/llm.ts` line ~226) — off by default. **The `overflow.ts`/V1 path
+quoted above is what ships by default in v1.18.27.** If this repo's OpenCode setup ever sets
+`OPENCODE_EXPERIMENTAL_NATIVE_LLM=1` (or a future release flips the default), the newer engine's
+`compactIfNeeded()` uses a structurally similar but not identical check:
+`estimatedPromptTokens <= context − max(output, config.buffer)` — same shape (context minus a
+reserved buffer, no percentage), different field name (`buffer` vs `reserved`). **Confidence:
+High** on which path ships by default; **Medium** on exact behavior differences if the experimental
+flag is ever turned on, since I did not exhaustively diff every line of the newer engine.
+
+## 3. Reasoning/thinking-token accounting — the Qwen3.8 angle
+
+This repo's `litellm-config.yaml` flags that Qwen3.8-27B (`qwen3.8-27b-local`) spends generation
+tokens on `reasoning_content` before `content`:
+
+```yaml
+# Qwen3 is a reasoning model — it spends output tokens on
+# reasoning_content before ever writing content. ...
+max_tokens: 16384
+```
+(`/home/haylan/Projects/LLM-Server/litellm-config.yaml`, `qwen3.8-27b-local` model block.)
+
+OpenCode's own token bookkeeping (`packages/opencode/src/session/session.ts`, ~lines 340–375, the
+function that turns a provider's raw `usage` object into the `tokens` struct stored on each
+assistant message) does this:
+
+```ts
+const inputTokens = safe(input.usage.inputTokens ?? 0)
+const outputTokens = safe(input.usage.outputTokens ?? 0)
+const reasoningTokens = safe(input.usage.reasoningTokens ?? 0)
+...
+const total = input.usage.totalTokens
+
+const tokens = {
+  total,
+  input: adjustedInputTokens,
+  output: safe(outputTokens - reasoningTokens),   // reasoning is subtracted OUT of "output"
+  reasoning: reasoningTokens,                      // tracked as its own field
+  cache: { write: cacheWriteInputTokens, read: cacheReadInputTokens },
+}
+```
+
+And the compaction trigger (`overflow.ts`, quoted in §2) computes:
+
+```ts
+const count = input.tokens.total || input.tokens.input + input.tokens.output + input.tokens.cache.read + input.tokens.cache.write
+```
+
+Two things follow:
+
+1. **`tokens.reasoning` is never added back in** by the fallback sum
+   (`input + output + cache.read + cache.write`) — that expression has no `+ reasoning` term. If
+   the compaction trigger ever fell back to this sum (i.e., the provider didn't return
+   `totalTokens`), reasoning tokens spent on `reasoning_content` would be **excluded** from the
+   overflow calculation, undercounting real context usage.
+2. **In the normal case, `total` is used instead of the fallback sum**, and `total = usage.totalTokens`
+   straight from the provider's raw response — computed by the provider/AI-SDK *before* OpenCode
+   splits `outputTokens` into `output`/`reasoning`. Since the AI SDK's OpenAI-compatible adapter (and
+   litellm/llama.cpp underneath it) counts every generated token — reasoning and content alike — as
+   part of `completion_tokens`/`total_tokens`, `total` **does include reasoning tokens** in this
+   normal path. So in practice, for a llama.cpp/litellm backend that reports `usage.total_tokens` on
+   every response (the OpenAI chat-completions spec requires this field), **reasoning tokens are
+   accounted for in the compaction trigger via `total`, not via the explicit `reasoning` field.**
+
+The edge case that would matter for this repo: if litellm or llama.cpp's OpenAI-compatible endpoint
+ever omitted `usage.total_tokens` from a response (malformed/incomplete usage block — this has
+happened with some llama.cpp server versions/flags), OpenCode's fallback sum would silently
+undercount by the full `reasoning` amount, delaying compaction past the point it should have
+triggered and increasing risk of a hard `context_length_exceeded` — exactly the failure mode
+reported by an unrelated user in
+https://github.com/anomalyco/opencode/issues/8089 ("Auto-compaction enabled by default, but
+context_length_exceeded errors still occur in agent workflows"), though that issue's cause was not
+confirmed to be this specific gap (it involved OpenAI's GPT-5.2 and multi-agent/subagent workflows,
+not a local llama.cpp backend, and the issue thread contains no maintainer diagnosis of root cause
+in the content fetched).
+
+**Confidence: High** on the source-code mechanics described (the `output = outputTokens −
+reasoningTokens` split, the `total || sum` fallback, and the missing `+ reasoning` term in the
+fallback). **Medium** on whether this repo's specific llama.cpp/litellm stack reliably returns
+`usage.total_tokens` on every response for the `qwen3.8-27b-local` model — this was not verified
+against a live request/response in this research pass (would need an empirical check: hit
+`http://localhost:8080/v1/chat/completions` directly or via the litellm proxy and inspect the
+`usage` block of an actual reasoning response). Recommend that empirical check as a fast follow if
+this matters operationally.
+
+## 4. Per-model tunability
+
+Confirmed absent, both from the schema (§1) and from community feature requests asking for exactly
+this and not getting it:
+
+- https://github.com/anomalyco/opencode/issues/11314 — "Feature Request: Configurable Context
+  Compaction Threshold" — requests a `compaction.threshold` with "optional per-model overrides."
+  Closed as not planned (per WebFetch of the issue).
+- https://github.com/anomalyco/opencode/issues/11930 — "Feature: Configurable compaction threshold
+  and model (global + per-model)" — explicitly requests global **and** per-model threshold config.
+  Closed as not planned, no maintainer reply visible in the content fetched.
+- https://github.com/anomalyco/opencode/issues/8140 — "Feature Request: Configurable context limit
+  and auto-compaction threshold" — same theme (title only confirmed via search; not individually
+  fetched in this pass).
+- https://github.com/anomalyco/opencode/issues/16375 — "[FEATURE]: Per-agent compaction config
+  (disable compaction for specific agents)" — same theme, per-agent instead of per-model (title only
+  confirmed via search; not individually fetched in this pass).
+
+All four are open/closed-not-planned as of 2026-09-03 — i.e., **as of this check, none of this has
+shipped**: compaction remains a single global on/off + buffer-size knob, with no per-model or
+per-agent threshold override. **Confidence: High** that the feature doesn't exist in the schema
+(direct source read); **Medium** on the exact current status of #8140/#16375 specifically since only
+their titles were confirmed via search results, not their full issue bodies.
+
+The one *indirect* per-model lever that does exist: since `usable()` (§2) reads
+`input.model.limit.context` / `input.model.limit.input` / `input.model.limit.output` — all
+per-model fields already documented in `opencode-cli-setup.md` §3/§5 for custom providers — setting
+those numbers differently per model in the `provider.<id>.models.<model-id>.limit` block changes
+where that model's compaction fires, without needing a dedicated per-model compaction key. This
+is bookkeeping-hint tuning, not a first-class "compaction threshold" feature.
+
+## 5. Is any of this specific to hosted/built-in providers vs. `@ai-sdk/openai-compatible`?
+
+**No.** The entire trigger path (`overflow.ts`, `compaction.ts`) operates only on `Provider.Model`
+(a normalized model descriptor with `limit.context`/`limit.input`/`limit.output`) and the message
+`tokens` struct built from the SDK's generic `usage` object (`session.ts`, §3) — nothing in the
+compaction code branches on `model.api.npm` or provider identity. The mechanism is provider-agnostic
+by construction: any provider adapter that populates `usage` (inputTokens/outputTokens/totalTokens)
+and any model entry that has a nonzero `limit.context` gets the same compaction behavior, including
+a hand-declared `@ai-sdk/openai-compatible` provider block like this repo's `llamacpp` provider in
+`opencode-cli-setup.md` §3. **Confidence: High** — read directly from the trigger/accounting source,
+which takes no provider-specific branch.
+
+The one place this repo needs to be careful about, restated from §2: for a custom
+`@ai-sdk/openai-compatible` provider, `limit.context` (and ideally `limit.output`) must be set by
+hand in `opencode.json` to match the real `--ctx-size` the llama.cpp container is launched with — if
+left unset (`limit.context` defaults to `0` for an unrecognized custom model), compaction is
+silently disabled for that model rather than silently misfiring.
+
+## Sources consulted (primary)
+
+- https://opencode.ai/docs/ — nav/sitemap fetch (2026-09-03); confirms no dedicated "context
+  management"/"compaction" page exists in the current docs nav — it lives only in the Config
+  reference.
+- https://opencode.ai/docs/config/ — fetched 2026-09-03; source of the `compaction.auto/prune/reserved`
+  descriptions and defaults quoted in §1.
+- https://opencode.ai/docs/models/ — fetched 2026-09-03; confirms no compaction/context-limit
+  content on that page (only `reasoningEffort`/`thinking` keys, unrelated to compaction).
+- `github.com/anomalyco/opencode` @ `b578b7261fc9ec4917fe272df5cc4bd8a056cd5d` (cloned 2026-09-03,
+  `package.json` version `1.18.27`) — primary source for all source-code claims:
+  - `packages/opencode/src/session/overflow.ts` — trigger formula (`usable`/`isOverflow`), §2
+  - `packages/opencode/src/session/compaction.ts` — shipped compaction service, tail-turn selection,
+    pruning, §2/§4
+  - `packages/opencode/src/session/session.ts` (~lines 340–375) — `usage` → `tokens` mapping,
+    reasoning-token split, §3
+  - `packages/core/src/v1/config/config.ts` (~lines 95–170) — config schema, field descriptions, §1
+  - `packages/core/src/config/compaction.ts`, `packages/core/src/session/compaction.ts`,
+    `packages/core/src/session/runner/llm.ts` — the experimental/newer compaction engine, gated
+    behind `OPENCODE_EXPERIMENTAL_NATIVE_LLM`, §2
+  - `packages/opencode/src/effect/runtime-flags.ts`, `packages/opencode/src/session/llm.ts` (~line
+    226) — confirms which engine ships by default, §2
+  - `packages/opencode/src/provider/transform.ts` (~line 1468, `maxOutputTokens`) — output-buffer
+    sizing used in the reserved-token default, §2
+- https://github.com/anomalyco/opencode/issues/11314 — "Configurable Context Compaction Threshold"
+  (closed, not planned) — source of the "hardcoded 75%" community claim, §2/§4
+- https://github.com/anomalyco/opencode/issues/11930 — "Configurable compaction threshold and model
+  (global + per-model)" (closed, not planned) — source of the conflicting "100% threshold" claim,
+  §2/§4
+- https://github.com/anomalyco/opencode/issues/8089 — "Auto-compaction enabled by default, but
+  context_length_exceeded errors still occur in agent workflows" (closed, not planned) — cited in §3
+  as a related-but-unconfirmed failure report
+- https://github.com/anomalyco/opencode/issues/8140, #16375 — titles only, confirmed via
+  `WebSearch`, not individually fetched; §4
+- This repo: `docs/research/opencode-cli-setup.md` — structural template, and source of the
+  `limit.context`/`limit.output` per-model config shape referenced throughout
+- This repo: `litellm-config.yaml` — `qwen3.8-27b-local` model block, `max_tokens` comment on
+  `reasoning_content`, §3
+
+## Confidence summary
+
+| Claim | Confidence |
+|---|---|
+| `compaction` is a single global top-level config key (`auto`/`prune`/`reserved`/`tail_turns`/`preserve_recent_tokens`) | High |
+| No percentage-threshold config key exists | High |
+| Trigger formula is `usedTokens >= context − reservedBuffer`, not a flat percentage | High |
+| "Hardcoded 75%" (issue #11314) and "100% threshold" (issue #11930) are unverified community claims, not confirmed mechanism | High (that they're unverified/conflicting); the actual mechanism per source is definitive |
+| `overflow.ts`/V1 path ships by default; newer `core` engine is gated behind `OPENCODE_EXPERIMENTAL_NATIVE_LLM` | High (default path); Medium (exact newer-engine behavior if enabled) |
+| Reasoning tokens counted via `usage.total_tokens` in the normal (non-fallback) path | High |
+| Reasoning tokens excluded from the fallback sum if `total_tokens` is ever absent | High (source); Medium (whether this repo's llama.cpp/litellm stack ever hits that fallback in practice) |
+| No per-model/per-agent compaction threshold override exists; confirmed by rejected feature requests | High |
+| Compaction mechanism is provider-agnostic — applies identically to `@ai-sdk/openai-compatible` custom providers | High |
+| `limit.context` unset/0 on a custom model silently disables compaction for it | High |
@@ -0,0 +1,64 @@
+# Ponytail audit — repo-wide over-engineering scan, 2026-09-09
+
+Whole-tree audit (ponytail-audit skill), not a diff review. The only real
+code in this repo is `scripts/update.sh` (253L) and `scripts/switch-model.sh`
+(50L), plus `docker-compose.yml` and `.env.example`; the rest is docs.
+`switch-model.sh` and the compose comments (ROCm GID workarounds,
+`GPU_MAX_HW_QUEUES` rationale, lazytainer label placement) are load-bearing
+and lean — left alone. Scope: over-engineering and complexity only;
+correctness/security/performance out of scope. Findings ranked biggest cut
+first. One-shot report — nothing was applied.
+
+## Findings
+
+1. **`delete:` the `downloader-fast` line — it references a service removed
+   in `5d6a17f` ("feat: remove llama-server-fast") and no longer exists in
+   `docker-compose.yml`.** Under `set -euo pipefail`,
+   `docker compose run downloader-fast` errors on the unknown service and
+   **aborts every `update.sh` run** right after config sync, before omniroute
+   comes up. Dead code that also breaks the mandatory deploy flow.
+   *Remove the line.* [scripts/update.sh:235]
+
+2. **`delete:` the entire gum path — `ensure_gum()` (~27L, L59–85), the
+   `GUM_VERSION`/`GUM_DIR`/`GUM_BIN` vars (L56–58), the vendored
+   `scripts/vendor/gum_0.14.5_Linux_x86_64.tar.gz` (4.4MB checked into git),
+   the download fallback, and the `if [ -n "$gum_bin" ]` branch (L123,
+   L127–130).** The plain-bash fallback (L131–148) already makes the
+   *identical* decision (which keys take the new value) whenever gum is
+   absent; the gum TUI is a speculative nicer prompt on top of a working
+   path. ~4.4MB in git + arch detection + a `.cache/gum` layer, all to
+   prettify a rare interactive conflict. *Replacement: nothing — always use
+   the plain-bash one-screen prompt.* [scripts/update.sh, scripts/vendor/]
+
+3. **`delete:` stale `llama-server-fast` / `fastModel` references — the fast
+   model was removed but the docs still describe a two-model Qwen Code
+   setup.** `.env.example:98` comment still lists it;
+   `docs/coding-cli-setup/index.md:32` says "2 models: chat + `fastModel`";
+   and `docs/coding-cli-setup/qwen-code.md` carries a whole fast-model
+   section (11 refs: the `fastModel` config block, `LLAMA_FAST_CTX_SIZE`
+   notes, Qwen3-4B). *Rewrite to single-model.* [docs/coding-cli-setup/
+   qwen-code.md, index.md, .env.example:98]
+
+4. **`shrink:` the 3× repeated `test -f … || curl …` blocks in
+   `downloader-comfyui` (YAML L85–99, ~15 lines) → a `for` loop over the 3
+   model files (~5 lines).** *Low confidence:* the env var names are
+   non-uniform (`COMFYUI_DIFFUSION_MODEL_FILE` / `TEXT_ENCODER_FILE` /
+   `VAE_FILE`), so the loop needs a small `case` — marginal win, and it
+   matches the house "one-off downloader" style. [docker-compose.yml]
+
+5. **`yagni:` (verify-first) qdrant + neo4j run with no consumer in the
+   stack yet** — added ahead of the RAG app via the `feat-rag-databases`
+   merge; nothing writes to them. Two always-on DBs for a feature that
+   isn't wired. *Confirm the RAG consumer is still on the roadmap before
+   keeping both; the compose comment already concedes neo4j "can absorb
+   qdrant's job later."* Low confidence — deliberate tracked decision, and
+   cheap to leave running. [docker-compose.yml]
+
+## Net
+
+`net: -45 lines script/compose (+~15 stale doc lines), -1 dep (gum, 4.4MB
+vendored binary) possible.`
+
+No out-of-scope (correctness/security/performance) findings. #1 is the one
+to fix first — it's not just bloat, it's the deploy script halting on every
+run.
@@ -93,10 +93,12 @@ conflict_old=()
 conflict_new=()
 while IFS='=' read -r key value; do
  [ -n "$value" ] || continue
-  current="$(grep -E "^${key}=" .env | head -1 | cut -d= -f2-)"
  if ! grep -qE "^${key}=" .env; then
    echo "${key}=${value}" >> .env
-  elif [ "$current" != "$value" ]; then
+    continue
+  fi
+  current="$(grep -E "^${key}=" .env | head -1 | cut -d= -f2-)"
+  if [ "$current" != "$value" ]; then
    conflict_keys+=("$key")
    conflict_old+=("$current")
    conflict_new+=("$value")
@@ -231,6 +233,7 @@ docker compose build --pull
 echo "==> ensuring models are downloaded (skips already-present files)"
 docker compose --profile tools run --rm downloader
 docker compose --profile tools run --rm downloader-fast
+docker compose --profile tools run --rm downloader-classifier
 docker compose --profile tools run --rm downloader-comfyui

 echo "==> bringing up omniroute"
Author	SHA1	Message	Date
haylan	edad6b2249	Merge branch 'research/opencode-auto-compact'	2026-09-14 07:44:32 +02:00
haylan	2250e804db	Implement code changes to enhance functionality and improve performance	2026-09-11 13:15:30 +02:00
haylanandClaude-Bot	e31647812a	fix(omniroute): raise REQUEST_TIMEOUT_MS and enable --cache-reuse to stop non-ping SSE stream aborts STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended llama-server prefill room to produce a first token, but qwen-code sessions kept hitting "Stream produced no non-ping SSE event within 95000ms" the very next morning. Per OmniRoute's own docs, that's the wrong timer: the first non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min, computed as remaining budget after retries/cooldowns), not STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming has already started). Two changes: - Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety margin, not the root-cause fix. - Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured, so every request reprefilled its full prompt from scratch even when most of a conversation's prefix was unchanged. This is the actual fix for why compact-prompt prefill was slow enough to hit the timeout in the first place. Documents the distinction and root cause in docs/research/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc	2026-09-10 10:12:02 +02:00
haylanandClaude-Bot	930e407053	docs(llm): document qwen-classifier reality, add --reasoning off safety net - docker-compose.yml: add --reasoning off to qwen-classifier per fast-model-choice.md's own recommendation (ggml-org/llama.cpp#20809 safety net) — missed in the original rollout, caught while writing this up. - docs/coding-cli-setup/qwen-code.md: rewritten to match what's actually deployed (qwen-classifier, partial GPU offload, 65536 ctx, Q4_K_XL) — previously described an unimplemented llama-server-fast/8192-ctx plan. Documents the non-interactive MCP tool allow-list gap found live-testing. - docs/coding-cli-setup/opencode.md: fix stale 65536 example that didn't match its own documented LLAMA_CTX_SIZE/LLAMA_PARALLEL formula (131072). - docs/research/fast-model-choice.md: implementation note recording where the actual rollout diverged from this doc's original recommendations (service name, quant, context size, CPU-first-then-GPU path). - docs/research/omniroute-account-semaphore-timeout.md: new — the hardcoded 30s per-connection semaphore timeout found during the pr-agent investigation, root-caused against OmniRoute's own source, and the maxConcurrent:null + providerSpecificData.timeoutMs fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 19:10:32 +02:00
haylanandClaude-Bot	76043e2c6f	tune(llm): partial GPU offload for qwen-classifier, real headroom Full offload (999 layers) left only ~768MB free VRAM regardless of batch-size/flash-attn tuning — that gap tracks roughly fixed regardless of those knobs, most likely ROCm's own per-process HIP context overhead (same ROCm#5706 quirk already noted for two HIP contexts sharing this card, see llama-server's GPU_MAX_HW_QUEUES comment above). Dropping to 28/36 layers on GPU (6 layers + their KV on CPU) trades a slice of speed for real freed VRAM — still >80% of layers on GPU, nowhere near CPU-only's unusable latency. Verifying live before locking this number in. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 18:10:39 +02:00
haylanandClaude-Bot	1102273384	fix(llm): enable flash-attn on qwen-classifier, real VRAM cause found Batch/ubatch reduction barely moved measured VRAM (~768MB free, same as before) — wrong lever. llama-server runs with --flash-attn on; this service didn't. Without it, the unfused attention compute buffer at 65536 ctx is far larger than flash-attn's fused workspace, which is what the naive weights+KV estimate missed. Matches llama-server's flag. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 17:04:05 +02:00
haylanandClaude-Bot	ba9ace6f71	fix(llm): shrink qwen-classifier's compute buffer for real VRAM headroom Measured live: full-offload weights+KV (~4.9GiB estimate) actually used ~5.85GiB, leaving only ~700MB free on the R9700 — too tight, real OOM risk for either GPU process. The gap was compute-buffer/graph overhead the naive estimate didn't account for. Drop --batch-size/--ubatch-size well below llama-server's defaults (2048/512) to shrink it — a single-request classifier has no batching throughput to lose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 17:02:44 +02:00
haylanandClaude-Bot	8e2650807c	fix(llm): move qwen-classifier to GPU, right-size context CPU-only was too slow in practice: real classification calls blew past OmniRoute's 60s timeout and retry-looped (504/499). Moved to GPU. Also traced qwen-code's actual classifier transcript cap in its source (MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000/message) — worst case is ~40-50K tokens, not the 131072 originally set in settings.json (copied from the main model's entry, not a real qwen-code requirement). Dropped ctx-size to 65536 (~1.5x margin) so Q4_K_XL weights + q4_0/q4_0 KV fit fully on GPU (~4.9GiB) inside the ~6.1GiB free on the R9700, instead of needing partial CPU/GPU offload. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 16:59:02 +02:00
haylanandClaude-Bot	4353e5c0e8	merge: follow-up fix for qwen-classifier model file Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 16:38:11 +02:00
haylanandClaude-Bot	20cc0bcc70	fix(llm): point qwen-classifier at the Q8 GGUF already on disk The Q4_K_M-class file this originally specced didn't exist yet on gameserver (classifier crash-looped: "No such file or directory"). A Q8_K_XL GGUF for the same model was already sitting in the models volume from something earlier — point at that instead of downloading a new file, and drop the KV cache quant to q4_0/q4_0 to keep total RAM comfortable now that the weights are the larger Q8 variant. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 16:37:54 +02:00
haylan	c1e30ec9bb	Merge pull request 'Fix/omniroute pr agent timeout' (#54 ) from fix/omniroute-pr-agent-timeout into main Reviewed-on: #54	2026-09-09 14:29:13 +00:00
haylan	b3a64fe4b5	chroe(chore): added markdown for harnesses	2026-09-09 16:28:42 +02:00
haylanandClaude-Bot	828bd4c046	feat(llm): dedicate a CPU-only backend for the qwen-code tool-call classifier fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was aliased onto llama-server's own 27B connection, so every tool-call safety check queued behind whatever heavy generation was already running on that model's 2 GPU slots. Add qwen-classifier: a separate llama.cpp instance, CPU-only, running Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context, qwen-code's requirement, without lossy RoPE scaling). Structurally isolated from llama-server's queue instead of sharing it. Sized for gameserver's ~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M- class weights (~2.3GiB) fits comfortably, with better KV quality than the q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-09 16:26:51 +02:00
haylan	16df051318	docs(research): add ponytail audit report highlighting over-engineering and complexity issues	2026-09-09 08:22:31 +02:00
haylan	128503b68a	docs: add AGENTS.md and QWEN.md with agent instructions and project overview	2026-09-09 07:32:09 +02:00
haylan	df900404c0	chore(config): fix ignored path .qwen/temp to .qwen/tmp	2026-09-08 12:46:32 +02:00
haylan	f4729ba704	docs(research): evaluate Colibrì for this stack Assess whether the Colibrì disk-streaming MoE inference engine fits the AMD ROCm single-GPU setup or fills a gap beyond the existing llama.cpp/OmniRoute/Qdrant/Neo4j/ComfyUI stack. Conclusion: worth a passing watch, not worth integrating today.	2026-09-08 12:46:16 +02:00
haylanandClaude-Bot	8f3feb4881	Merge branch 'feat-rag-databases' Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-08 12:30:27 +02:00
haylanandClaude-Bot	6f4e736da8	docs(coding-cli-setup): split per-CLI docs into their own files Move docs/coding-cli-setup.md to docs/coding-cli-setup/ with one file per CLI (claude-code, kimi-cli, opencode, qwen-code) plus a shared index.md for the gateway intro, tool-calling risk note, and summary table. Also fixes the qwen-code doc: context sizes are per-slot (LLAMA_CTX_SIZE / LLAMA_PARALLEL), not raw LLAMA_CTX_SIZE (same fix applied to OpenCode's limit.context); documents the fastModel classifier provider and its own context math; adds the omniroute-search MCP server (SearXNG web search) and Auto Mode permissions tuning that were missing from the original qwen-code section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-09-08 12:30:08 +02:00
haylan	4c8a9c039e	Merge pull request 'fix(update.sh): stop config sync loop from dying silently on a missing key' (#53 ) from feat-rag-databases into main Reviewed-on: #53	2026-09-07 18:10:23 +00:00
haylan	2ee308c1d9	Merge branch 'main' into feat-rag-databases	2026-09-07 18:10:16 +00:00
haylanandClaude-Bot	1fcf30e9a1	fix(update.sh): stop config sync loop from dying silently on a missing key current=$(grep ...) ran before checking whether the key existed in .env at all. Under set -euo pipefail, a key missing from .env (e.g. a new tunable this branch just added to .env.example) made that grep exit 1, pipefail propagated it, and set -e killed the script instantly with no output - looked exactly like a gum hang but never reached ensure_gum. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc	2026-09-07 20:09:31 +02:00
haylan	31e9aab1f3	Merge pull request 'Add Qdrant/Neo4j RAG storage + update.sh gum fixes' (#52 ) from feat-rag-databases into main Reviewed-on: #52	2026-09-07 18:07:48 +00:00
haylanandClaude-Bot	1eaa2a0d2e	docs(research): document opencode auto-compact trigger and config surface Resolves #27 (part of #26). Findings from opencode.ai/docs (fetched 2026-09-03) plus a fresh clone of anomalyco/opencode @ b578b72 (v1.18.27): - No percentage-threshold config key exists; compaction.{auto,prune, reserved,tail_turns,preserve_recent_tokens} is the full global config surface (opencode.json top-level, not per-provider/model). - Trigger is usedTokens >= context - reservedBuffer, not a hardcoded 75%/95% cutoff — contradicts an unverified claim in a closed GitHub feature request (#11314). - Reasoning tokens (Qwen3's reasoning_content) are counted via the provider's usage.total_tokens in the normal path, but excluded from the fallback sum if a provider ever omits total_tokens. - No per-model/per-agent threshold override exists (confirmed by several closed-not-planned feature requests); the only per-model lever is each model's own limit.context/limit.output. - Mechanism is provider-agnostic: applies identically to a hand- declared @ai-sdk/openai-compatible provider (this repo's llamacpp setup) as to hosted providers, provided limit.context is set. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m	2026-09-03 06:25:57 +02:00