docs(coding-cli-setup): split per-CLI docs into their own files

Move docs/coding-cli-setup.md to docs/coding-cli-setup/ with one file
per CLI (claude-code, kimi-cli, opencode, qwen-code) plus a shared
index.md for the gateway intro, tool-calling risk note, and summary
table.

Also fixes the qwen-code doc: context sizes are per-slot
(LLAMA_CTX_SIZE / LLAMA_PARALLEL), not raw LLAMA_CTX_SIZE (same fix
applied to OpenCode's limit.context); documents the fastModel
classifier provider and its own context math; adds the omniroute-search
MCP server (SearXNG web search) and Auto Mode permissions tuning that
were missing from the original qwen-code section.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.env.example

@@ -1,21 +1,148 @@
-# Copy to .env and adjust. All values below are defaults baked into
-# docker-compose.yml — only uncomment/change what you actually want to override.
+# Copy to .env and adjust — or just run ./scripts/update.sh, which creates
+# .env from this file and fills in every secret/key below it can generate
+# itself (see each var's comment). All values below are defaults baked into
+# docker-compose.yml — only uncomment/change what you actually want to
+# override.
 
 # --- llama.cpp / model ---
 LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
+# 999 = every layer on GPU (this model is dense, not MoE, and already fits
+# fully in 32GB VRAM — see docs/research/qwen3.8-27b-quant.md). Lower this
+# to leave that many fewer layers on GPU and push the rest to CPU/system RAM
+# if something else is contending for VRAM — llama.cpp has no separate
+# "RAM offload" flag, --n-gpu-layers *is* the RAM-offload knob for a dense
+# model. Don't reach for --n-cpu-moe/--cpu-moe/--override-tensor "exps" —
+# those target Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B), not this
+# one, and are no-ops here.
+# There's no separate "then SSD" tier to enable either: llama.cpp mmaps the
+# model file by default (no --no-mmap here), so if GPU+RAM ever can't hold
+# the working set, the OS pages the rest in from disk automatically — an
+# implicit, slow last resort, not a config knob. An explicit tiered SSD
+# offload has been an open llama.cpp feature request since 2025 (still
+# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
 LLAMA_GPU_LAYERS=999
-# 65536 (64K) fits comfortably in 32GB VRAM alongside the model weights.
-# Raise toward 131072 if you need more context; see docs/research/qwen3.8-27b-quant.md
-# for the VRAM math at larger context sizes.
-LLAMA_CTX_SIZE=65536
-LLAMA_PORT=8080
-
-# --- Open WebUI ---
-WEBUI_PORT=3000
-# Dummy key — llama.cpp's OpenAI-compatible endpoint doesn't check it, but
-# Open WebUI requires the field to be non-empty.
-OPENAI_API_KEY=local
+# 262144 = this model's true max (max_position_embeddings in Qwen/Qwen3.8-27B's
+# config.json) — the largest --ctx-size llama.cpp will even accept for it.
+# fp16 KV cache at full context would be ~16GB, on top of 17.6GB weights =
+# ~33.6GB, which does NOT fit the 32GB R9700 on its own. docker-compose.yml
+# now runs --cache-type-k/v q8_0, which roughly halves KV memory (~8GB at
+# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
+# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
+LLAMA_CTX_SIZE=262144
+# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
+# default) with no flag set — under concurrent subagent fan-out, 4 requests
+# split the same GPU compute, so a large-context prefill can queue behind
+# others long enough to blow past OmniRoute's stream-idle timeout, which then
+# cancels the request (see issue-tracker notes on the timeout/cancel loop).
+# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
+# raise back toward 4 if throughput (not latency) becomes the bottleneck
+# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
+# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
+# checking that per-slot number stays comfortably above observed usage.
+LLAMA_PARALLEL=2
 
 # --- Lazytainer ---
 # Seconds of inactivity before llama-server is stopped. 900 = 15 min.
 LAZYTAINER_INACTIVE_TIMEOUT=900
+
+# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
+# Resolved automatically by ./scripts/update.sh from search.home on this
+# host — leave blank. Only set by hand if that resolution fails (e.g.
+# search.home isn't a static DHCP reservation and its IP drifted).
+SEARXNG_LAN_IP=
+
+# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
+# OMNIROUTE_PORT is the host-published port (reverse-proxied by NPM) — kept
+# at 4000, same as the old LiteLLM setup, so existing NPM/firewall config
+# doesn't need to change. It's mapped via plain Docker port publishing onto
+# API_PORT, omniroute's own container-internal port (left at its default,
+# not reconfigured to match). The dashboard (DASHBOARD_PORT) is never
+# published at all — see docker-compose.yml's omniroute service comment.
+OMNIROUTE_API_PORT=20129
+OMNIROUTE_DASHBOARD_PORT=20128
+# SSE inactivity timeout before OmniRoute gives up on a streaming request and
+# cancels it (which cancels the matching llama-server task too). 180s gives
+# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
+OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
+# Random values, filled in automatically by ./scripts/update.sh — leave
+# blank. Bootstrap dashboard admin password (log in at the dashboard port,
+# change it there afterwards — this is only the first-boot value):
+OMNIROUTE_INITIAL_PASSWORD=
+# Signs dashboard session cookies:
+OMNIROUTE_JWT_SECRET=
+# Encrypts API key values at rest in omniroute's SQLite DB:
+OMNIROUTE_API_KEY_SECRET=
+# Encrypts the whole SQLite DB at rest. Do not change after first run —
+# existing encrypted data becomes unreadable if you do (same caveat as
+# LiteLLM's old LITELLM_SALT_KEY):
+OMNIROUTE_STORAGE_ENCRYPTION_KEY=
+# Per-deployment salts — random is fine, just needs to be stable:
+OMNIROUTE_MACHINE_ID_SALT=
+OMNIROUTE_CLI_SALT=
+# Required (production) — shared secret for the internal Codex Responses
+# WebSocket bridge. Random value, filled in automatically:
+OMNIROUTE_WS_BRIDGE_SECRET=
+# Per-workload virtual keys (one per client that calls the gateway) have no
+# scripted /key/generate equivalent yet — omniroute's key-creation endpoint
+# needs a dashboard login session, not a static bearer key (see issue #37).
+# Mint them by hand in the dashboard, add a KEY=value line here per workload
+# as you onboard one. See docs/proxy-key-onboarding.md.
+
+# --- ComfyUI (local image generation, see issue #38 wayfinder map) ---
+# yurisasc/comfyui-rocm7.1 manages GPU-group access via these GID/UID env
+# vars rather than relying solely on docker-compose.yml's group_add.
+# Resolved automatically from the host by ./scripts/update.sh — leave blank.
+COMFYUI_PUID=
+COMFYUI_PGID=
+
+# Shared by every GPU-touching service (llama-server, llama-server-fast,
+# comfyui) for group_add: — resolved to real host GIDs by ./scripts/update.sh
+# rather than left as plain group names in docker-compose.yml, because Docker
+# resolves a *named* group_add entry against the container's own /etc/group,
+# not the host's, and fails unpredictably when the image doesn't define one
+# (worse with multiple GPU services racing on the same lookup at once — see
+# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
+HOST_VIDEO_GID=
+HOST_RENDER_GID=
+
+# --- llama.cpp / fast model (second, always-resident instance — see
+# docs/research/fast-model-choice.md and issue #44) ---
+# Qwen3-4B-Instruct-2507: architecturally non-thinking (never emits
+# <think> blocks, unlike Qwen3-1.7B/0.6B which need a per-call toggle) —
+# picked specifically so it stays fast enough for qwen-code's Auto Mode
+# classifier (Stage 1 wants ~300ms). Same publisher (unsloth) as the main
+# model for consistency.
+LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
+# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
+# is dense too.
+LLAMA_FAST_GPU_LAYERS=999
+# --ctx-size is the TOTAL across every LLAMA_FAST_PARALLEL slot, not per
+# request — same halving already called out for the main model above.
+# Was PARALLEL=2, silently halving this to 4096/slot — too small: a real
+# classifier call (hints + environment + recent tool-call history) hit
+# "exceeds the available context size (4096 tokens)" in practice, which
+# qwen-code surfaces as "Auto Mode couldn't classify this action
+# (Classifier stage 1 unavailable)" — see issue #5. Fixed by dropping to
+# a single slot instead of raising ctx-size (no extra VRAM, and this
+# service doesn't need concurrent classifier calls the way the main
+# model needs concurrent chat sessions) — the full 8192 now goes to the
+# one slot. If hints.allow/softDeny/hardDeny ever approach their
+# 50-entries-each ceiling, raise LLAMA_FAST_CTX_SIZE instead — qwen-code
+# caps those at 200 chars x 150 entries plus 40,000 chars of
+# historical-action context, which can exceed 8192 tokens worst-case.
+LLAMA_FAST_CTX_SIZE=8192
+LLAMA_FAST_PARALLEL=1
+
+# --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
+# image-generation-model-choice.md and issue #42) ---
+# Three files: diffusion weights, text encoder, VAE — all from the official
+# Comfy-Org FP8 split, chosen specifically because it's the only candidate
+# with a ComfyUI workflow pre-validated on this exact GPU (gfx1201/R9700).
+COMFYUI_DIFFUSION_MODEL_FILE=qwen_image_fp8_e4m3fn.safetensors
+COMFYUI_TEXT_ENCODER_FILE=qwen_2.5_vl_7b_fp8_scaled.safetensors
+COMFYUI_VAE_FILE=qwen_image_vae.safetensors
+
+# --- RAG databases (qdrant + neo4j, see wayfinder notes) ---
+# No auth on qdrant (its default) — same trust boundary as llama-server:
+# ai-stack is not exposed off-box. Random, filled in automatically:
+NEO4J_PASSWORD=

.gitignore

@@ -1 +1,8 @@
 .claude/
+.env
+# Personal memory content ingested by scripts/ingest-memory.sh — not meant
+# to be committed to this repo.
+data/
+.leankg/
+.cache/
+.qwen/temp

CLAUDE.md

@@ -7,3 +7,7 @@ Issues live as Gitea issues on `git.arthurerlich.de` (repo `haylan/LLM-Server`);
 ### Domain docs
 
 Single-context: `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
+
+### Deploying changes
+
+The running stack lives on a separate box (the R9700 server), not wherever this repo is being edited. After **any** change to `docker-compose.yml`, `.env.example`, or a script under `scripts/`, commit/push it, then run `./scripts/update.sh` on the server to apply it — don't just describe the change as done. If this session doesn't have shell access to the server, say so explicitly and tell the user to run it themselves rather than leaving it unsaid.

README.md

@@ -1,21 +1,35 @@
 # LLM-Server
 
-Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by Open WebUI (RAG + Memory via Qdrant), with Lazytainer auto-suspending the inference container when idle.
+Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle.
 
 See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
 
 ## Quickstart
 
 ```bash
-cp .env.example .env      # adjust if needed
-./scripts/download-model.sh
-docker compose up -d
+./scripts/update.sh
 ```
 
-- Open WebUI: http://localhost:3000 (first signup becomes the admin account — `WEBUI_AUTH` is on)
-- llama.cpp OpenAI-compatible API: http://localhost:8080/v1
-- llama.cpp Anthropic Messages API (for Claude Code CLI): http://localhost:8080/v1/messages
+`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
 
-Pointing Claude Code CLI or Kimi CLI at the local endpoint is documented separately — see [issue #6](https://git.arthurerlich.de/haylan/LLM-Server/issues/6) once resolved.
+llama.cpp's own API is internal-only — everything routes through the AI gateway below.
+
+Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).
 
 **Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
+
+## AI gateway (OmniRoute)
+
+An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload API keys and usage tracking. As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) this is [OmniRoute](https://github.com/diegosouzapw/OmniRoute), replacing the original LiteLLM setup. `./scripts/update.sh` handles most of OmniRoute's secrets (see `.env.example`); per-workload API keys still need minting by hand in the dashboard — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
+
+- Gateway API: `http://<this-machine>:${OMNIROUTE_PORT:-4000}/v1` locally, or `proxy-ai.home` / `proxy-ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
+- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
+- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
+
+Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
+
+**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
+
+### Web search
+
+The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).

docker-compose.yml

@@ -5,12 +5,20 @@ services:
     devices:
       - /dev/kfd
       - /dev/dri
+    # Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
+    # .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
     group_add:
-      - video
-      - render
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
     security_opt:
       - seccomp=unconfined
     ipc: host
+    # Caps this process's HIP hardware-queue allocation — works around
+    # ROCm/ROCm#5706 (GPU pinned at 100%/boost-clock whenever two
+    # concurrent HIP contexts touch this card, confirmed on real hardware
+    # against llama-server-fast below). See the research doc above.
+    environment:
+      - GPU_MAX_HW_QUEUES=1
     volumes:
       - models:/models
     command: >
@@ -18,12 +26,17 @@ services:
       --host 0.0.0.0
       --port 8080
       --n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
-      --ctx-size ${LLAMA_CTX_SIZE:-65536}
+      --ctx-size ${LLAMA_CTX_SIZE:-262144}
+      --parallel ${LLAMA_PARALLEL:-2}
+      --flash-attn on
+      --cache-type-k q8_0
+      --cache-type-v q8_0
       --jinja
-    ports:
-      # published to the host so Claude Code CLI / Kimi CLI can reach it directly,
-      # bypassing Open WebUI.
-      - "${LLAMA_PORT:-8080}:8080"
+    # No published host port: llama-server is reached only via the omniroute
+    # gateway on the ai-stack docker network now — see issue #15. Its
+    # unauthenticated API no longer needs to be LAN-reachable directly.
+    expose:
+      - "8080"
     restart: unless-stopped
     networks: [ai-stack]
     labels:
@@ -34,60 +47,296 @@ services:
       - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
       - "lazytainer.group.llamaserver.minPacketThreshold=2"
 
+  llama-server-fast:
+    image: ghcr.io/ggml-org/llama.cpp:server-rocm
+    container_name: llama-server-fast
+    devices:
+      - /dev/kfd
+      - /dev/dri
+    group_add:
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
+    security_opt:
+      - seccomp=unconfined
+    ipc: host
+    # See llama-server's identical setting above — same fix, same bug.
+    environment:
+      - GPU_MAX_HW_QUEUES=1
+    volumes:
+      - models:/models
+    command: >
+      -m /models/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf}
+      --host 0.0.0.0
+      --port 8080
+      --n-gpu-layers ${LLAMA_FAST_GPU_LAYERS:-999}
+      --ctx-size ${LLAMA_FAST_CTX_SIZE:-8192}
+      --parallel ${LLAMA_FAST_PARALLEL:-1}
+      --flash-attn on
+      --cache-type-k q8_0
+      --cache-type-v q8_0
+      --reasoning off
+      --jinja
+    # Second, always-resident llama.cpp instance — small non-thinking model
+    # used as qwen-code's Auto Mode classifier fastModel, alongside the main
+    # 27B instance above. See docs/research/fast-model-choice.md and #44.
+    # Same ai-stack-only pattern as llama-server: no published host port.
+    expose:
+      - "8080"
+    restart: unless-stopped
+    networks: [ai-stack]
+    labels:
+      - "lazytainer.group.llamaserverfast.sleepMethod=stop"
+      - "lazytainer.group.llamaserverfast.ports=8080"
+      - "lazytainer.group.llamaserverfast.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
+      - "lazytainer.group.llamaserverfast.minPacketThreshold=2"
+
   # ponytail: one-off downloader, not a standing service — run via
-  # `docker compose --profile tools run --rm downloader` (see scripts/download-model.sh).
-  # Keeps the model file inside the named `models` volume instead of a host bind-mount.
+  # `docker compose --profile tools run --rm downloader`. Folded into
+  # scripts/update.sh, which runs this every time; the `test -f` guard is
+  # what makes that safe to re-run without re-downloading. Keeps the model
+  # files inside the named `models` volume instead of a host bind-mount.
   downloader:
     image: curlimages/curl:latest
     profiles: ["tools"]
+    # ponytail: named volume is created root-owned; curl_user (uid 100) can't
+    # write into it otherwise, so run as root for this one-off job.
+    user: root
     volumes:
       - models:/models
     entrypoint: ["sh", "-c"]
     command:
       - >
+        test -f /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf} &&
+        echo "already downloaded, skipping" ||
         curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
         https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
 
-  qdrant:
-    image: qdrant/qdrant:latest
-    container_name: qdrant
+  # Same test -f guard pattern as downloader above — fetches the second,
+  # smaller model for llama-server-fast. See issue #44.
+  downloader-fast:
+    image: curlimages/curl:latest
+    profiles: ["tools"]
+    user: root
     volumes:
-      - qdrant-data:/qdrant/storage
+      - models:/models
+    entrypoint: ["sh", "-c"]
+    command:
+      - >
+        test -f /models/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf} &&
+        echo "already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /models/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf}
+        https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_FAST_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf}
+
+  # Fetches the three Qwen-Image FP8 files ComfyUI needs (diffusion model,
+  # text encoder, VAE) — same test -f guard pattern as downloader/
+  # downloader-fast above. See docs/research/image-generation-model-choice.md
+  # and issue #42.
+  #
+  # ponytail: target paths assume ComfyUI's standard models/ layout under
+  # BASE_STORAGE_PATH (/storage) — same "not independently confirmed against
+  # the image's Dockerfile" caveat already flagged on the comfyui service
+  # below. If ComfyUI doesn't pick these up, check its actual models root
+  # first.
+  downloader-comfyui:
+    image: curlimages/curl:latest
+    profiles: ["tools"]
+    user: root
+    volumes:
+      - comfyui-data:/storage
+    entrypoint: ["sh", "-c"]
+    command:
+      - >
+        mkdir -p /storage/models/diffusion_models /storage/models/text_encoders /storage/models/vae &&
+        (test -f /storage/models/diffusion_models/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen_image_fp8_e4m3fn.safetensors} &&
+        echo "diffusion model already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /storage/models/diffusion_models/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen_image_fp8_e4m3fn.safetensors}
+        https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen_image_fp8_e4m3fn.safetensors}) &&
+        (test -f /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen_2.5_vl_7b_fp8_scaled.safetensors} &&
+        echo "text encoder already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen_2.5_vl_7b_fp8_scaled.safetensors}
+        https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen_2.5_vl_7b_fp8_scaled.safetensors}) &&
+        (test -f /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_vae.safetensors} &&
+        echo "vae already downloaded, skipping" ||
+        curl -L --fail --create-dirs -o /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_vae.safetensors}
+        https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/${COMFYUI_VAE_FILE:-qwen_image_vae.safetensors})
+
+  # Local image generation — see issue #38 (wayfinder map). yurisasc's image
+  # is gfx1201-tuned specifically (R9700's arch), unlike the official/AMD
+  # ComfyUI image which doesn't pin RDNA4 support — see
+  # docs/research/image-generation-options.md.
+  comfyui:
+    image: yurisasc/comfyui-rocm7.1:latest
+    container_name: comfyui
+    devices:
+      - /dev/kfd
+      - /dev/dri
+    # Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
+    # .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
+    group_add:
+      - "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
+      - "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
+    security_opt:
+      - seccomp=unconfined
+    ipc: host
+    environment:
+      - HSA_OVERRIDE_GFX_VERSION=12.0.1
+      - PYTORCH_ROCM_ARCH=gfx1201
+      # This image also wants GID env vars directly (its own README asks
+      # for both these and group_add above) — same HOST_VIDEO_GID/
+      # HOST_RENDER_GID resolved by scripts/update.sh, shared with
+      # llama-server/llama-server-fast now instead of comfyui-only vars.
+      - PUID=${COMFYUI_PUID}
+      - PGID=${COMFYUI_PGID}
+      - VIDEO_GID=${HOST_VIDEO_GID}
+      - RENDER_GID=${HOST_RENDER_GID}
+      - BASE_STORAGE_PATH=/storage
+    volumes:
+      - comfyui-data:/storage
+    # ponytail: exact internal storage path taken from the image's own
+    # BASE_STORAGE_PATH env var, not independently confirmed against its
+    # Dockerfile — if models/workflows don't persist across a recreate,
+    # check this against the image's actual entrypoint first.
+    #
+    # Published host port (unlike llama-server's ai-stack-only pattern):
+    # ComfyUI's own UI is meant to be reachable directly too, for a planned
+    # external nginx reverse-proxy route to comfy.home — not just through
+    # OmniRoute. Still also reachable at http://comfyui:8188 internally on
+    # ai-stack, which is the URL to register as OmniRoute's comfyui
+    # provider (dashboard or POST /api/providers, per docs/proxy-key-onboarding.md
+    # — same undocumented-in-repo manual flow already used for llama-server).
+    ports:
+      - "8138:8188"
     restart: unless-stopped
     networks: [ai-stack]
 
-  open-webui:
-    image: ghcr.io/open-webui/open-webui:main
-    container_name: open-webui
+  # Replaces litellm — see issue #31 (wayfinder map) for the full migration
+  # rationale/findings. No static config.yaml equivalent: provider routing
+  # (llama-server, searxng-search) is registered once through the dashboard
+  # or POST /api/providers after first boot, not checked into this repo —
+  # see docs/proxy-key-onboarding.md.
+  omniroute:
+    image: diegosouzapw/omniroute:latest
+    container_name: omniroute
     depends_on:
-      - qdrant
+      llama-server:
+        condition: service_started
+      llama-server-fast:
+        condition: service_started
     volumes:
-      - openwebui-data:/app/backend/data
+      - omniroute-data:/app/data
+    env_file: .env
     environment:
-      - WEBUI_AUTH=True
-      - OPENAI_API_BASE_URL=http://llama-server:8080/v1
-      - OPENAI_API_KEY=${OPENAI_API_KEY:-local}
-      - VECTOR_DB=qdrant
-      - QDRANT_URI=http://qdrant:6333
+      # Split-port mode: dashboard and API are fully separate ports (unlike
+      # LiteLLM's single :4000 for both /v1 and /ui) — both published
+      # directly below, unlike the old :4000-only host mapping.
+      - API_HOST=0.0.0.0
+      - API_PORT=${OMNIROUTE_API_PORT:-20129}
+      - DASHBOARD_PORT=${OMNIROUTE_DASHBOARD_PORT:-20128}
+      # Required to register llama-server/searxng-search as providers —
+      # their base URLs are LAN/container-internal addresses, blocked by
+      # default (SSRF guard against public-provider spoofing).
+      - OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true
+      - OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true
+      # Required (production) per docs/reference/ENVIRONMENT.md — shared
+      # secret for the internal Codex Responses WebSocket bridge. Missed on
+      # first pass; docker-compose config validated fine without it, but
+      # the docs are explicit this one's required, not optional.
+      - OMNIROUTE_WS_BRIDGE_SECRET=${OMNIROUTE_WS_BRIDGE_SECRET}
+      # Default heap (1024MB) is dashboard-only sized per OmniRoute's own
+      # Docker guide — every client here is a coding CLI, which needs the
+      # larger figure the guide recommends. Paired with mem_limit below.
+      - OMNIROUTE_MEMORY_MB=8192
+      # Default 300000 (5 min) per OmniRoute's own docs, but this deployment
+      # had it dialed down elsewhere (dashboard) to ~95s — too tight for a
+      # contended local llama-server: large-context prefill under multiple
+      # concurrent slots can outrun that before the first SSE token arrives,
+      # so OmniRoute cancels a request that was actually still working (see
+      # LLAMA_PARALLEL above for the other half of this fix). Raised here so
+      # it's tracked in git instead of a dashboard-only setting.
+      - STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
+    # Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
+    # network can't resolve search.home on its own.
+    extra_hosts:
+      - "search.home:${SEARXNG_LAN_IP}"
     ports:
-      - "${WEBUI_PORT:-3000}:8080"
+      - "${OMNIROUTE_API_PORT:-20129}:${OMNIROUTE_API_PORT:-20129}"
+      - "${OMNIROUTE_DASHBOARD_PORT:-20128}:${OMNIROUTE_DASHBOARD_PORT:-20128}"
+    # 10+ GiB ceiling per OmniRoute's Docker guide, matching
+    # OMNIROUTE_MEMORY_MB=8192 above.
+    mem_limit: 10g
+    # SQLite WAL needs time to checkpoint back into the main DB file on
+    # shutdown — the Docker guide's --stop-timeout 40 equivalent.
+    stop_grace_period: 40s
     restart: unless-stopped
     networks: [ai-stack]
+    # ponytail: TCP-connect check, not an HTTP /healthz GET — the image has
+    # no python3/curl/wget (confirmed live, `which` found only node), and
+    # OmniRoute's own Docker guide already treats a bare TCP probe on this
+    # port as an acceptable liveness check, not just the HTTP one. Simpler
+    # and avoids depending on /healthz's exact path/response shape.
+    healthcheck:
+      test:
+        - CMD-SHELL
+        - node -e "require('net').connect(${OMNIROUTE_API_PORT:-20129},'localhost').on('connect',function(){this.end();process.exit(0)}).on('error',()=>process.exit(1))"
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
 
   lazytainer:
     image: ghcr.io/vmorganp/lazytainer:master
     container_name: lazytainer
-    network_mode: host
+    # NOT network_mode: host — lazytainer identifies its own container by
+    # matching os.Hostname() against the Docker container-ID list
+    # (vmorganp/Lazytainer, configureFromLabels()); under host networking the
+    # container inherits the host's hostname instead of its own ID, so that
+    # match always fails and it panics with "Could not determine container ID
+    # of lazytainer" on every start. Host networking also can't see traffic
+    # to llama-server:8080 anyway — that port only exists on the ai-stack
+    # bridge network (no host port published, see issue #15 above). Joining
+    # ai-stack instead fixes both: hostname becomes the real container ID,
+    # and it's on the same network as the traffic it's watching.
+    networks: [ai-stack]
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
     restart: unless-stopped
     depends_on:
       - llama-server
 
+  # RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
+  # directly like comfyui above, not gatewayed through omniroute (it isn't an
+  # LLM provider).
+  qdrant:
+    image: qdrant/qdrant:latest
+    container_name: qdrant
+    volumes:
+      - qdrant-data:/qdrant/storage
+    ports:
+      - "6333:6333"
+    restart: unless-stopped
+    networks: [ai-stack]
+
+  # RAG graph store, native vector index too (can absorb qdrant's job later
+  # if the two-DB split proves unnecessary — see wayfinder notes).
+  neo4j:
+    image: neo4j:5-community
+    container_name: neo4j
+    environment:
+      - NEO4J_AUTH=neo4j/${NEO4J_PASSWORD:?run scripts/update.sh first to resolve this}
+    volumes:
+      - neo4j-data:/data
+    ports:
+      - "7474:7474" # browser UI
+      - "7687:7687" # bolt
+    restart: unless-stopped
+    networks: [ai-stack]
+
 networks:
   ai-stack:
 
 volumes:
   models:
+  omniroute-data:
+  comfyui-data:
   qdrant-data:
-  openwebui-data:
+  neo4j-data:

docs/agents/issue-tracker.md

@@ -9,7 +9,7 @@ Use the **`tea` CLI** (already installed and authenticated as `haylan` via `tea
 - **Create an issue**: `tea issues create --title "..." --description "..." --labels "..."`
 - **Read an issue**: `tea issues <index> --comments`
 - **List issues**: `tea issues list --state open --labels "..."` (add `-f` to control which fields print)
-- **Comment on an issue**: `tea comments create <index> --description "..."` (check `tea comments -h` for exact flags)
+- **Comment on an issue**: `tea comment <index> -d "..."` (check `tea comments -h` for exact flags — `tea comments create` is invalid, `add`/`a` is the subcommand)
 - **Apply / remove labels**: `tea issues edit <index> --add-labels "..."` / `--remove-labels "..."`
 - **Close**: `tea issues close <index>`
 - **Labels**: `tea labels create --name "..." --color "#hex" --description "..."`; `tea labels list`
@@ -48,4 +48,4 @@ Used by `/wayfinder`. This Gitea instance (1.27.2) has **no native sub-issue/par
 - **Blocking**: native issue dependencies via the raw API calls above. A ticket is unblocked when every dependency (`GET .../dependencies`) is closed.
 - **Frontier query**: `tea issues list --state open --labels "wayfinder:<type1>,wayfinder:<type2>,..."` scoped to the map's children (cross-check against the map's task list), drop any with an open dependency or an assignee.
 - **Claim**: `tea issues edit <n> --add-assignees haylan` — the session's first write.
-- **Resolve**: `tea comments create <n> --description "<answer>"`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far, and check off its line in the map's task list.
+- **Resolve**: `tea comment <n> -d "<answer>"`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far, and check off its line in the map's task list. Map edits are full-body replaces (`tea issues edit` has no append) — concurrent resolutions racing on the same map issue can clobber each other's Decisions-so-far lines; re-fetch the map immediately before editing it, not from an earlier read.

docs/coding-cli-setup/claude-code.md

@@ -0,0 +1,13 @@
+# Claude Code CLI
+
+[← back to overview](index.md)
+
+Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
+
+```bash
+export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
+export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
+claude
+```
+
+Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.

docs/coding-cli-setup/index.md

@@ -0,0 +1,34 @@
+# Pointing a coding-agent CLI at this stack
+
+[← back to README](../../README.md)
+
+This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
+
+- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
+- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
+
+Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
+
+`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
+
+**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
+
+> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
+
+## Per-CLI setup
+
+- [Claude Code CLI](claude-code.md)
+- [Kimi CLI](kimi-cli.md)
+- [OpenCode CLI](opencode.md)
+- [Qwen Code CLI](qwen-code.md)
+
+## Summary
+
+| CLI | Wire format | Endpoint | Config |
+|---|---|---|---|
+| [Claude Code](claude-code.md) | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
+| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
+| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
+| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` |
+
+Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`.

docs/coding-cli-setup/kimi-cli.md

@@ -0,0 +1,14 @@
+# Kimi CLI
+
+[← back to overview](index.md)
+
+Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
+
+```toml
+[providers.openai]
+type = "openai"
+base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
+api_key = "<kimi-cli virtual key>"
+```
+
+If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.

docs/coding-cli-setup/opencode.md

@@ -0,0 +1,45 @@
+# OpenCode CLI
+
+[← back to overview](index.md)
+
+Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
+
+**Install**:
+```bash
+curl -fsSL https://opencode.ai/install | bash
+```
+
+**Config** (`opencode.json`, project root or `~/.config/opencode/opencode.json`):
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "aiproxy": {
+      "npm": "@ai-sdk/openai-compatible",
+      "name": "AI proxy (local)",
+      "options": {
+        "baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "apiKey": "<opencode-cli virtual key>"
+      },
+      "models": {
+        "qwen3.8-27b-local": {
+          "name": "Qwen3.8-27B",
+          "limit": { "context": 65536, "output": 8192 }
+        }
+      }
+    }
+  }
+}
+```
+
+Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server.
+
+Select the model with `aiproxy/qwen3.8-27b-local`.
+
+**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)):
+- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
+- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
+- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
+
+Further reading: `docs/research/opencode-cli-setup.md`.

docs/coding-cli-setup/qwen-code.md

@@ -0,0 +1,116 @@
+# Qwen Code CLI
+
+[← back to overview](index.md)
+
+Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier (a separate, always-resident, always-fast instance so classification doesn't queue behind chat prefill; see `docker-compose.yml`'s `llama-server-fast` service and `docs/research/fast-model-choice.md`). Both are registered as separate providers in OmniRoute but reachable through the same gateway URL. Config lives in `~/.qwen/settings.json`:
+
+```json
+{
+  "modelProviders": {
+    "openai": [
+      {
+        "id": "<main-model-provider-id-in-omniroute>",
+        "name": "qwen3.8-27b-local",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "generationConfig": { "contextWindowSize": 131072 }
+      },
+      {
+        "id": "<fast-model-provider-id-in-omniroute>",
+        "name": "qwen3.8-27b-classifier",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
+        "generationConfig": {
+          "contextWindowSize": 8192,
+          "extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
+        }
+      }
+    ]
+  },
+  "security": { "auth": { "selectedType": "openai" } },
+  "model": {
+    "name": "<main-model-provider-id-in-omniroute>",
+    "baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
+  },
+  "fastModel": "<fast-model-provider-id-in-omniroute>"
+}
+```
+
+- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=<qwen-code-cli virtual key>` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
+- **`contextWindowSize` is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it per model from `.env`:
+  - Main model: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**.
+  - Fast model: `LLAMA_FAST_CTX_SIZE / LLAMA_FAST_PARALLEL` = `8192 / 1` = **8192**. Undersizing this one specifically breaks Auto Mode ("Classifier stage 1 unavailable") once `hints.allow`/`softDeny`/`hardDeny` entries and recent-action history push a classifier call past it — see the `LLAMA_FAST_CTX_SIZE` comment in `.env.example` before raising it instead of `LLAMA_FAST_PARALLEL`.
+- `enable_thinking: false` on the fast model matters: the fast model file (`Qwen3-4B-Instruct-2507`) is already non-thinking, but this also suppresses `<think>` output on any fast-model swap that isn't, keeping classifier responses parseable.
+- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
+
+## Web search via OmniRoute
+
+Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
+
+```js
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+
+const BASE_URL = process.env.OMNIROUTE_BASE_URL || "http://proxy-ai.home";
+const API_KEY = process.env.OMNIROUTE_API_KEY;
+
+if (!API_KEY) {
+  console.error("OMNIROUTE_API_KEY is not set in the environment.");
+  process.exit(1);
+}
+
+const server = new McpServer({ name: "omniroute-search", version: "1.0.0" });
+
+server.registerTool(
+  "search",
+  {
+    description: "Web/news search via OmniRoute's /v1/search endpoint.",
+    inputSchema: { query: z.string().describe("Search query") },
+  },
+  async ({ query }) => {
+    const res = await fetch(`${BASE_URL}/v1/search`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json", Authorization: `Bearer ${API_KEY}` },
+      body: JSON.stringify({ query }),
+    });
+    const text = await res.text();
+    if (!res.ok) return { content: [{ type: "text", text: `HTTP ${res.status}: ${text}` }], isError: true };
+    return { content: [{ type: "text", text }] };
+  }
+);
+
+await server.connect(new StdioServerTransport());
+```
+
+Register it in `~/.qwen/settings.json`:
+
+```json
+{
+  "mcpServers": {
+    "omniroute-search": { "command": "node", "args": ["<path-to>/index.mjs"] }
+  },
+  "tools": { "webSearch": { "enabled": false } }
+}
+```
+
+It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.
+
+## Auto Mode tuning
+
+Auto Mode's action classifier calls the fast model above — its own request can queue behind other stack traffic before the fast llama-server instance is warm, so the default classifier timeout is worth raising. And since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:
+
+```json
+{
+  "permissions": {
+    "autoMode": {
+      "classifier": { "timeouts": { "stage1Ms": 600000 } },
+      "hints": { "allow": ["Requests to proxy-ai.home, my own local omniroute model proxy"] }
+    }
+  }
+}
+```
+
+`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each (see the `LLAMA_FAST_CTX_SIZE` note above for why that ceiling matters).
+
+Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.

docs/network-access.md

@@ -0,0 +1,24 @@
+# Network access: proxy-ai.home / proxy-ai.haylan.ch
+
+This stack has no chat UI — every client is a coding CLI reaching the AI gateway (OmniRoute). It doesn't run its own reverse proxy — it publishes the gateway's API port to the host and relies on the **existing Nginx Proxy Manager (NPM)** instance already fronting other self-hosted services on this network.
+
+## llama.cpp's raw API stays LAN-only — deliberately
+
+The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else.
+
+If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.
+
+## The AI gateway (OmniRoute) — `proxy-ai.home` / `proxy-ai.haylan.ch`
+
+As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute:
+
+- **`proxy-ai.home`** and **`proxy-ai.haylan.ch`** both point only at `${OMNIROUTE_PORT:-4000}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `proxy-ai.home` internal-only, `proxy-ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual).
+- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
+
+**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname.
+
+## RAG knowledge graph (Neo4j) — `knowledge.proxy-ai.home`
+
+Set up as an NPM Proxy Host pointing at this machine's LAN IP on Neo4j's Browser port (`7474`, see `docker-compose.yml`'s `neo4j` service, [PR #50](https://git.arthurerlich.de/haylan/LLM-Server/pulls/50)). Internal-only, same as `proxy-ai.home` — no DMZ/external route, this is admin/dev tooling, not a client-facing endpoint. Bolt (`7687`, the actual query protocol) isn't proxied through NPM at all — clients on the LAN reach it directly at `<this-machine>:7687`.
+
+Qdrant's dashboard (`6333`) stays on its raw LAN IP/port for now — no hostname assigned yet.

docs/proxy-key-onboarding.md

@@ -0,0 +1,26 @@
+# Onboarding a workload onto the AI gateway
+
+How to issue a new per-workload API key against the OmniRoute gateway (see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) — the LiteLLM → OmniRoute migration; original gateway rationale in [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
+
+No workload in this stack itself needs a key right now — every client is external (a coding CLI, or another self-hosted service). There's no scripted mint yet either way: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key by hand for now, via the dashboard steps below.
+
+## Create the key
+
+1. Log into the omniroute dashboard. `DASHBOARD_PORT` (20128) is never published to the host (see `docker-compose.yml`'s `omniroute` service) — from the R9700 box itself, find the container's own address (`docker inspect -f '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`) and browse to `http://<that-ip>:20128` (the host can reach a container's bridge-network IP directly, published port or not). From elsewhere, SSH port-forward instead: `ssh -L 20128:<container-ip>:20128 <host>`, then browse `http://localhost:20128`.
+2. "Keys" → "Create API key".
+3. Label it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `claude-code-cli`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync.
+4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.
+
+Once `POST /api/keys`'s session-auth flow is worked out (issue #37), the equivalent `curl` here can replace this manual step, the way `update.sh` used to automate LiteLLM's `/key/generate`.
+
+## Hand it to the workload
+
+Drop the key into that workload's own `.env` (or equivalent config) — never into this repo. Each workload's config is the source of truth for its own credential, same pattern as `.env.example` in this repo for the existing stack.
+
+## Retiring or rotating a key
+
+No scheduled rotation. Revoke the key by hand in the dashboard ("Keys" → delete) only when:
+- the workload is retired, or
+- the key is suspected leaked/compromised.
+
+Then remove it from that workload's `.env`.

docs/proxy-request-priority.md

@@ -0,0 +1,28 @@
+# Request priority on the AI proxy
+
+**Stale as of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (LiteLLM → OmniRoute migration)** — the Mechanism section below describes LiteLLM's specific scheduler, which no longer applies. Whether OmniRoute has an equivalent priority/queueing mechanism hasn't been researched. The Tiers/problem statement below still holds; treat Mechanism onward as historical until this is revisited.
+
+One local model instance (llama.cpp on the single R9700) serves every workload through the AI gateway ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9)). Interactive usage shouldn't get stuck behind a batch job.
+
+## Tiers
+
+Two tiers, assigned per workload's virtual key (per `docs/proxy-key-onboarding.md`):
+
+- **High priority** (interactive — someone's waiting): coding CLIs (Claude Code / Kimi / OpenCode), Gitea code review.
+- **Low priority** (batch — nobody's watching a spinner): Paperless OCR/tagging, Nextcloud Memories face-recognition, AI watermark removal.
+
+## Mechanism
+
+Use LiteLLM's built-in request-prioritization scheduler ([docs](https://docs.litellm.ai/docs/scheduler)) — callers pass a `priority` value, LiteLLM's router queues and dispatches by priority. Per `docs/research/proxy-tool-choice.md`, this feature is real but **beta**: there's a known, closed-as-not-planned bug where the `priority` field can leak into the provider request. Treat it as unproven, not settled:
+
+- **#14 (compose authoring) must smoke-test the scheduler against llama.cpp specifically** before this is relied on — confirm the `priority` field doesn't leak into llama.cpp's request and actually reorders dispatch under load.
+- **If it's broken in practice**, fall back to a lightweight queuing shim in front of the proxy (a small sidecar) rather than reworking the gateway tool choice. Don't build this shim speculatively — only if the smoke test fails.
+- Single-instance deployment (this stack) doesn't need Redis for virtual keys/spend, but LiteLLM's scheduler does use Redis for cross-instance state — if the scheduler needs it even single-instance, add a `redis` service to docker-compose.yml at that point, not before.
+
+## Timeout behavior
+
+A request queued too long (burst of batch jobs, or the model just being slow) times out and returns an error to the caller — no indefinite waiting. Use LiteLLM's default request timeout unless testing shows it needs tuning.
+
+## Lazytainer interaction
+
+No new risk: queueing happens inside LiteLLM *before* it dispatches to llama.cpp. Lazytainer watches actual traffic reaching the llama.cpp container, so it still sees the first dispatched request and wakes the container normally — priority ordering only changes which queued request gets dispatched first, not whether Lazytainer sees traffic.

docs/research/connecting-cloud-providers.md

@@ -0,0 +1,149 @@
+# Connecting real cloud LLM providers (Anthropic Claude et al.) behind the LiteLLM gateway
+
+**Date:** 2026-08-26
+**Scope:** Can the existing $20/mo Claude Pro subscription be used as a backend behind our
+self-hosted LiteLLM gateway (`docker-compose.yml`, `litellm-config.yaml`) instead of, or alongside,
+llama.cpp? If not, what's the concrete path to add real Anthropic (and other) cloud models?
+
+## 1. Claude Pro subscription quota vs. Anthropic API — is it the same thing?
+
+**No. They are two separate products with separate billing, and Anthropic's own documentation and
+Terms of Service explicitly prohibit routing Pro/Max subscription credentials through third-party
+tools like a self-hosted gateway.**
+
+### 1a. Pro plan does not include API access
+
+Anthropic's help center is explicit:
+
+> "The Pro plan does not include API usage through the Claude Console. If you're interested in both
+> enhanced Claude features and the Claude API, you'll need to set up Console access to pay for API
+> usage separately."
+— https://support.claude.com/en/articles/8325606-what-is-the-pro-plan
+
+Pro/Max are flat-fee subscriptions to *products* (claude.ai web/desktop/mobile chat, and Claude Code)
+with rolling usage limits. The Anthropic API (Console/`platform.claude.com`) is a metered,
+pay-per-token product with its own separate billing account. There is no documented way to
+authenticate a non-Anthropic tool against Pro/Max quota "as if" it were an API key — no such
+integration exists.
+
+### 1b. How Claude Code itself authenticates, and why that path is closed off to other tools
+
+When you run `claude` and log in via `/login` with a claude.ai account, Claude Code obtains an
+**OAuth token scoped to the subscription** (standard OAuth 2.0 authorization-code flow against
+Anthropic's consent page). Anthropic also documents `claude setup-token`, which mints a **long-lived
+(1-year) OAuth token** (`CLAUDE_CODE_OAUTH_TOKEN`, prefixed `sk-ant-oat01-...`) explicitly *for use
+outside interactive login* — e.g. CI. This is real, officially documented, subscription-backed
+credential material that is technically exportable as an environment variable.
+— https://code.claude.com/docs/en/authentication
+
+**However, Anthropic's official Legal & Compliance page for Claude Code states this is authorized for
+Claude Code (and other native Anthropic apps) only, not for arbitrary third-party tools:**
+
+> "**OAuth authentication** is intended exclusively for purchasers of Claude Free, Pro, Max, Team, and
+> Enterprise subscription plans and is designed to support ordinary use of Claude Code and other
+> native Anthropic applications."
+>
+> "Anthropic does not permit third-party developers to offer Claude.ai login into their own
+> applications, or to route requests through Free, Pro, or Max plan credentials on behalf of their
+> users. Moreover, developers may not collect, store, or intermediate Claude.ai credentials or session
+> tokens — sign-in to a Claude account must complete through Anthropic's own flow."
+>
+> "Anthropic reserves the right to take measures to enforce these restrictions and may do so without
+> prior notice."
+— https://code.claude.com/docs/en/legal-and-compliance ("Authentication and credential use" section)
+
+That page cites the governing documents directly:
+- Consumer Terms of Service (Free/Pro/Max): https://www.anthropic.com/legal/consumer-terms
+- Commercial Terms of Service (Team/Enterprise/API): https://www.anthropic.com/legal/commercial-terms
+- Anthropic Usage Policy: https://www.anthropic.com/legal/aup
+
+**This is not just a ToS technicality — it's actively enforced.** In March 2026 Anthropic used
+server-side enforcement (in addition to legal action) to block third-party "harnesses" (e.g.
+OpenClaw and similar tools) from routing traffic through Claude subscription OAuth credentials,
+specifically citing that such traffic bypasses the telemetry/behavior the Claude Code harness
+provides. Coverage: https://www.theregister.com/software/2026/02/20/anthropic-clarifies-ban-on-third-party-tool-access-to-claude/
+(secondary source; the primary enforcement basis is the Legal & Compliance page above, which is
+current as fetched today).
+
+**Community finding (flagged as unofficial/against ToS):** technically, `claude setup-token`'s OAuth
+token *can* be handed to LiteLLM (some LiteLLM community docs / discussions describe pointing
+`ANTHROPIC_AUTH_TOKEN`/`CLAUDE_CODE_OAUTH_TOKEN` at a proxy, e.g.
+https://docs.litellm.ai/docs/tutorials/claude_code_max_subscription and
+https://github.com/BerriAI/litellm/discussions/30827). This works at the protocol level because the
+token is a bearer credential like any other. **It is exactly the pattern Anthropic's compliance page
+above says is not permitted** ("route requests through Free, Pro, or Max plan credentials on behalf
+of \[other] users" / outside "ordinary use of Claude Code and other native Anthropic applications").
+Doing this for **personal, single-user use** through the unmodified `claude` binary (e.g. just running
+`claude` itself pointed at your own gateway) is different from what's prohibited — the prohibition
+targets *intermediating* the subscription credential through a third-party tool/product on behalf of
+requests that aren't Claude Code itself. Routing arbitrary LiteLLM/OpenWebUI traffic through a
+subscription OAuth token extracted from `claude setup-token` falls squarely in the prohibited
+category. **Do not build this into the gateway.**
+
+### 1c. Bottom line on subscription reuse
+
+No documented, ToS-compliant way exists to point LiteLLM (or any third-party gateway) at Claude
+Pro/Max quota instead of a real API key. The $20/mo subscription is for claude.ai and Claude Code
+usage only.
+
+## 2. The real path: an Anthropic API key (Console, pay-per-token)
+
+This is the supported way to add Claude models to LiteLLM, and it's the same config shape already
+used for llama.cpp in `litellm-config.yaml`.
+
+LiteLLM's Anthropic provider docs (https://docs.litellm.ai/docs/providers/anthropic) give this
+`model_list` shape:
+
+```yaml
+model_list:
+  - model_name: claude-sonnet-5   # whatever alias you want callers to use
+    litellm_params:
+      model: anthropic/claude-sonnet-5    # "anthropic/<model-id>" tells LiteLLM which provider
+      api_key: os.environ/ANTHROPIC_API_KEY
+```
+
+Practical steps:
+1. Create a Console account / org at https://platform.claude.com (separate from claude.ai login),
+   generate an API key there.
+2. Add it to `.env` as `ANTHROPIC_API_KEY=sk-ant-api03-...` and reference it via
+   `os.environ/ANTHROPIC_API_KEY` in `litellm-config.yaml`, following the existing pattern used for
+   `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` in `docker-compose.yml`.
+3. Add a `model_list` entry per Claude model you want exposed (e.g. `claude-sonnet-5`,
+   `claude-haiku-4-5`), each with its own `input_cost_per_token`/`output_cost_per_token` if you want
+   LiteLLM's cost tracking to reflect real spend (today `litellm-config.yaml`'s cost fields are a
+   shadow estimate against Sonnet 5 pricing for the *local* model — real Anthropic model entries
+   should carry the model's *actual* published rate from
+   https://platform.claude.com/docs/en/about-claude/pricing).
+4. **This is billed separately from, and in addition to, the $20/mo subscription** — pay-per-token,
+   metered on the Console account, no relationship to Pro/Max usage limits.
+
+No other config changes are implied — the router/`general_settings` blocks already in place don't
+need to change for a second provider; LiteLLM routes per-`model_name` based on the `litellm_params`
+each entry declares.
+
+## 3. Other providers — same general pattern
+
+Every major consumer AI subscription (OpenAI's ChatGPT Plus, and by the same logic Google's Gemini
+subscription tiers) works identically to Anthropic's: **the consumer chat subscription and the
+pay-per-token developer API are separate products with separate billing, and the subscription does
+not unlock API access.** Confirmed for OpenAI: "ChatGPT Plus does not include API access... no API
+key in the box, no monthly API credits, and no discount on per-token prices. The OpenAI API is a
+separate product with its own billing." There's no equivalent OAuth-token-reuse loophole documented
+for OpenAI/Google either — this isn't an Anthropic-specific restriction, it's the standard shape of
+how these companies segment consumer vs. developer products. Adding OpenAI, Gemini, or any other
+cloud model to LiteLLM means the same recipe as Anthropic: get a real pay-per-token API key from that
+provider's own developer console and add a `model_list` entry with `litellm_params.model` set to
+that provider's LiteLLM prefix (`openai/...`, `gemini/...`, etc.) — see
+https://docs.litellm.ai/docs/providers for the full prefix list.
+
+## 4. Recommended next step
+
+- Do **not** attempt to feed the Claude Pro subscription's OAuth token into LiteLLM/Open WebUI — it's
+  against Anthropic's Consumer Terms of Service and Usage Policy, and Anthropic has shown it will
+  enforce this server-side without notice (per the March 2026 crackdown on third-party harnesses).
+- If real cloud Claude models behind the gateway are wanted, get a **separate Anthropic Console API
+  key** (pay-per-token, its own bill, independent of the $20/mo subscription) and add it as a
+  `model_list` entry per §2 above. Same recipe for any other provider (§3).
+- Keep using the existing $20/mo Claude Pro subscription only for what it's licensed for: the claude.ai
+  web app and the Claude Code CLI itself (this session included) — not as a backend behind the
+  self-hosted gateway.

docs/research/dashscope-websearch-selfhost-options.md

@@ -0,0 +1,412 @@
+# Research: self-hosted alternatives to DashScope for Qwen Code's built-in `web_search` tool
+
+**Question:** Qwen Code CLI's built-in `web_search` tool requires `tools.webSearch.model`
+to resolve to a "DashScope-compatible `modelProviders` entry." Is there any real,
+non-Alibaba-Cloud way to satisfy that requirement with something self-hosted —
+or is the already-working OmniRoute MCP + SearXNG path (`docs/research/omniroute-qwen-websearch.md`)
+the end of the road?
+
+**Answer, short version:** No. The client-side code that decides whether a
+`baseUrl` is "DashScope-compatible" checks the **literal hostname** against a
+hardcoded allowlist of Alibaba-owned domains, before any request is sent — it
+is not a protocol-compatibility check that a look-alike server could pass. A
+self-hosted server cannot satisfy it, full stop, unless you fork qwen-code and
+delete that check. Once you've done that, the actual wire protocol
+(OpenAI SDK `responses.create()`, SSE, specific item types) is buildable
+(a few hundred lines), but nothing you can install off the shelf implements it
+today. The already-working OmniRoute MCP + SearXNG path costs nothing further
+and does not have this problem. **Recommendation: don't build this — see
+§6.**
+
+## 1. What "DashScope Responses API" is, precisely
+
+Alibaba Cloud Model Studio (Bailian/DashScope) added an **OpenAI-compatible
+Responses API**, layered on top of its existing Chat Completions
+compatible-mode surface:
+
+- Endpoint (per Alibaba's own docs): `POST {baseUrl}/responses`, where
+  `baseUrl` is the region's compatible-mode base, e.g.
+  `https://dashscope.aliyuncs.com/compatible-mode/v1` (China/Beijing) or the
+  `-intl` / regional `*.maas.aliyuncs.com` variants.
+  Source: https://www.alibabacloud.com/help/en/model-studio/qwen-api-via-openai-responses
+  and https://www.alibabacloud.com/help/en/model-studio/compatibility-with-openai-responses-api
+- Request shape: standard Responses API (`model`, `input`, `stream`, `store`,
+  `instructions`) plus a `tools` array that can include
+  `{"type": "web_search"}`, `{"type": "web_extractor"}`, `{"type": "code_interpreter"}`
+  as **hosted, server-side tools** — the inference backend runs the search
+  itself and streams results back, the same hosted-tool pattern as OpenAI's
+  own Responses API `web_search_preview`, not a client-side function-calling
+  round trip.
+  Source: https://www.alibabacloud.com/help/en/model-studio/qwen-api-via-openai-responses
+- Response shape: an `output` array of typed items — a `web_search_call` item
+  carries `action: {type, query, sources: [{type: "url", url}]}`; narration
+  comes back as `message` items with `content: [{type, text}]`.
+  Source: same page.
+- Separately, DashScope's plain Chat Completions endpoint (not Responses)
+  has an older, unrelated `enable_search` boolean (passed via `extra_body`)
+  for models like Qwen3.8/Qwen3.6-Plus — the docs explicitly note this older
+  surface **does not return citations/sources**, which is exactly why
+  qwen-code's built-in tool uses the *Responses* API instead.
+  Source: https://docs.qwencloud.com/developer-guides/tool-calling/web-search
+
+This is the same hosted-tool pattern as OpenAI's Responses API
+`web_search_preview` (§4 confirms this directly from qwen-code's own client
+code — it literally reuses the OpenAI Node SDK's `responses.create()` call
+against a DashScope base URL).
+
+## 2. What qwen-code's own client code actually sends (ground truth)
+
+Fetched directly from `QwenLM/qwen-code`'s `main` branch,
+`packages/core/src/tools/web-search.ts` (1087 lines) and
+`packages/core/src/core/openaiContentGenerator/{constants,provider/dashscope}.ts`.
+This supersedes anything inferred from the docs pages — it's the literal
+implementation.
+
+**The request** (`web-search.ts` lines 651–699):
+
+```ts
+const client = new OpenAI({
+  apiKey,                    // from the resolved modelProviders entry's envKey
+  baseURL: backend.baseUrl,  // the modelProviders entry's baseUrl / WEB_SEARCH_BASE_URL
+  timeout: 60_000,
+  defaultHeaders: { 'User-Agent': `QwenCode/${version} (...)`, ...customHeaders },
+});
+
+const tools = [{ type: 'web_search' }];
+if (backend.webExtractor) tools.push({ type: 'web_extractor' });
+
+const requestParams = {
+  model: backend.modelId,
+  input: `Perform a web search for the query: ${query}`,
+  stream: true,
+  store: false,
+  instructions: SIDE_REQUEST_INSTRUCTIONS,   // a fixed system prompt, see source
+  tools,
+};
+
+const stream = await client.responses.create(requestParams, { signal });
+```
+
+This is the **official OpenAI Node SDK**, so `client.responses.create()`
+literally POSTs to `{baseURL}/responses` with that JSON body and reads back
+an SSE stream — there is no DashScope-specific SDK involved at all. Anything
+speaking real OpenAI Responses-API SSE syntax at that path, with these two
+extra tool types, is protocol-compatible.
+
+**What the client parses out of the SSE stream** (lines 359–509): event types
+`response.output_item.added`, `response.output_item.done`,
+`response.output_text.delta`, and terminal `response.completed` /
+`.failed` / `.incomplete` / `.cancelled`, each carrying a `response` object
+with `output: WsOutputItem[]` and `usage.x_tools.{web_search,web_extractor}.count`.
+Output items it understands: `web_search_call` (`action.query`/`action.queries`,
+`action.sources[].url`, `status`), `web_extractor_call` (`urls`, `goal`,
+`output`, `status`), and `message` (`content[].text`). It also defensively
+handles a DashScope-specific quirk: **request-level failures arrive as a bare
+SSE `event:error` with `{code, message, request_id}` and no `type`/`error`
+wrapper** — the OpenAI SDK doesn't recognize this shape, so qwen-code parses
+it itself (comment: "probe-verified"). Any replacement backend needs to emit
+exactly these item/event shapes, or qwen-code's parser silently ignores
+unrecognized item types and ultimately reports
+`WEB_SEARCH_NO_SEARCH_PERFORMED` (it treats zero `web_search_call` items as
+"no search happened," with one retry, before failing outright — see lines
+883–906).
+
+**The hard gate — this is the actual finding.** Before any request is sent,
+`evaluateWebSearchGate()` (lines 169–335) validates the resolved `baseUrl`
+through `classifyDashScopeBaseUrl()` (lines 122–157):
+
+```ts
+function classifyDashScopeBaseUrl(baseUrl: string): DashScopeBaseUrlIssue | null {
+  const url = new URL(baseUrl);
+  if (url.protocol !== 'https:') return 'insecure';
+  const hostname = url.hostname.toLowerCase();
+  const suffixes = [
+    ...DASHSCOPE_REGIONAL_HOSTS,   // dashscope.aliyuncs.com, dashscope-intl.aliyuncs.com, dashscope-us.aliyuncs.com
+    'maas.aliyuncs.com',
+    'alibaba-inc.com',
+    'aliyun-inc.com',
+  ];
+  return suffixes.some(s => hostname === s || hostname.endsWith('.' + s)) ? null : 'unknown-host';
+}
+```
+
+`DASHSCOPE_REGIONAL_HOSTS` is defined in
+`packages/core/src/core/openaiContentGenerator/provider/dashscope.ts` as
+exactly `['dashscope.aliyuncs.com', 'dashscope-intl.aliyuncs.com', 'dashscope-us.aliyuncs.com']`.
+
+**This means "DashScope-compatible" is not a protocol claim you can satisfy
+by implementing the right JSON shapes — it is a literal hostname allowlist
+checked client-side, before the request is even built.** A self-hosted server
+at `http://search.home`, `https://proxy-ai.home`, or any hostname you control
+will be rejected with *"WebSearch ... is not a DashScope-compatible
+endpoint"* regardless of what protocol it speaks, unless its hostname ends in
+one of `dashscope.aliyuncs.com`, `dashscope-intl.aliyuncs.com`,
+`dashscope-us.aliyuncs.com`, `*.maas.aliyuncs.com`, `*.alibaba-inc.com`, or
+`*.aliyun-inc.com` — domains Alibaba owns, that you cannot obtain a valid TLS
+certificate for. (There's also a separate, unrelated `DASHSCOPE_PROXY_BASE_URL`
+env var used by the *main* content generator's provider-detection code
+(`dashscope.ts` lines 244–262) for header/cache-control routing through a
+corporate proxy — it is not consulted by `classifyDashScopeBaseUrl()` at all,
+so it does not help here either.)
+
+The only way around this specific check is to **fork qwen-code and delete or
+relax `classifyDashScopeBaseUrl()`** — it's ~15 lines of open-source
+TypeScript, so this is not hard *code-wise*, but it means running a patched
+build of the CLI, not configuring the stock release.
+
+## 3. Any self-hostable server implementing this surface today? — No
+
+Checked the servers this task named:
+
+- **vLLM**: has a real `/v1/responses` implementation
+  (https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/responses/), and
+  for `gpt-oss` models specifically supports a **built-in `browser` tool**
+  with a pluggable, MCP-compliant external tool server in place of the
+  default Exa-backed reference implementation
+  (https://vllm.ai/blog/2025-08-05-gpt-oss;
+  https://github.com/vllm-project/recipes/blob/main/OpenAI/GPT-OSS.md). This
+  is the closest existing building block found — but it's gpt-oss/harmony
+  specific (not Qwen), and its tool/event shapes (`browser.search`,
+  `browser.open` harmony-channel messages) are **not** the same as DashScope's
+  `web_search_call`/`web_extractor_call` items qwen-code's parser expects, so
+  it is not drop-in — it would need a translation shim in front, at which
+  point you're building the shim anyway and don't need vLLM in the path.
+- **SGLang**: Responses API support is unclear/inconsistent per its own
+  issue tracker (https://github.com/sgl-project/sglang/issues/10038) — no
+  usable built-in web-search tool found.
+- **LiteLLM**: does expose `/v1/responses`, but has an **open bug**
+  rejecting the `web_search` tool type outright — "LiteLLM raises a
+  validation error... only `web_search_preview` is currently allowed"
+  (https://github.com/BerriAI/litellm/issues/14011). Its actual SearXNG
+  integration is the unrelated standalone `/v1/search` REST endpoint already
+  documented in `docs/research/litellm-searxng-search.md` (§1–3 there) — a
+  sibling API to chat/responses, not a Responses-API `tools:[{"type":"web_search"}]`
+  handler. It doesn't have a DashScope-mode either
+  (https://docs.litellm.ai/docs/providers/dashscope is a plain client wrapper
+  that calls the real dashscope.aliyuncs.com; nothing in it hosts a
+  DashScope-shaped server).
+- **LocalAI / Ollama**: no Responses API or DashScope-compatible mode found
+  in searches for either.
+- **A generic "OpenAI Responses API" self-hosted shim that could be relabeled**:
+  the closest match found, `teabranch/open-responses-server` (185 stars, 161
+  commits, wraps Ollama/vLLM as a Responses API with MCP support), **does not
+  implement `web_search` at all** — its own roadmap lists "Web search: crawl4ai"
+  as a *future* item, not shipped (verified live against the repo,
+  2026-09-05). No other candidate turned up in repeated GitHub searches for
+  "dashscope emulator/mock/fake server" or "responses api web_search
+  self-hosted."
+
+**Conclusion for §3: nothing installable off the shelf implements the
+DashScope Responses API's `web_search`/`web_extractor` hosted-tool surface.**
+Building it means writing your own small SSE server (see §5 sizing).
+
+## 4. Is DashScope's shape "OpenAI Responses API + web_search" reused wholesale?
+
+Yes, confirmed directly from source, not inference: qwen-code's client uses
+the **official `openai` npm package**'s `client.responses.create()` against a
+DashScope `baseURL` (§2 above) — it is not a DashScope-specific SDK or
+protocol. OpenAI's own Responses API supports a hosted `web_search_preview`
+tool with a similar `output[].type === "web_search_call"` item shape
+(OpenAI's public Responses API docs, referenced but not independently
+re-fetched here since qwen-code's source is authoritative for what it
+actually calls). DashScope's extension is the tool *name* (`web_search`
+rather than `web_search_preview` — the exact naming mismatch LiteLLM's own
+open bug in §3 stumbles on) plus the additional `web_extractor` tool and the
+`x_tools` usage-accounting field. No existing "OpenAI Responses API shim"
+project was found that already emulates `web_search_preview`/`web_search`
+server-side against a pluggable backend (see §3) — the two hosted-tool
+ecosystems (OpenAI's and DashScope's) both currently require literally
+calling out to the vendor's own cloud; nobody has open-sourced a
+self-hosted stand-in for either.
+
+## 5. LiteLLM specifically, re-examined against this exact requirement
+
+`docs/research/litellm-searxng-search.md` already established SearXNG is a
+first-class LiteLLM `search_provider` behind the **standalone** `/v1/search`
+REST endpoint (its own §1–2). That endpoint is irrelevant to qwen-code's
+`tools.webSearch.model` gate: qwen-code doesn't call an arbitrary search REST
+endpoint, it calls `POST {baseUrl}/responses` on an **OpenAI-SDK client**
+with `tools:[{type:"web_search"}]`, and gates `baseUrl` on the Alibaba
+hostname allowlist in §2. Even ignoring the hostname gate entirely (i.e.
+assuming a patched qwen-code build), LiteLLM's `/v1/responses` route
+currently **rejects** the `web_search` tool type per the open bug in §3 — so
+today, LiteLLM cannot terminate this request even as an internal component of
+a custom build. Nothing here changes the litellm-searxng-search.md
+recommendation; it remains correct and unrelated to this question.
+
+## 6. Effort assessment and recommendation
+
+**Option A — patch qwen-code + hand-roll a DashScope-Responses-shaped SSE
+server in front of SearXNG.** What it needs, concretely:
+1. Fork qwen-code, delete/relax `classifyDashScopeBaseUrl()` (§2) — trivial,
+   but means building and distributing a patched CLI, and re-patching on every
+   upstream update that touches this file or its surrounding gate logic.
+2. Write a small HTTP server exposing `POST /responses` that: accepts the
+   exact request shape in §2, calls SearXNG (`http://search.home`, already
+   reachable per `docs/research/litellm-searxng-search.md`'s `extra_hosts`
+   finding) for results, and streams back SSE events in the precise sequence
+   qwen-code's parser expects (`response.output_item.added` /
+   `.done` with a `web_search_call` item carrying `action.sources[].url`,
+   optionally a `message` item with narrated text, then
+   `response.completed`). No narration/LLM step is strictly required — an
+   empty or templated `message` still satisfies the parser as long as at
+   least one non-`failed` `web_search_call` item exists (§2's "no-search"
+   check only counts search-call items, not narration quality).
+   Realistically a few hundred lines (Node/Python + SSE), a day or so of
+   work plus debugging the exact event ordering, error-shape (`event:error`
+   quirk), and `store`/`instructions` fields the client sends but doesn't
+   strictly require echoing back.
+3. Register this server's URL as a `modelProviders` entry — except the
+   patched hostname check from step 1 is required for step 3 to pass at all,
+   so steps 1 and 2 are both mandatory, not alternatives.
+4. Maintain the fork indefinitely against upstream qwen-code releases.
+
+**Option B — do nothing further.** `docs/research/omniroute-qwen-websearch.md`
+already documents a **verified, working, fully self-hosted** path: OmniRoute's
+own `omniroute_web_search` MCP tool, backed by this stack's SearXNG instance,
+confirmed connected (`qwen mcp list` → Connected) and exercised end-to-end
+(`POST /v1/search` returned real results). This uses qwen-code's *documented,
+supported, unpatched* MCP-server extension point (`mcpServers` in
+`settings.json`) — no fork, no upstream-drift risk, no protocol shape to
+maintain.
+
+**Recommendation: do not build Option A.** The built-in `web_search` tool's
+"DashScope-compatible" requirement is, by design in qwen-code's own source, a
+hostname allowlist for Alibaba's cloud — it is not a compatibility surface
+meant to be reimplemented, and no one else has reimplemented it either (§3).
+Satisfying it self-hosted requires forking and permanently maintaining a
+patch to code whose only purpose is to *stop* you from doing that. The MCP
+path in `omniroute-qwen-websearch.md` already delivers the same end-user
+capability (web search, backed by this stack's own SearXNG, no external
+API) through qwen-code's actual supported extension point, with zero ongoing
+fork-maintenance burden. There is no functional gap Option A would close that
+Option B doesn't already close today.
+
+## Open questions / unknowns
+
+- Whether `DASHSCOPE_REGIONAL_HOSTS` or the extra suffixes
+  (`maas.aliyuncs.com`, `alibaba-inc.com`, `aliyun-inc.com`) ever change
+  across qwen-code releases — checked only against the current `main` branch
+  (fetched 2026-09-05); a future release could tighten or loosen this list.
+- Whether OpenAI's own `web_search_preview` Responses-API tool has a
+  publicly documented exact request/response JSON schema identical enough to
+  DashScope's `web_search`/`web_extractor` pair that a single shim could serve
+  both — not independently verified against OpenAI's own docs in this pass;
+  qwen-code's source (§2) is authoritative for the DashScope side only.
+- Whether `teabranch/open-responses-server`'s planned "Web search: crawl4ai"
+  roadmap item, if shipped, would end up emitting DashScope-shaped
+  `web_search_call` items or OpenAI-shaped `web_search_preview` ones — could
+  become relevant later but is speculative (unshipped) as of this research.
+
+## Sources
+
+- https://qwenlm.github.io/qwen-code-docs/en/developers/tools/web-search/ and
+  https://raw.githubusercontent.com/QwenLM/qwen-code/main/docs/developers/tools/web-search.md
+  — current built-in-tool vs. MCP options, settings keys, migration note.
+- `packages/core/src/tools/web-search.ts`,
+  `packages/core/src/core/openaiContentGenerator/constants.ts`,
+  `packages/core/src/core/openaiContentGenerator/provider/dashscope.ts` —
+  fetched directly from `QwenLM/qwen-code`'s `main` branch via
+  `raw.githubusercontent.com` on 2026-09-05; ground truth for the request
+  shape, SSE parsing, and the hostname gate (§2).
+- https://github.com/QwenLM/qwen-code/issues/3841 — prior (closed,
+  "not planned") community proposal for DashScope `enable_search` passthrough;
+  shows the feature that eventually shipped took a different path (Responses
+  API, not Chat Completions `enable_search`).
+- https://www.alibabacloud.com/help/en/model-studio/qwen-api-via-openai-responses
+  and https://www.alibabacloud.com/help/en/model-studio/compatibility-with-openai-responses-api
+  — Alibaba's own Responses API docs: endpoint, `tools` shape, `output` item
+  shape.
+- https://docs.qwencloud.com/developer-guides/tool-calling/web-search —
+  the older Chat-Completions `enable_search` mechanism and its
+  no-citations limitation.
+- https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/responses/,
+  https://vllm.ai/blog/2025-08-05-gpt-oss,
+  https://github.com/vllm-project/recipes/blob/main/OpenAI/GPT-OSS.md — vLLM's
+  Responses API and gpt-oss browser-tool/tool-server support.
+- https://github.com/sgl-project/sglang/issues/10038 — SGLang Responses API
+  support unclear.
+- https://github.com/BerriAI/litellm/issues/14011 — LiteLLM's `/v1/responses`
+  rejects the `web_search` tool type.
+- https://docs.litellm.ai/docs/providers/dashscope — LiteLLM's DashScope
+  provider is a plain client wrapper, no Responses API, no web_search.
+- https://github.com/teabranch/open-responses-server — closest
+  "self-hosted Responses API shim" found; web_search not implemented
+  (roadmap item only), checked live 2026-09-05.
+- `G:\_DEV\repos\LLM-Server\docs\research\omniroute-qwen-websearch.md` —
+  the already-working, verified self-hosted alternative this doc is weighed
+  against.
+- `G:\_DEV\repos\LLM-Server\docs\research\litellm-searxng-search.md` —
+  LiteLLM's actual (unrelated) SearXNG integration, re-confirmed as
+  orthogonal to this question in §5.
+
+## Tried it live (2026-09-05) — confirmed empirically, plus one new fact
+
+The user asked to actually run the experiment rather than stop at the analysis above.
+
+**What was done** (all local to the WSL install, reverted afterward — nothing in
+this repo or the live OmniRoute instance was left changed):
+- Patched the installed CLI file
+  `~/.local/lib/qwen-code/lib/chunks/web-search-K2FMOGS5.js` with a one-line
+  bypass in `classifyDashScopeBaseUrl()`: `if (baseUrl.includes("proxy-ai.home")) return null;`
+- Added a `tools.webSearch` block to `~/.qwen/settings.json` pointing
+  `model`/`baseUrl` at a new `qwen-experiment-websearch` `modelProviders` entry
+  using OmniRoute's existing `http://proxy-ai.home/v1` and the already-working
+  `OMNIROUTE_API_KEY`.
+- Ran `qwen` with a prompt forcing use of the built-in `web_search` tool.
+
+**Result — the client-side gate bypass worked**, confirming the research's
+read of `classifyDashScopeBaseUrl()` was accurate: qwen accepted the OmniRoute
+host as "DashScope-compatible" and attempted the tool call. It stopped at an
+interactive approval prompt first (expected — headless auto-approve wasn't
+attempted, since that flips on unrestricted auto-execution of every tool call
+at process privilege, not just this one).
+
+**New fact, not visible from static docs alone**: a direct `curl -X POST
+http://proxy-ai.home/v1/responses` (with a valid key, matching the request
+shape qwen would send) returned `{"error":{"message":"No active credentials
+for provider: codex.","type":"authentication_error","code":"invalid_api_key"}}`
+— **not** the generic "unknown route" error a nonexistent path returns (verified
+earlier in this same research thread against `/v1/search`-adjacent bogus
+paths). So `/v1/responses` **is a real, implemented OmniRoute route**, not
+merely undocumented — the earlier inference that it didn't exist was wrong;
+it exists but is hardcoded to proxy exclusively through a specific provider
+connection OmniRoute's catalog calls `codex`.
+
+**`codex` identified via `PROVIDER_REFERENCE.md`**: `id: codex`, alias `cx`,
+name "OpenAI Codex", **auth type: OAuth** — a real, personal
+ChatGPT/OpenAI-account connection, not a free/no-auth scraper provider like
+several others already connected in this instance (`felo-web`,
+`duckduckgo-web`, etc.). Checked `docs/reference/ENVIRONMENT.md` for any
+setting to redirect `/v1/responses` to a different provider — **none
+exists**; there is no `responsesProvider` or equivalent override.
+
+**Why routing isn't configurable, architecturally**: OpenAI's Responses API
+`web_search` is a *hosted* tool — the search executes inside the model
+backend's own infrastructure as part of generating the response, not as a
+client-visible round trip. Confirmed directly against llama.cpp's own
+`tools/server` docs (`github.com/ggml-org/llama.cpp/tree/master/tools/server`):
+it implements only `/v1/chat/completions` with client-side tool-calling
+(the model emits a `tool_call`; the *client* must execute it), has no
+`/v1/responses` endpoint, no hosted-tool execution, and its built-in
+`--tools` are local-only (`read_file`, `grep_search`, `exec_shell_command`,
+etc.) — none make outbound HTTP requests. So even with configurable routing,
+pointing `/v1/responses` at the local Qwen model wouldn't work: the upstream
+llama-server has nothing that could serve the hosted-tool half of the
+contract. Building that would mean OmniRoute (or a custom shim) intercepting
+the model's tool-call mid-generation and splicing in a real search — the
+same shim work priced out as not-worth-it earlier in this document, now
+confirmed to be the *only* way, not one option among several.
+
+**Conclusion holds, sharpened**: the dead end isn't just qwen-code's
+client-side hostname check anymore — even a fully self-hosted, hostname-gate-bypassed
+setup terminates at OmniRoute's `codex`-only `/v1/responses` routing, which
+itself terminates at needing a real OpenAI/ChatGPT OAuth account, which is
+exactly the kind of external paid dependency this whole line of inquiry was
+trying to avoid. `omniroute_web_search` via MCP (already working, already
+free, already self-hosted) remains the only path that actually satisfies the
+original goal.
+
+**Revert**: both the CLI patch and the `settings.json` changes were reverted
+after the test — `omniroute-search` MCP confirmed still `Connected` via
+`qwen mcp list` afterward. No lasting changes from this experiment.

docs/research/fast-model-choice.md

@@ -0,0 +1,221 @@
+# Which small model to run as the always-resident `fastModel` for qwen-code's Auto Mode classifier?
+
+**Date:** 2026-09-06
+**Budget:** ≤7GB VRAM, resident concurrently alongside the existing Qwen3.8-27B instance on the single
+32GB R9700, via the same `llama.cpp:server-rocm` image already in `docker-compose.yml`.
+**Answer: Qwen3-4B-Instruct-2507, Q8_0 GGUF (~4.3GB weights).** The prior quick pass's tentative pick
+holds up under primary-source verification, for a more specific reason than "same tokenizer family":
+it is the only strong candidate in the shortlist that is *architecturally* non-thinking (no `<think>`
+code path exists at all, vs. models that are thinking-by-default and rely on a per-call
+`enable_thinking:false` toggle that llama.cpp does not cleanly expose). It does carry one directly
+relevant, documented llama.cpp bug — but that bug is closed, has a one-flag workaround, and is
+strictly less severe than the still-open Qwen3.5/Qwen3.8-lineage bugs already documented against the
+27B model in this repo.
+
+## 1. What the classifier actually needs (grounding the requirement)
+
+Per qwen-code's own docs, Auto Mode's permission gate is a two-stage LLM classifier:
+
+- **Stage 1** — outputs only `{ shouldBlock: bool }`, ~300ms budget, thinking already disabled at the
+  request level. If `shouldBlock` is `false`, the action proceeds immediately.
+- **Stage 2** — only runs when Stage 1 blocks; uses chain-of-thought review to downgrade false
+  positives, ~3-5s budget.
+- Both stages use "your configured fast model (`/model --fast`)"; if none is configured, the full
+  session model is used instead — which is the current, too-slow state this second model is meant to
+  fix.
+
+Source: [Qwen Code docs — Auto Mode](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/),
+[QwenLM/qwen-code docs/users/features/auto-mode.md](https://github.com/QwenLM/qwen-code/blob/main/docs/users/features/auto-mode.md).
+
+A live qwen-code issue independently confirms the exact failure mode this repo already hit with the
+27B model — a model *thinking* inside the classifier path is a first-order latency problem, not a
+nice-to-have to tune later:
+
+> "for a latency-sensitive permission gate, thinking should be disabled in every stage" — enabling it
+> "makes the review path slower and more expensive, which directly worsens the timeout problem."
+
+That issue (timeouts tripping on slow inference) was closed by a PR that both loosened the stage
+timeout budgets *and* moved toward disabling thinking everywhere in the classifier.
+Source: [QwenLM/qwen-code issue #4676](https://github.com/QwenLM/qwen-code/issues/4676).
+
+Takeaway for model selection: the request-level "don't think" instruction already exists in
+qwen-code's own classifier code. What matters is whether the **model + llama.cpp combination actually
+honors it reliably** — which is precisely where the 27B model failed (see
+[`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md)) and where several shortlist candidates
+have their own version of the same problem.
+
+## 2. Candidates evaluated against primary sources
+
+| Model | Params | GGUF size (quant) | Context | License | Thinking behavior | Tool-calling | Verdict |
+|---|---|---|---|---|---|---|---|
+| **Qwen3-4B-Instruct-2507** | 4B | Q4_K_M 2.5GB / **Q8_0 4.28GB** | 262,144 native | Apache 2.0 | **Non-thinking only** — model card states it "does not generate `<think></think>` blocks in its output," full stop, no toggle needed | Yes, native `<tool_call>` format, BFCL-v3 61.9 | **Recommended** |
+| Qwen3-1.7B | 1.7B | ~1.1GB (Q4_K_M, typical) | 32,768 | Apache 2.0 | Thinking **on by default**; needs `enable_thinking:false` per call | Yes | Rejected — see §3 |
+| Qwen3-0.6B | 0.6B | ~0.4GB (Q4_K_M) | 32,768 | Apache 2.0 | Thinking on by default, same toggle issue as 1.7B | Yes, but weakest reasoning of the family | Rejected — undersized for reliability at this size, same toggle risk |
+| Llama-3.2-3B-Instruct | 3B | ~2GB (Q4_K_M, typical) | 128K | Llama 3.2 Community License — commercial use allowed, but text/EU carve-out language and an explicit >700M-MAU re-licensing clause | No thinking mode | Not natively documented on the model card fetched (no tool-call format called out) | Deprioritized — license has more fine print than Apache 2.0 for no clear benefit here |
+| Gemma-3-4b-it | 4B | Q4_K_M 2.49GB / Q8_0 4.13GB | 128K | Custom "Gemma" license (Google usage terms) | No documented thinking mode | Not documented on the model card fetched | Deprioritized — no confirmed native tool-calling story, non-Apache license |
+| Phi-4-mini-instruct | 3.8B | Q4_K_M 2.49GB / Q8_0 4.08GB | 128K | **MIT** | Not a reasoning model (that's the separate Phi-4-mini-**reasoning** model); no `<think>` tags by default | Yes — documented function-call format with dedicated tokens | Credible alternative — see §4 |
+| SmolLM3-3B | 3B | Q4_K_M ~1.9GB (typical) | 128K (64K trained + YaRN) | Apache 2.0 | **Thinking on by default** (`enable_thinking`), toggled via system-prompt flags | Yes (XML or Python-style tool calls) | Rejected — same thinking-by-default risk as Qwen3-1.7B |
+| Ministral-8B-Instruct-2410 | 8B | too large for budget at any useful quant with headroom | 128K | **Mistral Research License — commercial use requires contacting Mistral for a separate license** | Not documented as a reasoning model | Yes, documented function-calling with benchmark (31.6 vs Mistral-7B's 6.9) | Rejected — license restricts this repo's own dev-tooling use without contacting Mistral; also parameter count crowds the 7GB budget once Q8_0 + KV cache is counted |
+
+Sources (fetched directly from each model's own HF card / GGUF repo unless noted):
+[Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507),
+[unsloth/Qwen3-4B-Instruct-2507-GGUF](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF),
+[Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B),
+[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B),
+[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct),
+[google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it),
+[bartowski/google_gemma-3-4b-it-GGUF](https://huggingface.co/bartowski/google_gemma-3-4b-it-GGUF),
+[microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct),
+[bartowski/microsoft_Phi-4-mini-instruct-GGUF](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF),
+[HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B),
+[mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410).
+
+**Confidence note:** file sizes for Qwen3-1.7B/0.6B, Llama-3.2-3B, and SmolLM3-3B GGUF quants above are
+typical/approximate — those repos weren't individually re-verified against a specific GGUF file tree
+since all three families were eliminated on architectural grounds (§3) before size mattered. Sizes for
+the two models actually compared head-to-head (Qwen3-4B-Instruct-2507, Phi-4-mini-instruct) and
+Gemma-3-4b-it were pulled directly from each quantizer's own repo page.
+
+## 3. Why "thinking-by-default + per-call toggle" is disqualifying, not just a minor ding
+
+This is the deciding architectural distinction, and it's exactly the failure this second model exists
+to avoid. Qwen's own llama.cpp docs page states the toggle problem directly:
+
+> "the hard switch implemented in the chat template is not exposed in llama.cpp" for controlling
+> `enable_thinking` — the documented workaround is to supply "a custom chat template equivalent to
+> always `enable_thinking=False`" via `--chat-template-file`.
+Source: [Qwen — Run with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html).
+
+That means for Qwen3-1.7B, Qwen3-0.6B, and SmolLM3-3B — all thinking-on-by-default — reliably
+suppressing the reasoning phase in this llama.cpp/ROCm stack is not a request-body flag away; it needs
+a hand-maintained custom chat template file, which is exactly the kind of fragile, easy-to-silently-
+regress setup this task is trying to get away from (the 27B model's whole problem was reasoning_content
+being consumed before the answer). Qwen3-4B-Instruct-2507 has no such toggle to maintain in the first
+place — the model card states the non-thinking behavior as an unconditional property of the model, not
+a configurable default that has to be forced correctly on every request. This is a stronger claim than
+"same tokenizer family as the 27B" (the original quick-pass's reasoning) and is the actual basis for
+the recommendation.
+
+## 4. The one documented risk specific to Qwen3-4B-Instruct-2507 — and why it doesn't change the pick
+
+llama.cpp has its own closed, dated bug where server builds around **b8429** (March 2026)
+mis-detected Qwen3-Instruct-2507 models — the 4B included by name in the reporter's repro command — as
+thinking models, routing tool-call output into `reasoning_content` instead of `tool_calls`:
+
+> "llama.cpp b8429 incorrectly detects Qwen3-Instruct-2507 models as thinking models (`thinking = 1`).
+> This causes tool calls to be captured as `reasoning_content` instead of being parsed into the
+> `tool_calls` array."
+
+The documented, confirmed-working workaround is a single server flag:
+
+```
+llama-server -hf unsloth/Qwen3-4B-Instruct-2507-GGUF:Q4_K_M --jinja --port 8222 --reasoning off
+```
+
+which restores `thinking = 0` and correct `finish_reason: tool_calls` output. The issue is **closed**.
+Source: [ggml-org/llama.cpp issue #20809](https://github.com/ggml-org/llama.cpp/issues/20809).
+
+This is worth flagging honestly against the recommendation, but it's materially different from the
+open, only-partially-fixed Qwen3.5/Qwen3.8-lineage parser bugs already documented in this repo's
+[`qwen3.8-27b-tool-calling.md`](qwen3.8-27b-tool-calling.md) (issues #21158, #20837 — both open at time
+of that research): this is a llama.cpp *server-side misdetection* bug with a one-flag fix, not an
+unresolved upstream grammar/parser defect in the Qwen3.5 architecture family itself. Concretely: add
+`--reasoning off` to this second llama-server instance's command regardless — it's a no-cost safety net
+whether or not the current `ghcr.io/ggml-org/llama.cpp:server-rocm` build still has the bug, and it
+directly targets the exact failure mode (reasoning_content eating the completion) that ruled out the
+27B model for this role in the first place.
+
+## 5. VRAM math for the classifier role specifically
+
+Qwen3-4B-Instruct-2507 is a plain (non-hybrid) transformer — every layer is standard GQA attention, so
+unlike the 27B model's Gated-DeltaNet hybrid, KV cache scales with *all* layers, not a fraction of them.
+From the model's own `config.json`:
+
+- `num_hidden_layers`: 36, `num_key_value_heads`: 8, `head_dim`: 128
+Source: [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json).
+
+Per-token KV cache (fp16, both K and V):
+`36 layers × 2 (K+V) × 8 kv_heads × 128 head_dim × 2 bytes = 144 KiB/token`
+
+The classifier transcript is bounded by design — qwen-code's own two-stage design keeps Stage 1 to a
+`{shouldBlock}`-only judgment and Stage 2 to a chain-of-thought review of one blocked action, not an
+open-ended agent session — so a context window in the low thousands of tokens is generous headroom,
+not a tight fit:
+
+| Context | KV cache (fp16) | KV cache (q8_0, `--cache-type-k/v q8_0`) | Weights (Q8_0) | Total (q8_0 KV) | Headroom under 7GB |
+|---|---|---|---|---|---|
+| 4,096 tokens | ~0.56 GB | ~0.28 GB | 4.28 GB | **~4.56 GB** | ~2.4 GB |
+| 8,192 tokens | ~1.13 GB | ~0.56 GB | 4.28 GB | **~4.84 GB** | ~2.2 GB |
+| 32,768 tokens (generous ceiling) | ~4.5 GB | ~2.25 GB | 4.28 GB | **~6.53 GB** | ~0.5 GB (tight) |
+
+At any context length actually needed for a permission-gate classifier (thousands, not tens of
+thousands, of tokens), Q8_0 weights plus q8_0 KV cache comfortably clears the 7GB ceiling with headroom
+to spare for the compute buffer and batch overhead — matching the same `--cache-type-k q8_0
+--cache-type-v q8_0` pattern this repo already uses for the 27B instance. There's no need to drop to
+Q4_K_M (2.5GB) unless a much larger classifier context is anticipated later; Q8_0 is the better default
+here since it's a small model where quantization loss matters proportionally more, and the VRAM budget
+comfortably affords the higher-precision quant.
+
+## 6. What would change the answer
+
+- **If Phi-4-mini-instruct's MIT license matters more than matching the 27B model's tokenizer/template
+  family**, it's a legitimate second choice: confirmed non-thinking by default, confirmed native
+  function-calling format, comparable Q8_0 size (4.08GB), and a license with zero commercial-use fine
+  print (vs. Apache 2.0's still-permissive but slightly more conditional terms). It wasn't picked
+  because it has no llama.cpp-specific tool-calling track record verified in this pass (no equivalent
+  to the issue #20809 workaround search done for it), so its actual reliability on this exact
+  `llama.cpp:server-rocm` stack is less directly evidenced than Qwen3-4B-Instruct-2507's.
+- **If the classifier transcript ever needs to grow well past ~8K tokens routinely**, drop to Q4_K_M
+  (2.5GB) to keep well clear of the 7GB ceiling — the KV-cache math in §5 shows the crossover point.
+- **If llama.cpp's #20809 misdetection turns out to still reproduce** on the exact
+  `ghcr.io/ggml-org/llama.cpp:server-rocm` build this repo pulls, the fix is the one-flag
+  `--reasoning off` workaround already confirmed in that issue — not a reason to pick a different
+  model, since every thinking-capable alternative in this shortlist has an equal-or-worse version of
+  the same class of bug with less clean workarounds (custom chat-template files, per §3).
+
+## Sources
+
+- [Qwen Code docs — Auto Mode](https://qwenlm.github.io/qwen-code-docs/en/users/features/auto-mode/)
+- [QwenLM/qwen-code — docs/users/features/auto-mode.md](https://github.com/QwenLM/qwen-code/blob/main/docs/users/features/auto-mode.md)
+- [QwenLM/qwen-code issue #4676](https://github.com/QwenLM/qwen-code/issues/4676)
+- [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
+- [Qwen/Qwen3-4B-Instruct-2507 config.json](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507/raw/main/config.json)
+- [unsloth/Qwen3-4B-Instruct-2507-GGUF](https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF)
+- [Qwen — Run with llama.cpp](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html)
+- [ggml-org/llama.cpp issue #20809](https://github.com/ggml-org/llama.cpp/issues/20809)
+- [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
+- [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
+- [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
+- [google/gemma-3-4b-it](https://huggingface.co/google/gemma-3-4b-it)
+- [bartowski/google_gemma-3-4b-it-GGUF](https://huggingface.co/bartowski/google_gemma-3-4b-it-GGUF)
+- [microsoft/Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct)
+- [bartowski/microsoft_Phi-4-mini-instruct-GGUF](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF)
+- [HuggingFaceTB/SmolLM3-3B](https://huggingface.co/HuggingFaceTB/SmolLM3-3B)
+- [mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410)
+- [docs/research/qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) (this repo — cross-referenced
+  for the 27B model's own, still-open, tool-calling parser bugs)
+
+## Confidence/uncertainty summary
+
+- **High confidence:** Qwen3-4B-Instruct-2507's non-thinking-only status (direct model-card quote);
+  qwen-code's two-stage classifier timing/design and its use of `/model --fast` (direct docs quote);
+  the existence, exact symptom, and workaround of llama.cpp issue #20809 (direct issue quote); the
+  KV-cache architecture math (computed directly from the model's own `config.json`, same method as the
+  existing `qwen3.8-27b-quant.md` research in this repo).
+- **Medium confidence:** exact GGUF file sizes for Qwen3-1.7B, Qwen3-0.6B, Llama-3.2-3B-Instruct, and
+  SmolLM3-3B — not individually re-verified against a specific quantizer's file tree since these were
+  eliminated on architectural (thinking-toggle) grounds before size became the deciding factor; treat
+  as typical/approximate, not exact.
+- **Low confidence / not independently verified:** whether the current
+  `ghcr.io/ggml-org/llama.cpp:server-rocm` image (pulled fresh) still reproduces issue #20809's
+  misdetection — the issue is closed but no changelog/PR diff was fetched to confirm the underlying
+  detection logic was actually patched vs. the reporter simply adopting the `--reasoning off`
+  workaround. Recommend a live smoke test (send one tool-calling request, confirm the response lands in
+  `tool_calls` not `reasoning_content`, and time a trivial completion) before wiring this model in as
+  the production `fastModel`, the same caveat this repo's `qwen3.8-27b-tool-calling.md` already flags
+  for the 27B model.
+- Whether Phi-4-mini-instruct has a comparably clean llama.cpp tool-calling track record was **not**
+  deep-dived (no issue-tracker search run against it) — it's flagged in §6 as a live alternative rather
+  than fully evaluated, since Qwen3-4B-Instruct-2507's architectural non-thinking guarantee and
+  same-family template consistency with the existing 27B deployment made it the clearer pick without
+  needing that extra research pass.

docs/research/image-generation-model-choice.md

@@ -0,0 +1,220 @@
+# Research: which diffusion model to target with the full ~32GB R9700
+
+**Question:** With Qwen/llama-server fully stopped (per issue #39's premise — see
+map #38), ComfyUI has the whole ~32GB R9700 (ROCm/gfx1201) to itself instead
+of the ~6GB left over during concurrent operation
+(per [`image-generation-options.md`](image-generation-options.md)). Given that
+headroom, should the build stay on FLUX.1-schnell, or move up to
+FLUX.1-dev, SD3.5-large, Qwen-Image, HunyuanImage-3.0, or Krea-2?
+
+**Answer, short version:** move up to **Qwen-Image at FP8 precision**
+(`qwen_image_fp8_e4m3fn.safetensors` diffusion weights +
+`qwen_2.5_vl_7b_fp8_scaled.safetensors` text encoder, ~25GB combined). It's
+the only one of the five upgrade candidates with **direct, hardware-specific
+evidence of running on this exact GPU architecture** (gfx1201/R9700) rather
+than a generic "ComfyUI supports ROCm" inference, it carries the cleanest
+license of the group (Apache-2.0, no revenue threshold, no non-commercial
+clause), and its 20B MMDiT is a real capability step up from schnell's
+distilled 12B (notably for text rendering and prompt adherence), while still
+fitting with real margin inside 32GB.
+
+## Why not just re-derive the schnell/dev/SDXL/SD3.5 findings
+
+`docs/research/image-generation-options.md` already covers, with primary
+sources: ComfyUI's ROCm/gfx1201 story (official AMD docs + RDNA4 blog post +
+community gfx1201 Docker images), FLUX.1-schnell vs FLUX.1-dev vs SDXL vs
+SD3.5 licenses, and FLUX GGUF VRAM figures at the ~6GB-headroom scale. None
+of that is repeated here except where the ~32GB ceiling changes the
+conclusion. This doc adds: FLUX.1-dev/SD3.5 at the *larger* headroom, plus
+three models the prior doc didn't cover at all (Qwen-Image, HunyuanImage-3.0,
+Krea-2).
+
+## Candidate comparison
+
+| Model | License (primary source) | Params | Stated/typical VRAM | ROCm/gfx1201 evidence |
+|---|---|---|---|---|
+| FLUX.1-schnell (current) | Apache-2.0 | 12B | GGUF Q4_K_S ~7GB (per prior doc) | Confirmed on gfx1201 (prior doc) |
+| **Qwen-Image** | **Apache-2.0** | 20B (20.4B DiT + 8.3B Qwen2.5-VL text encoder) | fp8 ~16GB (diffusion) + ~9.4GB (fp8 text encoder) ≈ 25GB total; bf16 needs 24GB+ and "48GB+" per some quant write-ups | **Direct**: [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) ships a pre-validated "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow specifically for the R9700 AI Pro (gfx1201), on a ROCm 7 (TheRock nightlies) toolbox |
+| FLUX.1-dev | [FLUX.1-dev Non-Commercial License](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) — non-commercial weights, outputs usable commercially | 12B | bf16 ~24GB; GGUF Q8 ~12-13GB | Same *family* evidence as schnell (gfx1201 Docker images target FLUX generally), but no R9700-specific FLUX.1-dev report found |
+| SD3.5-large | [Stability Community License](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md) — free under $1M annual revenue | 8B | bf16 ~16GB; 4-bit NF4 fits small GPUs | AMD's own ComfyUI-ROCm doc lists an "SD3.5 Simple" template workflow (per prior doc) — vendor-blessed but not R9700-specific |
+| HunyuanImage-3.0 | [tencent-hunyuan-community license](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0) | 80B total / 13B active (MoE, 64 experts) | Official repo: "≥ 3 × 80GB" VRAM for the base model, "≥ 8 × 80GB" for -Instruct; ~177GB at fp16 | **None, and actively contraindicated**: setup requires CUDA 12.8 + FlashAttention2/FlashInfer, no AMD/ROCm mention anywhere in the official repo |
+| Krea-2 (Turbo) | [Krea 2 Community License](https://www.krea.ai/krea-2-licensing) — free under $1M annual revenue, homelab/personal explicitly covered | 12-13B DiT | No official VRAM figure; community reports (RTX hardware only) cite fp8 ~16GB, GGUF ~12GB | **None found** — released [June 22, 2026 per the HF model card](https://huggingface.co/krea/Krea-2-Turbo); ComfyUI added native support per [blog.comfy.org](https://blog.comfy.org/p/krea-2-open-source-models-are-now), and GGUF quants exist ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), but no R9700/gfx1201-specific report exists yet — too new for that evidence to have accumulated |
+
+## Per-model detail
+
+### Qwen-Image — recommended
+
+- **License**: Apache-2.0, stated directly on the model card, no revenue
+  threshold, no non-commercial clause, no attribution/naming requirement.
+  The cleanest license of every model considered in this doc or the prior
+  one. Source: [Qwen/Qwen-Image on Hugging Face](https://huggingface.co/Qwen/Qwen-Image).
+- **Architecture**: 20B-parameter MMDiT (Multimodal Diffusion Transformer)
+  combined with an 8.3B Qwen2.5-VL text encoder — notably larger and more
+  capable than FLUX.1-schnell's 12B distilled model, particularly for
+  multilingual text rendering and instruction-following, per the official
+  [QwenLM/Qwen-Image GitHub repo](https://github.com/QwenLM/Qwen-Image).
+- **ComfyUI support**: native, not a wrapper/custom-node integration —
+  landed August 2025 per [ComfyUI Wiki's native-support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image),
+  with an official FP8 checkpoint (`qwen_image_fp8_e4m3fn.safetensors`) and
+  FP8-scaled text encoder (`qwen_2.5_vl_7b_fp8_scaled.safetensors`, 9.38GB)
+  published under [Comfy-Org/Qwen-Image_ComfyUI](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders).
+- **VRAM at FP8**: the official Comfy-Org FP8 diffusion checkpoint plus FP8
+  text encoder land around ~25GB combined — comfortably inside the 32GB
+  ceiling with headroom for ComfyUI runtime/VAE overhead, versus 24GB+ (some
+  sources say 48GB+) for the unquantized bf16 model. Sources: community
+  VRAM write-ups aggregated via [Comfy-Org/ComfyUI issue #10852 ("Qwen-image in 24GB VRAM and 32GB RAM")](https://github.com/Comfy-Org/ComfyUI/issues/10852)
+  and the [official ComfyUI Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/) —
+  treat exact GB figures as secondary/community-sourced, consistent with how
+  the prior doc flagged FLUX's GGUF VRAM table.
+- **Direct R9700/gfx1201 evidence (the deciding factor)**:
+  [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) is a
+  Fedora-toolbox ROCm 7 (TheRock nightlies) environment built specifically
+  for the "AMD Radeon 9700 AI PRO (32GB)" and ships a pre-validated
+  "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow
+  (plus a Qwen-Image-Edit 2511 FP8 workflow) as one of only four workflows
+  in the whole repo. This is the only model in this comparison with
+  card-architecture-specific (not just "ComfyUI supports ROCm generically")
+  validation — everything else relies on family-level or vendor-generic
+  ROCm claims. A related community discussion
+  ([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
+  does flag that "Qwen image edit doesn't always work on AMD HIP/ROCm" in
+  some configurations — worth testing the specific workflow before assuming
+  zero friction, but this is a known-quantity, actively-discussed rough edge
+  rather than a documented hard blocker.
+- **4-step Lightning LoRA**: the R9700-validated workflow pairs Qwen-Image
+  with a "Lightning" LoRA for 4-step inference — the same fast-inference
+  pattern FLUX.1-schnell uses, so switching models doesn't have to mean
+  giving up the short GPU-resident-time-per-image property the prior doc
+  called out as valuable for time-sliced use alongside llama-server (even
+  though this ticket's premise is llama-server being stopped, that pattern
+  still helps if the two services are ever run in an overlapping window).
+
+### FLUX.1-dev — solid alternative, not the pick
+
+- License permits non-commercial use of the weights; BFL's own license page
+  states generated outputs are separately usable commercially (already
+  covered in the prior doc; unchanged here). Source:
+  [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).
+- At bf16 (~24GB) or GGUF Q8 (~12-13GB), it fits the 32GB ceiling with room
+  to spare — a legitimate move up from schnell's distilled quality.
+  Source: [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
+  and community Q8 VRAM reports.
+- Loses to Qwen-Image on two counts: license (non-commercial-weights clause
+  vs. Apache-2.0 — not a hard blocker for this homelab per the prior doc's
+  own reasoning, but strictly worse) and hardware evidence (FLUX's gfx1201
+  validation is at the *family* level — the prior doc's `yurisasc/comfyui-rocm-rdna4`
+  and `charlie12345/R9700AIProComfyUIPatch` sources are about running FLUX
+  models on this card generally, not a FLUX.1-dev-specific report the way
+  kyuz0's repo is Qwen-Image-specific).
+
+### SD3.5-large — no longer the best use of the freed headroom
+
+- License unchanged from the prior doc: Stability Community License, free
+  under $1M annual revenue (irrelevant threshold for this homelab). Source:
+  [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
+- At 8B params (bf16 ~16GB), it's the smallest of the upgrade candidates —
+  which mattered when 6GB was the ceiling, but with 32GB available there's
+  no VRAM reason to pick the model the prior doc already flagged as "lower
+  fidelity than FLUX/SD3.5 by current standards" over Qwen-Image or
+  FLUX.1-dev. AMD's own ComfyUI-ROCm docs do list it as a first-party
+  example template ("SD3.5 Simple", per the prior doc), so it remains a fine
+  fallback if Qwen-Image's FP8 path hits the ROCm rough edge noted above.
+
+### HunyuanImage-3.0 — ruled out
+
+- Official repo states VRAM requirements of "≥ 3 × 80GB" for the base model
+  and "≥ 8 × 80GB" for HunyuanImage-3.0-Instruct — i.e. multi-GPU
+  datacenter-class NVIDIA clusters, not a single 32GB consumer/workstation
+  card at any precision. Source:
+  [Tencent-Hunyuan/HunyuanImage-3.0 GitHub repo](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0).
+- Setup instructions require CUDA 12.8, PyTorch 2.8.0 built for CUDA, and
+  optionally FlashAttention2/FlashInfer for MoE routing speed — no AMD or
+  ROCm path is mentioned anywhere in the official repo. Even the
+  MoE-efficient "13B active" framing doesn't help here: the tooling itself
+  assumes an NVIDIA multi-GPU cluster, and the 80B total parameter set still
+  has to be resident somewhere.
+- 32GB of headroom on one AMD card doesn't move this model into reach at
+  any precision considered here; it's excluded regardless of how much VRAM
+  frees up on this specific box.
+
+### Krea-2 (Turbo) — promising, but unverified on this hardware
+
+- Verified directly against primary sources per the ticket's instruction
+  (this is a June 2026 release, past most training cutoffs): the
+  [Hugging Face model card](https://huggingface.co/krea/Krea-2-Turbo) states
+  a release date of **June 22, 2026**, a 12-billion-parameter single-stream
+  diffusion transformer, `torch.bfloat16` as the reference precision, and
+  the **Krea 2 Community License**.
+- License, per [krea.ai/krea-2-licensing](https://www.krea.ai/krea-2-licensing):
+  non-commercial (including explicitly personal/homelab) use is free;
+  commercial use is permitted royalty-free for entities under $1M
+  trailing-12-month revenue (same shape as SD3.5's and matching this
+  homelab's use case); content-filter and AI-disclosure obligations apply if
+  deployed publicly; derivative model names must start with "Krea".
+- ComfyUI added native support for both open-weight checkpoints (Krea 2 Raw
+  and Krea 2 Turbo) per [blog.comfy.org's announcement](https://blog.comfy.org/p/krea-2-open-source-models-are-now),
+  and community GGUF quants already exist
+  ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), with
+  reports (RTX hardware only) of fp8 fitting 16GB and GGUF fitting 12GB.
+- **No AMD/ROCm mention anywhere** in the model card, and no gfx1201/R9700
+  community report was found — unsurprising given the model is roughly
+  2.5 months old at the time of this research. Its architecture (a
+  standard-shaped DiT that ComfyUI loads through its normal diffusion-model
+  nodes, per the ComfyUI blog post) gives reasonable expectation it will run
+  on the same ROCm/PyTorch backend already proven for FLUX and Qwen-Image on
+  this card, but that's an inference, not a verified fact the way
+  kyuz0's Qwen-Image workflow is.
+- **Not the pick today**, precisely because Qwen-Image already offers a
+  hardware-verified path at a comparable parameter count and VRAM budget.
+  Worth a follow-up research ticket once R9700/gfx1201-specific Krea-2
+  reports exist — the license and ComfyUI support are both already in
+  place, so the only open question is real-world ROCm behavior.
+
+## Recommendation
+
+**Qwen-Image, FP8 precision** (`qwen_image_fp8_e4m3fn.safetensors` +
+`qwen_2.5_vl_7b_fp8_scaled.safetensors`, ~25GB combined), optionally paired
+with the 4-step Lightning LoRA the R9700-specific validated workflow uses.
+It wins on all three axes the ticket asked about:
+
+1. **License**: Apache-2.0 — no restriction at all, strictly better than
+   every other candidate including the current FLUX.1-schnell pick.
+2. **ROCm/gfx1201 compatibility**: the only candidate with a workflow
+   pre-validated specifically on this GPU architecture
+   ([kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy)),
+   not just "ComfyUI supports ROCm in general."
+3. **VRAM at the ~32GB ceiling**: ~25GB at FP8 leaves real margin for
+   ComfyUI runtime/VAE overhead, without needing the multi-step,
+   non-distilled FLUX.1-dev's full 24GB bf16 footprint or accepting
+   SD3.5's lower fidelity ceiling — and it's a genuine capability upgrade
+   over schnell (20B vs. 12B, non-distilled-quality text rendering) rather
+   than just a bigger file.
+
+If the known Qwen-Image/ROCm edit-mode rough edge
+([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
+turns out to affect plain text-to-image generation too, SD3.5-large (AMD's
+own first-party "SD3.5 Simple" ComfyUI-ROCm template workflow) is the
+fallback, with FLUX.1-dev as a second option. HunyuanImage-3.0 is excluded
+outright regardless of available VRAM (CUDA-only tooling, multi-GPU
+datacenter VRAM floor). Krea-2 is worth revisiting once R9700-specific
+field reports exist.
+
+## Sources consulted
+
+- [docs/research/image-generation-options.md](image-generation-options.md) (this repo — prior findings, not re-derived)
+- [Qwen/Qwen-Image (Hugging Face)](https://huggingface.co/Qwen/Qwen-Image)
+- [QwenLM/Qwen-Image (GitHub)](https://github.com/QwenLM/Qwen-Image)
+- [ComfyUI Wiki — Qwen-Image native support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image)
+- [Comfy-Org/Qwen-Image_ComfyUI (Hugging Face, FP8 checkpoints)](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders)
+- [Comfy-Org/ComfyUI issue #10852 — Qwen-Image VRAM](https://github.com/Comfy-Org/ComfyUI/issues/10852)
+- [ComfyUI official Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/)
+- [kyuz0/amd-r9700-comfy (R9700-specific ROCm ComfyUI toolbox)](https://github.com/kyuz0/amd-r9700-comfy)
+- [pollockjj/ComfyUI-MultiGPU discussion #133 (Qwen-Image-Edit ROCm rough edge)](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133)
+- [black-forest-labs/FLUX.1-dev (Hugging Face) + LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev)
+- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
+- [stabilityai/stable-diffusion-3.5-large (Hugging Face) + LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)
+- [Tencent-Hunyuan/HunyuanImage-3.0 (GitHub)](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0)
+- [krea/Krea-2-Turbo (Hugging Face)](https://huggingface.co/krea/Krea-2-Turbo)
+- [Krea 2 Community License Agreement (krea.ai)](https://www.krea.ai/krea-2-licensing)
+- [blog.comfy.org — Krea 2 open-source models in ComfyUI](https://blog.comfy.org/p/krea-2-open-source-models-are-now)
+- [molbal/krea2-gguf (Hugging Face)](https://huggingface.co/molbal/krea2-gguf)

docs/research/image-generation-options.md

@@ -0,0 +1,249 @@
+# Research: adding local image generation to the stack
+
+**Question:** What's the best way to add local image generation alongside
+the existing Qwen3.8-27B / llama.cpp text stack, given a single AMD Radeon
+R9700 (32GB VRAM, ROCm/gfx1201 — not CUDA), routed through the OmniRoute
+gateway on the `ai-stack` Docker network?
+
+**Answer, short version:** run **ComfyUI** (official AMD-blessed ROCm
+Docker path exists, and OmniRoute already has a first-class `comfyui`
+provider — no bespoke API wrapper needed) with **FLUX.1 [schnell]**
+(Apache-2.0, 4-step, GGUF-quantizable) as the default model, falling back to
+**SDXL** for anything schnell's distilled-step license/quality tradeoffs
+don't suit. VRAM headroom against the current llama-server footprint is too
+tight for both to be resident at once at any real image quality — plan for
+**time-sliced use** (llama-server's existing lazytainer stop-on-idle pattern,
+mirrored for the image-gen service, or a manual "stop one, start the other"
+toggle), not concurrent operation.
+
+## Current VRAM baseline (from this repo)
+
+Per `docker-compose.yml` and `.env.example`, llama-server runs
+`Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights) at `--ctx-size 262144` with
+`--cache-type-k q8_0 --cache-type-v q8_0`, landing at **~25.6 GB** total
+(weights + q8_0 KV cache), leaving **~6 GB** free on the 32GB card — this
+matches the math already recorded in
+[`docs/research/qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md). Per the auto-memory
+note on this repo, real measured VRAM use has run closer to ~75% (~24 GB) in
+practice versus the theoretical estimate, which doesn't change the
+conclusion below but means the ~6 GB figure is closer to a ceiling than a
+comfortable number.
+
+**Implication:** 6 GB is not enough for any current-generation image model at
+usable quality (see VRAM table below — even the smallest practical FLUX
+quant wants ~7 GB alone, before ComfyUI's own runtime/VAE overhead). Running
+image-gen *concurrently* with llama-server resident is not realistic on this
+card. The two need to time-share the GPU, not split it.
+
+## Backend evaluation (ROCm support, checked against primary sources)
+
+### ComfyUI — recommended
+
+- **Official AMD ROCm docs exist and are current.** AMD's own ROCm docs site
+  hosts a dedicated ComfyUI install guide with a prebuilt Docker image path
+  (recommended) or build-from-source, listing ROCm 7.2.0 and 7.1.0 as
+  supported versions, explicit `--device=/dev/kfd --device=/dev/dri
+  --group-add video` flags (same device-passthrough pattern this repo
+  already uses for llama-server), and template workflows including "SD3.5
+  Simple". Officially the guide only names AMD Instinct
+  MI355X/MI325X/MI300X (datacenter cards) as supported platforms.
+  Source: [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html).
+- **The upstream ComfyUI README itself documents AMD support directly**,
+  including consumer cards: stable ROCm install via
+  `pip install torch torchvision torchaudio --index-url
+  https://download.pytorch.org/whl/rocm7.2`, plus an experimental Windows
+  build explicitly naming **RDNA 3 (RX 7000), RDNA 3.5 (Strix Halo), and
+  RDNA 4 (RX 9000 series)** — i.e. the same RDNA4 generation as the R9700 —
+  and `HSA_OVERRIDE_GFX_VERSION` workarounds for older/unlisted cards.
+  Source: [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI).
+- **AMD has published a specific RDNA4/RX 9000 ComfyUI guide** (separate
+  from the Instinct-only install page above), confirming RDNA4 consumer
+  cards are an explicitly supported, first-party-documented target, not just
+  a community workaround.
+  Source: [ROCm blog — Getting Started with ComfyUI on AMD Radeon RX 9000 Series GPUs](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html).
+- **gfx1201 (R9700's arch) specifically has active community Docker images**:
+  `yurisasc/comfyui-rocm-rdna4` targets ROCm 7.1 + PyTorch 2.9.1 with
+  `HSA_OVERRIDE_GFX_VERSION=12.0.1` / `PYTORCH_ROCM_ARCH=gfx1201` baked in,
+  and there's a published community patch specifically for R9700 AI Pro +
+  ComfyUI video-gen speedups, evidence the card is being run today, not just
+  theoretically compatible.
+  Sources: [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4),
+  [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch).
+- **Known gfx1201 caveat:** AMD's own TransformerEngine repo has an open
+  issue confirming gfx1201 is missing from the FP8 architecture table, so
+  FP8 kernels silently fall back to FP32 with ~50% throughput loss
+  (18-22 vs. 35-40 tok/s in the reporter's LLM benchmark) — not a
+  correctness blocker, but relevant if planning to use FP8-quantized image
+  models expecting native FP8 speed on this card; GGUF/Q-quants (see below)
+  avoid this path entirely since they dequantize to bf16/fp16, not fp8.
+  Source: [ROCm/TransformerEngine issue #520](https://github.com/ROCm/TransformerEngine/issues/520).
+- **Actively maintained community Docker packaging** beyond AMD's own image:
+  `YanWenKun/ComfyUI-Docker` ships parallel `rocm` (PyTorch-build-based,
+  faster releases) and `rocm7` (AMD-build-based, more comprehensive)
+  variants, both targeting ROCm 7, with ~1000 commits of ongoing history —
+  a viable alternative to the official AMD image if it lags behind ComfyUI
+  releases.
+  Source: [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker).
+
+### AUTOMATIC1111 / Forge — usable but a step down for this hardware
+
+- ROCm support for A1111/Forge is real but community-patched, not
+  first-party. The upstream `lllyasviel/stable-diffusion-webui-forge` repo's
+  own discussion thread on AMD support points users to
+  `lshqqytiger/stable-diffusion-webui-amdgpu-forge`, a community fork
+  specifically maintained for AMD, "regarded as the go-to version" for
+  running FLUX-era models on AMD — i.e. the *mainline* Forge repo does not
+  claim ROCm support itself; you're expected to run a fork.
+  Source: [lllyasviel/stable-diffusion-webui-forge discussion #67](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67).
+- No first-party AMD vendor documentation (unlike ComfyUI's AMD-authored
+  ROCm/RDNA4 blog posts above) was found for A1111/Forge specifically.
+  Given ComfyUI already has an AMD-blessed path plus a first-class OmniRoute
+  provider (below), there's no reason to take on a community fork's
+  maintenance risk instead.
+
+### InvokeAI — usable but weaker AMD story for a new-generation card
+
+- InvokeAI documents ROCm support but flags it as second-tier: "AMD GPUs
+  are only supported on Linux," and "support for newer AMD GPUs is spotty
+  ... you may experience garbled images, black images, or long startup
+  delays." Its own install docs reference ROCm 5.4.2-era wheels, notably
+  older than the ROCm 7.x this stack's llama-server image already runs on
+  gfx1201.
+  Source: [InvokeAI installation docs (mauwii mirror)](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/).
+- No OpenAI-compatible-images angle either — same drawback as A1111/Forge.
+  Not recommended as primary given ComfyUI's stronger, more current AMD
+  documentation trail.
+
+## Model choice: FLUX.1 [schnell] vs FLUX.1 [dev] vs SDXL vs SD3.5
+
+| Model | License | Params | Notes |
+|---|---|---|---|
+| **FLUX.1 [schnell]** | **Apache-2.0** — fully open, no commercial restriction | 12B | Distilled for 1-4 step inference (fast); Black Forest Labs' own model card states this license directly |
+| FLUX.1 [dev] | [FLUX.1-dev Non-Commercial License](https://github.com/black-forest-labs/flux/blob/main/model_licenses/LICENSE-FLUX1-dev) | 12B | Non-commercial for the *model/weights*; generated *outputs* are explicitly usable commercially per BFL's license page. Higher quality than schnell (more steps, non-distilled) but the weights themselves can't be redistributed/used commercially |
+| SDXL | CreativeML OpenRAIL++ (permissive, commercial-friendly) | ~3.5B | Older (2023), lower fidelity than FLUX/SD3.5 by current standards, but lowest VRAM footprint and best long-standing tooling maturity |
+| SD3.5 (Large/Medium) | Stability AI Community License — free commercial use under $1M annual revenue, else enterprise license required | 8B / 2.5B | Free for this repo's non-commercial homelab use regardless; template already listed in AMD's own ComfyUI-ROCm doc ("SD3.5 Simple") as a first-party example workflow |
+
+Sources: [black-forest-labs/flux model cards](https://github.com/black-forest-labs/flux/blob/main/model_cards/FLUX.1-dev.md),
+[black-forest-labs/FLUX.1-schnell on Hugging Face](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
+(license: apache-2.0), [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md),
+[Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5),
+[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
+
+**Recommendation: FLUX.1 [schnell].** For a private homelab, license
+enforcement isn't the deciding factor by itself, but schnell's Apache-2.0
+status removes any future ambiguity if outputs or the setup are ever shared
+or repurposed, and its whole design point — good quality in 1-4 sampling
+steps — directly addresses the VRAM/time-slicing constraint below (less
+GPU-resident time per image than a 20-50 step dev/SDXL/SD3.5 run).
+Quantized via `city96/ComfyUI-GGUF` (an actively-referenced, community-
+trusted quantization node — its GGUF Q-quants dequantize to bf16/fp16 at
+runtime, sidestepping the gfx1201 FP8 dequant bug above entirely), FLUX fits
+in a fraction of its fp16 footprint:
+
+| Precision | Approx. VRAM (model only) |
+|---|---|
+| fp16 (baseline) | ~24 GB |
+| fp8 | ~12 GB |
+| GGUF Q5_K_S | ~12-15 GB (practical quality floor) |
+| GGUF Q4_K_S | ~7 GB (quality starts degrading on hands/text below Q4) |
+
+Source: aggregated VRAM figures from GGUF-quantization write-ups referencing
+city96's FLUX GGUF conversions — treat as secondary/community sourced
+(no single BFL-published VRAM table was found), consistent across multiple
+independent sources.
+[city96/ComfyUI-GGUF README](https://github.com/city96/ComfyUI-GGUF/blob/main/README.md),
+[city96/FLUX.1-dev-gguf model card](https://huggingface.co/city96/FLUX.1-dev-gguf).
+
+**Fallback pick: SDXL.** If schnell's distilled quality ceiling proves too
+low for some use case, SDXL is the safer second choice over FLUX.1 [dev] or
+SD3.5 specifically *because* of this card's tight headroom: it's the
+smallest of the four by a wide margin, has the longest production track
+record on ROCm of any of these models, and its OpenRAIL++ license carries no
+revenue-threshold clause to track (unlike SD3.5's Community License) or
+non-commercial weight restriction (unlike FLUX.1 [dev]).
+
+## OpenAI-compatible API / OmniRoute integration
+
+This is the best news in this research: **OmniRoute already ships a
+first-class, built-in `comfyui` provider** — not a generic "point it at an
+OpenAI base URL and hope" integration. Its provider reference documents it
+explicitly: *"No API key required. Configure the local ComfyUI base URL
+(default: http://localhost:8188)."* OmniRoute's own image-routing feature
+set (`/v1/images/generations`, `/v1/images/edits`, `/v1/images/variations`,
+with automatic provider fallback) is designed for exactly this pattern:
+register ComfyUI as a backend, then any client already calling OmniRoute's
+OpenAI-compatible images endpoints reaches it with no extra shim.
+Source: [diegosouzapw/OmniRoute PROVIDER_REFERENCE.md](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md),
+[diegosouzapw/OmniRoute repo description](https://github.com/diegosouzapw/OmniRoute).
+
+This means **ComfyUI does not need an extra OpenAI-API wrapper project** —
+the wrapper projects found during this research
+([`ComfyUI-OpenAI-Compatible-API`](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API))
+turned out to be a ComfyUI *custom node* for calling *outbound* to LLM APIs
+from within a workflow (the reverse direction), not something this stack
+needs — OmniRoute's own native ComfyUI provider is the actual integration
+point, one layer up.
+
+**Confidence note:** the provider-reference detail above was fetched via an
+automated summarizer against the raw doc rather than manually re-verified
+line-by-line; re-check `PROVIDER_REFERENCE.md`'s `comfyui` entry directly
+before wiring this up, in case ComfyUI's own `/prompt` API (a
+workflow-graph-shaped API, not a simple text-prompt-in/image-out call) needs
+a specific default workflow JSON configured on the OmniRoute side to produce
+a plain text-to-image call.
+
+## Integration sketch (not a full compose — see caveats above)
+
+- New service in `docker-compose.yml`, e.g. `comfyui`, image
+  `rocm/comfyui-rocm` (or `yurisasc/comfyui-rocm-rdna4` for a gfx1201-tuned
+  build) or built from AMD's own ROCm ComfyUI Dockerfile, same
+  `/dev/kfd` + `/dev/dri` + `group_add: [video, render]` device-passthrough
+  block already used for `llama-server`, joined to the same `ai-stack`
+  network so `omniroute` can reach it as `http://comfyui:8188` — no host
+  port needed (matches the existing llama-server pattern of no published
+  port, gateway-only access).
+- Register it in OmniRoute's dashboard as a `comfyui` provider pointing at
+  that internal URL, same manual-registration pattern already used for
+  llama-server and searxng-search per `docs/proxy-key-onboarding.md`.
+- **VRAM contention is the real design problem, not networking.** Given the
+  ~6 GB headroom, the two services can't both sit GPU-resident.
+  Two workable patterns, in order of how well they fit what's already in
+  this repo:
+  1. **Mirror the existing lazytainer stop-on-idle pattern** already applied
+     to `llama-server` (`docker-compose.yml`'s `lazytainer.group.*` labels) —
+     add an equivalent idle-timeout group for `comfyui`, and rely on the two
+     services naturally not being hit at the same time for a single-user
+     homelab. This doesn't *guarantee* mutual exclusion (both could still be
+     woken concurrently and both try to fit in 6 GB free), so it's a
+     reasonable-effort fit, not a hard guarantee.
+  2. **Explicit mutual exclusion**: a small script/compose profile that
+     stops `llama-server` before starting `comfyui` (and vice versa) rather
+     than relying on lazytainer's independent idle timers — worth doing if
+     the reasonable-effort version above causes a visible OOM in practice.
+  Either way, this is a "pick one, then the other" story, not "run both."
+- Given FLUX.1 [schnell]'s 1-4 step design, a cold-start-and-generate cycle
+  (wake ComfyUI from lazytainer sleep, generate, let it idle back down) is
+  a reasonably good fit for occasional image requests through the same
+  gateway that already does this for llama-server.
+
+## Sources consulted
+
+- [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html)
+- [ROCm blog — ComfyUI on AMD Radeon RX 9000 Series (RDNA4)](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html)
+- [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI)
+- [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker)
+- [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4)
+- [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch)
+- [ROCm/TransformerEngine issue #520 (gfx1201 FP8 fallback)](https://github.com/ROCm/TransformerEngine/issues/520)
+- [lllyasviel/stable-diffusion-webui-forge discussion #67 (AMD support)](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67)
+- [InvokeAI CUDA/ROCm install docs](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/)
+- [black-forest-labs/flux GitHub repo + model cards](https://github.com/black-forest-labs/flux)
+- [black-forest-labs/FLUX.1-schnell (Hugging Face, Apache-2.0)](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
+- [black-forest-labs/FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)
+- [Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5)
+- [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)
+- [city96/ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF)
+- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
+- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) and its `PROVIDER_REFERENCE.md`
+- [yeeyou/ComfyUI-OpenAI-Compatible-API](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API) (checked and ruled out — wrong direction)
+- This repo: `docker-compose.yml`, `.env.example`, `docs/research/qwen3.8-27b-quant.md`

docs/research/langchain-pgvector-vs-litellm-pgvector.md

@@ -0,0 +1,388 @@
+# Research: LangChain+pgvector-direct RAG vs. the litellm-pgvector connector (for Gitea issue #25)
+
+**Question:** Is a LangChain + pgvector-direct retrieval architecture (LangChain's
+`PGVector` class talking to Postgres directly, LiteLLM used only for
+provider-abstracted chat/embedding calls) a better fit for this stack than the
+already-built `litellm-pgvector` connector service (`vendor/litellm-pgvector/`,
+issue #23's resolution)?
+
+**Bottom line: yes — switch.** Replace `litellm-pgvector` /
+`vendor/litellm-pgvector/` with a small hand-written FastAPI service (roughly
+80–120 lines, one new dependency set: `langchain-postgres`, `langchain-openai`,
+`fastapi`, `uvicorn`) that wraps LangChain's `PGVector` class and calls
+LiteLLM's OpenAI-compatible `/v1/embeddings` endpoint for embeddings. This
+keeps "memory served from the gateway layer" in spirit (LiteLLM still does all
+chat/embedding model calls; only the retrieval *query* now goes through a
+LangChain-based service instead of a vendored 754-line Prisma-backed
+connector), removes 754 lines of unvetted vendored source and a Prisma
+migration dependency, and — critically — removes the *guessed and unverified*
+`vector_store_registry` / `custom_llm_provider: pg_vector` config that issue
+#24 flagged as never confirmed against a live deploy. `pgvector-db` (plain
+`pgvector/pgvector:pg16`) is reusable completely as-is; only the schema
+management strategy changes (LangChain manages its own two tables instead of
+Prisma migrations).
+
+## 1. Verifying the markaicode article against LangChain's actual docs
+
+The article (<https://markaicode.com/stack/litellm-langchain-stack/>) proposes
+three services — LangServe (port 8000, wraps a LangChain chain as REST),
+LiteLLM (port 4000), and Postgres+pgvector (port 5432) — and shows:
+
+```python
+vectorstore = PGVector(
+    connection_string="postgresql://user:pass@pgvector:5432/vectordb",
+    embedding_function=your_embeddings,
+    collection_name="documents",
+)
+```
+
+**This snippet uses the deprecated class.** `connection_string` and
+`embedding_function` are constructor args of
+`langchain_community.vectorstores.pgvector.PGVector`, the older integration
+that ships inside the general-purpose `langchain-community` package. The
+current, actively-maintained integration lives in its own package,
+`langchain-postgres`, as `langchain_postgres.vectorstores.PGVector`, with a
+different constructor:
+
+```python
+from langchain_postgres import PGVector
+from langchain_openai import OpenAIEmbeddings
+
+vector_store = PGVector(
+    embeddings=embeddings,          # not embedding_function
+    collection_name="my_docs",
+    connection="postgresql+psycopg://user:pass@host:5432/db",  # not connection_string; psycopg3 DSN
+    use_jsonb=True,
+)
+```
+
+Confirmed constructor signature (`embeddings`, `connection`, `collection_name`,
+`use_jsonb`, `create_extension`, `async_mode`) via
+[reference.langchain.com/python/langchain-postgres/vectorstores/PGVector](https://reference.langchain.com/python/langchain-postgres/vectorstores/PGVector)
+and the setup walkthrough at
+[docs.langchain.com/oss/python/integrations/vectorstores/pgvector](https://docs.langchain.com/oss/python/integrations/vectorstores/pgvector).
+Two concrete corrections to the article's shape:
+
+- **Package**: `langchain-postgres` is a separate pip install
+  (`pip install -qU langchain-postgres`), not part of `langchain_community`.
+  The docs page does not say the community class is formally deprecated, but
+  it is the older/legacy path — the actively-documented integration page for
+  Postgres+pgvector points at `langchain_postgres.vectorstores.PGVector`.
+- **Driver**: `langchain-postgres` requires the `psycopg` (psycopg3) driver
+  and a `postgresql+psycopg://` DSN, not `psycopg2` / plain `postgresql://`.
+  This repo's `pgvector-db` service doesn't care (it's just Postgres), but the
+  new service's `requirements.txt` needs `psycopg[binary]`, not
+  `psycopg2-binary` (which `vendor/litellm-pgvector/requirements.txt` uses).
+- **`create_extension`** defaults to `True` — `PGVector` runs `CREATE
+  EXTENSION IF NOT EXISTS vector` itself on first connect. Source: same API
+  reference page above. This is the same requirement `litellm-pgvector`'s
+  Prisma schema has (Postgres user needs `CREATE` privilege) — **no change in
+  privilege requirements**, just who issues the DDL.
+- The article's LangServe wrapper (`add_routes()`) is a real LangChain
+  component but adds a fourth abstraction (LangServe app → LangChain
+  chain/retriever → PGVector → Postgres) for what this stack needs from only
+  two HTTP endpoints (ingest, query). A plain FastAPI app calling `PGVector`
+  methods directly is simpler and is what's specced below (§5) — LangServe is
+  overkill for a two-endpoint internal service.
+
+There is also a v2, engine-based API, `langchain_postgres.v2.vectorstores.PGVectorStore`
+(with native async via `PGEngine`/`AsyncPGVectorStore`), documented at
+[reference.langchain.com/python/integrations/langchain_postgres](https://reference.langchain.com/python/integrations/langchain_postgres/).
+It's newer and adds first-class async, but the plain `PGVector` class is the
+one shown on LangChain's main pgvector integration page, is simpler to set up
+(no separate `PGEngine` object), and this service has no need for the async
+throughput `PGVectorStore` targets (single ingest script, occasional queries).
+**Recommendation: use `langchain_postgres.vectorstores.PGVector`, not
+`PGVectorStore`** — revisit only if query volume ever demands async
+throughput.
+
+## 2. Complexity comparison: vendor/litellm-pgvector vs. a LangChain-direct service
+
+`vendor/litellm-pgvector/` as it exists in this repo today:
+
+| File | Lines |
+|---|---|
+| `main.py` | 521 |
+| `embedding_service.py` | 89 |
+| `models.py` | 85 |
+| `config.py` | 59 |
+| `prisma/schema.prisma` | 39 |
+| **Total (Python + schema)** | **793** |
+
+Plus: a `Dockerfile` that runs `prisma generate` at build time, a
+`requirements.txt` pulling in `prisma==0.11.0` (which itself vendors a Rust
+query-engine binary download at install time — a second network dependency
+inside the Docker build, on top of the git-context build that already failed
+once per the vendoring rationale in `docker-compose.yml`/`VENDORED.md`), and
+hand-rolled raw-SQL string building for every endpoint (see `main.py` lines
+86–128, 219–320, 322–402, 405–511 — table/column names are
+f-string-interpolated from `settings.table_names`/`settings.db_fields`, not
+parameterized, though the *values* are parameterized). None of this has been
+smoke-tested against a live deploy (issue #24).
+
+A LangChain-direct replacement needs to expose exactly two HTTP endpoints
+(`POST /ingest`, `POST /query` — see §5) as a thin wrapper around library
+calls that already do the embedding + SQL work:
+
+- `langchain_postgres.PGVector.add_texts(texts, metadatas)` — ingestion,
+  replaces `main.py`'s `create_embedding`/`create_embeddings_batch` (176
+  lines) and all of `models.py`'s embedding request/response schemas.
+- `langchain_postgres.PGVector.similarity_search_with_relevance_scores(query,
+  k)` — query, replaces `search_vector_store` (100 lines) and
+  `generate_query_embedding`/`embedding_service.py` (89 lines) — LangChain's
+  `OpenAIEmbeddings` (from `langchain-openai`, pointed at
+  `base_url=http://litellm:4000/v1`) does the embedding call instead of a
+  hand-written `httpx` call to LiteLLM.
+- No `create_vector_store`/`list_vector_stores` needed — one Postgres
+  "collection" (LangChain's term, one row in its `langchain_pg_collection`
+  table) is enough for this repo's single `memory-and-notes` use case;
+  `collection_name="memory-and-notes"` passed once at construction covers it.
+- No Prisma, no generated query-engine binary, no `prisma/schema.prisma` —
+  `PGVector` creates and manages its own two tables
+  (`langchain_pg_collection`, `langchain_pg_embedding`) via SQLAlchemy on
+  first use.
+
+Estimated replacement size: **~90–120 lines of Python** across the FastAPI app
++ Pydantic request/response models (a `/ingest` and `/query` endpoint, an
+`Embeddings`-backed `PGVector` instance built once at startup, an API-key
+`Depends()` check reused almost verbatim from `vendor/litellm-pgvector/main.py`
+lines 50–55) — roughly an **85% reduction** from the vendored connector's 793
+lines, with zero Prisma/migration surface and no hand-built raw SQL (parameter
+binding, vector-string formatting, and JSONB metadata handling are all
+internal to `PGVector`, not re-implemented per endpoint as in `main.py`).
+
+**This does not eliminate the need for a small custom service** — a bare
+`PGVector.similarity_search()` call is a Python library call, not an HTTP
+endpoint, and this stack's callers (the ingest script, Claude Code hooks
+described in `docs/memory-knowledgebase.md`) need HTTP. Something still has to
+be that thin wrapper; it's just ~85% smaller and has no ORM/migration layer.
+
+## 3. Does this still count as "memory served from the gateway layer"?
+
+Yes, with the same division of responsibility issue #21 established, just a
+narrower one for the connector. LiteLLM (`litellm` container) remains the
+**only** thing every client (Open WebUI, Claude Code, anything else behind the
+proxy) talks to for chat and embedding *model* calls — nothing changes there.
+What moves is which service holds the *retrieval* logic: today it's
+`litellm-pgvector` (itself calling back into `litellm:4000` for embeddings,
+per `docker-compose.yml` lines 241–246); under this alternative it's the new
+LangChain-based service, which **also** calls back into `litellm:4000/v1` for
+embeddings (via `langchain-openai`'s `OpenAIEmbeddings(base_url="http://litellm:4000/v1")`,
+since LiteLLM exposes an OpenAI-compatible API — confirmed by this repo's
+existing `local-embedding` `model_list` entry in `litellm-config.yaml` already
+being OpenAI-route-compatible). The client-facing shape is identical: no
+client (Open WebUI, a Claude Code hook, `ingest-memory.sh`) talks to Postgres
+directly, and no client bypasses LiteLLM for the actual embedding/generation
+model calls. Only the internal retrieval connector's implementation changes,
+from a vendored third-party FastAPI+Prisma app to an in-repo FastAPI+LangChain
+app. This does not contradict #21's standing decision.
+
+One nuance: today, `vector_store_registry` makes LiteLLM itself the thing a
+chat completion's `tools: [{"type": "file_search", ...}]` call reaches (per
+`docs/research/litellm-knowledgebase.md` §1) — LiteLLM intercepts the tool
+call and proxies to the registered `pg_vector` backend. **Under the
+alternative, that specific mechanism goes away**: the new service is not a
+`vector_store_registry` provider (there is no `langchain_postgres`/custom
+provider type in LiteLLM's `vector_store_registry`, and building one would
+reintroduce exactly the custom-connector complexity this ticket is trying to
+shed). Retrieval becomes a direct call to the new service's `/query` endpoint
+(or a client-side RAG step: query the service, splice results into the prompt
+before calling `litellm`) rather than an in-band `file_search` tool call
+LiteLLM resolves itself. This is a real, if modest, capability reduction:
+`{"type": "file_search", "vector_store_ids": [...]}` inside a `/chat/completions`
+call to `litellm` stops working; callers move to a two-step "search then
+send" pattern instead. Given `vector_store_registry`'s exact field shape was
+never confirmed working (issue #24) and this is the only capability lost,
+this is judged an acceptable trade for the complexity and reliability win.
+
+## 4. Ingestion, embedding-server, pgvector-db reuse
+
+**`embedding-server`** (llama.cpp, `nomic-embed-text-v1.5`, `docker-compose.yml`
+lines 38–66): fully reusable as-is. It already serves an OpenAI-compatible
+`/v1/embeddings` route (llama.cpp's `--embeddings` flag, per
+`docs/research/litellm-knowledgebase.md` §3), and `litellm-config.yaml`
+already exposes it through LiteLLM as the `local-embedding` model
+(`api_base: http://embedding-server:8080/v1`). The new service reaches it the
+same way `litellm-pgvector` does today — indirectly, by calling
+`http://litellm:4000/v1/embeddings` with `model=local-embedding` — via
+`langchain_openai.OpenAIEmbeddings(model="local-embedding",
+base_url="http://litellm:4000/v1", api_key=<virtual key>)`. `OpenAIEmbeddings`
+is LangChain's standard `Embeddings` implementation for any OpenAI-compatible
+endpoint (no custom `Embeddings` subclass needed) — confirmed via the
+constructor example in
+[docs.langchain.com/oss/python/integrations/vectorstores/pgvector](https://docs.langchain.com/oss/python/integrations/vectorstores/pgvector),
+which pairs `OpenAIEmbeddings` with `PGVector` directly.
+
+**`scripts/ingest-memory.sh`** changes, but modestly: same per-line chunking
+of `data/memory.md`/`data/claude-legacy-memory.md` (the ponytail comment in
+the current script — one chunk per non-empty, non-heading line, no real
+chunking logic needed — still holds), but it now `curl`s the new service's
+`POST /ingest` instead of `litellm-pgvector`'s two-step
+`POST /v1/vector_stores` + `POST /v1/vector_stores/{id}/embeddings/batch`.
+Concretely: build one JSON array of `{content, metadata}` per file (same
+shape the script already builds) and POST it once to `/ingest`; the new
+service calls `PGVector.add_texts(texts, metadatas)` internally, which
+embeds and inserts in one call — no separate "create the store" step, since
+`collection_name` is fixed at service startup and `PGVector` creates the
+collection row implicitly on first `add_texts`. The no-dedup caveat in the
+current script's header comment carries over unchanged (`add_texts` always
+inserts; nothing in `PGVector` deduplicates by content).
+
+**`pgvector-db`** (`pgvector/pgvector:pg16`, `docker-compose.yml` lines
+198–214): reusable completely as-is — same image, same "separate Postgres
+instance from `litellm-db`" rationale, no compose changes needed to the
+service definition itself. Only the *schema* changes: `litellm-pgvector`'s
+Prisma schema (`vendor/litellm-pgvector/prisma/schema.prisma`, 39 lines)
+defines its own `vector_stores`/embeddings tables with configurable
+table/field names (`config.py`'s `settings.table_names`/`db_fields`), applied
+via Prisma migrations at container startup (the exact behavior of which,
+per issue #24's ponytail note, was never smoke-tested). `PGVector` instead
+auto-creates two fixed tables — `langchain_pg_collection` and
+`langchain_pg_embedding` — via SQLAlchemy `Base.metadata.create_all()`-style
+setup on first connection, no separate migration step or migration tool
+needed. This is a strictly simpler schema-management story: no Prisma CLI,
+no `prisma generate` build step, no migration files to track. The existing
+`litellm_pgvector`/`PGVECTOR_DB_PASSWORD` database and role stay — the new
+service just needs its own `DATABASE_URL` pointed at the same database (or a
+fresh one; reusing `litellm_pgvector` is simplest since no data has been
+proven-loaded into it yet per issue #24).
+
+**`litellm-pgvector` (container) and `vendor/litellm-pgvector/` (source
+tree)**: removed if this verdict is adopted. Nothing else in the repo depends
+on the vendored source once the new service replaces it.
+
+## 5. Concrete replacement spec (not implemented — spec only)
+
+**New compose service**, replacing the `litellm-pgvector` block:
+
+```yaml
+  memory-retrieval:                     # name TBD; "litellm-pgvector" freed up
+    build:
+      context: ./services/memory-retrieval    # new small in-repo dir, not vendored
+    container_name: memory-retrieval
+    depends_on:
+      pgvector-db:
+        condition: service_healthy
+      litellm:
+        condition: service_healthy
+    environment:
+      - DATABASE_URL=postgresql+psycopg://litellm_pgvector:${PGVECTOR_DB_PASSWORD}@pgvector-db:5432/litellm_pgvector
+      - LITELLM_BASE_URL=http://litellm:4000/v1
+      - LITELLM_API_KEY=${MEMORY_RETRIEVAL_EMBEDDING_KEY}   # new virtual key, same pattern as LITELLM_PGVECTOR_EMBEDDING_KEY today
+      - EMBEDDING_MODEL=local-embedding
+      - SERVER_API_KEY=${MEMORY_RETRIEVAL_API_KEY}          # replaces LITELLM_PGVECTOR_API_KEY
+      - COLLECTION_NAME=memory-and-notes
+    expose:
+      - "8000"
+    restart: unless-stopped
+    networks: [ai-stack]
+```
+
+Note `DATABASE_URL` changes scheme to `postgresql+psycopg://` (psycopg3 DSN
+`langchain-postgres` requires — see §1), not the `postgresql://` scheme
+`litellm-pgvector` uses today.
+
+**Dockerfile shape** (`services/memory-retrieval/Dockerfile`): same base image
+family as today's connector — `python:3.11-slim` — `pip install` the
+requirements, copy the app, `CMD ["uvicorn", "main:app", "--host", "0.0.0.0",
+"--port", "8000"]`. No `prisma generate` step, no `build-essential`/`postgresql-client`
+needed at the OS-package level (no Prisma binary to build against; `psycopg[binary]`
+ships prebuilt wheels).
+
+**`requirements.txt`**: `fastapi`, `uvicorn[standard]`, `langchain-postgres`,
+`langchain-openai`, `psycopg[binary]`, `pydantic` — five packages vs. the
+current ten in `vendor/litellm-pgvector/requirements.txt`, and no `prisma`.
+
+**Endpoints** (2, vs. today's 5 — create store, list stores, search, create
+embedding, create embeddings batch):
+
+- `POST /ingest` — body: `{"chunks": [{"content": "...", "metadata": {...}}]}`.
+  Calls `PGVector.add_texts(texts=[...], metadatas=[...])`. Replaces both
+  `ingest-memory.sh`'s "create store" call (no longer needed — collection is
+  implicit) and its batch-embeddings call.
+- `POST /query` — body: `{"query": "...", "k": 5}`. Calls
+  `PGVector.similarity_search_with_relevance_scores(query, k=k)`, returns
+  `{"results": [{"content": ..., "metadata": ..., "score": ...}]}`.
+- `GET /health` — same trivial liveness check the current connector has
+  (`main.py` lines 514–517), kept for compose `depends_on`/`healthcheck` use
+  if desired.
+
+Bearer-token auth via `SERVER_API_KEY`, same `Depends()` pattern as
+`vendor/litellm-pgvector/main.py` lines 47–55 — that part of the existing
+connector is fine as-is and can be copied over close to verbatim.
+
+**`litellm-config.yaml` change**: remove the `vector_store_registry` block
+entirely (lines 47–61 today) — per §3, the new service isn't a
+`vector_store_registry` provider, so there's nothing to register. The
+`local-embedding` `model_list` entry (lines 25–35) is unchanged and still
+needed — it's what the new service's `OpenAIEmbeddings` client calls through
+`litellm`.
+
+## 6. Verdict
+
+**Switch to LangChain-direct.** This does invalidate part of issue #23's
+resolution and the `litellm-pgvector` container currently in
+`docker-compose.yml` — say so explicitly for scoping a follow-up
+implementation ticket:
+
+- **Torn out**: `litellm-pgvector` service block in `docker-compose.yml`,
+  `vendor/litellm-pgvector/` (entire tree), the `vector_store_registry` block
+  in `litellm-config.yaml`, `LITELLM_PGVECTOR_API_KEY`/
+  `LITELLM_PGVECTOR_EMBEDDING_KEY` env vars (replaced by new-service
+  equivalents).
+- **Survives unchanged**: `pgvector-db` (compose block as-is), `embedding-server`
+  (compose block as-is), the `local-embedding` `model_list` entry in
+  `litellm-config.yaml`, the overall "LiteLLM does all model calls" gateway
+  boundary from issue #21.
+- **New**: a small in-repo `services/memory-retrieval/` FastAPI+LangChain
+  app (~90–120 lines, spec in §5), a new compose service block, a rewritten
+  `scripts/ingest-memory.sh` pointed at `/ingest` instead of
+  `litellm-pgvector`'s two-step API, and one new virtual key
+  (`MEMORY_RETRIEVAL_EMBEDDING_KEY`) issued the same way
+  `LITELLM_PGVECTOR_EMBEDDING_KEY` was per `docs/proxy-key-onboarding.md`.
+- **Capability trade-off** (§3): the OpenAI Assistants-style `file_search`
+  tool call directly against `litellm`'s `/chat/completions` stops working
+  (no `vector_store_registry` entry to resolve it); callers move to querying
+  the new service directly and splicing results into the prompt themselves.
+  This mechanism was never confirmed working in the first place (issue #24),
+  so it's a loss of unverified capability, not working functionality.
+
+This is not implemented here — this document is research/spec only, per the
+ticket's scope. A follow-up implementation ticket should: write
+`services/memory-retrieval/`, update `docker-compose.yml` and
+`litellm-config.yaml` per §5, rewrite `scripts/ingest-memory.sh`, delete
+`vendor/litellm-pgvector/`, and update `docs/memory-knowledgebase.md`.
+
+## 7. Repo context read for this research
+
+- `docker-compose.yml` — `embedding-server`, `pgvector-db`, `litellm-pgvector`
+  service definitions (build context, env vars, network).
+- `litellm-config.yaml` — `local-embedding` model entry, `vector_store_registry`
+  block.
+- `docs/research/litellm-knowledgebase.md` — prior research establishing
+  LiteLLM's native vector-store feature has no Qdrant provider and that
+  `litellm-pgvector` was the only self-hosted path found at the time; its §1,
+  §3, §4 are cited above for the `file_search` tool-call mechanism and
+  embedding-model wiring.
+- `docs/memory-knowledgebase.md` — current user-facing setup doc for the
+  knowledgebase/memory feature.
+- `scripts/ingest-memory.sh` — current ingestion mechanism and its no-dedup
+  caveat.
+- `vendor/litellm-pgvector/{main.py,embedding_service.py,models.py,config.py,
+  prisma/schema.prisma,requirements.txt,Dockerfile}` — read in full to
+  characterize size (793 total lines Python+schema) and shape (raw-SQL
+  f-string-built queries per endpoint, Prisma-managed schema, calls back to
+  LiteLLM's `/embeddings` for query embedding).
+- Gitea issue #25 (this ticket) and its parent #21 — standing decision that
+  memory is served from the gateway layer, not a client.
+- Gitea issue #23/#24 (referenced by #25) — original `litellm-pgvector`
+  adoption and its unverified-against-live-deploy caveat.
+
+## 8. Primary sources consulted
+
+- [markaicode.com/stack/litellm-langchain-stack/](https://markaicode.com/stack/litellm-langchain-stack/) — the alternative surfaced by the user; verified and corrected above (§1).
+- [reference.langchain.com/python/langchain-postgres/vectorstores/PGVector](https://reference.langchain.com/python/langchain-postgres/vectorstores/PGVector) — `PGVector` constructor signature, `create_extension` default.
+- [docs.langchain.com/oss/python/integrations/vectorstores/pgvector](https://docs.langchain.com/oss/python/integrations/vectorstores/pgvector) — install instructions, `postgresql+psycopg://` DSN requirement, `OpenAIEmbeddings` + `PGVector` pairing example.
+- [github.com/langchain-ai/langchain-postgres](https://github.com/langchain-ai/langchain-postgres) — package README; async-function-pairing note; `PGVector` deprecated in v0.0.14+ in favor of `PGVectorStore` (v2 API).
+- [reference.langchain.com/python/integrations/langchain_postgres/](https://reference.langchain.com/python/integrations/langchain_postgres/) — v2 `PGVectorStore`/`PGEngine`/`AsyncPGVectorStore` API, compared against `PGVector` in §1.
+- `vendor/litellm-pgvector/README.md` and source files (this repo) — connector's existing endpoint/config shape, cited for the LOC comparison in §2.

docs/research/lazytainer-omniroute-idle-stop.md

@@ -0,0 +1,194 @@
+# Research: Why lazytainer's idle-stop on llama-server doesn't fire, and what switch-model.sh should do about it
+
+**Question:** lazytainer is configured on `llama-server` (`docker-compose.yml`
+`lazytainer.group.llamaserver.*` labels) but its idle-stop never triggers in
+practice — OmniRoute appears to keep the container looking "active" to
+lazytainer's packet-threshold detector. Confirm the mechanism, find root
+cause, and recommend how the future `scripts/switch-model.sh` (#43, blocked)
+should handle GPU-residency swaps between `llama-server` and a new `comfyui`
+service given this.
+
+**Answer:** Confirmed. lazytainer's detector is a dumb per-port packet
+counter with no traffic classification — it cannot tell OmniRoute's
+background provider health-check pings apart from real inference traffic,
+and there is no config knob in lazytainer or a per-provider one in OmniRoute
+that fixes this. **`switch-model.sh` should bypass lazytainer entirely** for
+the swap: drive `docker compose stop`/`up -d` directly on both services,
+rather than trying to make lazytainer's idle-stop cooperate.
+
+## Current config (`docker-compose.yml`)
+
+```yaml
+labels:
+  - "lazytainer.group.llamaserver.sleepMethod=stop"
+  - "lazytainer.group.llamaserver.ports=8080"
+  - "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
+  - "lazytainer.group.llamaserver.minPacketThreshold=2"
+```
+
+`ports=8080` matches the container's real internal port (`expose: ["8080"]`,
+confirmed in the same file) — not a misconfiguration. `minPacketThreshold=2`
+is already far *below* lazytainer's own documented default of `30`, i.e. this
+deployment already tried loosening the threshold to make idle-stop easier to
+reach, not harder.
+
+## How lazytainer's detector actually works (primary source: `vmorganp/Lazytainer`)
+
+Confirmed against the project's README and Go source
+(`src/group.go`) on [github.com/vmorganp/Lazytainer](https://github.com/vmorganp/Lazytainer):
+
+- It captures packets with **gopacket/libpcap directly on the configured
+  `netInterface`** (default `eth0`), applying a BPF filter built from the
+  group's `ports` list (`"port 8080"` here, per the source's filter-string
+  construction, e.g. `"port 80 or port 81 or etc."` in the general case).
+- The filter matches **every packet to or from the port** — SYN, ACK,
+  data, FIN, everything. It is not restricted to new-connection SYNs.
+- Every `pollRate` seconds (default `30`; not overridden in this repo's
+  config) it samples a rolling packet counter (`rxHistory`) and compares the
+  delta against `minPacketThreshold`:
+  `rxHistory[0]+minPacketThreshold > rxHistory[len(rxHistory)-1]` → treated as
+  active, `inactiveSeconds` resets to 0.
+- `ignoreActiveClients` (default `false`, not set here) only changes whether
+  an ESTABLISHED-connection count is also checked; it does not add any
+  content- or source-based filtering.
+- **There is no mechanism anywhere in lazytainer to exclude specific traffic
+  (by source IP, path, header, or request type) from the packet count.** The
+  README's config table (`ports`, `inactiveTimeout`, `minPacketThreshold`,
+  `ignoreActiveClients`, `pollRate`, `sleepMethod`, `netInterface`) is
+  exhaustive — nothing else exists to tune this per-caller.
+
+Consequence: a single TCP connection to port 8080 — a bare connect + one
+small HTTP exchange + close — already produces well over `minPacketThreshold=2`
+packets purely from the handshake and teardown (SYN, SYN-ACK, ACK, ..., FIN,
+ACK), regardless of payload size or purpose. At this threshold, essentially
+*any* connection to the port counts as "active" and resets `inactiveTimeout`.
+Raising the threshold wouldn't help either — the fix would need to be
+"ignore packets from OmniRoute's health-checker," which the tool has no way
+to express; it only counts packets on a port, source-blind.
+
+## How OmniRoute actually touches registered providers (primary source: `diegosouzapw/OmniRoute`)
+
+Confirmed against
+[`docs/reference/ENVIRONMENT.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/ENVIRONMENT.md)
+in the OmniRoute repo:
+
+- OmniRoute runs a **background credential/connection health-check
+  scheduler** (`src/lib/credentialHealth/scheduler.ts`) on
+  `CREDENTIAL_HEALTH_CHECK_INTERVAL`, default `300000` ms (5 min), minimum
+  `10000` ms (10s) — this periodically re-tests each registered provider's
+  connection, which for a provider like `llama-server` (a plain HTTP base
+  URL, no API key) means an actual request/connection to
+  `llama-server:8080`.
+- Results are cached for `CREDENTIAL_HEALTH_CACHE_TTL` (default also 5 min).
+- **Only one exclusion exists, and it's hardcoded by provider category, not
+  configurable per-provider**: search providers
+  (`SEARCH_VALIDATOR_CONFIGS` in
+  `src/lib/providers/validation/searchProviders.ts`, e.g. `tavily-search`)
+  are permanently skipped because their validation call is a real billed
+  upstream query. `llama-server` is an inference provider, not a search
+  provider — it is not in this exclusion list.
+- The only toggle that actually stops the sweep is global:
+  `OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK=1`/`true`, which "disable[s]
+  background periodic testing of provider connections" for **every**
+  registered provider at once. There is no documented per-provider
+  disable/pause flag in
+  [`docs/reference/PROVIDER_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md) —
+  the dashboard's `/dashboard/providers` page is described only as where you
+  "enable, configure, and test each provider," with no documented
+  independent "pause health checks for this one provider" control.
+
+So: OmniRoute is not the sole cause, but it is a live, recurring cause. Every
+5 minutes (at most — could also be triggered ad hoc by dashboard/API use) it
+opens a connection to `llama-server:8080` purely to check the provider is
+alive, which is exactly the kind of traffic lazytainer's port-level counter
+cannot distinguish from real inference calls. With `inactiveTimeout=900`
+(15 min) and a health-check every ≤300s, the container practically always
+sees qualifying traffic before its idle timer would expire.
+
+## Root cause
+
+Two independent, both-true facts combine to defeat idle-stop:
+
+1. **lazytainer's detector is fundamentally traffic-blind** — it counts raw
+   packets on a port with no way to exclude any specific caller or traffic
+   class. This is a property of the tool, not a misconfiguration in this
+   repo (`ports=8080` is correct; `minPacketThreshold=2` is already at the
+   permissive end).
+2. **OmniRoute periodically pings every registered non-search provider**
+   (default every ≤5 min) to keep its health/availability status current,
+   and that ping is indistinguishable, at the packet level, from a real
+   inference request.
+
+Neither side offers a targeted fix: lazytainer has no allowlist/denylist by
+source, and OmniRoute's only "stop pinging" lever
+(`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK`) is all-or-nothing across every
+provider, not scoped to just `llama-server`. Tuning `minPacketThreshold`
+higher or lower doesn't change the outcome either way, since the health-check
+traffic and real traffic land on the exact same port with no distinguishing
+packet-level signature.
+
+## Recommendation for `scripts/switch-model.sh` (#43)
+
+**Bypass lazytainer entirely for the GPU-residency swap.** Drive both
+services directly:
+
+```bash
+docker compose stop llama-server
+docker compose up -d comfyui
+# ...and the reverse when swapping back
+```
+
+Justification:
+
+- The swap is a **deliberate, scripted, known-in-advance** event — the
+  script always knows exactly which service should go up and which should
+  go down. Idle-stop detection exists to handle the case where nobody knows
+  when a service last had traffic; that's not this case, so routing the
+  swap through a passive heuristic (lazytainer's idle timer) that this
+  research shows is already unreliable for `llama-server` adds a point of
+  failure for no benefit. Direct `docker compose stop`/`up -d` is
+  deterministic and immune to the packet-counting confound described above.
+- Reconfiguring lazytainer's thresholds was considered and rejected: no
+  threshold value fixes a detector that cannot distinguish OmniRoute's
+  keepalive traffic from real traffic on the same port (see Root cause).
+  This is a ceiling in the tool itself, not a tuning problem.
+- Pausing OmniRoute's polling for the swap window was also considered.
+  It's the one lever available (`OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK`),
+  but it is global — it would blind OmniRoute's health status for *every*
+  provider (including `searxng-search`, if registered) for the duration of
+  the swap, and adds an extra env-toggle-and-restart step to the script for
+  a problem that direct compose control sidesteps completely. It's worth
+  flagging for #43's implementation as a *secondary* safety measure — briefly
+  disabling the sweep (or accepting that OmniRoute may show `llama-server` as
+  errored/offline for up to `CREDENTIAL_HEALTH_CHECK_INTERVAL` after it's
+  stopped) — but it should not be the primary mechanism the swap relies on.
+- This does **not** require removing the existing `lazytainer.group.llamaserver.*`
+  labels — they can stay for whatever idle-stop benefit they still provide
+  between swaps (e.g. genuinely idle periods where nothing, including
+  OmniRoute, has recently touched the container long enough to matter) while
+  `switch-model.sh` simply never depends on lazytainer to do the actual
+  stop/start for a swap.
+
+## Bottom line for #43 (blocked ticket, once unblocked)
+
+- `switch-model.sh` should call `docker compose stop <from-service>` /
+  `docker compose up -d <to-service>` directly — never rely on lazytainer's
+  idle-stop to free the GPU as part of a swap.
+- No lazytainer config change (threshold, ports, poll rate) is a viable fix;
+  the detector has no way to exclude OmniRoute's traffic by source.
+- Optionally, as a secondary hygiene step, the script may toggle
+  `OMNIROUTE_DISABLE_CREDENTIAL_HEALTH_CHECK` around the swap (or simply
+  tolerate a stale "errored" status in OmniRoute's dashboard for up to one
+  `CREDENTIAL_HEALTH_CHECK_INTERVAL`) to avoid OmniRoute flagging the
+  just-stopped provider as failed mid-swap — but this is cosmetic/status
+  hygiene, not what makes the swap itself work.
+
+Sources: [`vmorganp/Lazytainer`](https://github.com/vmorganp/Lazytainer)
+(README config table; `src/group.go` packet-capture and threshold-comparison
+logic), [`diegosouzapw/OmniRoute` —
+`docs/reference/ENVIRONMENT.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/ENVIRONMENT.md)
+(credential health-check scheduler env vars), [`diegosouzapw/OmniRoute` —
+`docs/reference/PROVIDER_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md)
+(provider dashboard controls), this repo's `docker-compose.yml`
+(`lazytainer.group.llamaserver.*` labels, `llama-server`/`omniroute` service
+definitions).

docs/research/litellm-knowledgebase.md

@@ -0,0 +1,305 @@
+# Research: LiteLLM vector store / knowledgebase feature (for Gitea issue #23)
+
+**Question:** Can LiteLLM's built-in "knowledgebase" / vector store feature be
+used to load two Markdown fact files (`data/memory.md`,
+`data/claude-legacy-memory.md`) into a queryable knowledge base, and if so,
+what does that require from this stack (config shape, backend, embedding
+model, ingestion mechanism)?
+
+**Bottom line: no, not against this stack's existing Qdrant instance, and not
+without adding a dedicated embedding model.** LiteLLM's vector store feature
+is a *routing/registry* layer over a small set of natively-integrated
+backends (Bedrock Knowledge Bases, OpenAI Vector Stores, Azure
+Vector/AI-Search, Vertex AI RAG/Search, Gemini File Search, RAGFlow) plus one
+self-hosted option — Postgres+pgvector, via a **separate companion service**
+(`BerriAI/litellm-pgvector`), not Qdrant. There is no Qdrant provider at all.
+Ingesting the two fact files would also require running a dedicated
+embedding-capable model, which this stack doesn't currently have (the one
+llama.cpp instance serves a chat model, not started with `--embeddings`).
+
+## 1. Config shape in `config.yaml`
+
+Top-level block is **`vector_store_registry`** (this is the current key name;
+an earlier PR that introduced the feature used `vector_stores` — see
+Provenance note below), a list of entries:
+
+```yaml
+vector_store_registry:
+  - vector_store_name: "my-knowledgebase"      # optional friendly name
+    litellm_params:
+      vector_store_id: "T37J8R4WTM"             # required, provider's own ID
+      custom_llm_provider: "bedrock"            # required — selects backend
+      vector_store_description: "..."           # optional
+      vector_store_metadata: {}                 # optional
+      litellm_credential_name: "..."            # optional, reuse a named credential
+      embedding_model: "..."                    # backend-dependent — see §3
+```
+
+Source: [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase)
+(fetched directly). Earlier shape (same idea, older field names) documented
+in [BerriAI/litellm PR #10448](https://github.com/BerriAI/litellm/pull/10448)
+("[Feat] Vector Stores/KnowledgeBases - Allow defining Vector Store Configs"):
+
+```yaml
+vector_stores:
+  - vector_store_name: "bedrock-litellm-website-knowledgebase"
+    litellm_params:
+      custom_llm_provider: "bedrock"
+      id: "T37J8R4WTM"
+```
+
+This is a **registry of pointers to knowledge bases that already exist on
+the backend provider** — it is not itself a vector database. `model_list`
+entries are not where a store is attached; `vector_store_registry` is its
+own top-level sibling of `model_list`.
+
+### Referencing a vector store from a request
+
+Not via a `model_list` entry — via the **`tools`** array of a
+`/chat/completions` (or `/v1/responses`) request, OpenAI Assistants-style
+`file_search` tool:
+
+```json
+{
+  "model": "gpt-4o",
+  "messages": [...],
+  "tools": [
+    {
+      "type": "file_search",
+      "vector_store_ids": ["T37J8R4WTM"]
+    }
+  ]
+}
+```
+
+LiteLLM intercepts the tool call, looks up the referenced ID in
+`vector_store_registry`, calls that backend's native search, and injects the
+retrieved chunks into the prompt before calling the target model. Source:
+[docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
+
+There are also direct, OpenAI-compatible proxy endpoints for managing/using a
+store outside of a chat completion — see §4.
+
+## 2. Backend providers — Qdrant is NOT one of them
+
+Per the official docs page and the PR that introduced the feature, the
+natively-supported `custom_llm_provider` values are:
+
+| Provider | `custom_llm_provider` value | Notes |
+|---|---|---|
+| AWS Bedrock Knowledge Bases | `bedrock` | Original/reference implementation ([PR #10448](https://github.com/BerriAI/litellm/pull/10448)) |
+| OpenAI Vector Stores | `openai` | Wraps OpenAI's own vector store API |
+| Azure Vector Stores | `azure` | Assistants-API-only per docs |
+| Azure AI Search | `azure_ai_search` (vector search capability) | |
+| Vertex AI RAG Engine | `vertex_ai` | Added in [PR #15781](https://github.com/BerriAI/litellm/pull/15781) |
+| Vertex AI Search API | `vertex_ai/search_api` | |
+| Gemini File Search | — | |
+| RAGFlow Datasets | — | Dataset mgmt only; search unsupported per docs |
+| Postgres + pgvector | `pg_vector` (via a **separate** connector service) | See below — not built into the litellm proxy image |
+
+Source: [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase),
+cross-checked against [PR #12595](https://github.com/BerriAI/litellm/pull/12595)
+("[Feat] Vector Stores - Add Vertex RAG Engine API as a provider") and
+[PR #15781](https://github.com/BerriAI/litellm/pull/15781) ("(feat) Vector
+Stores: support Vertex AI Search API").
+
+**Qdrant is not listed anywhere** in the knowledgebase docs, the vector
+store provider PRs, or the pgvector connector's own README. LiteLLM does use
+Qdrant in one unrelated feature — **semantic response caching**
+(`docs.litellm.ai/docs/caching/all_caches`, `qdrant_api_base` /
+`qdrant_api_key` / `qdrant_collection_name` config) — but that is a cache for
+LLM *responses*, not the vector-store/knowledgebase (RAG) feature, and shares
+no config or code path with `vector_store_registry`. It would not let
+LiteLLM search Qdrant-held documents as a knowledge base.
+
+### The pgvector option is a separate microservice, not a built-in backend
+
+[`BerriAI/litellm-pgvector`](https://github.com/BerriAI/litellm-pgvector) is
+its own repo/container: a FastAPI app that exposes OpenAI-compatible vector
+store endpoints backed by Postgres with the `pgvector` extension, calling
+back out to a LiteLLM proxy's `/embeddings` endpoint to generate embeddings.
+It is registered into `vector_store_registry` like any other backend (with
+`custom_llm_provider: pg_vector` pointed at this companion service's URL),
+but it is **not Qdrant** and **not part of the main `litellm` proxy image**
+already running in this stack — it would mean deploying and operating a
+fourth service (on top of `litellm`, `litellm-db`, and `llama-server`), with
+its own Postgres database using the pgvector extension (the existing
+`litellm-db` Postgres image, `postgres:16-alpine`, does not have pgvector
+installed).
+
+### Verdict on the existing Qdrant instance
+
+**LiteLLM's knowledgebase/vector_store feature cannot point at this repo's
+existing standalone `qdrant` service.** There is no Qdrant provider type for
+`vector_store_registry`. To use LiteLLM's native feature at all, this stack
+would need to either integrate with a cloud-native backend (Bedrock/Vertex/
+Azure/OpenAI — none of which apply, this stack is local-only) or stand up
+the separate `litellm-pgvector` + pgvector-enabled Postgres stack — an
+entirely different vector store technology from the Qdrant already running
+for Open WebUI. The existing Qdrant collection Open WebUI uses for its own
+RAG/Memory feature is unrelated to and unreachable from LiteLLM's
+knowledgebase feature.
+
+## 3. Embedding model requirement
+
+LiteLLM's vector store feature **does call an embedding endpoint itself**
+when a backend needs one (pgvector explicitly; the managed cloud backends
+handle embedding server-side). The field is `embedding_model` inside a
+`vector_store_registry` entry's `litellm_params`, confirmed in
+[BerriAI/litellm issue #23980](https://github.com/BerriAI/litellm/issues/23980)
+("[Bug]: Vector store creation fails when using model mapping public model
+name for embedding_model"), which shows:
+
+```json
+"litellm_params": {
+  "vector_bucket_name": "my-embeddings",
+  "index_name": "test-index",
+  "aws_region_name": "us-east-1",
+  "embedding_model": "test-vector-store/bedrock/amazon.nova-2-multimodal-embeddings-v1:0"
+}
+```
+
+For the pgvector connector specifically, the embedding model is configured
+via its own env vars (not `vector_store_registry`, since it's a separate
+service): `EMBEDDING__MODEL`, `EMBEDDING__BASE_URL` (pointed at a LiteLLM
+proxy), `EMBEDDING__API_KEY`, `EMBEDDING__DIMENSIONS` — e.g.
+`EMBEDDING__MODEL=text-embedding-ada-002`,
+`EMBEDDING__BASE_URL=http://litellm:4000`. Source:
+[BerriAI/litellm-pgvector README](https://github.com/BerriAI/litellm-pgvector/blob/main/README.md).
+This confirms: **yes, the embedding model must be reachable as a model
+LiteLLM's proxy can call** — i.e. it needs its own `model_list` entry with
+`mode: embedding` (LiteLLM's standard way of declaring an embedding-capable
+model — see [docs.litellm.ai/docs/embedding/supported_embedding](https://github.com/BerriAI/litellm/blob/main/docs/my-website/docs/embedding/supported_embedding.md)),
+so the pgvector service (or LiteLLM itself for backends that embed
+internally) can call `POST /embeddings` against it through the proxy.
+
+### Does llama.cpp (this repo's model server) support `/embeddings`?
+
+Yes, but not enabled as currently configured, and not well-suited to the
+model already loaded. llama.cpp's server (`tools/server`, the same
+`ghcr.io/ggml-org/llama.cpp:server-rocm` image this repo uses per
+`docker-compose.yml`) exposes an OpenAI-compatible `POST /v1/embeddings`
+route (and a native `/embedding` route), **but only when started with the
+`--embeddings` flag** — source:
+[ggml-org/llama.cpp tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
+This repo's `llama-server` command in `docker-compose.yml` (lines 16–22) does
+not pass `--embeddings`, so the currently-running instance does not serve
+embeddings at all.
+
+Even if the flag were added, llama.cpp loads **one model per server
+process** — the flag would make the already-loaded Qwen3.8-27B **chat**
+model (per `litellm-config.yaml`, `qwen3.8-27b-local` /
+`Qwen3.8-27B-UD-Q4_K_XL.gguf`) emit pooled hidden-state vectors as
+"embeddings," but a generalist instruction-tuned chat model is not what
+that's trained for — embedding quality from a non-embedding-trained model is
+materially worse than a purpose-trained embedding model (e.g. BGE, Nomic
+Embed, mxbai-embed, gte). **A separate, dedicated embedding model/server
+would be needed** — either a second `llama-server` container loaded with a
+small GGUF embedding model (`--embeddings` flag on), or a different
+embedding-serving stack — and it would need its own `model_list` entry in
+`litellm-config.yaml` with `mode: embedding` for LiteLLM/litellm-pgvector to
+call it.
+
+## 4. How documents actually get ingested
+
+Two ingestion paths exist depending on backend, both are HTTP APIs — there
+is no admin-UI "add document" flow beyond store *creation*, and no CLI:
+
+**a) OpenAI-compatible vector-store file API** (used for the `openai`
+backend, and shown as the general pattern in the docs):
+
+- `POST /v1/vector_stores` — create a store. Body:
+  ```json
+  {
+    "name": "My Document Store",
+    "file_ids": ["file-abc123"],
+    "chunking_strategy": {
+      "type": "static",
+      "static": {"max_chunk_size_tokens": 800, "chunk_overlap_tokens": 400}
+    },
+    "metadata": {"key": "value"}
+  }
+  ```
+  Requires files already uploaded through a separate Files API to obtain
+  `file_id`s first — the docs page does not show a files-upload endpoint
+  under the vector-store docs directly (this is OpenAI's own two-step
+  upload-then-attach flow, proxied through). Source:
+  [docs.litellm.ai/docs/vector_stores/create](https://docs.litellm.ai/docs/vector_stores/create).
+- `POST /v1/vector_stores/{vector_store_id}/search` — query it:
+  ```json
+  {
+    "query": "What is the capital of France?",
+    "filters": {"file_ids": ["file-abc123"]},
+    "max_num_results": 5,
+    "ranking_options": {"score_threshold": 0.7},
+    "rewrite_query": true
+  }
+  ```
+  Source: [docs.litellm.ai/docs/vector_stores/search](https://docs.litellm.ai/docs/vector_stores/search).
+- `GET /vector_store/list` — list registered stores. Source:
+  [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
+- LiteLLM Admin UI: **Experimental → Vector Stores → Create Vector Store**
+  exists for registering/creating stores, and the **Logs** page shows
+  vector-store search queries/scores after use — but this is store
+  management and observability, not a bulk document-ingestion UI. Source:
+  [docs.litellm.ai/docs/completion/knowledgebase](https://docs.litellm.ai/docs/completion/knowledgebase).
+
+**b) `litellm-pgvector` connector's own embeddings API** (only path relevant
+if this repo went the self-hosted pgvector route, since Qdrant isn't
+supported at all): direct chunk-level ingestion, no file upload step —
+
+- `POST /v1/vector_stores/{id}/embeddings` — single chunk:
+  ```json
+  {"content": "...", "embedding": [/* optional, else computed server-side */], "metadata": {}}
+  ```
+- `POST /v1/vector_stores/{id}/embeddings/batch` — array of the same shape,
+  for bulk loading.
+
+Source: [BerriAI/litellm-pgvector README](https://github.com/BerriAI/litellm-pgvector/blob/main/README.md).
+
+For a follow-up ticket that wants to programmatically load
+`data/memory.md` / `data/claude-legacy-memory.md` (dated fact lists) into a
+knowledge base, the realistic path through LiteLLM's native feature would be
+the pgvector connector's batch-embeddings endpoint (chunk the Markdown into
+facts/sections client-side, POST each as a batch) — **but that requires
+first standing up**: (1) a pgvector-enabled Postgres, (2) the
+`litellm-pgvector` service, and (3) a dedicated embedding model server
+registered in `litellm-config.yaml`. None of that reuses the Qdrant instance
+already running in this stack.
+
+## 5. Repo context read for this research
+
+- `litellm-config.yaml` — current config has one `model_list` entry
+  (`qwen3.8-27b-local`, chat-only, via llama.cpp), `router_settings`
+  (priority scheduling), `general_settings.master_key`. No
+  `vector_store_registry` block exists yet.
+- `docker-compose.yml` — confirms `qdrant` service (image `qdrant/qdrant`,
+  network `ai-stack`, no host port, only `open-webui` currently consumes it
+  via `VECTOR_DB=qdrant` / `QDRANT_URI=http://qdrant:6333`); confirms
+  `llama-server` command has no `--embeddings` flag and loads a single GGUF
+  (`Qwen3.8-27B-UD-Q4_K_XL.gguf`); confirms `litellm-db` is plain
+  `postgres:16-alpine` (no pgvector extension installed).
+- `CLAUDE.md` — points to `docs/agents/issue-tracker.md` (Gitea issues via
+  `tea`) and `docs/agents/domain.md` (domain docs convention).
+- `docs/agents/domain.md` — says to check `CONTEXT.md` and `docs/adr/` at
+  repo root before exploring, but "if any of these files don't exist,
+  proceed silently." Neither `CONTEXT.md` nor `docs/adr/` exist in this repo
+  yet, so there's no glossary/ADR conflict to flag.
+- `docs/research/` — repeated existing precedent (e.g.
+  `docs/research/proxy-shadow-pricing.md`, `docs/research/voidllm-evaluation.md`,
+  `docs/research/qwen3.8-27b-tool-calling.md`) confirms this is the
+  established location and Markdown format for this kind of investigation;
+  this file follows that convention.
+
+## Provenance note on the top-level config key name
+
+The docs page fetched live (`docs.litellm.ai/docs/completion/knowledgebase`)
+shows `vector_store_registry` as the current top-level key. The original
+feature PR ([#10448](https://github.com/BerriAI/litellm/pull/10448)) used
+`vector_stores` in its example YAML. If implementing against a specific
+pinned LiteLLM version, verify the exact key against that version's docs/
+source rather than assuming either name — this repo's `docker-compose.yml`
+pins `ghcr.io/berriai/litellm:main-stable`, a rolling tag, so the schema in
+whatever image is actually pulled should be spot-checked (e.g. `GET /openapi.json`
+against the running proxy, or grepping the image's installed
+`litellm/types/router.py` / proxy schema) before writing config against it.

docs/research/litellm-searxng-search.md

@@ -0,0 +1,260 @@
+# Research: Wiring the local SearXNG instance into LiteLLM's web-search feature
+
+**Question:** How does LiteLLM's web-search integration
+(https://docs.litellm.ai/docs/search) actually work, and what does wiring the
+local SearXNG instance (`http://search.home/`) into this repo's
+`litellm-config.yaml` require?
+
+**Answer, short version:** SearXNG **is** a natively supported provider for
+LiteLLM's `/v1/search` feature — no custom-endpoint workaround needed. But the
+feature is **not** a model-callable tool and **not** automatic
+context-injection into chat completions either — it's a **separate REST API**
+(`/v1/search/{search_tool_name}`) that a caller (Open WebUI, a script, a
+future MCP wrapper) must invoke directly, independent of any LLM call. That
+sidesteps this project's known-flaky Qwen3.8-27B tool-calling entirely, as
+long as nothing wraps the endpoint back into model-driven tool-calling.
+Reachability is the real blocker: `search.home` is a LAN mDNS/local-DNS name
+that the `litellm` container cannot resolve by default — needs an
+`extra_hosts` entry in `docker-compose.yml`.
+
+## 1. What LiteLLM's search feature actually is
+
+LiteLLM ships a **unified search API** (`/v1/search` and
+`/v1/search/{search_tool_name}`) that wraps multiple search-provider backends
+behind one Perplexity-compatible request/response shape.
+Source: https://docs.litellm.ai/docs/search
+
+Config shape in `config.yaml` (LiteLLM's own documented example, Perplexity
+shown, same shape for every provider):
+
+```yaml
+search_tools:
+  - search_tool_name: perplexity-search
+    litellm_params:
+      search_provider: perplexity
+      api_key: os.environ/PERPLEXITYAI_API_KEY
+```
+
+Call shape:
+
+```bash
+curl http://0.0.0.0:4000/v1/search/searxng-search \
+  -H "Authorization: Bearer sk-1234" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "latest AI developments", "max_results": 5}'
+```
+
+Source: https://docs.litellm.ai/docs/search
+
+**18 providers are listed as supported**, including Perplexity, Tavily, Exa
+AI, Brave, Parallel AI, Google PSE, DataForSEO, Firecrawl, **SearXNG**,
+Linkup, Serper, DuckDuckGo, SearchAPI.io, You.com, APISerpent, Bedrock
+AgentCore, Nimble, and Bing Grounding.
+Source: https://docs.litellm.ai/docs/search
+
+## 2. Is SearXNG natively supported? — Yes
+
+SearXNG is one of the 18 built-in `search_provider` values, added by
+BerriAI/litellm PR #16259 ("[Feat] add serxng search API provider").
+Source: https://github.com/BerriAI/litellm/pull/16259
+
+Config shape for SearXNG specifically:
+
+```yaml
+search_tools:
+  - search_tool_name: searxng-search
+    litellm_params:
+      search_provider: searxng
+      api_base: https://your-searxng-instance.com
+```
+
+Equivalently, the base URL can be supplied via the `SEARXNG_API_BASE`
+environment variable instead of an inline `api_base` key — SearXNG has no API
+key of its own (it's an unauthenticated local meta-search engine), so this is
+the one provider in the list where `litellm_params` doesn't need a secret.
+Sources: https://github.com/BerriAI/litellm/pull/16259,
+https://docs.litellm.ai/docs/search
+
+**For this repo**, the addition to `litellm-config.yaml` (research only — not
+applied here) would look like:
+
+```yaml
+search_tools:
+  - search_tool_name: searxng-search
+    litellm_params:
+      search_provider: searxng
+      api_base: http://search.home/
+```
+
+No custom-endpoint or "generic OpenAI-compatible /v1/web_search" fallback is
+needed — the concern in the ticket that LiteLLM's docs "may assume a hosted
+provider like Tavily/Serper" turned out not to apply; SearXNG is a first-class
+`search_provider` value, same shape as every hosted one.
+
+## 3. Tool-call vs. automatic injection vs. a third thing
+
+The ticket asked to determine whether this is (a) a tool the model must
+explicitly call, or (b) automatic pre-retrieval/context-injection like
+Perplexity's own search-augmented answers. **It's neither** — it's a
+**standalone REST endpoint** that sits alongside `/v1/chat/completions`, not
+wired into it:
+
+> The documentation indicates this is a separate REST endpoint the
+> application calls directly. The page presents `/search` as a standalone API
+> endpoint alongside chat completions, not as an automatic injection feature.
+> Users explicitly invoke the search endpoint; LiteLLM does not automatically
+> inject search results into completions.
+
+Source: https://docs.litellm.ai/docs/search (fetched content, describing the
+`/v1/search` and `/v1/search/{search_tool_name}` endpoints as siblings of
+`/v1/chat/completions`, not a chat-completion parameter or automatic
+retrieval step)
+
+Practical effect: whatever calls this endpoint — Open WebUI's own web-search
+feature, a shell script, a future MCP server — does so with a plain HTTP call.
+**LiteLLM's model routing and Qwen3.8-27B's tool-calling reliability are not
+in that path at all**, unless something downstream chooses to expose this
+endpoint back to the model *as* a function-calling tool (e.g. an MCP wrapper
+that hands the model a `web_search` tool definition backed by this endpoint —
+that would reintroduce the model-must-emit-a-correct-tool-call problem, but
+that's a choice made one layer up, not something LiteLLM's `/v1/search`
+feature forces).
+
+## 4. Network reachability: `search.home` from inside the `litellm` container
+
+`docker-compose.yml`'s `litellm` service joins only the `ai-stack` bridge
+network (`networks: [ai-stack]`, line 116) and gets DNS resolution from
+Docker's embedded DNS server for that network — which resolves other
+containers by service/container name (`llama-server`, `qdrant`, etc., as
+already used at `api_base: http://llama-server:8080/v1` in
+`litellm-config.yaml` line 8) but has **no visibility into the LAN's mDNS/
+local-DNS namespace** that resolves `search.home` on the host machine or on
+LAN clients. So `http://search.home/` will not resolve from inside the
+`litellm` container as configured today — this matches the ticket's
+suspicion, and is standard Docker bridge-networking behavior, not specific to
+this repo.
+Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` (litellm service, lines
+94–124; `networks:` block, lines 154–156)
+
+No `extra_hosts`, `host.docker.internal`, or `network_mode: host` pattern
+exists yet anywhere in this compose file to crib from — this would be the
+first. (One service, `lazytainer`, already uses `network_mode: host`, but for
+an unrelated reason — Docker-socket/host-port introspection — and switching
+`litellm` to host networking would be a much bigger blast-radius change than
+this ticket needs, dropping the `ai-stack` network isolation for every other
+port `litellm` exposes.)
+Source: `g:\_DEV\repos\LLM-Server\docker-compose.yml` lines 144–152
+
+**Recommendation: `extra_hosts` on the `litellm` service**, mapping
+`search.home` to its LAN IP, e.g.:
+
+```yaml
+litellm:
+  ...
+  extra_hosts:
+    - "search.home:192.0.2.10"   # replace with SearXNG's actual LAN IP
+```
+
+This is the smallest, most local fix: one line, scoped to the one service
+that needs it, no change to network topology or isolation, and it keeps
+`litellm-config.yaml`'s `api_base: http://search.home/` value human-readable
+(matching this repo's existing preference for symbolic hostnames like
+`ai.home` / `proxy.ai.home` documented in `docs/network-access.md`) rather
+than hardcoding the LAN IP directly into the YAML config. The IP needs to stay
+in sync if SearXNG's host ever gets a new DHCP lease — same caveat that would
+apply to any hardcoded-IP alternative, just isolated to one `extra_hosts`
+line instead of buried in the search config.
+
+`host.docker.internal` is not applicable here: that special hostname
+resolves to the Docker **host's** own IP (useful for reaching a service
+running directly on the host machine's loopback), not to arbitrary LAN mDNS
+names — it wouldn't help resolve `search.home` unless SearXNG happens to run
+on the same physical host as this compose stack.
+
+## 5. Interaction with this project's known-flaky Qwen3.8-27B tool-calling
+
+`docs/research/qwen3.8-27b-tool-calling.md` (2026-08-24) found, with medium-
+to-high confidence, that Qwen3.8-27B's tool-calling through llama.cpp
+inherits open/partially-fixed upstream parser bugs from the Qwen3.5 lineage
+(issues #21158, #20837 in `ggml-org/llama.cpp`) — tool calls can be emitted
+but not recognized, or land as inert XML inside a reasoning block, especially
+with thinking enabled.
+Source: `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md`
+(section 3, "Bottom line" section)
+
+Given section 3 above (`/v1/search` is a standalone endpoint, not a
+model-tool), **that flakiness has no bearing on the recommended integration
+path**: nothing about calling `POST /v1/search/searxng-search` from Open
+WebUI or a script asks Qwen3.8-27B to emit a tool call at all. The search
+happens (or doesn't) independent of the model's tool-calling grammar/parser
+entirely.
+
+The risk **would** resurface only if a *different* design choice is made
+later — e.g. wrapping this same SearXNG-backed endpoint as an MCP tool or a
+`tools=[...]` function definition handed to Qwen3.8-27B in a chat-completion
+request, so the model itself decides when to search. That path would inherit
+every bug documented in `qwen3.8-27b-tool-calling.md` (calls silently dropped,
+calls trapped inside `<think>` blocks, etc.) and would need the live smoke
+test that doc recommends before being trusted unattended. **That is not what
+LiteLLM's `/v1/search` feature itself requires** — it's a choice a caller
+could additionally make on top of it.
+
+## Recommendation
+
+1. Add a `search_tools` block to `litellm-config.yaml` using
+   `search_provider: searxng` and `api_base: http://search.home/` (see
+   section 2) — no custom/generic-endpoint workaround needed, this is a
+   first-class supported provider.
+2. Add `extra_hosts: ["search.home:<LAN IP>"]` to the `litellm` service in
+   `docker-compose.yml` (see section 4) so the container can resolve the
+   hostname; confirm the IP is stable (static DHCP reservation) since
+   `extra_hosts` is a static mapping baked in at container start.
+3. Treat `/v1/search` as a plain HTTP integration point, not a model tool —
+   whatever calls it (Open WebUI, a script) should call the REST endpoint
+   directly rather than exposing it to Qwen3.8-27B as a function-calling
+   tool, to avoid inheriting this project's documented tool-calling
+   flakiness (section 5). If model-driven search-tool-calling is wanted
+   later, that's a separate decision that should be smoke-tested against the
+   caveats in `qwen3.8-27b-tool-calling.md` first.
+
+This is research only — `litellm-config.yaml` and `docker-compose.yml` are
+not modified by this doc.
+
+## Sources
+
+- https://docs.litellm.ai/docs/search — LiteLLM search feature docs:
+  endpoints, `search_tools` config shape, provider list, standalone-endpoint
+  behavior.
+- https://github.com/BerriAI/litellm/pull/16259 — SearXNG provider
+  implementation: `search_provider: searxng`, `api_base` /
+  `SEARXNG_API_BASE` config.
+- `g:\_DEV\repos\LLM-Server\docker-compose.yml` — `litellm` service
+  definition (lines 94–124), `ai-stack` network block (lines 154–156),
+  `lazytainer`'s `network_mode: host` precedent (lines 144–152).
+- `g:\_DEV\repos\LLM-Server\litellm-config.yaml` — current proxy config
+  conventions (`model_list`, `litellm_params`, `api_base` usage at line 8).
+- `g:\_DEV\repos\LLM-Server\docs\network-access.md` — this repo's existing
+  `*.home` / NPM hostname conventions.
+- `g:\_DEV\repos\LLM-Server\docs\research\qwen3.8-27b-tool-calling.md` —
+  prior findings on Qwen3.8-27B tool-calling reliability via llama.cpp.
+
+## Confidence/uncertainty summary
+
+- **High confidence:** SearXNG is a native, first-class `search_provider` in
+  LiteLLM's `/v1/search` feature (directly documented and confirmed via the
+  implementing PR); the feature is a standalone REST endpoint separate from
+  chat completions, not automatic context-injection and not itself a
+  model-callable tool.
+- **Medium confidence:** the exact field name (`api_base` vs. relying solely
+  on `SEARXNG_API_BASE`) — two independent fetch passes against
+  docs.litellm.ai returned slightly different renderings of the same example
+  (one showed `api_key: os.environ/SEARXNG_API_BASE`, the other and the PR
+  fetch showed `api_base: <url>`); the PR-sourced `api_base` form is treated
+  as authoritative here since it comes from the implementing code change, but
+  this should be smoke-tested against the actual deployed LiteLLM image
+  (`ghcr.io/berriai/litellm:main-stable`) before being relied on verbatim.
+- **Not independently verified:** SearXNG's actual LAN IP/hostname stability
+  on this network, and whether the deployed LiteLLM version
+  (`main-stable`, per `docker-compose.yml` line 95) already includes PR
+  #16259 — worth a quick `docker exec litellm pip show litellm` / changelog
+  check before wiring this in for real.

docs/research/omniroute-qwen-websearch.md

@@ -0,0 +1,330 @@
+# OmniRoute + Qwen Code CLI web search — setup research
+
+Investigates how to (a) confirm/complete OmniRoute's routing to this stack's local
+Qwen model, and (b) enable Qwen Code CLI's web-search tool, for a user running
+`qwen` from WSL against this repo's docker-compose stack.
+
+## What's already configured (verified live in WSL, 2026-09-05)
+
+Checked via `wsl.exe -- bash -lc '...'` against `~/.qwen/`:
+
+- **qwen-code CLI is installed**: `which qwen` → `/home/haylan/.local/bin/qwen`, `qwen --version` → `0.23.0`.
+- **`~/.qwen/settings.json` already points at this stack's OmniRoute gateway**, in the exact shape OmniRoute's own `setup-qwen` command produces (see below):
+  ```json
+  "modelProviders": {
+    "openai": [
+      {
+        "id": "qwen3.8-27b-local//models/Qwen3.8-27B-UD-Q4_K_XL.gguf",
+        "name": "qwen3.8-27b-local",
+        "envKey": "OMNIROUTE_API_KEY",
+        "baseUrl": "http://proxy-ai.home/v1",
+        "generationConfig": { "contextWindowSize": 131072 }
+      }
+    ]
+  },
+  "security": { "auth": { "selectedType": "openai" } },
+  "model": {
+    "name": "qwen3.8-27b-local//models/Qwen3.8-27B-UD-Q4_K_XL.gguf",
+    "baseUrl": "http://proxy-ai.home/v1"
+  }
+  ```
+  This targets `http://proxy-ai.home/v1` (this repo's OmniRoute gateway hostname per `docs/network-access.md`), reads the API key from the `OMNIROUTE_API_KEY` env var, and matches `docs/coding-cli-setup.md`'s convention of naming the registered provider `qwen3.8-27b-local`. Two backup files (`settings.json.bak-cbm-*`, `settings.json.save`) show earlier iterations of the same config — this was set up deliberately, not a stray default.
+  - **Not a gap — verified correct**: `contextWindowSize: 131072` matches `LLAMA_CTX_SIZE / LLAMA_PARALLEL` (`262144 / 2`), not half of it. `docker-compose.yml` (lines 21–22) runs llama-server with `--ctx-size ${LLAMA_CTX_SIZE:-262144} --parallel ${LLAMA_PARALLEL:-2}`, and `.env.example` (line 39) spells out that each of the two concurrent slots gets `LLAMA_CTX_SIZE / LLAMA_PARALLEL` tokens — i.e. 131072 per slot, matching commit `23e90fe` ("cap concurrent slots at 2"). So `~/.qwen/settings.json`'s value is correctly sized to what one slot actually offers; no fix needed here.
+  - **Unverified**: whether `OMNIROUTE_API_KEY` is actually set in the WSL environment or in a `~/.qwen/.env` file — `env | grep -i qwen` in the same session showed no `OMNIROUTE_API_KEY` in the *current* shell (only `PATH` entries matched `qwen`), and `~/.qwen/.env` wasn't checked (missed in the executed probe — see Open questions). If it's unset, `qwen` calls will fail auth against OmniRoute regardless of the `web_search` setup below.
+- **No web-search config exists yet**: `env | grep -i tavily` and `env | grep -i search` both returned nothing; `settings.json` has no `tools.webSearch` key and no `mcpServers` entry for Tavily/Bailian/GLM search or for OmniRoute's own MCP server (it does have an unrelated `mcpServers.codebase-memory-mcp` stdio entry for this repo's own codebase-memory tool). `grep -ril "tavily\|websearch\|web_search\|web-search" ~/.qwen` matched only unrelated project chat-log files (from an unrelated `shopware-420-seeds` project), not any config.
+- **Conclusion**: model routing (a) is already done. Web search (b) is not configured at all — no API key, no MCP server, no built-in-tool setting.
+
+## (a) OmniRoute → local Qwen model routing
+
+Sources: this repo's `docker-compose.yml` (lines 63–133) and `.env.example`
+(lines 54–90); `README.md` §"AI gateway (OmniRoute)"; `docs/coding-cli-setup.md`;
+OmniRoute's own docs at `github.com/mckazzy/OmniRoute-run-qwen`, ref
+`release/v3.8.50`.
+
+**Current repo state**: `docker-compose.yml`'s `omniroute` service comment (lines 63–67)
+states routing is registered "once through the dashboard or `POST /api/providers`
+after first boot, not checked into this repo." `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true`
+and `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true` are already set (lines 84–88) so the
+dashboard/API will accept `llama-server`'s container-internal address instead of
+rejecting it as a private URL.
+
+**Confirmed against OmniRoute's own reference docs**
+(`docs/reference/ENVIRONMENT.md` at the pinned ref):
+- `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` — default `false`; the doc says it is
+  **"REQUIRED for self-hosted providers"** (it names LM Studio, Ollama, vLLM,
+  Llamafile, Triton, SearXNG). Confirms the repo's own comment is correct and necessary.
+- `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS` — default `true` ("local-first"); `false`
+  would block localhost/LAN/private ranges outright (cloud-metadata IPs stay
+  blocked either way).
+- `OMNIROUTE_WS_BRIDGE_SECRET` — "REQUIRED in production — when unset, all WS
+  bridge requests are rejected," generated via `openssl rand -base64 32` — matches
+  this repo's comment (lines 89–93) and `scripts/update.sh` autofill.
+
+`docs/reference/PROVIDER_REFERENCE.md` (same ref) lists **`llama-cpp`** as a
+built-in "Local, self-hosted" provider ID:
+
+> "Configure the OpenAI-compatible base URL (default: `http://127.0.0.1:8080/v1`)"
+
+This is a good match for this stack's `llama-server` container, which exposes
+port 8080 only on the internal `ai-stack` Docker network (`docker-compose.yml`
+lines 30–33, "No published host port"). Inside that network the service is
+reachable by its Compose service name, so the base URL to register should be
+`http://llama-server:8080/v1`, not `127.0.0.1` (127.0.0.1 inside the OmniRoute
+container would mean OmniRoute itself, not llama-server — they're different
+containers on the same bridge network).
+
+**Concrete steps** (dashboard, matching `docs/proxy-key-onboarding.md`'s
+existing pattern for reaching the dashboard):
+
+1. Reach the dashboard: from the R9700 box, `docker inspect -f
+   '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`, then browse
+   `http://<that-ip>:20128`; from elsewhere, SSH-tunnel
+   `ssh -L 20128:<container-ip>:20128 <host>` then browse `localhost:20128`.
+2. Providers → Add provider → **llama.cpp** (`llama-cpp` provider ID per
+   `PROVIDER_REFERENCE.md`).
+3. Set base URL to `http://llama-server:8080/v1` (the Compose service name — both
+   containers share the `ai-stack` network per `docker-compose.yml`'s `networks:
+   [ai-stack]` on both services). No API key needed (llama-server's endpoint is
+   unauthenticated internally, per `docs/network-access.md`).
+4. Register the model under that provider using the naming this repo already
+   assumes downstream (`qwen3.8-27b-local`, per `docs/coding-cli-setup.md` line 8)
+   — pick a model ID/name here and keep it consistent everywhere a CLI config
+   references it (`~/.qwen/settings.json`'s existing entry already assumes this name).
+5. Mint or reuse a virtual API key for the `qwen-code-cli` workload per
+   `docs/proxy-key-onboarding.md` (label `qwen-code-cli`), and confirm it's the
+   value behind `OMNIROUTE_API_KEY` in the WSL environment (or `~/.qwen/.env` —
+   see Open questions) that `~/.qwen/settings.json`'s `envKey` references.
+
+**OmniRoute's own automation for this exact CLI** — `docs/guides/CLI-INTEGRATIONS.md`
+at the pinned ref documents a dedicated `omniroute setup-qwen` command:
+
+> `omniroute setup-qwen --model qwen/qwen3.8-max-preview` — writes
+> `~/.qwen/settings.json` (V4 `modelProviders.openai` array) and stores
+> `OMNIROUTE_API_KEY` in `~/.qwen/.env`; supports `--yes` (non-interactive),
+> `--config-path` / `--env-path` (custom locations), and works in local or remote mode.
+
+The `~/.qwen/settings.json` found on this machine has exactly the V4
+`modelProviders.openai` shape this command produces, and the two `.bak`/`.save`
+files back that up — this was very likely already run once, pointed at whichever
+model ID was registered in the dashboard at the time (the `id` field embeds the
+GGUF filename, `qwen3.8-27b-local//models/Qwen3.8-27B-UD-Q4_K_XL.gguf`, matching
+`.env.example`'s `LLAMA_MODEL_FILE`). Re-running it after registering/renaming
+the provider in step 2–4 above is the fastest way to refresh this file if the
+registered model ID ever changes (`contextWindowSize: 131072` itself is already
+correct — see note above on `--parallel`).
+
+## (b) Qwen Code CLI web search
+
+Sources: `qwenlm.github.io/qwen-code-docs/en/developers/tools/web-search/`,
+`.../en/developers/tools/mcp-server/`, `.../en/users/configuration/settings/`;
+OmniRoute's `docs/frameworks/MCP-SERVER.md` and `docs/reference/PROVIDER_REFERENCE.md`
+at `release/v3.8.50`.
+
+**Qwen Code's web-search docs page states plainly**: the *original* built-in
+`web_search` tool ("Tavily/Google/GLM/DashScope multi-provider") **"and its
+configuration were removed."** Current options, per that same page:
+
+1. **New built-in `web_search` tool** — DashScope-only now, not multi-provider.
+   Needs `tools.webSearch.enabled: true` and `tools.webSearch.model` (e.g.
+   `"qwen3.6-plus"`) in `settings.json`, or equivalent env vars if `settings.json`
+   can't be edited; requires a `DASHSCOPE_API_KEY` (Alibaba Cloud). It "issues a
+   self-contained search request to a small auxiliary model with DashScope's
+   server-side `web_search` (and `web_extractor`) tools, and returns the
+   narrated findings plus source URLs" — i.e. it calls out to Alibaba's cloud,
+   not this stack's local model or SearXNG.
+   - **Caveat**: `users/configuration/settings/` (the canonical settings-schema
+     page) does **not** list `tools.webSearch` anywhere among its documented
+     `tools.*` keys — only `tools.sandbox`, `tools.shell`, `tools.core`,
+     `tools.exclude`, `tools.disabled`. This key may be genuinely undocumented
+     there, or newer than that page's last update. Treat `tools.webSearch` as
+     unconfirmed against the settings schema itself — verify with `qwen --help`
+     or by testing once a `DASHSCOPE_API_KEY` is available (see Open questions).
+2. **MCP-based search** — three named services: Alibaba Cloud Bailian WebSearch,
+   Tavily WebSearch, GLM WebSearch Prime — each added as an `mcpServers` entry
+   in `settings.json`. Confirmed schema from `developers/tools/mcp-server/`:
+   HTTP/SSE servers use `httpUrl` (or `url` for SSE) plus an optional `headers`
+   object, e.g.:
+   ```json
+   { "mcpServers": { "tavily": {
+       "httpUrl": "https://mcp.tavily.com/mcp/?tavilyApiKey=${TAVILY_API_KEY}"
+   } } }
+   ```
+   (stdio servers instead use `command`/`args`/`env`/`cwd`, as the existing
+   `codebase-memory-mcp` entry in this machine's `~/.qwen/settings.json` does.)
+
+**Neither of Qwen Code's own two paths uses this stack's existing SearXNG
+integration.** But OmniRoute — already in front of this stack's model — has its
+own MCP server with a **built-in multi-provider web-search tool**, and this
+repo already wires SearXNG through OmniRoute (`README.md` §"Web search":
+"The gateway also fronts SearXNG-backed web search"; `.env.example`'s
+`SEARXNG_LAN_IP` / `search.home` extra_hosts entry in `docker-compose.yml`
+lines 106–109). OmniRoute's `docs/frameworks/MCP-SERVER.md` (pinned ref):
+
+> "Web search through OmniRoute search gateway
+> (Serper/Brave/Perplexity/Exa/Tavily/Google PSE/Linkup/SearchAPI/SearXNG) with
+> failover" — exposed as an `omniroute_web_search` tool requiring the
+> `execute:search` scope.
+
+And `docs/reference/PROVIDER_REFERENCE.md` lists `searxng-search` as one of its
+12 built-in search-provider IDs: **"API key is optional. Set your SearXNG base
+URL. Some instances may require a bearer token for access."** — meaning
+SearXNG can be registered as a search provider in the OmniRoute dashboard the
+same way `llama-cpp` is registered as a model provider, no separate API key
+needed for a self-hosted SearXNG instance.
+
+**This means the path that reuses what's already deployed in this stack (SearXNG,
+already reachable from OmniRoute via `search.home`) is: connect qwen-code to
+OmniRoute's MCP server, not to Tavily/DashScope/GLM directly.** Concrete steps:
+
+1. In the OmniRoute dashboard, register SearXNG as a search provider
+   (`searxng-search`), pointing at `http://search.home` (already resolvable
+   inside the OmniRoute container via the `extra_hosts` entry in
+   `docker-compose.yml`). This may already be done — `README.md` implies the
+   gateway already fronts SearXNG-backed search, but confirm live in the
+   dashboard since, per the same `docker-compose.yml` comment (lines 63–67),
+   provider registration isn't checked into this repo.
+2. Mint an API key scoped for MCP search use — OmniRoute's `MCP-SERVER.md`
+   names `execute:search` (to actually call the search tool) and `mcp:connect`
+   (narrow, MCP-connect-only) as the relevant scopes; `manage`/`admin` also work
+   but are broader than needed.
+3. Add an `mcpServers` entry to `~/.qwen/settings.json` pointing at OmniRoute's
+   MCP endpoint, using the same `httpUrl`/`headers` shape Qwen Code already
+   supports for Tavily:
+   ```json
+   {
+     "mcpServers": {
+       "omniroute-search": {
+         "httpUrl": "http://proxy-ai.home/api/mcp/stream",
+         "headers": { "Authorization": "Bearer ${OMNIROUTE_SEARCH_KEY}" }
+       }
+     }
+   }
+   ```
+   (`proxy-ai.home` matches the hostname the model-provider entry already uses
+   in this same file; swap in whatever host:port actually fronts OmniRoute's API
+   port from WSL — `docs/network-access.md` says `proxy-ai.home` points at
+   `${OMNIROUTE_PORT:-4000}`, the *API* port, and `docker-compose.yml`/`.env.example`
+   separately track `OMNIROUTE_API_PORT` (default `20129`) as the
+   container-internal port — confirm which one NPM actually proxies to before
+   trusting the `/api/mcp/stream` path resolves through `proxy-ai.home` unchanged;
+   this wasn't independently verified against a live instance, see Open questions.)
+4. Set `OMNIROUTE_SEARCH_KEY` in the WSL shell profile (or in `~/.qwen/.env`,
+   consistent with how `setup-qwen` already stores `OMNIROUTE_API_KEY` there).
+5. Restart `qwen`; the model should now see an MCP tool for web search backed by
+   this stack's own SearXNG, routed and rate-limited the same way its LLM calls
+   already are.
+
+If instead the goal is simply "get *any* web search working fastest, reuse
+nothing," the plain Tavily-MCP or DashScope built-in-tool paths above are
+simpler (one API key, no dashboard provider registration) — but they bypass
+this stack's OmniRoute/SearXNG setup entirely and send queries to an external
+paid API instead.
+
+## Follow-up verification (2026-09-05, live checks)
+
+- **`OMNIROUTE_API_KEY` — confirmed set and working.** It's exported from
+  `~/.bashrc` (line 133), *not* `~/.qwen/.env` — invisible to a non-interactive
+  `bash -lc` probe because `.bashrc`'s standard top-of-file guard
+  (`case $- in *i*) ;; *) return;; esac`) skips the rest of the file for
+  non-interactive shells; a real interactive shell (`bash -ic`, or `wsl` +
+  `qwen` as actually run) sources it fine. Verified: `curl -H "Authorization:
+  Bearer $OMNIROUTE_API_KEY" http://proxy-ai.home/v1/models` → `200`. Routing (a)
+  is confirmed end-to-end, no action needed.
+- **MCP endpoint located from primary source** — fetched OmniRoute's
+  `docs/frameworks/MCP-SERVER.md` at `release/v3.8.50` directly. Resolves the
+  port ambiguity above: the MCP server runs on **port 20128** (dashboard/API
+  port), paths `/api/mcp/stream` (streamable HTTP), `/api/mcp/sse`, and
+  `/api/mcp/status`. It states: `/api/mcp/*` is in OmniRoute's `LOCAL_ONLY` authz
+  tier (`src/server/authz/routeGuard.ts`) — loopback-only by default; a
+  non-loopback client needs a key carrying the `manage` scope or the narrower
+  `mcp:connect` scope (added v3.8.0), and the server's Settings must have
+  `mcpEnabled` on with the matching `mcpTransport` selected. `omniroute_web_search`
+  itself additionally needs `execute:search`. No separate "MCP key type" exists —
+  same provider keys, different scopes.
+- **Live probe result**: `curl http://proxy-ai.home:20128/api/mcp/status` returns
+  `{"error":{"code":"AUTH_001","message":"Authentication required"}}` **identically
+  with or without** the `Authorization: Bearer $OMNIROUTE_API_KEY` header — the
+  existing model-routing key isn't recognized on this route at all, consistent
+  with it lacking `mcp:connect`/`manage`/`execute:search` scope and/or
+  `mcpEnabled` not yet being turned on in the dashboard. This is dashboard-side
+  state (not in git, no session credentials available from this environment) —
+  genuinely needs a human with dashboard access, not another probe.
+- **Config prepared** to unblock as soon as that's done: added an
+  `omniroute-search` entry to `~/.qwen/settings.json`'s `mcpServers` (backed up
+  the prior file first as `settings.json.bak-wayfinder-<timestamp>`):
+  ```json
+  "omniroute-search": {
+    "httpUrl": "http://proxy-ai.home:20128/api/mcp/stream",
+    "headers": { "Authorization": "Bearer ${OMNIROUTE_SEARCH_KEY}" }
+  }
+  ```
+  Deliberately a separate env var (`OMNIROUTE_SEARCH_KEY`), not reusing
+  `OMNIROUTE_API_KEY`, so the search-scoped key stays distinct from the
+  model-routing key — matches `docs/proxy-key-onboarding.md`'s per-workload
+  key pattern.
+
+## Resolution (2026-09-05, completed)
+
+The dashboard steps above turned out to need a different diagnosis than
+originally guessed — walked through live with a `oma_live_...` management
+token and a rotating set of `sk-...` provider keys the user supplied:
+
+- **`/api/providers` (management API) showed zero search providers at all**
+  — not a misconfigured `searxng-search` entry, it simply didn't exist as a
+  connection anymore (9 connections total, all LLM/chat providers). The
+  CHANGELOG at the pinned ref was checked and shows `/v1/search` under active
+  feature development (a `feat(search)` entry adding Firecrawl support), so
+  this wasn't an OmniRoute-side removal of the search system — the row was
+  just gone from this instance's own database (reason unconfirmed: update
+  migration vs. prior manual removal).
+- **Created it via the API**, not the dashboard UI — `POST /api/providers`
+  turned out to accept the same generic connection schema used for LLM
+  providers: `{"provider":"searxng-search","name":"searxng"}` (Zod-validated;
+  an empty-body POST surfaced the required fields). Then
+  `PATCH /api/providers/<id>` with `{"providerSpecificData":{"baseUrl":"http://search.home/search"}}`
+  set the real URL, replacing the catalog default.
+- **Verified end-to-end**: `POST /v1/search` with `provider: "searxng-search"`
+  returned real results (5 hits, `search_cost_usd: 0`, `cached: false`,
+  `response_time_ms: 4495`) — confirms `search.home`'s `extra_hosts` mapping
+  in `docker-compose.yml` resolves correctly from inside the OmniRoute
+  container and the whole chain (OmniRoute → SearXNG → results) works.
+- **`/api/mcp/status` confirmed `scopesEnforced: false`** on this instance —
+  the `mcp:connect`/`execute:search` scope requirement documented upstream
+  isn't actually being enforced here, so any valid provider key connects.
+  `mcpEnabled: true` already, transport `streamable-http`.
+- **Key rotation caveat hit live**: the first `sk-...` key the user shared
+  went from working to a flat 401 on *every* route (including `/v1/models`)
+  partway through testing — consistent with it having been revoked/rotated
+  server-side. A second key worked immediately. If this setup stops working
+  later, check for exactly this before re-diagnosing the whole chain.
+- **Final `~/.qwen/settings.json` `mcpServers` entry** (confirmed connected
+  via `qwen mcp list` → `✓ omniroute-search: ... - Connected`):
+  ```json
+  "omniroute-search": {
+    "httpUrl": "http://proxy-ai.home/api/mcp/stream",
+    "headers": { "Authorization": "Bearer ${OMNIROUTE_SEARCH_KEY}" }
+  }
+  ```
+  `OMNIROUTE_SEARCH_KEY` is exported in `~/.bashrc` alongside the existing
+  `OMNIROUTE_API_KEY`, holding the second (working) `sk-...` key.
+
+**Status: done.** `qwen` in WSL has a connected `omniroute-search` MCP server
+backed by this stack's own SearXNG instance — no external search API, no
+Alibaba DashScope dependency. Not yet exercised: an actual `qwen` chat turn
+that triggers the `omniroute_web_search` tool call (only the MCP handshake
+and the raw `/v1/search` call were verified directly).
+
+## Open questions / unverified
+
+- **`tools.webSearch.*` settings.json schema** — described on Qwen Code's
+  web-search doc page but absent from the canonical settings-schema page; not
+  independently confirmed (e.g. via `qwen --help` or source) — moot for this
+  setup since the MCP path (above) is what's being wired in, not the
+  DashScope-only built-in tool.
+- **DashScope vs SearXNG data-residency/cost tradeoffs** — out of scope here,
+  but worth noting the built-in `web_search` tool sends queries to Alibaba
+  Cloud regardless of this stack being otherwise fully self-hosted.
+- OmniRoute's own docs (already flagged in this repo's `README.md`) describe
+  stealth/anti-detection and TLS-interception features elsewhere in its repo;
+  none of that is exercised by anything in this note, but it's the same caveat
+  `README.md` already carries forward from issue #31.

docs/research/opencode-cli-setup.md

@@ -0,0 +1,227 @@
+# OpenCode CLI: install + local llama.cpp provider config
+
+**Date:** 2026-08-24
+**Scope:** How to install OpenCode CLI and point it at this repo's llama.cpp OpenAI-compatible server
+(`http://localhost:8080/v1`, serving Qwen3.8-27B) instead of a hosted provider.
+
+## 0. Which "opencode" is this?
+
+Confirmed project: **the OpenCode coding agent**, docs at https://opencode.ai/docs/, repo currently at
+**github.com/anomalyco/opencode**.
+
+- The repo was **formerly `sst/opencode`** and was moved to `anomalyco/opencode` as part of the SST
+  team's org rebrand to "Anomaly." Multiple in-repo issues track fallout from the move (stale
+  `ghcr.io/sst/opencode` Docker tips, Homebrew tap migration, tool integrations still pointing at the
+  old path). Sources:
+  https://github.com/anomalyco/opencode/issues/8390 ,
+  https://github.com/anomalyco/opencode/issues/6841 ,
+  https://github.com/anomalyco/opencode/issues/16440 ,
+  https://news.ycombinator.com/item?id=46552218 (community confirmation of the org move).
+  **Confidence: high** — corroborated by several independent primary-source issues in the same repo,
+  though I did not find an explicit "we renamed the org" announcement post to cite as the single
+  authoritative statement (the docs site itself just says "OpenCode" with no history section).
+- **Naming collision to avoid**: `opencode-ai/opencode` (a different, unrelated Go/Bubble Tea TUI
+  project, MIT licensed, also calling itself "a powerful AI coding agent") turned up in search results
+  and is **not** this project. The ticket's target — `sst/opencode` — is the TypeScript/Bun project now
+  at `anomalyco/opencode`, confirmed by docs domain `opencode.ai` matching the install script URL used
+  by both the old and current repo. Do not install `opencode-ai/opencode` by mistake — check `opencode
+  --version` output / package name (`opencode-ai` on npm) if unsure.
+  Source: https://github.com/opencode-ai/opencode (surfaced in search, cross-checked and excluded).
+
+## 1. Install
+
+Official methods, from https://opencode.ai/docs/ and https://github.com/anomalyco/opencode README
+(fetched directly):
+
+```bash
+# curl install script (recommended by the docs)
+curl -fsSL https://opencode.ai/install | bash
+
+# npm
+npm i -g opencode-ai@latest
+
+# Homebrew
+brew install anomalyco/tap/opencode
+```
+
+Also documented as available: Bun, pnpm, Yarn, Arch `pacman`/`paru`, Windows Chocolatey/Scoop, Mise,
+Docker. Source: https://opencode.ai/docs/ (Installation section).
+
+**Confidence: high** — these are the exact commands returned by fetching the docs and GitHub README
+directly, not a paraphrase from a third-party blog.
+
+## 2. Config file: location and format
+
+Format: **JSON or JSONC** (JSON with comments). File name: `opencode.json` (or `opencode.jsonc`).
+
+- **Global**: `~/.config/opencode/opencode.json` — user-wide settings.
+- **Project**: `opencode.json` in the project root — picked up per-project, overrides/merges with
+  global.
+- A separate `tui.json` (same locations) holds TUI-only settings (themes, keybinds) — not needed for
+  provider config.
+- Schema for editor validation/autocomplete: `"$schema": "https://opencode.ai/config.json"`.
+
+Source: https://opencode.ai/docs/config/ (fetched directly).
+**Confidence: high.**
+
+## 3. Declaring a local OpenAI-compatible provider (llama.cpp)
+
+Two-part setup per the docs:
+
+1. Optionally run `/connect` inside OpenCode, choose "Other", and give the provider a unique ID — this
+   only stores a credential (which can be left blank/dummy for a local no-auth server) and does **not**
+   generate the provider config block itself.
+2. Hand-write the provider block in `opencode.json` (project or global). Exact structure from
+   https://opencode.ai/docs/providers/ (fetched directly, JSON quoted verbatim from the docs):
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "provider": {
+    "llamacpp": {
+      "npm": "@ai-sdk/openai-compatible",
+      "name": "llama.cpp (local)",
+      "options": {
+        "baseURL": "http://localhost:8080/v1",
+        "apiKey": "sk-local-not-checked"
+      },
+      "models": {
+        "qwen3.8-27b": {
+          "name": "Qwen3.8-27B",
+          "limit": {
+            "context": 131072,
+            "output": 8192
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+Key fields:
+- **`npm`**: `"@ai-sdk/openai-compatible"` — the Vercel AI SDK adapter OpenCode uses for any server
+  speaking `/v1/chat/completions`. This is what makes it work against llama.cpp's OpenAI-compatible
+  endpoint (this repo's `/v1`), not the Anthropic shim (`/v1/messages` — see §4/§5 below for why the
+  OpenAI-compatible endpoint is the correct target, not the Anthropic shim, for OpenCode specifically).
+- **`options.baseURL`**: the server URL up to and including `/v1` — for this repo,
+  `http://localhost:8080/v1`.
+- **`options.apiKey`**: docs show it as *optional*, using `"{env:VAR_NAME}"` syntax to pull from an
+  env var in the general/hosted-provider example. For a local server that doesn't check the key,
+  supplying any non-empty string (or omitting it) both appear to work per the docs' own local-provider
+  example, which omits `apiKey` entirely for a bare local baseURL. Because the AI SDK's OpenAI-compatible
+  client sometimes still requires a non-empty `Authorization` header to be well-formed, the safer choice
+  is a dummy literal string (e.g. `"sk-local-not-checked"`) rather than omitting the field —
+  **this specific claim (dummy string vs. omit) is not directly confirmed by a docs quote either way**;
+  treat as a practical recommendation, not a documented requirement. **Confidence: medium** on the
+  omit-vs-dummy distinction; **high** on baseURL/npm/models structure itself.
+- **`models.<model-id>`**: the model ID key must match what the server exposes (check
+  `curl http://localhost:8080/v1/models`). `limit.context` / `limit.output` are optional token-limit
+  hints OpenCode uses for context-window bookkeeping — not enforced by the server itself.
+- Then select the model in OpenCode with `<providerId>/<modelId>`, e.g. `llamacpp/qwen3.8-27b`
+  (format documented on the troubleshooting page: "Models should be referenced like so:
+  `<providerId>/<modelId>`"). Source: https://opencode.ai/docs/troubleshooting/.
+
+Source for the whole block: https://opencode.ai/docs/providers/ (fetched directly).
+**Confidence: high** on the JSON shape; the exact llama.cpp `baseURL`/port is this repo's own
+docker-compose (`8080`), not an assumption.
+
+## 4. Documented compatibility notes: self-hosted OpenAI-compatible servers generally
+
+Two relevant, **currently open/closed-not-planned** primary-source issues in `anomalyco/opencode`:
+
+- **Issue #20669** — "Default agent is brittle against local OpenAI-compatible tool-call quirks."
+  Documents two concrete failure modes against local backends (llama.cpp, LM Studio) in OpenCode's
+  agent loop:
+  1. The `bash` tool throws a hard error ("expected string, received undefined") if the model emits a
+     valid `command` but omits the `description` field — some local models don't reliably fill optional
+     tool-call fields the way hosted models do.
+  2. Some local backends return `finish_reason: "tool_calls"` together with an **empty** `tool_calls: []`
+     array; OpenCode's loop doesn't treat this as a normal stop and can loop/hang instead.
+  **Resolution: closed as "not planned"** (maintainers declined to add compatibility shims for this),
+  referencing PR #26419. No documented workaround from the maintainers.
+  Source: https://github.com/anomalyco/opencode/issues/20669 (fetched directly).
+  **Confidence: high** on the issue content; the "not planned" resolution means this is a **live,
+  acknowledged, unfixed risk** for any local OpenAI-compatible backend, not specific to llama.cpp.
+
+- **Issue #1890** — "OpenCode sends tools (and Jinja tool template) to llama.cpp results in 500 error
+  unless `--jinja`; with `--jinja`, template crashes (reject filter)." llama.cpp-specific:
+  - OpenCode unconditionally includes `tools` scaffolding in the request even for plain-chat agents
+    with no tools configured.
+  - Without llama.cpp's `--jinja` server flag: llama.cpp rejects the request outright
+    (`"tools param requires --jinja flag"`).
+  - With `--jinja`: llama.cpp's Jinja renderer crashes on some models' chat templates
+    (`"Value is not callable: null"`, a `reject` filter llama.cpp's Jinja implementation doesn't
+    support) — a 500 error.
+  - Tested against OpenCode v0.4.40/v0.4.41 and a Qwen3-30B model.
+  - Workarounds discussed: run llama.cpp with `--jinja` and a minimal template that ignores tools, or
+    proxy requests to strip `tools`/`tool_choice` before they reach llama.cpp.
+  Source: https://github.com/anomalyco/opencode/issues/1890 (fetched directly).
+  **Confidence: high** on content; **medium** on current relevance to this repo's exact llama.cpp/model
+  version since the issue is against an older OpenCode build and doesn't confirm current-build behavior.
+
+**Practical implication for this repo**: this repo's llama.cpp container should be launched with
+`--jinja` (check `docker-compose.yml` / entrypoint flags) or tool-calling via OpenCode will hard-fail
+with a 500 immediately — and even with `--jinja`, Qwen-family chat templates have a documented history
+of crashing llama.cpp's Jinja renderer (see also this repo's own
+`docs/research/qwen3.8-27b-tool-calling.md`, which independently found llama.cpp's tool-call
+parser/grammar for the Qwen3.5 lineage — which Qwen3.8-27B shares — to be broken in multiple
+still-open llama.cpp upstream issues, unrelated to OpenCode).
+
+## 5. Model-specific notes for Qwen3.8-27B vs. what OpenCode expects from cloud models
+
+- **Tool-calling reliability is the dominant risk, not context/thinking-mode handling.** This repo's
+  own prior research (`docs/research/qwen3.8-27b-tool-calling.md`) independently concluded, via
+  llama.cpp upstream issues, that Qwen3.8-27B's architecture lineage (Qwen3.5, hybrid Gated
+  DeltaNet/attention) has **documented, still-open llama.cpp tool-call parser bugs** — this is a
+  llama.cpp-side problem that affects both the Anthropic shim and the plain OpenAI-compatible endpoint
+  identically, since the Anthropic shim just re-uses llama.cpp's normal chat-completions tool-calling
+  pipeline (source: https://github.com/ggml-org/llama.cpp/pull/17570, per that same file). Combined with
+  OpenCode's own §4 issues, expect tool-calling from OpenCode against this specific model to be
+  **unreliable until upstream llama.cpp fixes land**, independent of OpenCode config correctness.
+- **Long context**: OpenCode's `limit.context`/`limit.output` fields (§3) are just bookkeeping hints for
+  OpenCode's own context-management UI/truncation logic — they don't change what the server actually
+  accepts. Set them to match whatever `--ctx-size` this repo's llama.cpp container is actually launched
+  with (check `docker-compose.yml`), not a value assumed from the model card, or OpenCode may
+  miscalculate when to compact/summarize the conversation.
+- **Thinking/reasoning-mode handling**: OpenCode's documented reasoning-effort config
+  (`options.reasoningEffort`, `options.thinking.budgetTokens` — https://opencode.ai/docs/models/,
+  fetched directly) is written for providers with a **native reasoning-effort API parameter** (OpenAI's
+  `reasoningEffort`, Anthropic's `thinking` block) — i.e., the hosted provider itself understands the
+  knob and returns reasoning separately from the answer. **The docs do not document any handling for
+  models that instead emit inline `<think>...</think>` tags in the raw completion text** (the pattern
+  Qwen3-family "hybrid thinking" models typically use when reasoning isn't a first-class API field).
+  For a llama.cpp-served Qwen3.8-27B, `reasoningEffort`/`thinking` config keys almost certainly have
+  **no effect** (llama.cpp's OpenAI-compatible endpoint doesn't expose or consume those keys) — thinking
+  mode for this model would need to be controlled at the llama.cpp/prompt level (e.g., a
+  `/no_think` marker or template-level toggle, if Qwen3.8-27B's chat template supports one), and
+  whether OpenCode strips or garbles `<think>` blocks in the response was **not found documented
+  anywhere in the primary sources checked**. **Confidence: low** on this specific sub-claim — flagged
+  as an open question, not a verified fact. Recommend testing empirically once tool-calling is unblocked.
+
+## Sources consulted (primary)
+
+- https://opencode.ai/docs/ — install methods, docs nav
+- https://opencode.ai/docs/config/ — config file location/format
+- https://opencode.ai/docs/providers/ — custom OpenAI-compatible provider JSON
+- https://opencode.ai/docs/models/ — reasoning effort / thinking config
+- https://opencode.ai/docs/troubleshooting/ — model ID reference format
+- https://github.com/anomalyco/opencode — repo README, install commands, license
+- https://github.com/anomalyco/opencode/issues/20669 — local-backend tool-call brittleness (closed, not planned)
+- https://github.com/anomalyco/opencode/issues/1890 — llama.cpp `--jinja` / template crash
+- https://github.com/anomalyco/opencode/issues/8390, #6841, #16440 — org rename (`sst` → `anomalyco`) fallout
+- https://github.com/ggml-org/llama.cpp/pull/17570 — Anthropic Messages shim reuses OpenAI tool-calling pipeline (also cited in this repo's `docs/research/qwen3.8-27b-tool-calling.md`)
+- This repo: `docs/research/qwen3.8-27b-tool-calling.md` — independent finding of open llama.cpp tool-call parser bugs for the Qwen3.5/Qwen3.8 lineage
+
+## Confidence summary
+
+| Claim | Confidence |
+|---|---|
+| Correct project = `anomalyco/opencode` (formerly `sst/opencode`) | High |
+| Install commands | High |
+| Config file location/format | High |
+| Provider JSON shape (npm/baseURL/models) | High |
+| apiKey omit-vs-dummy for no-auth servers | Medium (not directly confirmed) |
+| OpenCode local-backend tool-call brittleness (#20669) | High (content); this is a live unfixed issue |
+| llama.cpp `--jinja` requirement / template crash (#1890) | High (content); Medium (currency vs. latest builds) |
+| Qwen3.8-27B thinking-tag handling in OpenCode | Low — undocumented, flagged as open question |

docs/research/proxy-shadow-pricing.md

@@ -0,0 +1,210 @@
+# Research: Reference cloud model/pricing for the shadow-cost estimate
+
+**Question:** Which reference cloud model/API pricing should the shadow-cost
+estimate use (per #9's Destination), and how does that rate get wired into
+LiteLLM (per #10's tool choice) to compute "what this local usage would have
+cost on a real cloud API"?
+
+**Answer:** Use a single fixed reference — **Claude Sonnet 5**, at its
+current published API price, hardcoded into one `model_info` block in
+LiteLLM's `config.yaml`. Skip a multi-tier pricing table.
+
+## Reference model: single fixed price, not a tier table
+
+Two options were weighed, per #11's framing:
+
+**Option A — single fixed reference (recommended).** Pick one current Claude
+model and price the local model against it always.
+
+**Option B — small pricing table across a couple of tiers** (e.g. price the
+same local usage against both a Haiku-tier and a Sonnet-tier rate
+simultaneously, showing a range).
+
+Recommendation: **Option A**, using **Claude Sonnet 5** — the model this
+Claude Code CLI session itself runs on, and the model this repo's own
+`docs/coding-cli-setup.md` documents pointing coding CLIs at this stack's
+local Qwen3.8-27B model. Reasoning:
+
+- **Matches what the user already tracks.** Map #9's Notes says the user
+  already tracks Claude pricing day-to-day. Sonnet is the tier actually used
+  for coding-CLI work against Claude directly (and is what this session is
+  running as), not a hypothetical comparison tier — a single number tied to
+  "what I'd have paid on the tool I actually use" is more meaningful than an
+  abstract low/high band.
+- **Simple to maintain.** One `model_info` block, one number to update when
+  Anthropic changes Sonnet pricing (rare — pricing on this page has been
+  stable for months at a time; see staleness section below), versus a table
+  that needs every row kept current. This is explicitly "for fun" (map #9's
+  Destination/Notes), not real accounting — a table adds bookkeeping
+  overhead the use case doesn't need.
+- **Local model's size is Sonnet-comparable, not flagship-comparable.** The
+  locally hosted model is `Qwen3.8-27B` (`docs/coding-cli-setup.md`) — a
+  ~27B-class model. Pricing it against Anthropic's flagship (Opus tier,
+  $5/$25 per MTok) would overstate the shadow cost for what's actually a
+  mid-tier local model; pricing it against Sonnet ($2/$10 per MTok, the
+  mid-tier) is the closer-fitting comparison, and it's also the tier this
+  repo already documents pointing coding CLIs at when using the *real*
+  Claude API (as opposed to the local shim) is desired.
+- A tier table would matter if the goal were "estimate real cloud spend
+  across scenarios," but map #9 explicitly frames this as a shadow/for-fun
+  estimate with no real external routing wired in — one clear number serves
+  that better than a range that needs interpreting.
+
+**Current price** (as of 2026-08-25, per Anthropic's official pricing page):
+
+| Model | Input | Output |
+|---|---|---|
+| **Claude Sonnet 5** | **$2.00 / MTok** ($0.000002/token) | **$10.00 / MTok** ($0.00001/token) |
+
+Source: [Claude Platform docs — Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
+(Model pricing table). Note: this $2/$10 rate was originally introductory
+pricing through 2026-08-31 with a scheduled increase to $3/$15 on
+2026-09-01; Anthropic's pricing page (fetched today) states that increase
+"will not occur" and $2/$10 is now the standard price — so this is a stable
+number, not a rate about to change out from under the config.
+
+Ignore prompt-caching, batch, and tool-use-overhead pricing modifiers for
+this shadow estimate — llama.cpp's local usage has none of those API
+features, so there's nothing to map them onto; base input/output token
+pricing is the only thing that has a clean local-usage analogue.
+
+## LiteLLM config shape: `model_info` custom pricing
+
+Confirmed against LiteLLM's docs (same mechanism #10 already found; this
+plugs the confirmed rate straight in). Add a `model_info` block to the local
+model's entry in `config.yaml`:
+
+```yaml
+model_list:
+  - model_name: qwen3.8-27b-local          # or whatever this stack names it
+    litellm_params:
+      model: openai/qwen3.8-27b            # or the provider shim used to reach llama.cpp
+      api_base: http://llama-server:8080/v1
+    model_info:
+      input_cost_per_token: 0.000002       # $2 / 1,000,000 — Claude Sonnet 5 input rate
+      output_cost_per_token: 0.00001       # $10 / 1,000,000 — Claude Sonnet 5 output rate
+```
+
+Confirmed details:
+
+- **Exact keys**: `model_info.input_cost_per_token` and
+  `model_info.output_cost_per_token`, both plain decimal USD-per-token
+  floats. LiteLLM also supports `input_cost_per_second` (time-based, e.g.
+  SageMaker-style billing) and `input_cost_per_character` /
+  `input_cost_per_image` / `input_cost_per_audio_token` /
+  `input_cost_per_video_per_second` for other modalities — none needed here
+  since this is a plain text chat model priced token-for-token.
+- **Input vs. output distinguished**: yes — separate keys, matching how
+  Claude's own pricing (and llama.cpp's own `usage.prompt_tokens` /
+  `usage.completion_tokens` split) is already input/output-separated.
+- **Per-model override**: yes — `model_info` is set per entry in
+  `model_list`, so only the local model entry needs it; LiteLLM's own
+  built-in cost map for 100+ known providers is untouched for any other
+  model added to the proxy later.
+- **Where the resulting cost surfaces**: computed via LiteLLM's internal
+  `completion_cost()` function (the same path used for every provider,
+  built-in or custom-priced) on every `/chat/completions` /
+  `/v1/messages` call. It surfaces in:
+  - **Per-key spend**: `/key/info` returns a `spend` field (cumulative USD)
+    per virtual key — this is the per-workload number map #9 wants.
+  - **Admin UI dashboard** (`/ui`, confirmed present in #10's research):
+    the Usage tab visualizes spend by key/team, sourced from the same
+    spend ledger.
+  - **Spend logs**: written to LiteLLM's Postgres-backed
+    `LiteLLM_SpendLogs` / verification-token table, queryable via
+    `/team/info` and `/user/info` for aggregation.
+  - **Per-call logging object**: `kwargs["response_cost"]` on each
+    completion call, for anyone hooking custom logging/callbacks later.
+
+Source: [LiteLLM — Custom Pricing docs](https://docs.litellm.ai/docs/proxy/custom_pricing),
+[LiteLLM — Virtual Keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys)
+(`/key/info` spend field example), [LiteLLM — completion_cost /
+model_prices_and_context_window.json](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
+(the built-in cost map that `model_info` overrides for a given entry).
+
+## Token-count mapping: token-for-token is fine here
+
+Claude and Qwen3.8-27B use different tokenizers, so the *same text* produces
+different token counts on each — a genuinely rigorous "what would this
+exact conversation have cost on Claude" would need to re-tokenize the local
+conversation text with Claude's tokenizer and price *that* count, not
+llama.cpp's own token count.
+
+LiteLLM does not do this re-tokenization for a custom-priced model: its
+`completion_cost()` multiplies the token counts reported in the backend's
+own `usage.prompt_tokens` / `usage.completion_tokens` response by whatever
+`input_cost_per_token` / `output_cost_per_token` is configured for that
+`model_list` entry — it does not re-count tokens against a different
+provider's tokenizer for cost purposes (LiteLLM's own token-counting
+utilities, e.g. `token_counter()`, exist as a fallback for providers that
+don't return usage at all, not as a re-tokenization step for cost
+calculation on a provider that does).
+
+**This is fine, and doesn't need fixing**, given map #9's own framing: this
+is explicitly a shadow/for-fun estimate, not real accounting. Treating
+llama.cpp's reported token count as if it were "Claude tokens" and applying
+Claude's per-token rate directly is a reasonable, cheap approximation —
+token counts between modern tokenizers for English text are typically
+within a similar order of magnitude (roughly comparable, not identical), so
+the estimate is order-of-magnitude meaningful ("this conversation would
+have cost about $X on Claude") without claiming precision it doesn't have.
+Building actual re-tokenization against Claude's tokenizer purely to feed a
+for-fun number would be effort disproportionate to the destination. If this
+ever needs to be exact, the fix is a small conversion factor applied at
+config time (e.g. inflate the configured per-token rate by a fudge factor
+to roughly account for tokenizer differences) — not worth doing now.
+
+One tokenizer wrinkle worth noting for future-proofing, not action:
+Anthropic's pricing page notes Claude 4.7-and-later models (which includes
+Sonnet 5) use a newer tokenizer producing "approximately 30% more tokens for
+the same text" than earlier Claude models. This doesn't change the
+recommendation (Sonnet 5 is still the right reference), it's just a reminder
+that "tokens" are already an approximate, provider-specific unit even within
+Anthropic's own model lineup — reinforcing that treating llama.cpp's token
+count as directly billable at Claude's rate is consistent with how loosely
+"a token" is already defined across models, not a special-case shortcut
+being taken here.
+
+## Config staleness risk
+
+LiteLLM ships a built-in default pricing table
+([`model_prices_and_context_window.json`](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json))
+covering 100+ known providers/models, refreshed via the LiteLLM project's
+own releases — that table is what could silently drift out of date for any
+model relying on it. **This doesn't apply to the local model here**: because
+the local llama.cpp model isn't a real named provider model, its pricing is
+only ever set via the explicit `model_info` block in this repo's own
+`config.yaml`, which LiteLLM never overwrites or auto-refreshes — the only
+way the shadow-cost number goes stale is if Anthropic changes Sonnet 5's
+published price and nobody updates the two numbers in this repo's config.
+
+Given Anthropic's pricing page explicitly states the Sonnet 5 rate is now
+locked in as the standard price (the previously-scheduled 2026-09-01
+increase was cancelled), staleness risk here is low and infrequent — but
+not zero, since Anthropic can still change prices with future model
+launches or repricing. Practical mitigation for implementation (#14): put
+the reference price in `config.yaml` with a comment noting the source URL
+and date it was last checked, so a future price change is a one-line
+`config.yaml` edit plus a comment-date bump — no code change, no migration.
+No need for anything more automated (e.g. scraping Anthropic's pricing page
+at startup) — that's more machinery than a for-fun estimate warrants.
+
+## Bottom line for #14 (compose/config authoring)
+
+- Use **Claude Sonnet 5** as the fixed shadow-pricing reference:
+  `input_cost_per_token: 0.000002`, `output_cost_per_token: 0.00001` in the
+  local model's `model_info` block in LiteLLM's `config.yaml`.
+- No pricing table, no per-request tier selection — one rate, one config
+  block, matches map #9's "for fun not real accounting" framing and the
+  user's existing day-to-day use of Claude Sonnet for coding-CLI work.
+- Token counts come straight from llama.cpp's own reported
+  `usage.prompt_tokens`/`completion_tokens` via LiteLLM's normal
+  `completion_cost()` path — no re-tokenization against Claude's tokenizer,
+  which is an acceptable approximation for a shadow estimate.
+- Resulting spend is visible per-key via `/key/info`, aggregated via
+  `/team/info`/`/user/info`, and in the Admin UI's Usage dashboard — no
+  extra tracking code needed, this is the same spend-tracking path LiteLLM
+  uses for any other provider.
+- Staleness risk is low (Sonnet 5's rate is currently locked as standard
+  pricing) and, if it ever drifts, is a one-line `config.yaml` edit — worth
+  a source-URL-and-date comment in the config, nothing more elaborate.

docs/research/proxy-tool-choice.md

@@ -0,0 +1,241 @@
+# Research: Which self-hosted AI gateway/proxy tool fits this effort's needs?
+
+**Question:** Which self-hosted AI gateway/proxy tool should front llama.cpp,
+given the requirements in issue #10 (Anthropic/OpenAI-compatible routing,
+per-workload virtual keys with separate usage views, a spend dashboard,
+custom cost-per-token pricing for a local model, docker-compose
+self-hosting alongside the existing stack, room to add backends later, and
+a plus for native queue/priority support relevant to #16)?
+
+**Answer: LiteLLM proxy.** It is the only candidate that meets every
+hard requirement out of the box, self-hosted, in its free/MIT tier. Portkey's
+open-source gateway is disqualified on the dashboard/virtual-key/budget
+requirement (those are cloud-only). Helicone is a weaker fit (maintenance
+mode, unclear virtual-key/custom-pricing story, feature-reduced self-host
+build, observability-first rather than budget/gateway-first). A hand-rolled
+nginx+script layer would mean re-building LiteLLM's virtual-key store, spend
+DB, dashboard, and Anthropic↔OpenAI translation from scratch — a maintenance
+trap, not a shortcut.
+
+## Candidate: LiteLLM proxy
+
+**License / project health:** MIT-licensed, with a separate `enterprise/`
+subdirectory under its own license for a small set of add-on features (SSO,
+audit logs, guaranteed-capacity priority reservation — see below). Widely
+deployed, 100+ provider integrations.
+Source: [BerriAI/litellm LICENSE](https://raw.githubusercontent.com/BerriAI/litellm/main/LICENSE),
+[BerriAI/litellm GitHub repo](https://github.com/BerriAI/litellm).
+
+**Anthropic/OpenAI-compatible routing:** LiteLLM proxy exposes a unified
+`/v1/messages` endpoint that accepts Anthropic-format requests and
+translates them to whatever backend format the target model needs (and
+translates the response back), so Anthropic-format clients (coding CLIs)
+and OpenAI-format clients (Open WebUI) can both hit the same proxy against
+the same `model_list` entry pointing at llama.cpp's OpenAI-compatible
+`/v1/chat/completions`. This means llama.cpp's own native `/v1/messages`
+shim doesn't strictly need to be reached directly through the proxy — LiteLLM
+does its own Anthropic↔OpenAI translation in front of llama.cpp's OpenAI
+endpoint, which is one plausible wiring; routing straight through to
+llama.cpp's native shim as a passthrough is a second option worth checking
+at implementation time (issue #14/#15 territory, not this ticket).
+Source: [LiteLLM /v1/messages unified endpoint docs](https://docs.litellm.ai/docs/anthropic_unified/),
+[LiteLLM — Claude Code with non-Anthropic models](https://docs.litellm.ai/docs/tutorials/claude_non_anthropic_models).
+
+**Virtual keys / per-workload accounts:** `/key/generate` issues a virtual
+key with its own `max_budget`, `budget_duration`, and `tpm_limit`/`rpm_limit`.
+Keys can be owned by a `user_id` or a `team_id`, so each workload (Open
+WebUI, a coding CLI, Gitea code review, Paperless OCR, etc.) gets its own
+key with its own budget and its own spend record, queryable via
+`/key/info` and aggregated per team via `/team/info`. Spend is written to
+the `LiteLLM_VerificationTokenTable` and computed via LiteLLM's own
+`completion_cost()` on every call.
+Source: [LiteLLM — Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys).
+
+**Spend dashboard:** The Admin UI (`/ui`) ships in the open-source build —
+key/model management plus a Usage tab showing spend tracked per key/team.
+Enterprise adds SSO/SAML, audit logs, and a more advanced UI on top, but
+core spend-by-key visualization is not gated.
+Source: [LiteLLM — Proxy UI docs](https://docs.litellm.ai/docs/proxy/ui),
+cross-checked via [LiteLLM GitHub repo description](https://github.com/BerriAI/litellm)
+("100+ LLM integrations, budgets, rate limits, logging, and virtual keys" in
+the free edition).
+
+**Custom cost-per-token pricing (shadow cloud-cost estimate):** Add a
+`model_info` block per model in `config.yaml`:
+
+```yaml
+model_list:
+  - model_name: my-local-model
+    litellm_params:
+      model: openai/local-model   # or whatever provider shim fits llama.cpp
+      api_base: http://llama-server:8080/v1
+    model_info:
+      input_cost_per_token: 0.000001
+      output_cost_per_token: 0.000002
+```
+
+This is exactly the mechanism #11 (reference cloud pricing) needs: once #11
+picks a reference cloud model/price, its per-token rate goes straight into
+this block and LiteLLM computes "what this local usage would have cost"
+using its normal `completion_cost()` path — no separate cost-tracking code
+needed.
+Source: [LiteLLM — Custom pricing docs](https://docs.litellm.ai/docs/proxy/custom_pricing).
+
+**Docker-compose deployment:** The documented quickstart is a two-service
+compose (LiteLLM gateway + Postgres, Postgres storing keys/models/spend
+logs); Redis is optional and only needed for multi-instance state (rate
+limiting, cross-instance priority queueing) — a single-instance deployment
+alongside the existing llama.cpp/Open WebUI/Qdrant/Lazytainer stack doesn't
+require it. `LITELLM_SALT_KEY` must be set to something real before
+production use (it encrypts stored provider keys); the documented example
+otherwise uses `sk-1234` as a placeholder master key that must be replaced.
+A YAML-only, no-database mode exists but drops budget enforcement — not
+useful here since budgets/spend-per-key are a hard requirement.
+Source: [LiteLLM — Docker Quick Start](https://docs.litellm.ai/docs/proxy/docker_quick_start).
+
+**Room for more backends later:** LiteLLM's whole design point is a
+`model_list` of arbitrary provider entries behind one routing layer (100+
+providers documented) — adding a second/third backend later is a config
+edit, not an architecture change.
+Source: [BerriAI/litellm GitHub repo](https://github.com/BerriAI/litellm).
+
+**Queuing/priority (relevant to #16, not decided here):** LiteLLM has an
+open-source (beta) request-prioritization scheduler: callers pass a
+`priority` field (lower number = higher priority) and a Router-level queue
+polls until a slot opens; multi-instance deployments need Redis to share
+queue state. This is a real, if beta-quality, building block for #16's
+priority queue and would mean #16 doesn't need a separate queuing
+component in front of the proxy. However:
+- The stricter **`priority_reservation`** feature (hard-reserving a % of
+  TPM/RPM capacity per priority tier — not just soft-prioritizing) is
+  gated behind the enterprise license.
+- The plain `priority` parameter had a real reported bug (leaking the
+  `priority` field into the provider request, breaking calls to some
+  providers) that maintainers closed as "not planned" rather than fixed —
+  worth a smoke test against llama.cpp specifically before #16 relies on
+  it, and worth treating the whole feature as "beta, verify before
+  depending on it" rather than a settled capability.
+Sources: [LiteLLM — Request Prioritization (scheduler) docs](https://docs.litellm.ai/docs/scheduler),
+[BerriAI/litellm issue #7144 — "Priority feature is broken"](https://github.com/BerriAI/litellm/issues/7144)
+(closed not-planned),
+[BerriAI/litellm issue #6867 — scheduler polling bug](https://github.com/BerriAI/litellm/issues/6867),
+[BerriAI/litellm issue #13405 — feature request for priority-based request
+handling via API keys, i.e. the simpler ergonomic form isn't fully built
+yet either](https://github.com/BerriAI/litellm/issues/13405).
+
+## Candidate: Portkey — disqualified
+
+Portkey's core gateway (`portkey-ai/gateway`) went fully open-source
+(Apache 2.0) and is self-hostable via Docker with routing, fallbacks,
+retries, load balancing, and a local logging console. But per the repo's
+own docs, **usage analytics/cost tracking, budget enforcement, and the
+spend dashboard are explicitly listed as "available in hosted and
+enterprise versions"** — i.e. they require Portkey's cloud control plane,
+not the self-hosted gateway alone. That fails this ticket's hard
+requirement for a self-hosted spend dashboard and per-key usage views, so
+Portkey is out regardless of its otherwise-solid routing feature set.
+Source: [Portkey-AI/gateway GitHub repo](https://github.com/portkey-ai/gateway).
+
+## Candidate: Helicone — weaker fit
+
+Helicone is self-hostable via a documented docker-compose stack (web
+dashboard, a combined API+proxy service, Postgres, ClickHouse, MinIO for
+S3-compatible storage), Apache-2.0, so it can be run independent of
+Helicone's own cloud. But:
+- **Project status:** Mintlify acquired Helicone (2026-03-03) and the
+  product is reported to be in maintenance mode; the repo is still active
+  and Apache-2.0 so self-hosting isn't blocked, but it's a weaker bet for
+  a component meant to grow (more backends, more workloads) over time.
+- **Feature parity gap:** the self-hosted docs explicitly note other
+  providers (Vertex AI, Bedrock, Azure OpenAI) aren't supported in the
+  self-hosted build the way they are in the cloud version — a signal the
+  self-host path is the less-maintained one.
+- **Virtual keys / custom pricing:** not clearly documented for the
+  self-hosted build in the pages checked — Helicone's primary framing is
+  LLM *observability* (logging, tracing, cost dashboards computed from
+  known-provider pricing tables) rather than a virtual-key issuing/budget
+  gateway; injecting a custom per-token price for an unlisted local model
+  isn't documented the way LiteLLM's `model_info.input_cost_per_token` is.
+- **Format translation:** self-host docs show separate `oai/` and
+  `anthropic/` proxy paths rather than a documented single endpoint that
+  translates Anthropic-format calls to an OpenAI-format backend the way
+  LiteLLM's unified `/v1/messages` does.
+
+Sources: [Helicone — Docker Compose self-deploy docs](https://docs.helicone.ai/getting-started/self-deploy-docker),
+[Helicone — self-hosting launch announcement](https://www.helicone.ai/blog/self-hosting-launch),
+[Helicone/helicone GitHub repo](https://github.com/helicone/helicone).
+**Confidence note:** these self-host feature-gap claims come from
+docs-page summaries rather than a hands-on deployment; if Helicone is
+ever reconsidered, verify virtual-key/custom-pricing support directly
+against a running self-hosted instance rather than trusting this summary
+alone.
+
+## Candidate: hand-rolled nginx + scripts — rejected as a maintenance trap
+
+A thin reverse proxy plus custom scripts could technically satisfy each
+bullet in isolation (issue an API key = generate a token and check it in
+an nginx `map`/Lua script; track spend = write to a DB on each request;
+dashboard = a small custom UI; custom pricing = a config file the script
+reads; Anthropic↔OpenAI translation = hand-written request/response
+transformers). But that's rebuilding LiteLLM's virtual-key store, spend
+ledger, dashboard, and format-translation layer from scratch, in a
+piece this repo would then own and maintain indefinitely, against a
+target (LiteLLM) that already does all of it, is MIT-licensed, and is
+a straightforward docker-compose service. Not adopted.
+
+## Risks / gaps to carry into later tickets
+
+1. **Lazytainer idle-suspend interaction (flagged in #9's "Not yet
+   specified").** Lazytainer decides to stop `llama-server` based on
+   network packet activity on its published port
+   (`lazytainer.group.llamaserver.minPacketThreshold` /
+   `inactiveTimeout` in this repo's `docker-compose.yml`). If LiteLLM
+   proxy performs periodic background health checks against configured
+   models (a common gateway behavior), that traffic could look like
+   real usage to Lazytainer and prevent it from ever idling the
+   container down. This needs to be checked against LiteLLM's actual
+   health-check config (there are documented options to disable/tune
+   background health checks) once the compose service is authored in
+   #14, and verified on real hardware per #9's "Not yet specified" note
+   on Lazytainer + multi-workload proxy interaction.
+2. **llama.cpp's native `/v1/messages` shim vs. LiteLLM's own Anthropic
+   translation.** Two wiring options exist (LiteLLM translates
+   Anthropic→OpenAI itself and hits llama.cpp's OpenAI endpoint; or
+   LiteLLM passes Anthropic-format requests straight through to
+   llama.cpp's own native shim). This ticket confirms both are
+   plausible per LiteLLM's docs but doesn't pick one — that's
+   implementation detail for #14/#15, and should be smoke-tested against
+   the actual coding-CLI flows in `docs/coding-cli-setup.md` once wired.
+3. **Priority queueing for #16 is a real but beta/rough-edged LiteLLM
+   feature**, with at least one reported-and-declined bug in the exact
+   `priority` mechanism, and the stronger reserved-capacity variant is
+   enterprise-gated. #16 should treat LiteLLM's scheduler as a
+   starting point to validate hands-on, not an assumed solved problem —
+   if it doesn't hold up under test, #16 may need a lightweight queuing
+   shim in front of the proxy (e.g. a small request-queue sidecar) rather
+   than reworking the whole gateway choice.
+4. **Redis need is deferred, not eliminated.** A single-instance LiteLLM
+   deployment (the right size for this effort) doesn't need Redis for
+   virtual keys/spend/dashboard, but the priority scheduler's
+   multi-instance behavior and rate-limit sharing do use it — if #16
+   ends up needing Redis-backed prioritization even in a single-instance
+   deployment, add a `redis` service to the compose file at that point;
+   no need to provision it speculatively now.
+
+## Bottom line for the wayfinder map
+
+- Adopt **LiteLLM proxy** (MIT-licensed, `BerriAI/litellm`) as the
+  gateway/proxy in front of llama.cpp for this effort.
+- It meets every hard requirement in #10 in its free/OSS build: virtual
+  keys with per-key spend, a self-hosted Admin UI spend dashboard,
+  config-driven custom per-token pricing (feeds #11 directly), a
+  documented two-service docker-compose deployment, and an
+  arbitrary-provider `model_list` that keeps future backends a config
+  change away.
+- Its beta priority/queueing feature is a promising but unproven fit for
+  #16 — validate it hands-on rather than assuming it's settled.
+- Portkey's self-hosted OSS gateway is disqualified (no self-hosted
+  dashboard/budgets). Helicone is a workable but weaker fallback
+  (maintenance-mode signal, feature-reduced self-host build, less clearly
+  documented virtual-key/custom-pricing support) if LiteLLM turns out to
+  be a poor fit during implementation.

docs/research/rocm-gpu-pin-and-render-group.md

@@ -0,0 +1,95 @@
+# Research: GPU pinned at 100% with two concurrent llama.cpp containers, and the intermittent "render" group startup error
+
+**Question:** After adding `llama-server-fast` (#44), real-hardware testing on the R9700
+showed `rocm-smi` pinned at 100% GPU / ~73-101W whenever both `llama-server` and
+`llama-server-fast` run concurrently, dropping to 3% / ~25-60W the moment either one
+alone is stopped. Separately, `docker compose up` intermittently failed with
+`Error response from daemon: unable to find group render: no matching entries in group file`
+— confirmed new since the second GPU service was added. See issue #5's comment thread
+for the raw `rocm-smi`/`free -h` output this doc is diagnosing.
+
+## GPU pin: root cause and fix
+
+**Confirmed via #5's own data**: either container alone is fine (3% GPU, low power).
+The pin only appears with two concurrent HIP-context-holding processes on the same
+GPU. This matches `ROCm/ROCm#5706` (already flagged as a risk in map #1) — full
+comment thread confirms:
+
+- Root cause: an AMD MES (Micro Engine Scheduler) firmware bug triggered by HIP
+  hardware-queue creation, pinning the GPU at boost clock the moment ROCm
+  initializes a queue. Not llama.cpp-specific — reproduced with vLLM and bare
+  PyTorch ROCm too. Source: [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706)
+  (`tcgu-amd`, AMD engineer, confirms MES firmware root cause; closed as
+  "fixed" in March, but a report as recent as May 24 shows it recurring even on
+  patched firmware/kernel).
+- **Validated workaround**: `GPU_MAX_HW_QUEUES=1` as a container env var. One
+  report ran a controlled before/after on the exact image this stack uses
+  (`ghcr.io/ggml-org/llama.cpp:server-rocm`, R9700/gfx1201):
+  baseline 100% GPU / 95W → with the var set, 3% GPU / 22W, VRAM unchanged.
+  Source: same thread, `interconnectedMe`'s comment.
+- **Semantics** (why this should apply to our two-container case, not just the
+  single-process case tested above): `GPU_MAX_HW_QUEUES` is a **per-process**
+  HIP runtime setting — it caps how many HSA/hardware queues *that process's*
+  HIP runtime allocates, default higher (over-subscription is what causes the
+  penalty). Source: [AMD ROCm workload-optimization docs](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html).
+  Since it's per-process, setting it on *each* container independently is the
+  correct scope — it should reduce total concurrent hardware-queue creation
+  across both processes, which is the trigger condition MES chokes on.
+  **Caveat**: no primary source explicitly tested two concurrent containers
+  both set to `GPU_MAX_HW_QUEUES=1` — this is a well-grounded extrapolation
+  from confirmed per-process semantics and the same root-cause mechanism, not
+  a directly-reproduced fix for our exact topology. Verify with `rocm-smi`
+  after applying, both containers up.
+
+## "unable to find group render" — a real Docker bug, not flaky hardware
+
+This is a known, documented Docker limitation, not something specific to this
+stack: `group_add` by **name** requires Docker to resolve that name against
+the **container's own** `/etc/group` file — if the image doesn't define a
+`render` entry there (common for minimal/slim base images), resolution fails.
+Source: [docker/cli#4714](https://github.com/docker/cli/issues/4714)
+("`docker run --group-add` by name doesn't add group from host as
+documented") and [docker/compose#7277](https://github.com/docker/compose/issues/7277)
+(same "no matching entries in group file" error).
+
+Confirms why it's now intermittent rather than always-broken: this repo's
+`docker-compose.yml` uses `group_add: [video, render]` (plain names) on
+**three** GPU services now (`llama-server`, `llama-server-fast`, `comfyui`).
+Docker Compose starts containers concurrently, and each does its own
+name-resolution lookup independently — with only one GPU service before #44,
+the resolution almost always won its race; with two (soon three, once
+`comfyui`'s downloader/model land per #46) the odds of losing that race and
+hitting the unresolved-name path go up. This is consistent with the user's
+own observation that it's new since the second GPU service.
+
+**Fix, already precedented in this repo**: `scripts/update.sh` already
+resolves the host's real `video`/`render` **numeric GIDs** for the `comfyui`
+service (`COMFYUI_VIDEO_GID`/`COMFYUI_RENDER_GID`, passed as app-level env
+vars) — but `comfyui`'s own `group_add:` still uses plain names too, so it
+isn't actually protected by that either. The correct fix per the Docker
+issues above: use the resolved **numeric GIDs** in `group_add:` itself
+(Compose accepts numeric strings directly), not names, on all three GPU
+services. Numeric GIDs skip the name-resolution step entirely, eliminating
+both the flakiness and the race.
+
+## Recommendation
+
+1. Add `GPU_MAX_HW_QUEUES=1` to both `llama-server` and `llama-server-fast`'s
+   `environment:` blocks. Verify with `rocm-smi` after redeploy, both
+   containers up — this is the one part of this doc that's extrapolated
+   rather than directly reproduced, so real confirmation matters here.
+2. Resolve host `video`/`render` GIDs once (generalize the existing
+   `COMFYUI_VIDEO_GID`/`COMFYUI_RENDER_GID` pattern in `scripts/update.sh`
+   to shared `HOST_VIDEO_GID`/`HOST_RENDER_GID` vars), and switch
+   `group_add:` on all three GPU services (`llama-server`,
+   `llama-server-fast`, `comfyui`) from `[video, render]` (names) to the
+   resolved numeric GIDs. Removes the race entirely rather than reducing its
+   odds.
+
+## Sources
+
+- [ROCm/ROCm#5706 — full comment thread](https://github.com/ROCm/ROCm/issues/5706)
+- [AMD ROCm — MI300/MI350 workload optimization docs (GPU_MAX_HW_QUEUES)](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
+- [docker/cli#4714 — group_add by name doesn't work as documented](https://github.com/docker/cli/issues/4714)
+- [docker/compose#7277 — "no matching entries in group file"](https://github.com/docker/compose/issues/7277)
+- This repo's issue #5 (real-hardware `rocm-smi`/`free -h` evidence this doc diagnoses)

docs/research/voidllm-evaluation.md

@@ -0,0 +1,133 @@
+# Evaluation: VoidLLM as a replacement for LiteLLM proxy
+
+**Question:** Does [voidmind-io/voidllm](https://github.com/voidmind-io/voidllm)
+(the user asked us to look at it) beat the already-chosen tool (LiteLLM proxy,
+see [`docs/research/proxy-tool-choice.md`](https://git.arthurerlich.de/haylan/LLM-Server/raw/branch/research/proxy-tool-choice/docs/research/proxy-tool-choice.md)
+on branch `research/proxy-tool-choice`, and [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10))
+against this effort's requirements ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9))?
+
+**Headline: the repo is real and is a genuine, functioning AI gateway/proxy** —
+not a placeholder, not something unrelated to the name. It is young
+(created March 2026), effectively a one-person project, and it has a
+documented, explicit gap that disqualifies it for this repo's actual usage
+pattern: it cannot proxy the coding CLIs' LLM traffic at all.
+
+**Answer: stay on LiteLLM.** VoidLLM fails one hard requirement outright
+(coding-CLI routing) and has no equivalent to LiteLLM's priority-queue
+building block. It matches or is comparable on virtual keys, dashboard, and
+custom pricing, but that isn't enough to justify a switch, let alone the
+migration cost of re-doing #12–#15's completed work.
+
+## Does the repo exist and is it what it claims to be?
+
+Yes on both counts, confirmed directly against the GitHub API and repo
+content (not a blog post or secondhand summary):
+
+- Repo: `voidmind-io/voidllm`, public, not a fork, not archived. Description:
+  "Privacy-first LLM proxy and AI gateway - load balancing, multi-provider
+  routing, API key management, usage tracking, rate limiting. Self-hosted.
+  Zero knowledge of your prompts." Created 2026-03-17, last pushed
+  2026-08-25 (same week as this evaluation). Language: Go, 129 stars, 15
+  forks, 28 open issues.
+  Source: `https://api.github.com/repos/voidmind-io/voidllm` (fetched
+  directly).
+- README confirms it is exactly what the description says: a self-hosted
+  proxy sitting in front of OpenAI/Anthropic/Azure/Ollama/vLLM/custom
+  providers, with virtual API keys, RBAC (org/team/user/key), rate limits,
+  token budgets, a web dashboard (usage, keys, playground), and an MCP
+  gateway feature.
+  Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/README.md`.
+- It has real release artifacts (Linux/Windows/macOS binaries), a Helm
+  chart, CI/codecov/Go-report-card/OpenSSF-scorecard/Snyk badges, and a
+  documented `docs/` tree with real content behind every link checked
+  (providers, load balancing, API reference) — not stub pages.
+  Source: same README; `docs/models/providers.md`, `docs/api/overview.md`,
+  `docs/models/load-balancing.md`, `docs/index.md`, all fetched from
+  `raw.githubusercontent.com/voidmind-io/voidllm/main/`.
+
+**Maturity/health caveat:** this is a small, young project. Contributors
+per the GitHub API: `christianromeni` (151 commits — the sole real author),
+`dependabot[bot]` (36, automated), and two accounts with 1 commit each
+(`martinsotirov`, `SAY-5`). Effectively a solo maintainer, ~5 months old.
+The README itself discloses "This project was built with significant
+assistance from AI (Claude by Anthropic)." None of this makes it fake, but
+it is a materially less-established project than LiteLLM (widely deployed,
+100+ integrations, multi-year history) and carries the usual small-project
+risks: bus factor, slower security response, less community troubleshooting
+history.
+Source: `https://api.github.com/repos/voidmind-io/voidllm/contributors`.
+
+**License:** Business Source License 1.1, not OSI open source. Self-hosting
+for internal/production use is explicitly and unconditionally permitted
+("regardless of the number of instances, users, or volume of traffic");
+the restriction is only on reselling it as a competing hosted/managed
+service. Converts to Apache 2.0 four years after each release. Fine for
+this repo's private homelab use, but a step down from LiteLLM's plain MIT.
+Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/LICENSE`.
+
+It also has a paid tier structure (Pro €49/mo, Enterprise €149/mo, one-time
+"Founding Member" €999) gating cross-org analytics, SSO/OIDC, audit logs,
+OpenTelemetry, and Redis-backed multi-instance state behind payment. The
+features this evaluation needs (virtual keys, per-key usage, dashboard,
+custom pricing, docker-compose deploy) are all listed under the free
+Community tier, so the paywall doesn't block this repo's use case — but it's
+a different project shape than LiteLLM's free/MIT-with-optional-enterprise-
+addon model.
+Source: README "Features" table.
+
+## Requirement-by-requirement
+
+| Requirement | LiteLLM (current) | VoidLLM |
+|---|---|---|
+| OpenAI-compatible routing | Yes | Yes — `/v1/chat/completions`, embeddings, images, audio, streaming |
+| Anthropic-compatible / unified Anthropic Messages endpoint | Yes — native `/v1/messages` unified endpoint accepts Anthropic-format requests, translates to any backend | **No.** No `/v1/messages` or any Anthropic-shaped *inbound* endpoint exists. VoidLLM only accepts OpenAI-format requests and can translate *outbound* to an Anthropic-format upstream (`provider: anthropic` in config) — the reverse direction of what's needed |
+| Coding CLIs routed through the proxy | Yes — Claude Code, Kimi, OpenCode all point at LiteLLM today (issue #15, `docs/coding-cli-setup.md`) | **No — explicitly unsupported.** `docs/models/providers.md`: *"Claude Code talks directly to Anthropic's API for LLM access - you can't route its LLM requests through VoidLLM."* VoidLLM can only be added as an MCP server to Claude Code, not as its LLM backend |
+| Per-workload virtual keys with separate usage views | Yes | Yes — `vl_uk_`/`vl_tk_`/`vl_sa_`/`vl_sk_` key types, org→team→user→key RBAC hierarchy, per-key and per-team usage (`GET /api/v1/usage/me`, `GET /api/v1/orgs/:org_id/usage`) |
+| Usage/spend dashboard (not logs-only) | Yes — Admin UI `/ui`, Usage tab, free tier | Yes — Web UI with dashboard/usage/keys/playground screens, listed as Community (free) tier |
+| Custom cost-per-token pricing for local model | Yes — `model_info.input_cost_per_token`/`output_cost_per_token` in `config.yaml` (already wired in this repo's `litellm-config.yaml` against Claude Sonnet 5's published rate) | Yes, equivalent mechanism — per-model `pricing.input_per_1m`/`output_per_1m` in `voidllm.yaml` |
+| docker-compose self-hostable alongside existing stack | Yes — already running (`litellm` + `litellm-db` services in `docker-compose.yml`) | Yes — documented `docker-compose up` quick start, single Go binary, SQLite by default or Postgres |
+| Native request queuing/priority | Beta, real but flaky — scheduler with a `priority` field, known bug (leaks into provider request, closed not-planned); needs smoke test (issue #17) | **Not found.** No queuing/priority-scheduling doc page exists in VoidLLM's docs index. The only "priority" concept is a *load-balancing* strategy (which upstream **deployment** to prefer/fail over to) — not request-level queue ordering for concurrent callers hitting one backend. Rate limiting is reject-on-429, not queue-and-wait. Concretely weaker than even LiteLLM's beta scheduler for this repo's actual need (one local GPU, interactive vs. batch tiers) |
+| Room to add more LLM backends later | Yes — `model_list` of arbitrary provider entries, 100+ providers | Yes — 6 built-in provider types (OpenAI, Anthropic, Azure, Ollama, vLLM, custom-OpenAI-compatible), multi-deployment load balancing/failover per model |
+| Project health/maturity | MIT, multi-year, widely deployed, 100+ integrations | Real project, ~5 months old, effectively solo-maintained, BSL 1.1, 129 stars |
+
+Sources for the VoidLLM column: `README.md`, `docs/models/providers.md`,
+`docs/api/overview.md`, `docs/models/load-balancing.md`, `docs/index.md`
+(all `raw.githubusercontent.com/voidmind-io/voidllm/main/...`, fetched
+directly during this evaluation). LiteLLM column sourced from
+`docs/research/proxy-tool-choice.md` on branch `research/proxy-tool-choice`
+and this repo's live `litellm-config.yaml` / `docker-compose.yml`.
+
+## Why this disqualifies VoidLLM here
+
+Two failures, not one, and they hit the requirements list at its hardest
+points:
+
+1. **Coding-CLI routing is a hard requirement this repo already depends on.**
+   Issue #15 migrated Claude Code, Kimi, and OpenCode to route through the
+   proxy (`docs/coding-cli-setup.md`), and issue #9's destination explicitly
+   lists coding CLIs as one of the gateway's fronted consumers. VoidLLM's own
+   docs say plainly that Claude Code's LLM traffic cannot go through it.
+   Even setting Claude Code aside, VoidLLM has no inbound Anthropic
+   Messages-shaped endpoint at all — any Anthropic-format client (present or
+   future) is unsupported, only OpenAI-format inbound is. LiteLLM's
+   `/v1/messages` unified endpoint is a direct, working answer to this same
+   need today.
+2. **No request-priority/queuing story**, which issue #16 already settled on
+   using LiteLLM's beta scheduler for. VoidLLM has nothing documented in
+   this space beyond reject-on-limit rate limiting and load-balancer
+   deployment ordering. Switching would mean giving up even LiteLLM's shaky
+   beta feature for nothing.
+
+On top of both dealbreakers, VoidLLM is a much younger, single-maintainer
+project against an already-integrated, working LiteLLM deployment (#14/#15
+done, only the scheduler smoke test in #17 outstanding). There's no
+requirement VoidLLM meets that LiteLLM doesn't already meet as well or
+better, so there's no upside to weigh against the migration cost and the
+two outright gaps.
+
+## Recommendation
+
+**Stick with LiteLLM.** Do not switch. VoidLLM is worth a second look in the
+future only if it adds an Anthropic-format inbound endpoint (making
+coding-CLI routing possible) and a real request-queuing/priority mechanism —
+neither exists today.

docs/server-planing.md

@@ -0,0 +1,108 @@
+# 4x Radeon AI PRO R9700 — Rackmount LLM Server Build Plan
+
+**Goal:** Local LLM inference/training rig running 3-4x AMD Radeon AI PRO R9700 (32GB) GPUs, rack-mounted in a full-size 19" rack.
+
+---
+
+## GPU
+
+| Spec | Value |
+|---|---|
+| Model | AMD Radeon AI PRO R9700 32GB |
+| VRAM | 32GB GDDR6, 256-bit bus |
+| Interface | PCIe 5.0 x16 (backward-compatible with PCIe 4.0 slots at x16 bandwidth) |
+| Form factor | Dual-slot, blower-style cooler |
+| TDP | 300W |
+| Quantity | 4x → ~1,200W GPU load alone |
+
+Blower coolers exhaust heat out the back of the case rather than into it — important for stacking 4 cards close together.
+
+---
+
+## Motherboard
+
+**Gigabyte MZ32-AR0 (Rev 1.0)**
+
+| Spec | Value |
+|---|---|
+| Socket | Single SP3, EPYC 7002/7001 series |
+| Form factor | E-ATX (305mm x 330mm) |
+| PCIe slots | 4x full PCIe Gen4 x16 (Slot_3, Slot_4, Slot_6, Slot_7) + additional Gen4 x8 / Gen3 slots |
+| Memory | 8-channel DDR4 RDIMM/LRDIMM, 16 DIMM slots |
+| Management | ASPEED AST2500 BMC (IPMI) |
+
+**Note:** CPU/memory socket sits close to the PCIe slots — plan for PCIe riser cables to give the R9700's clearance and to route to rear chassis slot cutouts.
+
+---
+
+## CPU
+
+**AMD EPYC 7282** (Rome / 2nd Gen, SP3)
+
+- 16 cores — sufficient since the workload is GPU-bound, not CPU-bound
+- Full access to Rome's 128 PCIe lanes on this single-socket board
+- Budget-friendly used-market option vs. Threadripper PRO
+
+---
+
+## Memory
+
+- **Type:** DDR4 RDIMM or LRDIMM only — unbuffered (UDIMM) will not POST on SP3
+- Size to your model/dataset needs; 8-channel config gives strong bandwidth for CPU-side preprocessing and system overhead
+
+---
+
+## Power Supply
+
+- Estimated system draw: ~1,200W (GPUs) + CPU + overhead → **1,600-2,000W** PSU needed
+- At 4U rack size, use a **redundant CRPS-style server PSU** rather than a single consumer ATX unit — better suited to 24/7 operation
+
+---
+
+## Chassis
+
+**4U rackmount, E-ATX compatible, full-depth (420mm+)**
+
+Candidates:
+- Rosewill RSV-L4620-class (supports up to 4 GPUs, E-ATX compatible, hot-swap bays)
+- Rackchoice/RackOwl 4U E-ATX (7 full-height expansion slots, 420mm depth)
+
+Requirements:
+- Front-to-back airflow (matches R9700 blower exhaust direction)
+- 7 full-height PCIe slot cutouts at rear
+- Riser card support if GPU clearance near CPU socket is tight
+
+---
+
+## Rack
+
+**42U full-size rack, 19" width, ≥1,000-1,100mm usable depth**
+
+| Item | Space |
+|---|---|
+| GPU server (4U chassis) | 4U |
+| UPS | ~2-3U |
+| Network switch | 1U |
+| Cable management / PDU | 1-2U |
+| **Used now** | **~8-10U** |
+| **Free for expansion** | **~32-34U** |
+
+- Choose an **enclosed rack** (side panels + front/rear doors) for dust control and directed airflow, appropriate for a dedicated server room/garage.
+- Confirm listed depth explicitly supports full GPU servers — many "42U" racks are sized for shallow network gear, not 420mm+ deep GPU chassis.
+
+---
+
+## Sourcing links
+
+- [Netzwerkschrank, 37HE 600x1000](https://www.ebay.de/itm/314598915865?var=613137180576) — candidate rack; maybe go taller than 37HE
+- [Gigabyte MZ32-AR0 Rev1.0 + EPYC 7282 16-Kern 2.8GHz](https://www.ebay.de/itm/358886890399) — mainboard+CPU bundle matching the spec above. Still needs RAM — prefer older/used DDR4 RDIMM over new, it's cheaper.
+
+---
+
+## Open Items / Next Steps
+
+- [ ] Verify current ROCm/Linux driver compatibility for R9700 on Rome-era (SP3) PCIe topology
+- [ ] Confirm riser cable part numbers/compatibility with MZ32-AR0 + chosen 4U chassis
+- [ ] Size UPS capacity against full system load (GPU server + switch + growth headroom)
+- [ ] Source EPYC 7282 + MZ32-AR0 (used/refurb market)
+- [ ] Confirm chassis rail kit compatibility with target rack

scripts/download-model.sh

@@ -1,12 +0,0 @@
-#!/usr/bin/env bash
-# Downloads the model GGUF straight into the `models` named volume via a
-# one-off container — no huggingface-cli or host bind-mount needed.
-#
-# ponytail: hardcodes the one model this stack is built for (see the
-# `downloader` service in docker-compose.yml for the actual URL/filename).
-# Set LLAMA_MODEL_FILE in .env first if you're using a different quant.
-set -euo pipefail
-cd "$(dirname "$0")/.."
-
-docker compose --profile tools run --rm downloader
-echo "Model downloaded into the 'models' volume."

scripts/switch-model.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Swap GPU residency between llama-server (Qwen) and comfyui — they never
+# run concurrently, VRAM doesn't fit both (see issue #38's map). Manual
+# invocation only, no auto-switching.
+#
+# Bypasses lazytainer entirely and drives docker compose directly — its
+# idle-stop can't be used for this. Root cause (see
+# docs/research/lazytainer-omniroute-idle-stop.md, issue #40): lazytainer's
+# packet-threshold detector is source-blind and can't tell OmniRoute's
+# periodic health-check pings apart from real traffic on the same port, so
+# it never reliably sleeps a service on its own. A scripted swap always
+# knows which service should go up/down, so it doesn't need that heuristic.
+#
+# llama-server-fast (the small classifier model, issue #44) is NOT part of
+# this swap — it's meant to stay always-resident. Worst case with comfyui up
+# is comfyui (~25GB, Qwen-Image FP8) + llama-server-fast (~5GB) ≈ 30GB,
+# still under the 32GB card but tight — unverified on real hardware, check
+# `docker compose ps` / VRAM usage after the first real swap.
+#
+# OmniRoute may show the just-stopped provider as errored/offline in its
+# dashboard for up to CREDENTIAL_HEALTH_CHECK_INTERVAL (default 5 min) after
+# a swap — cosmetic, not a functional problem (see the research doc above).
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+usage() {
+  echo "Usage: $0 {qwen|comfyui}" >&2
+  echo "  qwen    - stop comfyui, start llama-server" >&2
+  echo "  comfyui - stop llama-server, start comfyui" >&2
+  exit 1
+}
+
+[ $# -eq 1 ] || usage
+
+case "$1" in
+  qwen)
+    from=comfyui
+    to=llama-server
+    ;;
+  comfyui)
+    from=llama-server
+    to=comfyui
+    ;;
+  *)
+    usage
+    ;;
+esac
+
+echo "==> stopping $from"
+docker compose stop "$from"
+
+echo "==> starting $to"
+docker compose up -d "$to"
+
+echo "==> status"
+docker compose ps

scripts/update.sh

@@ -0,0 +1,253 @@
+#!/usr/bin/env bash
+# The one command to run after any change to this repo (compose file, .env,
+# or a git pull) to bring the running stack in sync. Ensures secrets/keys
+# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
+# changed — safe to run any time, including with nothing to do.
+#
+# Tunable config values (LLAMA_*, ports, timeouts — anything with a real
+# default in .env.example) are synced from .env.example every run. A value
+# already matching is left alone silently. A value that DIFFERS from the
+# server's current .env is a conflict: interactively, you're shown every
+# conflict on one screen (via gum) and choose which to accept — unpicked
+# keys keep the server's current value. Non-interactively (no TTY — cron,
+# CI, piped), any conflict is a hard error unless --force is passed, which
+# accepts every new value automatically. Secrets and host-resolved values
+# (blank in .env.example — OMNIROUTE_*_SECRET/_KEY/_SALT/_PASSWORD,
+# SEARXNG_LAN_IP, COMFYUI_PUID/PGID, HOST_VIDEO_GID/RENDER_GID) are never
+# touched by this — they keep going through set_if_blank as before.
+#
+# omniroute's own routing/provider config (llama-server, search) lives in
+# its dashboard, not a checked-in file like the old litellm-config.yaml —
+# see issue #31 and docs/proxy-key-onboarding.md.
+#
+# ponytail: no rollback/backup logic — this is a single-user homelab box,
+# not a fleet. If a bad config lands, `git revert` + re-run is the recovery
+# path, not this script.
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+FORCE=false
+for arg in "$@"; do
+  case "$arg" in
+    --force) FORCE=true ;;
+    *) echo "Usage: $0 [--force]" >&2; exit 1 ;;
+  esac
+done
+
+# Must run before anything else touches a file this script itself reads
+# (docker-compose.yml, .env.example, this script's own remaining lines) —
+# a self-updating script isn't guaranteed atomic against its own file
+# changing mid-run, so pulling later can execute a mix of old and new
+# script/compose content in one pass. Bit us for real: old GID-resolution
+# code ran, then this pulled in new var names docker-compose.yml now
+# requires, and nothing re-ran the (now-current) resolution step for
+# them — see issue #5's thread.
+echo "==> git pull"
+git pull --ff-only
+
+[ -f .env ] || cp .env.example .env
+
+echo "==> syncing tracked config values from .env.example"
+# ponytail: gum (charmbracelet/gum) is a single static binary. Vendored as
+# a release tarball under scripts/vendor/ (checked into git) for the R9700
+# box, which has no outbound internet access — the download fallback below
+# is only for other archs / when the vendored copy is missing or stale.
+# Cached under .cache/gum/ (gitignored) so repeat runs don't re-extract.
+GUM_VERSION="0.14.5"
+GUM_DIR="$(pwd)/.cache/gum"
+GUM_BIN="$GUM_DIR/gum"
+ensure_gum() {
+  command -v gum >/dev/null 2>&1 && { echo "gum"; return; }
+  [ -x "$GUM_BIN" ] && { echo "$GUM_BIN"; return; }
+  mkdir -p "$GUM_DIR"
+  local arch tmpdir vendored
+  case "$(uname -m)" in
+    x86_64) arch="x86_64" ;;
+    aarch64|arm64) arch="arm64" ;;
+    *) echo "no gum build for $(uname -m), falling back to plain prompts" >&2; echo ""; return ;;
+  esac
+  tmpdir="$(mktemp -d)"
+  vendored="$(pwd)/scripts/vendor/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
+  if [ -f "$vendored" ]; then
+    tar -xz -C "$tmpdir" -f "$vendored"
+  else
+    local url="https://github.com/charmbracelet/gum/releases/download/v${GUM_VERSION}/gum_${GUM_VERSION}_Linux_${arch}.tar.gz"
+    if ! curl -fsSL "$url" | tar -xz -C "$tmpdir" 2>/dev/null; then
+      echo "no vendored gum for $arch and couldn't download from $url (no internet egress? falling back to plain prompts)" >&2
+    fi
+  fi
+  if [ -n "$(find "$tmpdir" -name gum -type f 2>/dev/null)" ]; then
+    find "$tmpdir" -name gum -type f -exec cp {} "$GUM_BIN" \;
+    chmod +x "$GUM_BIN" 2>/dev/null || true
+  fi
+  rm -rf "$tmpdir"
+  [ -x "$GUM_BIN" ] && echo "$GUM_BIN" || echo ""
+}
+
+# Collect every key where .env.example has a real (non-blank) default:
+# missing from .env -> just add it (no conflict, nothing to decide);
+# present and identical -> leave alone silently; present and different ->
+# a conflict to resolve below.
+conflict_keys=()
+conflict_old=()
+conflict_new=()
+while IFS='=' read -r key value; do
+  [ -n "$value" ] || continue
+  if ! grep -qE "^${key}=" .env; then
+    echo "${key}=${value}" >> .env
+    continue
+  fi
+  current="$(grep -E "^${key}=" .env | head -1 | cut -d= -f2-)"
+  if [ "$current" != "$value" ]; then
+    conflict_keys+=("$key")
+    conflict_old+=("$current")
+    conflict_new+=("$value")
+  fi
+done < <(grep -E '^[A-Za-z_][A-Za-z0-9_]*=.+' .env.example)
+
+if [ "${#conflict_keys[@]}" -gt 0 ]; then
+  if [ "$FORCE" = true ]; then
+    for i in "${!conflict_keys[@]}"; do
+      key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
+      sed -i "s|^${key}=.*|${key}=${new}|" .env
+      echo "${key}: ${conflict_old[$i]} -> ${new} (--force)"
+    done
+  elif [ ! -t 0 ] || [ ! -t 1 ]; then
+    echo "ERROR: ${#conflict_keys[@]} config value(s) in .env differ from .env.example, and this isn't an interactive terminal:" >&2
+    for i in "${!conflict_keys[@]}"; do
+      echo "  ${conflict_keys[$i]}: ${conflict_old[$i]} (current) vs ${conflict_new[$i]} (.env.example)" >&2
+    done
+    echo "Re-run interactively to choose per-key, or pass --force to accept every new value." >&2
+    exit 1
+  else
+    gum_bin="$(ensure_gum)"
+    labels=()
+    for i in "${!conflict_keys[@]}"; do
+      labels+=("${conflict_keys[$i]}: ${conflict_old[$i]} -> ${conflict_new[$i]}")
+    done
+    if [ -n "$gum_bin" ]; then
+      selected="$(printf '%s\n' "${labels[@]}" | "$gum_bin" choose --no-limit --selected "$(printf '%s\n' "${labels[@]}" | paste -sd,)" --header "Config differs from .env.example — selected keys take the new value, unselected keep the server's current value:")"
+    else
+      # ponytail: plain-bash fallback if gum couldn't be fetched (offline,
+      # unsupported arch) — same one-screen-of-conflicts idea, cruder UI.
+      echo "Config differs from .env.example. Enter space-separated numbers to KEEP the server's current value (all others take the new value), or press enter to take every new value:"
+      for i in "${!conflict_keys[@]}"; do
+        echo "  $((i+1))) ${labels[$i]}"
+      done
+      read -r -p "> " keep_nums
+      selected=""
+      for i in "${!conflict_keys[@]}"; do
+        case " $keep_nums " in
+          *" $((i+1)) "*) ;;
+          *) selected="${selected}${labels[$i]}"$'\n' ;;
+        esac
+      done
+    fi
+    for i in "${!conflict_keys[@]}"; do
+      key="${conflict_keys[$i]}"; new="${conflict_new[$i]}"
+      if printf '%s\n' "$selected" | grep -qxF "${labels[$i]}"; then
+        sed -i "s|^${key}=.*|${key}=${new}|" .env
+        echo "${key}: ${conflict_old[$i]} -> ${new}"
+      else
+        echo "${key}: kept ${conflict_old[$i]} (server value)"
+      fi
+    done
+  fi
+fi
+
+# Handles all three cases: the KEY=value line is missing entirely (.env
+# predates that var being added to .env.example — sed can't fix what isn't
+# there, so this appends it), present but blank, or already set.
+set_if_blank() {
+  local key="$1" value="$2"
+  if grep -qE "^${key}=.*[^[:space:]]" .env; then
+    echo "${key}: already set, skipping."
+  elif grep -qE "^${key}=" .env; then
+    sed -i "s|^${key}=.*|${key}=${value}|" .env
+    echo "${key}: set."
+  else
+    echo "${key}=${value}" >> .env
+    echo "${key}: added (was missing from .env)."
+  fi
+}
+
+echo "==> filling in missing secrets"
+# Random values — safe to re-run, never overwrites what's already set.
+# OMNIROUTE_STORAGE_ENCRYPTION_KEY especially: never change it after first
+# run, existing encrypted data becomes unreadable if you do (same caveat as
+# LiteLLM's old LITELLM_SALT_KEY).
+set_if_blank OMNIROUTE_INITIAL_PASSWORD "$(openssl rand -hex 16)"
+set_if_blank OMNIROUTE_JWT_SECRET "$(openssl rand -base64 48)"
+set_if_blank OMNIROUTE_API_KEY_SECRET "$(openssl rand -hex 32)"
+set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
+set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
+set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
+set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
+set_if_blank NEO4J_PASSWORD "$(openssl rand -hex 16)"
+
+echo "==> resolving SEARXNG_LAN_IP"
+# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
+# not from inside the omniroute container (see docs/research/litellm-searxng-search.md,
+# still the relevant background even though omniroute replaced litellm — see issue #31).
+searxng_ip="$(getent hosts search.home 2>/dev/null | awk '{print $1}' | head -1)"
+if [ -n "$searxng_ip" ]; then
+  set_if_blank SEARXNG_LAN_IP "$searxng_ip"
+else
+  echo "SEARXNG_LAN_IP: couldn't resolve search.home from this host, set it manually if still blank."
+fi
+
+echo "==> resolving ComfyUI host UID"
+# yurisasc/comfyui-rocm7.1 wants these as env vars, not just group_add in
+# compose — resolve from this host, same pattern as SEARXNG_LAN_IP.
+set_if_blank COMFYUI_PUID "$(id -u)"
+set_if_blank COMFYUI_PGID "$(id -g)"
+
+echo "==> resolving host video/render GIDs (shared by every GPU service)"
+# Numeric GIDs, not names, in docker-compose.yml's group_add: — Docker
+# resolves a *named* group_add entry against the container's own /etc/group,
+# not the host's, and fails unpredictably (worse with multiple GPU services
+# starting concurrently and racing on the same lookup) — see
+# docs/research/rocm-gpu-pin-and-render-group.md and issue #5.
+video_gid="$(getent group video 2>/dev/null | cut -d: -f3)"
+render_gid="$(getent group render 2>/dev/null | cut -d: -f3)"
+if [ -n "$video_gid" ]; then
+  set_if_blank HOST_VIDEO_GID "$video_gid"
+else
+  echo "HOST_VIDEO_GID: no 'video' group on this host, set it manually if still blank."
+fi
+if [ -n "$render_gid" ]; then
+  set_if_blank HOST_RENDER_GID "$render_gid"
+else
+  echo "HOST_RENDER_GID: no 'render' group on this host, set it manually if still blank."
+fi
+
+echo "==> validating compose config"
+docker compose config -q
+
+echo "==> pulling images"
+docker compose pull --ignore-buildable
+
+echo "==> rebuilding local-build services"
+docker compose build --pull
+
+echo "==> ensuring models are downloaded (skips already-present files)"
+docker compose --profile tools run --rm downloader
+docker compose --profile tools run --rm downloader-fast
+docker compose --profile tools run --rm downloader-comfyui
+
+echo "==> bringing up omniroute"
+docker compose up -d --wait omniroute
+
+# ponytail: no scripted key-minting yet, unlike the old LiteLLM /key/generate
+# flow — omniroute's POST /api/keys needs a dashboard login session
+# (ManagementSessionAuth), not a static bearer key, and that flow hasn't
+# been verified against a live instance (see issue #37). No in-stack
+# workload needs a key right now (nothing left calls the gateway besides
+# coding CLIs, which mint their own by hand per docs/proxy-key-onboarding.md)
+# — revisit this script once that flow is automatable.
+
+echo "==> recreating changed services"
+docker compose up -d --remove-orphans
+
+echo "==> status"
+docker compose ps