docs/agents/writing-docs.md holds the doc/comment rules and done-check,
reached via a pointer in CLAUDE.md (AGENTS.md and QWEN.md import it).
docs/agents/domain.md gains the procedure for new or conflicting terms.
Refs #55 (agent guide #62)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explanations from docker-compose.yml, .env.example and
scripts/update.sh now live in docs/architecture.md, gateway.md,
network-access.md and operations.md. Code keeps `# see docs/...`
pointers, one-line warnings (storage key, SSRF guard, WAL grace period,
git pull ordering) and "auto-filled — leave blank" markers.
Refs #55 (comment destination map #61)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CONTEXT.md glossary
- README reduced to intro, quickstart and doc index
- new architecture.md, gateway.md (from proxy-key-onboarding.md),
operations.md; network-access.md rewritten
- coding-cli-setup: stale OMNIROUTE_PORT:-4000 endpoint, terms and
duplicated per-slot/classifier explanations fixed
- research/README.md index; proxy-request-priority.md moved there as
superseded
Every doc names keys instead of restating values and links its source.
Refs #55 (#56, #57, #59, #60)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the comfyui, downloader-comfyui, open-webui, lazytainer and neo4j
services with their volumes, env keys and update.sh steps, plus
scripts/switch-model.sh and the two image-generation research docs.
Refs #55 (removal inventory #58, intent vs. source #63)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OmniRoute's built-in comfyui provider hardcodes a CheckpointLoaderSimple
(single-file-checkpoint) workflow with no override (confirmed by
reading open-sse/handlers/imageGeneration/providers/comfyUI.ts
upstream), so it can't drive Qwen-Image 2.1's split unet/text-encoder/
vae GGUF loader chain. Open WebUI's own ComfyUI image backend is fully
generic instead — any exported workflow_api.json with arbitrary node
IDs mapped to parameters — so it can point straight at comfyui:8188.
Chat still goes through omniroute like every other client, via a
per-workload key (OPENWEBUI_OMNIROUTE_KEY, minted by hand same as
always). Published on host port 8008 per user request.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
main.py crashed on boot (FileNotFoundError on /storage/custom_nodes) -
--base-directory fully relocates custom_nodes/input/output/user/temp
under it, it doesn't create them or fall back to the image's own
ComfyUI/custom_nodes (which has ComfyUI-Manager baked in). Seed
/storage/custom_nodes from the image's copy on first boot only (skips
once populated), so Manager survives the base-directory switch and
persists across recreates.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
abenzerps/Qwen-Image-2.1-Uncensored-GGUF keeps the diffusion GGUFs at
repo root but the companion text encoder and VAE under text_encoders/
and vae/ subfolders — confirmed via the HF API's file listing, not
present in the file list used when these URLs were first written.
404'd on first real run. Local storage paths are unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
That service was merged into llama-server a while back (see the
comment on llama-server-fast's old alias), but update.sh's set -euo
pipefail meant every run since has hard-exited there — silently
skipping downloader-comfyui and the final `docker compose up -d`
recreate step on every invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yurisasc/comfyui-rocm7.1 has no entrypoint/CMD of its own (drops into
bash) and never reads PUID/PGID/VIDEO_GID/RENDER_GID/BASE_STORAGE_PATH
- confirmed by inspecting the image directly. Without an explicit
command the container just exits 0 immediately and restart-loops
forever with zero log output, which is what was actually blocking
comfyui.home/8138, unrelated to the earlier GGUF model swap.
Add `command: python main.py --listen 0.0.0.0 --port 8188
--base-directory /storage` (the real way to point it at the storage
volume) and remove COMFYUI_PUID/COMFYUI_PGID, since nothing consumes
them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the Comfy-Org FP8 safetensors split with abenzerps'
Qwen-Image-2.1-Uncensored-GGUF (Q4_K_M diffusion weights + int8 text
encoder + bf16 VAE). GGUF weights land in models/unet per
ComfyUI-GGUF's convention instead of models/diffusion_models.
Flagged as unconfirmed: whether yurisasc/comfyui-rocm7.1 already
bundles the ComfyUI-GGUF custom node needed for the "Unet Loader
(GGUF)" node — check on deploy and install manually if absent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).
Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
so every request reprefilled its full prompt from scratch even when most
of a conversation's prefix was unchanged. This is the actual fix for why
compact-prompt prefill was slow enough to hit the timeout in the first
place.
Documents the distinction and root cause in docs/research/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
- docker-compose.yml: add --reasoning off to qwen-classifier per
fast-model-choice.md's own recommendation (ggml-org/llama.cpp#20809
safety net) — missed in the original rollout, caught while writing
this up.
- docs/coding-cli-setup/qwen-code.md: rewritten to match what's actually
deployed (qwen-classifier, partial GPU offload, 65536 ctx, Q4_K_XL) —
previously described an unimplemented llama-server-fast/8192-ctx plan.
Documents the non-interactive MCP tool allow-list gap found live-testing.
- docs/coding-cli-setup/opencode.md: fix stale 65536 example that didn't
match its own documented LLAMA_CTX_SIZE/LLAMA_PARALLEL formula (131072).
- docs/research/fast-model-choice.md: implementation note recording where
the actual rollout diverged from this doc's original recommendations
(service name, quant, context size, CPU-first-then-GPU path).
- docs/research/omniroute-account-semaphore-timeout.md: new — the
hardcoded 30s per-connection semaphore timeout found during the
pr-agent investigation, root-caused against OmniRoute's own source,
and the maxConcurrent:null + providerSpecificData.timeoutMs fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Full offload (999 layers) left only ~768MB free VRAM regardless of
batch-size/flash-attn tuning — that gap tracks roughly fixed regardless
of those knobs, most likely ROCm's own per-process HIP context overhead
(same ROCm#5706 quirk already noted for two HIP contexts sharing this
card, see llama-server's GPU_MAX_HW_QUEUES comment above). Dropping to
28/36 layers on GPU (6 layers + their KV on CPU) trades a slice of
speed for real freed VRAM — still >80% of layers on GPU, nowhere near
CPU-only's unusable latency. Verifying live before locking this number in.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Batch/ubatch reduction barely moved measured VRAM (~768MB free, same as
before) — wrong lever. llama-server runs with --flash-attn on; this
service didn't. Without it, the unfused attention compute buffer at
65536 ctx is far larger than flash-attn's fused workspace, which is
what the naive weights+KV estimate missed. Matches llama-server's flag.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Measured live: full-offload weights+KV (~4.9GiB estimate) actually used
~5.85GiB, leaving only ~700MB free on the R9700 — too tight, real OOM
risk for either GPU process. The gap was compute-buffer/graph overhead
the naive estimate didn't account for. Drop --batch-size/--ubatch-size
well below llama-server's defaults (2048/512) to shrink it — a
single-request classifier has no batching throughput to lose.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CPU-only was too slow in practice: real classification calls blew past
OmniRoute's 60s timeout and retry-looped (504/499). Moved to GPU.
Also traced qwen-code's actual classifier transcript cap in its source
(MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000/message) —
worst case is ~40-50K tokens, not the 131072 originally set in
settings.json (copied from the main model's entry, not a real qwen-code
requirement). Dropped ctx-size to 65536 (~1.5x margin) so Q4_K_XL
weights + q4_0/q4_0 KV fit fully on GPU (~4.9GiB) inside the ~6.1GiB
free on the R9700, instead of needing partial CPU/GPU offload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Q4_K_M-class file this originally specced didn't exist yet on
gameserver (classifier crash-looped: "No such file or directory").
A Q8_K_XL GGUF for the same model was already sitting in the models
volume from something earlier — point at that instead of downloading a
new file, and drop the KV cache quant to q4_0/q4_0 to keep total RAM
comfortable now that the weights are the larger Q8 variant.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
fastModel in ~/.qwen/settings.json (permissions.autoMode.classifier) was
aliased onto llama-server's own 27B connection, so every tool-call safety
check queued behind whatever heavy generation was already running on that
model's 2 GPU slots.
Add qwen-classifier: a separate llama.cpp instance, CPU-only, running
Qwen3-4B-Instruct-2507 (the smallest Qwen3 with native >=131072 context,
qwen-code's requirement, without lossy RoPE scaling). Structurally isolated
from llama-server's queue instead of sharing it. Sized for gameserver's
~17GiB free system RAM: q8_0/q8_0 KV at full 131072 ctx (~9.8GiB) + Q4_K_M-
class weights (~2.3GiB) fits comfortably, with better KV quality than the
q4_0 that would've been needed to fit this on the GPU's ~6GiB free VRAM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Assess whether the Colibrì disk-streaming MoE inference engine fits the AMD ROCm single-GPU setup or fills a gap beyond the existing llama.cpp/OmniRoute/Qdrant/Neo4j/ComfyUI stack. Conclusion: worth a passing watch, not worth integrating today.
Move docs/coding-cli-setup.md to docs/coding-cli-setup/ with one file
per CLI (claude-code, kimi-cli, opencode, qwen-code) plus a shared
index.md for the gateway intro, tool-calling risk note, and summary
table.
Also fixes the qwen-code doc: context sizes are per-slot
(LLAMA_CTX_SIZE / LLAMA_PARALLEL), not raw LLAMA_CTX_SIZE (same fix
applied to OpenCode's limit.context); documents the fastModel
classifier provider and its own context math; adds the omniroute-search
MCP server (SearXNG web search) and Auto Mode permissions tuning that
were missing from the original qwen-code section.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
current=$(grep ...) ran before checking whether the key existed in .env
at all. Under set -euo pipefail, a key missing from .env (e.g. a new
tunable this branch just added to .env.example) made that grep exit 1,
pipefail propagated it, and set -e killed the script instantly with no
output - looked exactly like a gum hang but never reached ensure_gum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
The server has no outbound internet access, so the curl download in
ensure_gum always failed silently and fell back to plain prompts.
Vendor the x86_64 release tarball under scripts/vendor/ and check it
before attempting a download - download stays as a fallback for other
archs or a version bump without a re-vendor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
curl -fsSL | tar swallowed curl errors entirely, so a network failure
fetching gum looked identical to a successful skip - just silently
dropped into the plain-prompt fallback with no explanation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bs8xf8Dt8X6tRBvBiLCXYc
Documents the NPM route for the RAG knowledge graph alongside the
existing proxy-ai.home/search.home entries. Neo4j (not Qdrant) gets
the hostname — it's the human-facing graph browser; Qdrant's
dashboard stays on its raw port for now.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
Drops the second always-resident llama.cpp instance, its downloader,
and the omniroute depends_on entry. Also strips the now-dead
LLAMA_FAST_* block from .env.example and the stale VRAM-budget comment
in scripts/switch-model.sh that assumed this service was always up.
Note: this was qwen-code's Auto Mode Stage 1 classifier (fastModel) —
see docs/research/fast-model-choice.md and issue #44. Auto Mode will
lose that classifier until/unless it's reconfigured to route
elsewhere or fall back to prompt-only classification.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
haylan
