STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).
Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
so every request reprefilled its full prompt from scratch even when most
of a conversation's prefix was unchanged. This is the actual fix for why
compact-prompt prefill was slow enough to hit the timeout in the first
place.
Documents the distinction and root cause in docs/research/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
- docker-compose.yml: add --reasoning off to qwen-classifier per
fast-model-choice.md's own recommendation (ggml-org/llama.cpp#20809
safety net) — missed in the original rollout, caught while writing
this up.
- docs/coding-cli-setup/qwen-code.md: rewritten to match what's actually
deployed (qwen-classifier, partial GPU offload, 65536 ctx, Q4_K_XL) —
previously described an unimplemented llama-server-fast/8192-ctx plan.
Documents the non-interactive MCP tool allow-list gap found live-testing.
- docs/coding-cli-setup/opencode.md: fix stale 65536 example that didn't
match its own documented LLAMA_CTX_SIZE/LLAMA_PARALLEL formula (131072).
- docs/research/fast-model-choice.md: implementation note recording where
the actual rollout diverged from this doc's original recommendations
(service name, quant, context size, CPU-first-then-GPU path).
- docs/research/omniroute-account-semaphore-timeout.md: new — the
hardcoded 30s per-connection semaphore timeout found during the
pr-agent investigation, root-caused against OmniRoute's own source,
and the maxConcurrent:null + providerSpecificData.timeoutMs fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Assess whether the Colibrì disk-streaming MoE inference engine fits the AMD ROCm single-GPU setup or fills a gap beyond the existing llama.cpp/OmniRoute/Qdrant/Neo4j/ComfyUI stack. Conclusion: worth a passing watch, not worth integrating today.
Two real-hardware findings from issue #5, both researched and fixed
together (docs/research/rocm-gpu-pin-and-render-group.md):
- GPU_MAX_HW_QUEUES=1 on llama-server and llama-server-fast. Confirmed
on real hardware: either container alone is fine (3% GPU, low
power), only two concurrent HIP contexts pin the R9700 at 100%/
boost-clock (ROCm/ROCm#5706, an MES firmware bug). The env var is
validated on the exact image this stack uses, per-process by design
— applying it to both containers is the correct scope.
- group_add switched from plain names (video/render) to resolved
numeric GIDs (HOST_VIDEO_GID/HOST_RENDER_GID) on all three GPU
services. The "unable to find group render: no matching entries in
group file" error confirmed new since the second GPU service was
added is a known Docker bug (docker/cli#4714): group_add by name
resolves against the container's own /etc/group, not the host's,
and multiple GPU services starting concurrently race on that lookup.
Numeric GIDs skip resolution entirely. scripts/update.sh's existing
comfyui-only GID resolution is generalized to resolve these once for
all three services.
docker compose config -q validated (fails fast with a clear error if
the GIDs aren't resolved yet, passes once they are).
Refs #5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507,
Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for
use as qwen-code CLI's Auto Mode classifier fastModel. Model choice
researched in docs/research/fast-model-choice.md: architecturally
non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added
defensively per a known (closed) llama.cpp misdetection bug.
- docker-compose.yml: llama-server-fast + downloader-fast services,
omniroute depends_on updated
- .env.example: LLAMA_FAST_* vars
- scripts/update.sh: runs the new downloader profile
Refs #44
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
Closes issue #39 — with llama-server stopped and the full ~32GB R9700
available, Qwen-Image (Apache-2.0, 20B MMDiT) at FP8 precision (~25GB)
is the recommended upgrade from FLUX.1-schnell: it has the cleanest
license of the candidates and is the only one with a ComfyUI workflow
pre-validated specifically on this GPU architecture (gfx1201/R9700),
per kyuz0/amd-r9700-comfy. HunyuanImage-3.0 is ruled out (CUDA-only,
multi-GPU datacenter VRAM floor); Krea-2 flagged as promising but too
new for R9700-specific field evidence.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015zCwaWJQuKgXDUfRPBqDS7
Verdict: switch. See docs/research/langchain-pgvector-vs-litellm-pgvector.md
for full reasoning, LOC comparison, and a concrete replacement service spec.
Research/doc only — no compose/config/vendor changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers whether SearXNG is natively supported by LiteLLM's /v1/search
feature, whether it's a model-tool or automatic retrieval, what
docker-compose.yml networking change is needed for the litellm container
to reach the LAN's search.home host, and how it interacts with this
project's known-flaky Qwen3.8-27B tool-calling.
Research only — litellm-config.yaml and docker-compose.yml are unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers whether the $20/mo Claude Pro subscription can be used as a
LiteLLM backend instead of a pay-per-token Anthropic API key. Finding:
no — Pro/Max OAuth credentials are contractually restricted to Claude
Code / native Anthropic apps (Consumer ToS + Usage Policy, actively
enforced server-side since March 2026). Documents the supported path:
a separate Anthropic Console API key as a model_list entry, same shape
as the existing llama.cpp entry. Same pattern applies to other
providers (ChatGPT Plus, Gemini subscriptions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified against the live GitHub repo/docs (not secondhand): real,
active, functioning AI gateway, ~5 months old, effectively
single-maintainer, BSL 1.1. Fails two of the requirements from
docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages
inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic
can't be routed through it) and no request-priority/queuing feature.
Recommendation: stay on LiteLLM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers issue #7. Confirms sst/opencode moved to anomalyco/opencode,
documents install methods, opencode.json provider config for a local
OpenAI-compatible server, and known tool-calling compatibility issues
against self-hosted backends (llama.cpp --jinja requirement, empty
tool_calls handling) plus Qwen3.8-27B-specific caveats.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships
UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB
RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude-Bot