STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).
Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
so every request reprefilled its full prompt from scratch even when most
of a conversation's prefix was unchanged. This is the actual fix for why
compact-prompt prefill was slow enough to hit the timeout in the first
place.
Documents the distinction and root cause in docs/research/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K
fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700,
~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table.
Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense
model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/
--cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that
there's no explicit SSD offload tier to enable — llama.cpp's default mmap
already falls back to disk implicitly if GPU+RAM run out.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
Qwen3 spends output tokens on reasoning_content before writing content.
Open WebUI's default chat request doesn't set max_tokens, so it fell
through to llama.cpp's low default and the model ran out mid-thought,
returning finish_reason=length with empty content — no reply shown in
Open WebUI. Confirmed via a manual /v1/chat/completions call: works with
max_tokens=2000, fails without it.
litellm_params.max_tokens is a default, not a cap — any caller (or Open
WebUI's per-model Advanced Params) that sets its own max_tokens still
overrides it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Without these, LiteLLM's /ui falls back to username 'admin' and the
master key as the password, forcing the master key to be typed into
a browser. generate-secrets.sh now fills UI_PASSWORD like the other
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
The curlimages/curl image drops to non-root curl_user (uid 100) by
default, but the models named volume is created root-owned by Docker,
so writes into it failed with 'Permission denied'.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Answers whether the $20/mo Claude Pro subscription can be used as a
LiteLLM backend instead of a pay-per-token Anthropic API key. Finding:
no — Pro/Max OAuth credentials are contractually restricted to Claude
Code / native Anthropic apps (Consumer ToS + Usage Policy, actively
enforced server-side since March 2026). Documents the supported path:
a separate Anthropic Console API key as a model_list entry, same shape
as the existing llama.cpp entry. Same pattern applies to other
providers (ChatGPT Plus, Gemini subscriptions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified against the live GitHub repo/docs (not secondhand): real,
active, functioning AI gateway, ~5 months old, effectively
single-maintainer, BSL 1.1. Fails two of the requirements from
docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages
inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic
can't be routed through it) and no request-priority/queuing feature.
Recommendation: stay on LiteLLM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open WebUI now points at litellm instead of llama-server directly, using a
provisioned virtual key. llama-server's host port is dropped (internal-only
on the ai-stack network) since the proxy is the only intended entry point
now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with
per-CLI virtual keys instead of the old shared dummy key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with
custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a
priority-scheduling stub (per #16, needs real-hardware smoke test), and
required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real
hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp
directly, migration is #15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #8. The stack is reached over LAN/WAN hostnames,
not localhost:
- Open WebUI: ai.home (LAN) and ai.haylan.ch (external, via DMZ) routed
through the existing Nginx Proxy Manager instance to the published
WEBUI_PORT — no new reverse-proxy service needed in this repo.
- llama.cpp's raw API stays LAN-only and unregistered in NPM (no auth of
its own); coding CLIs reach it via the box's LAN IP or ai.home directly.
New docs/network-access.md documents the NPM proxy-host setup and the
LAN-only decision. coding-cli-setup.md and README updated to stop
referencing localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #6. Covers the two server endpoints (Anthropic
shim vs OpenAI-compatible), exact config for all three CLIs, and surfaces
the shared Qwen3.8-27B tool-calling risk plus OpenCode's own documented
local-backend brittleness up front rather than burying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers issue #7. Confirms sst/opencode moved to anomalyco/opencode,
documents install methods, opencode.json provider config for a local
OpenAI-compatible server, and known tool-calling compatibility issues
against self-hosted backends (llama.cpp --jinja requirement, empty
tool_calls handling) plus Qwen3.8-27B-specific caveats.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #4. Wires up the locked decisions from the map:
- llama.cpp (ghcr.io/ggml-org/llama.cpp:server-rocm, gfx1201) serving
Qwen3.8-27B-UD-Q4_K_XL.gguf, port published for direct Claude Code CLI /
Kimi CLI access alongside Open WebUI.
- Open WebUI with WEBUI_AUTH on, RAG+Memory wired to a standalone Qdrant
service.
- Lazytainer labels on llama-server for a 15 min idle-stop.
- Named Docker volumes only (models, qdrant-data, openwebui-data) — no host
bind-mounts.
- One-off 'downloader' compose profile instead of a host-side script with
its own dependencies, wrapped by scripts/download-model.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships
UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB
RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude-Bot