DFlash2 truncates its draft when selector confidence drops below p_min,
cutting wasted verify work on long, low-acceptance thinking output.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move all speculative-decoding flags into LLAMA_SPEC_ARGS (blank = off)
and the KV cache quant into LLAMA_KV_CACHE_TYPE, so token-corruption
A/B tests are a single .env edit instead of a compose edit.
Replaces LLAMA_DRAFT_MODEL_FILE and LLAMA_SPEC_N_MAX.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The URL used the filename as the Hugging Face repo name. Also align
the compose fallback for LLAMA_DRAFT_MODEL_FILE with Q8_0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the per-model downloader services with a single downloader
that reads file/url pairs from models.yml. Downloads now go to a
.part file first, so an interrupted run can't leave a truncated
model that later runs skip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs llama.cpp's SPEED-Bench client (pinned commit) against
llama-server over the compose network, in the tools profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/agents/writing-docs.md holds the doc/comment rules and done-check,
reached via a pointer in CLAUDE.md (AGENTS.md and QWEN.md import it).
docs/agents/domain.md gains the procedure for new or conflicting terms.
Refs #55 (agent guide #62)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explanations from docker-compose.yml, .env.example and
scripts/update.sh now live in docs/architecture.md, gateway.md,
network-access.md and operations.md. Code keeps `# see docs/...`
pointers, one-line warnings (storage key, SSRF guard, WAL grace period,
git pull ordering) and "auto-filled — leave blank" markers.
Refs #55 (comment destination map #61)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CONTEXT.md glossary
- README reduced to intro, quickstart and doc index
- new architecture.md, gateway.md (from proxy-key-onboarding.md),
operations.md; network-access.md rewritten
- coding-cli-setup: stale OMNIROUTE_PORT:-4000 endpoint, terms and
duplicated per-slot/classifier explanations fixed
- research/README.md index; proxy-request-priority.md moved there as
superseded
Every doc names keys instead of restating values and links its source.
Refs #55 (#56, #57, #59, #60)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the comfyui, downloader-comfyui, open-webui, lazytainer and neo4j
services with their volumes, env keys and update.sh steps, plus
scripts/switch-model.sh and the two image-generation research docs.
Refs #55 (removal inventory #58, intent vs. source #63)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OmniRoute's built-in comfyui provider hardcodes a CheckpointLoaderSimple
(single-file-checkpoint) workflow with no override (confirmed by
reading open-sse/handlers/imageGeneration/providers/comfyUI.ts
upstream), so it can't drive Qwen-Image 2.1's split unet/text-encoder/
vae GGUF loader chain. Open WebUI's own ComfyUI image backend is fully
generic instead — any exported workflow_api.json with arbitrary node
IDs mapped to parameters — so it can point straight at comfyui:8188.
Chat still goes through omniroute like every other client, via a
per-workload key (OPENWEBUI_OMNIROUTE_KEY, minted by hand same as
always). Published on host port 8008 per user request.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
main.py crashed on boot (FileNotFoundError on /storage/custom_nodes) -
--base-directory fully relocates custom_nodes/input/output/user/temp
under it, it doesn't create them or fall back to the image's own
ComfyUI/custom_nodes (which has ComfyUI-Manager baked in). Seed
/storage/custom_nodes from the image's copy on first boot only (skips
once populated), so Manager survives the base-directory switch and
persists across recreates.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
abenzerps/Qwen-Image-2.1-Uncensored-GGUF keeps the diffusion GGUFs at
repo root but the companion text encoder and VAE under text_encoders/
and vae/ subfolders — confirmed via the HF API's file listing, not
present in the file list used when these URLs were first written.
404'd on first real run. Local storage paths are unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
That service was merged into llama-server a while back (see the
comment on llama-server-fast's old alias), but update.sh's set -euo
pipefail meant every run since has hard-exited there — silently
skipping downloader-comfyui and the final `docker compose up -d`
recreate step on every invocation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
yurisasc/comfyui-rocm7.1 has no entrypoint/CMD of its own (drops into
bash) and never reads PUID/PGID/VIDEO_GID/RENDER_GID/BASE_STORAGE_PATH
- confirmed by inspecting the image directly. Without an explicit
command the container just exits 0 immediately and restart-loops
forever with zero log output, which is what was actually blocking
comfyui.home/8138, unrelated to the earlier GGUF model swap.
Add `command: python main.py --listen 0.0.0.0 --port 8188
--base-directory /storage` (the real way to point it at the storage
volume) and remove COMFYUI_PUID/COMFYUI_PGID, since nothing consumes
them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the Comfy-Org FP8 safetensors split with abenzerps'
Qwen-Image-2.1-Uncensored-GGUF (Q4_K_M diffusion weights + int8 text
encoder + bf16 VAE). GGUF weights land in models/unet per
ComfyUI-GGUF's convention instead of models/diffusion_models.
Flagged as unconfirmed: whether yurisasc/comfyui-rocm7.1 already
bundles the ComfyUI-GGUF custom node needed for the "Unet Loader
(GGUF)" node — check on deploy and install manually if absent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
STREAM_IDLE_TIMEOUT_MS was raised to 180s on 2026-09-09 to give contended
llama-server prefill room to produce a first token, but qwen-code sessions
kept hitting "Stream produced no non-ping SSE event within 95000ms" the very
next morning. Per OmniRoute's own docs, that's the wrong timer: the first
non-ping SSE event's deadline inherits REQUEST_TIMEOUT_MS (default 10 min,
computed as remaining budget after retries/cooldowns), not
STREAM_IDLE_TIMEOUT_MS (which only bounds gaps between chunks once streaming
has already started).
Two changes:
- Add REQUEST_TIMEOUT_MS=1800000 (30 min) on the omniroute service, exposed
as OMNIROUTE_REQUEST_TIMEOUT_MS like the existing stream-idle var. Safety
margin, not the root-cause fix.
- Add --cache-reuse 256 to llama-server: it had no KV-cache reuse configured,
so every request reprefilled its full prompt from scratch even when most
of a conversation's prefix was unchanged. This is the actual fix for why
compact-prompt prefill was slow enough to hit the timeout in the first
place.
Documents the distinction and root cause in docs/research/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WqWBahogLCkrXNfzCSvcVc
LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K
fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700,
~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table.
Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense
model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/
--cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that
there's no explicit SSD offload tier to enable — llama.cpp's default mmap
already falls back to disk implicitly if GPU+RAM run out.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
Resolves#27 (part of #26). Findings from opencode.ai/docs (fetched
2026-09-03) plus a fresh clone of anomalyco/opencode @ b578b72
(v1.18.27):
- No percentage-threshold config key exists; compaction.{auto,prune,
reserved,tail_turns,preserve_recent_tokens} is the full global
config surface (opencode.json top-level, not per-provider/model).
- Trigger is usedTokens >= context - reservedBuffer, not a hardcoded
75%/95% cutoff — contradicts an unverified claim in a closed
GitHub feature request (#11314).
- Reasoning tokens (Qwen3's reasoning_content) are counted via the
provider's usage.total_tokens in the normal path, but excluded
from the fallback sum if a provider ever omits total_tokens.
- No per-model/per-agent threshold override exists (confirmed by
several closed-not-planned feature requests); the only per-model
lever is each model's own limit.context/limit.output.
- Mechanism is provider-agnostic: applies identically to a hand-
declared @ai-sdk/openai-compatible provider (this repo's llamacpp
setup) as to hosted providers, provided limit.context is set.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
Qwen3 spends output tokens on reasoning_content before writing content.
Open WebUI's default chat request doesn't set max_tokens, so it fell
through to llama.cpp's low default and the model ran out mid-thought,
returning finish_reason=length with empty content — no reply shown in
Open WebUI. Confirmed via a manual /v1/chat/completions call: works with
max_tokens=2000, fails without it.
litellm_params.max_tokens is a default, not a cap — any caller (or Open
WebUI's per-model Advanced Params) that sets its own max_tokens still
overrides it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Without these, LiteLLM's /ui falls back to username 'admin' and the
master key as the password, forcing the master key to be typed into
a browser. generate-secrets.sh now fills UI_PASSWORD like the other
secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
The curlimages/curl image drops to non-root curl_user (uid 100) by
default, but the models named volume is created root-owned by Docker,
so writes into it failed with 'Permission denied'.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q2CR8yawSf7pwAVYnjwFea
Answers whether the $20/mo Claude Pro subscription can be used as a
LiteLLM backend instead of a pay-per-token Anthropic API key. Finding:
no — Pro/Max OAuth credentials are contractually restricted to Claude
Code / native Anthropic apps (Consumer ToS + Usage Policy, actively
enforced server-side since March 2026). Documents the supported path:
a separate Anthropic Console API key as a model_list entry, same shape
as the existing llama.cpp entry. Same pattern applies to other
providers (ChatGPT Plus, Gemini subscriptions).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified against the live GitHub repo/docs (not secondhand): real,
active, functioning AI gateway, ~5 months old, effectively
single-maintainer, BSL 1.1. Fails two of the requirements from
docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages
inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic
can't be routed through it) and no request-priority/queuing feature.
Recommendation: stay on LiteLLM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open WebUI now points at litellm instead of llama-server directly, using a
provisioned virtual key. llama-server's host port is dropped (internal-only
on the ai-stack network) since the proxy is the only intended entry point
now. docs/coding-cli-setup.md repointed at the proxy's endpoints/ports with
per-CLI virtual keys instead of the old shared dummy key.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds litellm + litellm-db to docker-compose.yml, litellm-config.yaml with
custom shadow-cost pricing (Claude Sonnet 5 reference, per #11) and a
priority-scheduling stub (per #16, needs real-hardware smoke test), and
required LITELLM_MASTER_KEY/SALT_KEY/DB_PASSWORD env vars. Untested on real
hardware — that's #17. Open WebUI/coding CLIs still talk to llama.cpp
directly, migration is #15.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #8. The stack is reached over LAN/WAN hostnames,
not localhost:
- Open WebUI: ai.home (LAN) and ai.haylan.ch (external, via DMZ) routed
through the existing Nginx Proxy Manager instance to the published
WEBUI_PORT — no new reverse-proxy service needed in this repo.
- llama.cpp's raw API stays LAN-only and unregistered in NPM (no auth of
its own); coding CLIs reach it via the box's LAN IP or ai.home directly.
New docs/network-access.md documents the NPM proxy-host setup and the
LAN-only decision. coding-cli-setup.md and README updated to stop
referencing localhost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #6. Covers the two server endpoints (Anthropic
shim vs OpenAI-compatible), exact config for all three CLIs, and surfaces
the shared Qwen3.8-27B tool-calling risk plus OpenCode's own documented
local-backend brittleness up front rather than burying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Answers issue #7. Confirms sst/opencode moved to anomalyco/opencode,
documents install methods, opencode.json provider config for a local
OpenAI-compatible server, and known tool-calling compatibility issues
against self-hosted backends (llama.cpp --jinja requirement, empty
tool_calls handling) plus Qwen3.8-27B-specific caveats.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves wayfinder ticket #4. Wires up the locked decisions from the map:
- llama.cpp (ghcr.io/ggml-org/llama.cpp:server-rocm, gfx1201) serving
Qwen3.8-27B-UD-Q4_K_XL.gguf, port published for direct Claude Code CLI /
Kimi CLI access alongside Open WebUI.
- Open WebUI with WEBUI_AUTH on, RAG+Memory wired to a standalone Qdrant
service.
- Lazytainer labels on llama-server for a 15 min idle-stop.
- Named Docker volumes only (models, qdrant-data, openwebui-data) — no host
bind-mounts.
- One-off 'downloader' compose profile instead of a host-side script with
its own dependencies, wrapped by scripts/download-model.sh.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves the wayfinder research question: unsloth/Qwen3.8-27B-GGUF ships
UD-Q4_K_XL.gguf (17.6GB) directly. Documents VRAM footprint on 32GB
RDNA4 (R9700/gfx1201) at 32K and 128K context via llama.cpp ROCm/HIP.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
haylanandClaude Bot