haylanandClaude-Bot
52a92f6508
Real failure: "Auto Mode couldn't classify this action (Classifier
stage 1 unavailable)". Reproduced directly against the server:
{"error":{"message":"[400]: request (6186 tokens) exceeds the
available context size (4096 tokens)"...
LLAMA_FAST_CTX_SIZE=8192 is the TOTAL across every LLAMA_FAST_PARALLEL
slot, not per-request — the main model's own .env.example comment
already calls this out, missed it when llama-server-fast was set up
(#44). With PARALLEL=2 that's 4096/slot, too small for a real
classifier call (hints + environment + recent tool-call history).
Fixed by dropping to a single slot (LLAMA_FAST_PARALLEL=1) rather than
raising ctx-size — this service doesn't need concurrent classifier
calls the way the main model needs concurrent chat sessions, so this
costs no extra VRAM. The full 8192 now goes to the one slot.
docker compose config -q validated.
Refs #5
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
144 lines
8.1 KiB
Bash
144 lines
8.1 KiB
Bash
# Copy to .env and adjust — or just run ./scripts/update.sh, which creates
|
|
# .env from this file and fills in every secret/key below it can generate
|
|
# itself (see each var's comment). All values below are defaults baked into
|
|
# docker-compose.yml — only uncomment/change what you actually want to
|
|
# override.
|
|
|
|
# --- llama.cpp / model ---
|
|
LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
|
|
# 999 = every layer on GPU (this model is dense, not MoE, and already fits
|
|
# fully in 32GB VRAM — see docs/research/qwen3.8-27b-quant.md). Lower this
|
|
# to leave that many fewer layers on GPU and push the rest to CPU/system RAM
|
|
# if something else is contending for VRAM — llama.cpp has no separate
|
|
# "RAM offload" flag, --n-gpu-layers *is* the RAM-offload knob for a dense
|
|
# model. Don't reach for --n-cpu-moe/--cpu-moe/--override-tensor "exps" —
|
|
# those target Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B), not this
|
|
# one, and are no-ops here.
|
|
# There's no separate "then SSD" tier to enable either: llama.cpp mmaps the
|
|
# model file by default (no --no-mmap here), so if GPU+RAM ever can't hold
|
|
# the working set, the OS pages the rest in from disk automatically — an
|
|
# implicit, slow last resort, not a config knob. An explicit tiered SSD
|
|
# offload has been an open llama.cpp feature request since 2025 (still
|
|
# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
|
|
LLAMA_GPU_LAYERS=999
|
|
# 262144 = this model's true max (max_position_embeddings in Qwen/Qwen3.8-27B's
|
|
# config.json) — the largest --ctx-size llama.cpp will even accept for it.
|
|
# fp16 KV cache at full context would be ~16GB, on top of 17.6GB weights =
|
|
# ~33.6GB, which does NOT fit the 32GB R9700 on its own. docker-compose.yml
|
|
# now runs --cache-type-k/v q8_0, which roughly halves KV memory (~8GB at
|
|
# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
|
|
# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
|
|
LLAMA_CTX_SIZE=262144
|
|
# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
|
|
# default) with no flag set — under concurrent subagent fan-out, 4 requests
|
|
# split the same GPU compute, so a large-context prefill can queue behind
|
|
# others long enough to blow past OmniRoute's stream-idle timeout, which then
|
|
# cancels the request (see issue-tracker notes on the timeout/cancel loop).
|
|
# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
|
|
# raise back toward 4 if throughput (not latency) becomes the bottleneck
|
|
# instead. Each slot gets LLAMA_CTX_SIZE / LLAMA_PARALLEL tokens of context —
|
|
# real sessions have hit ~66K tokens, so don't drop LLAMA_CTX_SIZE without
|
|
# checking that per-slot number stays comfortably above observed usage.
|
|
LLAMA_PARALLEL=2
|
|
|
|
# --- Lazytainer ---
|
|
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
|
LAZYTAINER_INACTIVE_TIMEOUT=900
|
|
|
|
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
|
|
# Resolved automatically by ./scripts/update.sh from search.home on this
|
|
# host — leave blank. Only set by hand if that resolution fails (e.g.
|
|
# search.home isn't a static DHCP reservation and its IP drifted).
|
|
SEARXNG_LAN_IP=
|
|
|
|
# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
|
|
# OMNIROUTE_PORT is the host-published port (reverse-proxied by NPM) — kept
|
|
# at 4000, same as the old LiteLLM setup, so existing NPM/firewall config
|
|
# doesn't need to change. It's mapped via plain Docker port publishing onto
|
|
# API_PORT, omniroute's own container-internal port (left at its default,
|
|
# not reconfigured to match). The dashboard (DASHBOARD_PORT) is never
|
|
# published at all — see docker-compose.yml's omniroute service comment.
|
|
OMNIROUTE_API_PORT=20129
|
|
OMNIROUTE_DASHBOARD_PORT=20128
|
|
# SSE inactivity timeout before OmniRoute gives up on a streaming request and
|
|
# cancels it (which cancels the matching llama-server task too). 180s gives
|
|
# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
|
|
OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
|
|
# Random values, filled in automatically by ./scripts/update.sh — leave
|
|
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
|
|
# change it there afterwards — this is only the first-boot value):
|
|
OMNIROUTE_INITIAL_PASSWORD=
|
|
# Signs dashboard session cookies:
|
|
OMNIROUTE_JWT_SECRET=
|
|
# Encrypts API key values at rest in omniroute's SQLite DB:
|
|
OMNIROUTE_API_KEY_SECRET=
|
|
# Encrypts the whole SQLite DB at rest. Do not change after first run —
|
|
# existing encrypted data becomes unreadable if you do (same caveat as
|
|
# LiteLLM's old LITELLM_SALT_KEY):
|
|
OMNIROUTE_STORAGE_ENCRYPTION_KEY=
|
|
# Per-deployment salts — random is fine, just needs to be stable:
|
|
OMNIROUTE_MACHINE_ID_SALT=
|
|
OMNIROUTE_CLI_SALT=
|
|
# Required (production) — shared secret for the internal Codex Responses
|
|
# WebSocket bridge. Random value, filled in automatically:
|
|
OMNIROUTE_WS_BRIDGE_SECRET=
|
|
# Per-workload virtual keys (one per client that calls the gateway) have no
|
|
# scripted /key/generate equivalent yet — omniroute's key-creation endpoint
|
|
# needs a dashboard login session, not a static bearer key (see issue #37).
|
|
# Mint them by hand in the dashboard, add a KEY=value line here per workload
|
|
# as you onboard one. See docs/proxy-key-onboarding.md.
|
|
|
|
# --- ComfyUI (local image generation, see issue #38 wayfinder map) ---
|
|
# yurisasc/comfyui-rocm7.1 manages GPU-group access via these GID/UID env
|
|
# vars rather than relying solely on docker-compose.yml's group_add.
|
|
# Resolved automatically from the host by ./scripts/update.sh — leave blank.
|
|
COMFYUI_PUID=
|
|
COMFYUI_PGID=
|
|
|
|
# Shared by every GPU-touching service (llama-server, llama-server-fast,
|
|
# comfyui) for group_add: — resolved to real host GIDs by ./scripts/update.sh
|
|
# rather than left as plain group names in docker-compose.yml, because Docker
|
|
# resolves a *named* group_add entry against the container's own /etc/group,
|
|
# not the host's, and fails unpredictably when the image doesn't define one
|
|
# (worse with multiple GPU services racing on the same lookup at once — see
|
|
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
|
|
HOST_VIDEO_GID=
|
|
HOST_RENDER_GID=
|
|
|
|
# --- llama.cpp / fast model (second, always-resident instance — see
|
|
# docs/research/fast-model-choice.md and issue #44) ---
|
|
# Qwen3-4B-Instruct-2507: architecturally non-thinking (never emits
|
|
# <think> blocks, unlike Qwen3-1.7B/0.6B which need a per-call toggle) —
|
|
# picked specifically so it stays fast enough for qwen-code's Auto Mode
|
|
# classifier (Stage 1 wants ~300ms). Same publisher (unsloth) as the main
|
|
# model for consistency.
|
|
LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
|
|
# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
|
|
# is dense too.
|
|
LLAMA_FAST_GPU_LAYERS=999
|
|
# --ctx-size is the TOTAL across every LLAMA_FAST_PARALLEL slot, not per
|
|
# request — same halving already called out for the main model above.
|
|
# Was PARALLEL=2, silently halving this to 4096/slot — too small: a real
|
|
# classifier call (hints + environment + recent tool-call history) hit
|
|
# "exceeds the available context size (4096 tokens)" in practice, which
|
|
# qwen-code surfaces as "Auto Mode couldn't classify this action
|
|
# (Classifier stage 1 unavailable)" — see issue #5. Fixed by dropping to
|
|
# a single slot instead of raising ctx-size (no extra VRAM, and this
|
|
# service doesn't need concurrent classifier calls the way the main
|
|
# model needs concurrent chat sessions) — the full 8192 now goes to the
|
|
# one slot. If hints.allow/softDeny/hardDeny ever approach their
|
|
# 50-entries-each ceiling, raise LLAMA_FAST_CTX_SIZE instead — qwen-code
|
|
# caps those at 200 chars x 150 entries plus 40,000 chars of
|
|
# historical-action context, which can exceed 8192 tokens worst-case.
|
|
LLAMA_FAST_CTX_SIZE=8192
|
|
LLAMA_FAST_PARALLEL=1
|
|
|
|
# --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
|
|
# image-generation-model-choice.md and issue #42) ---
|
|
# Three files: diffusion weights, text encoder, VAE — all from the official
|
|
# Comfy-Org FP8 split, chosen specifically because it's the only candidate
|
|
# with a ComfyUI workflow pre-validated on this exact GPU (gfx1201/R9700).
|
|
COMFYUI_DIFFUSION_MODEL_FILE=qwen_image_fp8_e4m3fn.safetensors
|
|
COMFYUI_TEXT_ENCODER_FILE=qwen_2.5_vl_7b_fp8_scaled.safetensors
|
|
COMFYUI_VAE_FILE=qwen_image_vae.safetensors
|