chore: move code comments into docs, leave one-line pointers
Explanations from docker-compose.yml, .env.example and scripts/update.sh now live in docs/architecture.md, gateway.md, network-access.md and operations.md. Code keeps `# see docs/...` pointers, one-line warnings (storage key, SSRF guard, WAL grace period, git pull ordering) and "auto-filled — leave blank" markers. Refs #55 (comment destination map #61) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a0e5e9704b
commit
05effe5c88
4 files changed
+31
-265
No files matched your search
+11
-84
@@ -1,108 +1,35 @@
|
||||
# Copy to .env and adjust — or just run ./scripts/update.sh, which creates
|
||||
# .env from this file and fills in every secret/key below it can generate
|
||||
# itself (see each var's comment). All values below are defaults baked into
|
||||
# docker-compose.yml — only uncomment/change what you actually want to
|
||||
# override.
|
||||
# Template for .env, managed by ./scripts/update.sh — see docs/operations.md#env-sync
|
||||
|
||||
# --- llama.cpp / model ---
|
||||
# --- llama.cpp / models --- see docs/architecture.md#llama-server
|
||||
LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
|
||||
# 999 = every layer on GPU (this model is dense, not MoE, and already fits
|
||||
# fully in 32GB VRAM — see docs/research/qwen3.8-27b-quant.md). Lower this
|
||||
# to leave that many fewer layers on GPU and push the rest to CPU/system RAM
|
||||
# if something else is contending for VRAM — llama.cpp has no separate
|
||||
# "RAM offload" flag, --n-gpu-layers *is* the RAM-offload knob for a dense
|
||||
# model. Don't reach for --n-cpu-moe/--cpu-moe/--override-tensor "exps" —
|
||||
# those target Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B), not this
|
||||
# one, and are no-ops here.
|
||||
# There's no separate "then SSD" tier to enable either: llama.cpp mmaps the
|
||||
# model file by default (no --no-mmap here), so if GPU+RAM ever can't hold
|
||||
# the working set, the OS pages the rest in from disk automatically — an
|
||||
# implicit, slow last resort, not a config knob. An explicit tiered SSD
|
||||
# offload has been an open llama.cpp feature request since 2025 (still
|
||||
# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
|
||||
LLAMA_GPU_LAYERS=999
|
||||
# 262144 = this model's true max (max_position_embeddings in Qwen/Qwen3.8-27B's
|
||||
# config.json) — the largest --ctx-size llama.cpp will even accept for it.
|
||||
# fp16 KV cache at full context would be ~16GB, on top of 17.6GB weights =
|
||||
# ~33.6GB, which does NOT fit the 32GB R9700 on its own. docker-compose.yml
|
||||
# now runs --cache-type-k/v q8_0, which roughly halves KV memory (~8GB at
|
||||
# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
|
||||
# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
|
||||
LLAMA_CTX_SIZE=262144
|
||||
# Concurrent request slots — the real hardware ceiling for this GPU, not a
|
||||
# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
|
||||
# default; dropped to 2 because more contended prefill was blowing requests
|
||||
# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
|
||||
# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
|
||||
# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
|
||||
# the timeout that matters moved to OmniRoute's per-connection
|
||||
# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
|
||||
# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
|
||||
# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
|
||||
# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
|
||||
# comfortably above observed usage.
|
||||
LLAMA_PARALLEL=2
|
||||
|
||||
# Dedicated GPU-resident backend for qwen-code's tool-call harmfulness
|
||||
# classifier (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
|
||||
# qwen-classifier service comment for the why and the VRAM/context math.
|
||||
# see docs/architecture.md#qwen-classifier
|
||||
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
|
||||
|
||||
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
|
||||
# Resolved automatically by ./scripts/update.sh from search.home on this
|
||||
# host — leave blank. Only set by hand if that resolution fails (e.g.
|
||||
# search.home isn't a static DHCP reservation and its IP drifted).
|
||||
# --- SearXNG web search --- see docs/gateway.md#web-search
|
||||
# Auto-filled by update.sh — leave blank.
|
||||
SEARXNG_LAN_IP=
|
||||
|
||||
# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
|
||||
# OMNIROUTE_PORT is the host-published port (reverse-proxied by NPM) — kept
|
||||
# at 4000, same as the old LiteLLM setup, so existing NPM/firewall config
|
||||
# doesn't need to change. It's mapped via plain Docker port publishing onto
|
||||
# API_PORT, omniroute's own container-internal port (left at its default,
|
||||
# not reconfigured to match). The dashboard (DASHBOARD_PORT) is never
|
||||
# published at all — see docker-compose.yml's omniroute service comment.
|
||||
# --- OmniRoute gateway --- see docs/network-access.md#gateway
|
||||
OMNIROUTE_API_PORT=20129
|
||||
OMNIROUTE_DASHBOARD_PORT=20128
|
||||
# SSE inactivity timeout before OmniRoute gives up on a streaming request and
|
||||
# cancels it (which cancels the matching llama-server task too). 180s gives
|
||||
# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
|
||||
# see docs/gateway.md#timeouts
|
||||
OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
|
||||
# Wait budget for the *first* SSE token specifically (distinct from the
|
||||
# inter-chunk timeout above) — see
|
||||
# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a
|
||||
# contended, large-context prefill even after retries eat into the budget.
|
||||
OMNIROUTE_REQUEST_TIMEOUT_MS=1800000
|
||||
# Random values, filled in automatically by ./scripts/update.sh — leave
|
||||
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
|
||||
# change it there afterwards — this is only the first-boot value):
|
||||
# Secrets: auto-filled by update.sh — leave blank. See docs/operations.md#secrets
|
||||
OMNIROUTE_INITIAL_PASSWORD=
|
||||
# Signs dashboard session cookies:
|
||||
OMNIROUTE_JWT_SECRET=
|
||||
# Encrypts API key values at rest in omniroute's SQLite DB:
|
||||
OMNIROUTE_API_KEY_SECRET=
|
||||
# Encrypts the whole SQLite DB at rest. Do not change after first run —
|
||||
# existing encrypted data becomes unreadable if you do (same caveat as
|
||||
# LiteLLM's old LITELLM_SALT_KEY):
|
||||
# NEVER change after first run — OmniRoute's existing data becomes unreadable.
|
||||
OMNIROUTE_STORAGE_ENCRYPTION_KEY=
|
||||
# Per-deployment salts — random is fine, just needs to be stable:
|
||||
OMNIROUTE_MACHINE_ID_SALT=
|
||||
OMNIROUTE_CLI_SALT=
|
||||
# Required (production) — shared secret for the internal Codex Responses
|
||||
# WebSocket bridge. Random value, filled in automatically:
|
||||
OMNIROUTE_WS_BRIDGE_SECRET=
|
||||
# Per-workload virtual keys (one per client that calls the gateway) have no
|
||||
# scripted /key/generate equivalent yet — omniroute's key-creation endpoint
|
||||
# needs a dashboard login session, not a static bearer key (see issue #37).
|
||||
# Mint them by hand in the dashboard, add a KEY=value line here per workload
|
||||
# as you onboard one. See docs/proxy-key-onboarding.md.
|
||||
|
||||
# --- GPU access (group_add GIDs) ---
|
||||
# Shared by every GPU-touching service (llama-server, qwen-classifier)
|
||||
# for group_add: — resolved to real host GIDs by ./scripts/update.sh
|
||||
# rather than left as plain group names in docker-compose.yml, because Docker
|
||||
# resolves a *named* group_add entry against the container's own /etc/group,
|
||||
# not the host's, and fails unpredictably when the image doesn't define one
|
||||
# (worse with multiple GPU services racing on the same lookup at once — see
|
||||
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
|
||||
# --- GPU access --- see docs/architecture.md#gpu-access
|
||||
# Auto-filled by update.sh — leave blank.
|
||||
HOST_VIDEO_GID=
|
||||
HOST_RENDER_GID=
|
||||
+11
-114
@@ -5,17 +5,13 @@ services:
|
||||
devices:
|
||||
- /dev/kfd
|
||||
- /dev/dri
|
||||
# Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
|
||||
# .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
|
||||
# see docs/architecture.md#gpu-access (group_add GIDs, GPU_MAX_HW_QUEUES)
|
||||
group_add:
|
||||
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
|
||||
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
|
||||
security_opt:
|
||||
- seccomp=unconfined
|
||||
ipc: host
|
||||
# Caps this process's HIP hardware-queue allocation — works around
|
||||
# ROCm/ROCm#5706 (GPU pinned at 100%/boost-clock whenever two
|
||||
# concurrent HIP contexts touch this card). See the research doc above.
|
||||
environment:
|
||||
- GPU_MAX_HW_QUEUES=1
|
||||
volumes:
|
||||
@@ -32,63 +28,13 @@ services:
|
||||
--cache-type-v q8_0
|
||||
--cache-reuse 256
|
||||
--jinja
|
||||
# --cache-reuse 256: reuse cached KV for any matching prompt chunk of at
|
||||
# least 256 tokens (KV-shift, no reprocessing) instead of reprefilling
|
||||
# from scratch every request. Directly targets the actual root cause
|
||||
# behind the OmniRoute non-ping SSE timeout, not just the symptom — see
|
||||
# docs/research/omniroute-non-ping-sse-stream-timeout.md. Pairs with
|
||||
# OmniRoute's promptCacheAffinityEnabled (dashboard default), which keeps
|
||||
# a conversation's requests pinned to the same slot so there's a matching
|
||||
# prefix to reuse.
|
||||
# No published host port: llama-server is reached only via the omniroute
|
||||
# gateway on the ai-stack docker network now — see issue #15. Its
|
||||
# unauthenticated API no longer needs to be LAN-reachable directly.
|
||||
# see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port)
|
||||
expose:
|
||||
- "8080"
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
# Dedicated backend for qwen-code's tool-call harmfulness classifier
|
||||
# (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
|
||||
# 27B connection — every classification call then queued behind whatever
|
||||
# heavy generation was already running on that model's 2 GPU slots (issue
|
||||
# tracker: OmniRoute semaphore/pr-agent investigation).
|
||||
#
|
||||
# Tried CPU-only first (own process avoids the GPU queue entirely) — too
|
||||
# slow in practice: real classification calls blew past OmniRoute's 60s
|
||||
# timeout and retry-looped (504→499→504...). Moved to GPU instead.
|
||||
#
|
||||
# Context sizing: qwen-code's classifier transcript is hard-capped in its
|
||||
# own source (MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000
|
||||
# per message, packages/core/src/permissions/classifier-transcript.ts) —
|
||||
# worst case is ~40-50K tokens, nowhere near the 131072 originally set in
|
||||
# settings.json (that number was copied from the main model's entry, not
|
||||
# a real qwen-code requirement). 65536 ctx gives ~1.5x margin over that
|
||||
# worst case. Qwen3-4B-Instruct-2507 is still the model choice — smallest
|
||||
# Qwen3 with long native context (262144) without RoPE-scaling, in case
|
||||
# that margin ever needs to grow.
|
||||
#
|
||||
# VRAM: weights+KV math (~4.9GiB) predicted comfortable headroom in the
|
||||
# ~6.1GiB free on the R9700, but measured live it actually used ~5.85GiB —
|
||||
# left only ~700MB free, too tight. Dropping --batch-size/--ubatch-size
|
||||
# barely moved it (~768MB free) — wrong lever. Actual cause: llama-server
|
||||
# runs with --flash-attn on but this service was missing it — without
|
||||
# flash attention the unfused attention compute buffer at 65536 ctx is
|
||||
# much larger (roughly O(n^2) intermediate buffers vs flash-attn's fused,
|
||||
# near-linear workspace), dwarfing the naive weights+KV estimate. Added
|
||||
# --flash-attn on to match llama-server; re-verify with rocm-smi after
|
||||
# deploy before trusting any of these numbers again. GPU_MAX_HW_QUEUES=1
|
||||
# carried over from llama-server's comment above — same ROCm/ROCm#5706
|
||||
# clock-pinning bug applies now that two HIP contexts (this +
|
||||
# llama-server) share the card.
|
||||
#
|
||||
# --reasoning off is a no-cost safety net, not a confirmed-needed fix:
|
||||
# ggml-org/llama.cpp#20809 (closed) documents some server builds
|
||||
# misdetecting Qwen3-Instruct-2507 models as thinking models, routing
|
||||
# tool-call output into reasoning_content instead of tool_calls — exactly
|
||||
# the failure mode that ruled out the 27B model for this role in the
|
||||
# first place. Whether the current image build still has it was never
|
||||
# independently confirmed (see docs/research/fast-model-choice.md §4/§6).
|
||||
# see docs/architecture.md#qwen-classifier
|
||||
qwen-classifier:
|
||||
image: ghcr.io/ggml-org/llama.cpp:server-rocm
|
||||
container_name: qwen-classifier
|
||||
@@ -101,6 +47,7 @@ services:
|
||||
security_opt:
|
||||
- seccomp=unconfined
|
||||
ipc: host
|
||||
# see docs/architecture.md#gpu-access
|
||||
environment:
|
||||
- GPU_MAX_HW_QUEUES=1
|
||||
volumes:
|
||||
@@ -124,16 +71,10 @@ services:
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
|
||||
# ponytail: one-off downloader, not a standing service — run via
|
||||
# `docker compose --profile tools run --rm downloader`. Folded into
|
||||
# scripts/update.sh, which runs this every time; the `test -f` guard is
|
||||
# what makes that safe to re-run without re-downloading. Keeps the model
|
||||
# files inside the named `models` volume instead of a host bind-mount.
|
||||
# see docs/operations.md#model-downloads (also covers downloader-classifier)
|
||||
downloader:
|
||||
image: curlimages/curl:latest
|
||||
profiles: ["tools"]
|
||||
# ponytail: named volume is created root-owned; curl_user (uid 100) can't
|
||||
# write into it otherwise, so run as root for this one-off job.
|
||||
user: root
|
||||
volumes:
|
||||
- models:/models
|
||||
@@ -145,8 +86,6 @@ services:
|
||||
curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
|
||||
|
||||
# Same pattern as downloader above, separate service so this one small
|
||||
# file doesn't get re-checked/re-pulled by the big model's job.
|
||||
downloader-classifier:
|
||||
image: curlimages/curl:latest
|
||||
profiles: ["tools"]
|
||||
@@ -161,11 +100,7 @@ services:
|
||||
curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||
https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||
|
||||
# Replaces litellm — see issue #31 (wayfinder map) for the full migration
|
||||
# rationale/findings. No static config.yaml equivalent: provider routing
|
||||
# (llama-server, searxng-search) is registered once through the dashboard
|
||||
# or POST /api/providers after first boot, not checked into this repo —
|
||||
# see docs/proxy-key-onboarding.md.
|
||||
# see docs/architecture.md#omniroute; runtime config lives in its dashboard, see docs/gateway.md
|
||||
omniroute:
|
||||
image: diegosouzapw/omniroute:latest
|
||||
container_name: omniroute
|
||||
@@ -176,64 +111,28 @@ services:
|
||||
- omniroute-data:/app/data
|
||||
env_file: .env
|
||||
environment:
|
||||
# Split-port mode: dashboard and API are fully separate ports (unlike
|
||||
# LiteLLM's single :4000 for both /v1 and /ui) — both published
|
||||
# directly below, unlike the old :4000-only host mapping.
|
||||
- API_HOST=0.0.0.0
|
||||
- API_PORT=${OMNIROUTE_API_PORT:-20129}
|
||||
- DASHBOARD_PORT=${OMNIROUTE_DASHBOARD_PORT:-20128}
|
||||
# Required to register llama-server/searxng-search as providers —
|
||||
# their base URLs are LAN/container-internal addresses, blocked by
|
||||
# default (SSRF guard against public-provider spoofing).
|
||||
# Security: relaxes OmniRoute's SSRF guard — see docs/gateway.md#provider-connections
|
||||
- OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true
|
||||
- OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true
|
||||
# Required (production) per docs/reference/ENVIRONMENT.md — shared
|
||||
# secret for the internal Codex Responses WebSocket bridge. Missed on
|
||||
# first pass; docker-compose config validated fine without it, but
|
||||
# the docs are explicit this one's required, not optional.
|
||||
- OMNIROUTE_WS_BRIDGE_SECRET=${OMNIROUTE_WS_BRIDGE_SECRET}
|
||||
# Default heap (1024MB) is dashboard-only sized per OmniRoute's own
|
||||
# Docker guide — every client here is a coding CLI, which needs the
|
||||
# larger figure the guide recommends. Paired with mem_limit below.
|
||||
- OMNIROUTE_MEMORY_MB=8192
|
||||
# Default 300000 (5 min) per OmniRoute's own docs, but this deployment
|
||||
# had it dialed down elsewhere (dashboard) to ~95s — too tight for a
|
||||
# contended local llama-server: large-context prefill under multiple
|
||||
# concurrent slots can outrun that before the first SSE token arrives,
|
||||
# so OmniRoute cancels a request that was actually still working (see
|
||||
# LLAMA_PARALLEL above for the other half of this fix). Raised here so
|
||||
# it's tracked in git instead of a dashboard-only setting.
|
||||
# see docs/gateway.md#timeouts (both timeouts below)
|
||||
- STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
|
||||
# Different timer than STREAM_IDLE_TIMEOUT_MS above — that one only
|
||||
# bounds gaps *between* SSE chunks once streaming has started.
|
||||
# REQUEST_TIMEOUT_MS bounds the wait for the *first* non-ping SSE
|
||||
# event, and it's what was still firing ("Stream produced no non-ping
|
||||
# SSE event within 95000ms") the morning after the timeout above was
|
||||
# raised — see docs/research/omniroute-non-ping-sse-stream-timeout.md.
|
||||
# Default 600000 (10 min) per OmniRoute's own docs, but the effective
|
||||
# deadline is remaining budget after retries/cooldowns eat into it, not
|
||||
# a flat timer, so raised well past the default for headroom.
|
||||
- REQUEST_TIMEOUT_MS=${OMNIROUTE_REQUEST_TIMEOUT_MS:-1800000}
|
||||
# Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
|
||||
# network can't resolve search.home on its own.
|
||||
# see docs/gateway.md#web-search
|
||||
extra_hosts:
|
||||
- "search.home:${SEARXNG_LAN_IP}"
|
||||
ports:
|
||||
- "${OMNIROUTE_API_PORT:-20129}:${OMNIROUTE_API_PORT:-20129}"
|
||||
- "${OMNIROUTE_DASHBOARD_PORT:-20128}:${OMNIROUTE_DASHBOARD_PORT:-20128}"
|
||||
# 10+ GiB ceiling per OmniRoute's Docker guide, matching
|
||||
# OMNIROUTE_MEMORY_MB=8192 above.
|
||||
mem_limit: 10g
|
||||
# SQLite WAL needs time to checkpoint back into the main DB file on
|
||||
# shutdown — the Docker guide's --stop-timeout 40 equivalent.
|
||||
# Don't shorten: SQLite needs it to checkpoint its WAL, or recent writes are lost — see docs/architecture.md#omniroute
|
||||
stop_grace_period: 40s
|
||||
restart: unless-stopped
|
||||
networks: [ai-stack]
|
||||
# ponytail: TCP-connect check, not an HTTP /healthz GET — the image has
|
||||
# no python3/curl/wget (confirmed live, `which` found only node), and
|
||||
# OmniRoute's own Docker guide already treats a bare TCP probe on this
|
||||
# port as an acceptable liveness check, not just the HTTP one. Simpler
|
||||
# and avoids depending on /healthz's exact path/response shape.
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
@@ -243,9 +142,7 @@ services:
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
# RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
|
||||
# directly, not gatewayed through omniroute (it isn't an
|
||||
# LLM provider).
|
||||
# see docs/architecture.md#qdrant, docs/network-access.md#qdrant
|
||||
qdrant:
|
||||
image: qdrant/qdrant:latest
|
||||
container_name: qdrant
|
||||
|
||||
@@ -23,7 +23,7 @@ The main model's backend. It is always running. Lazytainer used to idle-stop it,
|
||||
- **`LLAMA_GPU_LAYERS`**: the default puts every layer on the GPU. The model is dense, not Mixture-of-Experts, so this is the only RAM-offload knob. `--n-cpu-moe`, `--cpu-moe` and `--override-tensor "exps"` do nothing here. There is no SSD tier to enable either. llama.cpp mmaps the model file, so if GPU and RAM ever can't hold it, the OS pages the rest in from disk as a slow, implicit last resort. Explicit tiered offload is an [open llama.cpp request](https://github.com/ggml-org/llama.cpp/discussions/12507).
|
||||
- **`LLAMA_CTX_SIZE`**: set to the model's own maximum (`max_position_embeddings`). It only fits beside the weights because the KV cache is quantized (`--cache-type-k/v`). An fp16 KV cache at that size would not fit the card. The numbers are in the quant research doc.
|
||||
- **`LLAMA_PARALLEL`** is the backend's number of **slots**. It's kept low on purpose: more slots meant more contended prefill, which pushed requests past the gateway's timeouts ([gateway → Timeouts](gateway.md#timeouts)). Requests beyond the slot count wait in llama.cpp's own queue, which has no timeout. So the timeout that matters is the gateway's per-connection one.
|
||||
- **Per-slot context = `LLAMA_CTX_SIZE / LLAMA_PARALLEL`.** llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions have come close to it, so check that the per-slot figure stays above observed usage before lowering either value.
|
||||
- **Per-slot context = `LLAMA_CTX_SIZE / LLAMA_PARALLEL`.** llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions already fill a large share of it, so check that the per-slot figure stays comfortably above observed usage before lowering either value.
|
||||
- **`--cache-reuse`** reuses cached KV for any matching prompt chunk instead of re-prefilling from scratch. That fixes the root cause behind the gateway's non-ping SSE timeout ([`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md)). It pairs with OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot so that there is a matching prefix to reuse.
|
||||
- **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
|
||||
- **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).
|
||||
|
||||
+8
-66
@@ -1,28 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# The one command to run after any change to this repo (compose file, .env,
|
||||
# or a git pull) to bring the running stack in sync. Ensures secrets/keys
|
||||
# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
|
||||
# changed — safe to run any time, including with nothing to do.
|
||||
#
|
||||
# Tunable config values (LLAMA_*, ports, timeouts — anything with a real
|
||||
# default in .env.example) are synced from .env.example every run. A value
|
||||
# already matching is left alone silently. A value that DIFFERS from the
|
||||
# server's current .env is a conflict: interactively, you're shown every
|
||||
# conflict on one screen (via gum) and choose which to accept — unpicked
|
||||
# keys keep the server's current value. Non-interactively (no TTY — cron,
|
||||
# CI, piped), any conflict is a hard error unless --force is passed, which
|
||||
# accepts every new value automatically. Secrets and host-resolved values
|
||||
# (blank in .env.example — OMNIROUTE_*_SECRET/_KEY/_SALT/_PASSWORD,
|
||||
# SEARXNG_LAN_IP, HOST_VIDEO_GID/RENDER_GID) are never
|
||||
# touched by this — they keep going through set_if_blank as before.
|
||||
#
|
||||
# omniroute's own routing/provider config (llama-server, search) lives in
|
||||
# its dashboard, not a checked-in file like the old litellm-config.yaml —
|
||||
# see issue #31 and docs/proxy-key-onboarding.md.
|
||||
#
|
||||
# ponytail: no rollback/backup logic — this is a single-user homelab box,
|
||||
# not a fleet. If a bad config lands, `git revert` + re-run is the recovery
|
||||
# path, not this script.
|
||||
# Brings the running stack in sync with this repo; safe to re-run any time.
|
||||
# What it does and why: docs/operations.md
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
@@ -34,25 +12,14 @@ for arg in "$@"; do
|
||||
esac
|
||||
done
|
||||
|
||||
# Must run before anything else touches a file this script itself reads
|
||||
# (docker-compose.yml, .env.example, this script's own remaining lines) —
|
||||
# a self-updating script isn't guaranteed atomic against its own file
|
||||
# changing mid-run, so pulling later can execute a mix of old and new
|
||||
# script/compose content in one pass. Bit us for real: old GID-resolution
|
||||
# code ran, then this pulled in new var names docker-compose.yml now
|
||||
# requires, and nothing re-ran the (now-current) resolution step for
|
||||
# them — see issue #5's thread.
|
||||
# Keep this first — see docs/operations.md#self-update
|
||||
echo "==> git pull"
|
||||
git pull --ff-only
|
||||
|
||||
[ -f .env ] || cp .env.example .env
|
||||
|
||||
echo "==> syncing tracked config values from .env.example"
|
||||
# ponytail: gum (charmbracelet/gum) is a single static binary. Vendored as
|
||||
# a release tarball under scripts/vendor/ (checked into git) for the R9700
|
||||
# box, which has no outbound internet access — the download fallback below
|
||||
# is only for other archs / when the vendored copy is missing or stale.
|
||||
# Cached under .cache/gum/ (gitignored) so repeat runs don't re-extract.
|
||||
# see docs/operations.md#gum
|
||||
GUM_VERSION="0.14.5"
|
||||
GUM_DIR="$(pwd)/.cache/gum"
|
||||
GUM_BIN="$GUM_DIR/gum"
|
||||
@@ -84,10 +51,7 @@ ensure_gum() {
|
||||
[ -x "$GUM_BIN" ] && echo "$GUM_BIN" || echo ""
|
||||
}
|
||||
|
||||
# Collect every key where .env.example has a real (non-blank) default:
|
||||
# missing from .env -> just add it (no conflict, nothing to decide);
|
||||
# present and identical -> leave alone silently; present and different ->
|
||||
# a conflict to resolve below.
|
||||
# see docs/operations.md#env-sync
|
||||
conflict_keys=()
|
||||
conflict_old=()
|
||||
conflict_new=()
|
||||
@@ -128,8 +92,6 @@ if [ "${#conflict_keys[@]}" -gt 0 ]; then
|
||||
if [ -n "$gum_bin" ]; then
|
||||
selected="$(printf '%s\n' "${labels[@]}" | "$gum_bin" choose --no-limit --selected "$(printf '%s\n' "${labels[@]}" | paste -sd,)" --header "Config differs from .env.example — selected keys take the new value, unselected keep the server's current value:")"
|
||||
else
|
||||
# ponytail: plain-bash fallback if gum couldn't be fetched (offline,
|
||||
# unsupported arch) — same one-screen-of-conflicts idea, cruder UI.
|
||||
echo "Config differs from .env.example. Enter space-separated numbers to KEEP the server's current value (all others take the new value), or press enter to take every new value:"
|
||||
for i in "${!conflict_keys[@]}"; do
|
||||
echo " $((i+1))) ${labels[$i]}"
|
||||
@@ -155,9 +117,6 @@ if [ "${#conflict_keys[@]}" -gt 0 ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# Handles all three cases: the KEY=value line is missing entirely (.env
|
||||
# predates that var being added to .env.example — sed can't fix what isn't
|
||||
# there, so this appends it), present but blank, or already set.
|
||||
set_if_blank() {
|
||||
local key="$1" value="$2"
|
||||
if grep -qE "^${key}=.*[^[:space:]]" .env; then
|
||||
@@ -172,10 +131,7 @@ set_if_blank() {
|
||||
}
|
||||
|
||||
echo "==> filling in missing secrets"
|
||||
# Random values — safe to re-run, never overwrites what's already set.
|
||||
# OMNIROUTE_STORAGE_ENCRYPTION_KEY especially: never change it after first
|
||||
# run, existing encrypted data becomes unreadable if you do (same caveat as
|
||||
# LiteLLM's old LITELLM_SALT_KEY).
|
||||
# see docs/operations.md#secrets
|
||||
set_if_blank OMNIROUTE_INITIAL_PASSWORD "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_JWT_SECRET "$(openssl rand -base64 48)"
|
||||
set_if_blank OMNIROUTE_API_KEY_SECRET "$(openssl rand -hex 32)"
|
||||
@@ -185,9 +141,7 @@ set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
|
||||
set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
|
||||
|
||||
echo "==> resolving SEARXNG_LAN_IP"
|
||||
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
|
||||
# not from inside the omniroute container (see docs/research/litellm-searxng-search.md,
|
||||
# still the relevant background even though omniroute replaced litellm — see issue #31).
|
||||
# see docs/gateway.md#web-search
|
||||
searxng_ip="$(getent hosts search.home 2>/dev/null | awk '{print $1}' | head -1)"
|
||||
if [ -n "$searxng_ip" ]; then
|
||||
set_if_blank SEARXNG_LAN_IP "$searxng_ip"
|
||||
@@ -196,11 +150,7 @@ else
|
||||
fi
|
||||
|
||||
echo "==> resolving host video/render GIDs (shared by every GPU service)"
|
||||
# Numeric GIDs, not names, in docker-compose.yml's group_add: — Docker
|
||||
# resolves a *named* group_add entry against the container's own /etc/group,
|
||||
# not the host's, and fails unpredictably (worse with multiple GPU services
|
||||
# starting concurrently and racing on the same lookup) — see
|
||||
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5.
|
||||
# see docs/architecture.md#gpu-access
|
||||
video_gid="$(getent group video 2>/dev/null | cut -d: -f3)"
|
||||
render_gid="$(getent group render 2>/dev/null | cut -d: -f3)"
|
||||
if [ -n "$video_gid" ]; then
|
||||
@@ -230,14 +180,6 @@ docker compose --profile tools run --rm downloader-classifier
|
||||
echo "==> bringing up omniroute"
|
||||
docker compose up -d --wait omniroute
|
||||
|
||||
# ponytail: no scripted key-minting yet, unlike the old LiteLLM /key/generate
|
||||
# flow — omniroute's POST /api/keys needs a dashboard login session
|
||||
# (ManagementSessionAuth), not a static bearer key, and that flow hasn't
|
||||
# been verified against a live instance (see issue #37). No in-stack
|
||||
# workload needs a key right now (nothing left calls the gateway besides
|
||||
# coding CLIs, which mint their own by hand per docs/proxy-key-onboarding.md)
|
||||
# — revisit this script once that flow is automatable.
|
||||
|
||||
echo "==> recreating changed services"
|
||||
docker compose up -d --remove-orphans
|
||||
|
||||
|
||||
Reference in new issue
Block a user