haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Docs cleanup: restructure, move code comments into docs, remove image generation #65

Merged
haylan merged 5 commits from docs/cleanup-and-restructure into main 2026-09-24 20:01:31 +00:00
Conversation 0 Commits 5 Files Changed 23
+406 -1119
23 changed files with 406 additions and 1119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.env.example
+11 -115
View File
@@ -1,139 +1,35 @@
# Copy to .env and adjust — or just run ./scripts/update.sh, which creates
# .env from this file and fills in every secret/key below it can generate
# itself (see each var's comment). All values below are defaults baked into
# docker-compose.yml — only uncomment/change what you actually want to
# override.
# Template for .env, managed by ./scripts/update.sh — see docs/operations.md#env-sync
# --- llama.cpp / model ---
# --- llama.cpp / models --- see docs/architecture.md#llama-server
LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
# 999 = every layer on GPU (this model is dense, not MoE, and already fits
# fully in 32GB VRAM — see docs/research/qwen3.8-27b-quant.md). Lower this
# to leave that many fewer layers on GPU and push the rest to CPU/system RAM
# if something else is contending for VRAM — llama.cpp has no separate
# "RAM offload" flag, --n-gpu-layers *is* the RAM-offload knob for a dense
# model. Don't reach for --n-cpu-moe/--cpu-moe/--override-tensor "exps" —
# those target Mixture-of-Experts models (e.g. Qwen3.8-2.4T-A95B), not this
# one, and are no-ops here.
# There's no separate "then SSD" tier to enable either: llama.cpp mmaps the
# model file by default (no --no-mmap here), so if GPU+RAM ever can't hold
# the working set, the OS pages the rest in from disk automatically — an
# implicit, slow last resort, not a config knob. An explicit tiered SSD
# offload has been an open llama.cpp feature request since 2025 (still
# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
LLAMA_GPU_LAYERS=999
# 262144 = this model's true max (max_position_embeddings in Qwen/Qwen3.8-27B's
# config.json) — the largest --ctx-size llama.cpp will even accept for it.
# fp16 KV cache at full context would be ~16GB, on top of 17.6GB weights =
# ~33.6GB, which does NOT fit the 32GB R9700 on its own. docker-compose.yml
# now runs --cache-type-k/v q8_0, which roughly halves KV memory (~8GB at
# this size) — total ~25.6GB, ~6GB headroom, the same footprint the old
# 131072 fp16 setting used. See docs/research/qwen3.8-27b-quant.md.
LLAMA_CTX_SIZE=262144
# Concurrent request slots — the real hardware ceiling for this GPU, not a
# tunable to raise for throughput (was implicitly 4, llama.cpp's compiled-in
# default; dropped to 2 because more contended prefill was blowing requests
# past OmniRoute's idle timeout — see OMNIROUTE_STREAM_IDLE_TIMEOUT_MS below).
# The 3rd+ request now queues on llama.cpp itself instead — its own queue has
# no timeout (tools/server/server-queue.cpp), it just waits for a slot — so
# the timeout that matters moved to OmniRoute's per-connection
# providerSpecificData.timeoutMs (dashboard/API only, not in this file; see
# handoff notes in the issue tracker). Each slot gets LLAMA_CTX_SIZE /
# LLAMA_PARALLEL tokens of context — real sessions have hit ~66K tokens, so
# don't drop LLAMA_CTX_SIZE without checking that per-slot number stays
# comfortably above observed usage.
LLAMA_PARALLEL=2
# Dedicated GPU-resident backend for qwen-code's tool-call harmfulness
# classifier (fastModel in ~/.qwen/settings.json) — see docker-compose.yml's
# qwen-classifier service comment for the why and the VRAM/context math.
# see docs/architecture.md#qwen-classifier
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
# --- Lazytainer ---
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
LAZYTAINER_INACTIVE_TIMEOUT=900
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
# Resolved automatically by ./scripts/update.sh from search.home on this
# host — leave blank. Only set by hand if that resolution fails (e.g.
# search.home isn't a static DHCP reservation and its IP drifted).
# --- SearXNG web search --- see docs/gateway.md#web-search
# Auto-filled by update.sh — leave blank.
SEARXNG_LAN_IP=
# --- OmniRoute gateway (see docs/proxy-key-onboarding.md, docs/network-access.md) ---
# OMNIROUTE_PORT is the host-published port (reverse-proxied by NPM) — kept
# at 4000, same as the old LiteLLM setup, so existing NPM/firewall config
# doesn't need to change. It's mapped via plain Docker port publishing onto
# API_PORT, omniroute's own container-internal port (left at its default,
# not reconfigured to match). The dashboard (DASHBOARD_PORT) is never
# published at all — see docker-compose.yml's omniroute service comment.
# --- OmniRoute gateway --- see docs/network-access.md#gateway
OMNIROUTE_API_PORT=20129
OMNIROUTE_DASHBOARD_PORT=20128
# SSE inactivity timeout before OmniRoute gives up on a streaming request and
# cancels it (which cancels the matching llama-server task too). 180s gives
# contended prefill (see LLAMA_PARALLEL above) room to produce a first token.
# see docs/gateway.md#timeouts
OMNIROUTE_STREAM_IDLE_TIMEOUT_MS=180000
# Wait budget for the *first* SSE token specifically (distinct from the
# inter-chunk timeout above) — see
# docs/research/omniroute-non-ping-sse-stream-timeout.md. 30 min covers a
# contended, large-context prefill even after retries eat into the budget.
OMNIROUTE_REQUEST_TIMEOUT_MS=1800000
# Random values, filled in automatically by ./scripts/update.sh — leave
# blank. Bootstrap dashboard admin password (log in at the dashboard port,
# change it there afterwards — this is only the first-boot value):
# Secrets: auto-filled by update.sh — leave blank. See docs/operations.md#secrets
OMNIROUTE_INITIAL_PASSWORD=
# Signs dashboard session cookies:
OMNIROUTE_JWT_SECRET=
# Encrypts API key values at rest in omniroute's SQLite DB:
OMNIROUTE_API_KEY_SECRET=
# Encrypts the whole SQLite DB at rest. Do not change after first run —
# existing encrypted data becomes unreadable if you do (same caveat as
# LiteLLM's old LITELLM_SALT_KEY):
# NEVER change after first run — OmniRoute's existing data becomes unreadable.
OMNIROUTE_STORAGE_ENCRYPTION_KEY=
# Per-deployment salts — random is fine, just needs to be stable:
OMNIROUTE_MACHINE_ID_SALT=
OMNIROUTE_CLI_SALT=
# Required (production) — shared secret for the internal Codex Responses
# WebSocket bridge. Random value, filled in automatically:
OMNIROUTE_WS_BRIDGE_SECRET=
# Per-workload virtual keys (one per client that calls the gateway) have no
# scripted /key/generate equivalent yet — omniroute's key-creation endpoint
# needs a dashboard login session, not a static bearer key (see issue #37).
# Mint them by hand in the dashboard, add a KEY=value line here per workload
# as you onboard one. See docs/proxy-key-onboarding.md.
# --- ComfyUI (local image generation, see issue #38 wayfinder map) ---
# Shared by every GPU-touching service (llama-server, llama-server-fast,
# comfyui) for group_add: — resolved to real host GIDs by ./scripts/update.sh
# rather than left as plain group names in docker-compose.yml, because Docker
# resolves a *named* group_add entry against the container's own /etc/group,
# not the host's, and fails unpredictably when the image doesn't define one
# (worse with multiple GPU services racing on the same lookup at once — see
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
# --- GPU access --- see docs/architecture.md#gpu-access
# Auto-filled by update.sh — leave blank.
HOST_VIDEO_GID=
HOST_RENDER_GID=
# --- ComfyUI diffusion model (Qwen-Image 2.1, uncensored GGUF — see
# docs/research/image-generation-model-choice.md and issue #42) ---
# Three files, all from abenzerps/Qwen-Image-2.1-Uncensored-GGUF on
# HuggingFace: GGUF diffusion weights, safetensors text encoder, safetensors
# VAE. Q4_K_M is the repo's own recommended size/quality balance (~4.6GB);
# the int8 text encoder trades some quality for ~8GB less RAM than bf16.
#
# Requires the ComfyUI-GGUF custom node (github.com/city96/ComfyUI-GGUF) —
# NOT confirmed bundled in yurisasc/comfyui-rocm7.1. If the comfyui
# container doesn't show a "Unet Loader (GGUF)" node, `git clone` it into
# the image's custom_nodes dir and reinstall its requirements.
COMFYUI_DIFFUSION_MODEL_FILE=qwen-image-2.1-UC-Q4_K_M.gguf
COMFYUI_TEXT_ENCODER_FILE=qwen3vl_8b_int8_convrot.safetensors
COMFYUI_VAE_FILE=qwen_image_2.1_vae_bf16.safetensors
# --- Open WebUI (chat + ComfyUI image-gen frontend) ---
# Chat backend goes through omniroute like every other client — mint this
# key by hand in the omniroute dashboard per docs/proxy-key-onboarding.md
# (label it openwebui-chat), no scripted mint yet. Image generation talks
# to comfyui directly, no key needed there.
OPENWEBUI_OMNIROUTE_KEY=
# --- RAG databases (qdrant + neo4j, see wayfinder notes) ---
# No auth on qdrant (its default) — same trust boundary as llama-server:
# ai-stack is not exposed off-box. Random, filled in automatically:
NEO4J_PASSWORD=
CLAUDE.md
+4
View File
@@ -8,6 +8,10 @@ Issues live as Gitea issues on `git.arthurerlich.de` (repo `haylan/LLM-Server`);
Single-context: `CONTEXT.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
### Writing docs
Before editing `README.md`, `docs/`, or a comment in `docker-compose.yml` / `.env.example` / `scripts/`, read `docs/agents/writing-docs.md`.
### Deploying changes
The running stack lives on a separate box (the R9700 server), not wherever this repo is being edited. After **any** change to `docker-compose.yml`, `.env.example`, or a script under `scripts/`, commit/push it, then run `./scripts/update.sh` on the server to apply it — don't just describe the change as done. If this session doesn't have shell access to the server, say so explicitly and tell the user to run it themselves rather than leaving it unsaid.
CONTEXT.md
+59
View File
@@ -0,0 +1,59 @@
# LLM-Server
A self-hosted inference stack: local models served on one GPU machine, reached by other tools through a single authenticated entry point. This file is a glossary only — for how things are configured, follow the links in the docs to the source.
## Language
### Machine
**Server**:
The one machine that runs the stack.
_Avoid_: AI box, R9700 box, this machine
**Host**:
The Docker host, as opposed to a container — only used for that distinction.
_Avoid_: using it as a synonym for Server
### Serving models
**Backend**:
A model server process that serves one model to the gateway (`llama-server`, `qwen-classifier`).
_Avoid_: provider, model server
**Slot**:
One concurrent request lane inside a backend; a backend handles as many requests at once as it has slots.
_Avoid_: parallel request, lane
**Main model**:
The large model that does the actual work for workloads.
_Avoid_: chat model
**Classifier model**:
The small model that judges a coding CLI's tool calls before they run.
_Avoid_: fast model (`fastModel` only when quoting qwen-code's config key)
### Reaching the models
**Gateway**:
The single entry point every model call goes through (OmniRoute).
_Avoid_: proxy, AI proxy
**Provider connection**:
The gateway's registration of one backend.
_Avoid_: provider (alone), connection (alone)
**API key**:
A credential issued by the gateway to exactly one workload.
_Avoid_: virtual key, workload key
**Workload**:
Anything that calls the gateway with its own API key.
_Avoid_: client, consumer
**Coding CLI**:
A workload that is an interactive coding agent (Claude Code, Kimi CLI, OpenCode, Qwen Code).
_Avoid_: harness
**Reverse proxy**:
The existing Nginx Proxy Manager instance that publishes the gateway under hostnames; not part of this stack.
_Avoid_: proxy (alone)
README.md
+17 -23
View File
@@ -1,35 +1,29 @@
# LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle.
> **Source of truth:** [`docker-compose.yml`](docker-compose.yml) (what runs) and [`scripts/update.sh`](scripts/update.sh) (how it's deployed). The docs explain *why*; if they disagree with the source, the source wins.
See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
A self-hosted inference stack on one AMD GPU server. Two llama.cpp **backends**, the main model and the classifier model, sit behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches them through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
## Quickstart
On the server:
```bash
./scripts/update.sh
```
`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
It creates and syncs `.env`, generates secrets, downloads the models, and brings the stack up. It's safe to re-run at any time. What it does and why: [`docs/operations.md`](docs/operations.md).
llama.cpp's own API is internal-only — everything routes through the AI gateway below.
## Docs
Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
## AI gateway (OmniRoute)
An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload API keys and usage tracking. As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) this is [OmniRoute](https://github.com/diegosouzapw/OmniRoute), replacing the original LiteLLM setup. `./scripts/update.sh` handles most of OmniRoute's secrets (see `.env.example`); per-workload API keys still need minting by hand in the dashboard — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
- Gateway API: `http://<this-machine>:${OMNIROUTE_PORT:-4000}/v1` locally, or `proxy-ai.home` / `proxy-ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
### Web search
The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
| Doc | What it covers |
|---|---|
| [`CONTEXT.md`](CONTEXT.md) | Glossary: the terms every doc uses |
| [`docs/architecture.md`](docs/architecture.md) | Each service, why it exists, and why it's configured the way it is |
| [`docs/gateway.md`](docs/gateway.md) | OmniRoute: dashboard, provider connections, API keys, timeouts, web search |
| [`docs/network-access.md`](docs/network-access.md) | Who can reach which port, and why |
| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads |
| [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md) | Pointing Claude Code, Kimi CLI, OpenCode and Qwen Code at the gateway |
| [`docs/research/`](docs/research/README.md) | Dated research and decision records, each marked current or superseded |
| [`docs/server-planing.md`](docs/server-planing.md) | Background research: a 4× GPU rack build. Not an active plan |
| [`docs/agents/`](docs/agents/) | Instructions for coding agents: [issue tracker](docs/agents/issue-tracker.md), [domain docs](docs/agents/domain.md), [writing docs](docs/agents/writing-docs.md) |
docker-compose.yml
+12 -298
View File
@@ -5,17 +5,13 @@ services:
devices:
- /dev/kfd
- /dev/dri
# Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
# .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
# see docs/architecture.md#gpu-access (group_add GIDs, GPU_MAX_HW_QUEUES)
group_add:
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
security_opt:
- seccomp=unconfined
ipc: host
# Caps this process's HIP hardware-queue allocation — works around
# ROCm/ROCm#5706 (GPU pinned at 100%/boost-clock whenever two
# concurrent HIP contexts touch this card). See the research doc above.
environment:
- GPU_MAX_HW_QUEUES=1
volumes:
@@ -32,70 +28,13 @@ services:
--cache-type-v q8_0
--cache-reuse 256
--jinja
# --cache-reuse 256: reuse cached KV for any matching prompt chunk of at
# least 256 tokens (KV-shift, no reprocessing) instead of reprefilling
# from scratch every request. Directly targets the actual root cause
# behind the OmniRoute non-ping SSE timeout, not just the symptom — see
# docs/research/omniroute-non-ping-sse-stream-timeout.md. Pairs with
# OmniRoute's promptCacheAffinityEnabled (dashboard default), which keeps
# a conversation's requests pinned to the same slot so there's a matching
# prefix to reuse.
# No published host port: llama-server is reached only via the omniroute
# gateway on the ai-stack docker network now — see issue #15. Its
# unauthenticated API no longer needs to be LAN-reachable directly.
# see docs/architecture.md#llama-server (flags), docs/network-access.md#backends (no host port)
expose:
- "8080"
restart: unless-stopped
networks: [ai-stack]
labels:
# ponytail: idle-timeout tuning lives here, not in a separate lazytainer config file —
# one place to look. Raise LAZYTAINER_INACTIVE_TIMEOUT if 15 min proves too eager.
- "lazytainer.group.llamaserver.sleepMethod=stop"
- "lazytainer.group.llamaserver.ports=8080"
- "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
- "lazytainer.group.llamaserver.minPacketThreshold=2"
# Dedicated backend for qwen-code's tool-call harmfulness classifier
# (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
# 27B connection — every classification call then queued behind whatever
# heavy generation was already running on that model's 2 GPU slots (issue
# tracker: OmniRoute semaphore/pr-agent investigation).
#
# Tried CPU-only first (own process avoids the GPU queue entirely) — too
# slow in practice: real classification calls blew past OmniRoute's 60s
# timeout and retry-looped (504→499→504...). Moved to GPU instead.
#
# Context sizing: qwen-code's classifier transcript is hard-capped in its
# own source (MAX_TRANSCRIPT_MESSAGES=40, MAX_HISTORICAL_ACTION_CHARS=4000
# per message, packages/core/src/permissions/classifier-transcript.ts) —
# worst case is ~40-50K tokens, nowhere near the 131072 originally set in
# settings.json (that number was copied from the main model's entry, not
# a real qwen-code requirement). 65536 ctx gives ~1.5x margin over that
# worst case. Qwen3-4B-Instruct-2507 is still the model choice — smallest
# Qwen3 with long native context (262144) without RoPE-scaling, in case
# that margin ever needs to grow.
#
# VRAM: weights+KV math (~4.9GiB) predicted comfortable headroom in the
# ~6.1GiB free on the R9700, but measured live it actually used ~5.85GiB —
# left only ~700MB free, too tight. Dropping --batch-size/--ubatch-size
# barely moved it (~768MB free) — wrong lever. Actual cause: llama-server
# runs with --flash-attn on but this service was missing it — without
# flash attention the unfused attention compute buffer at 65536 ctx is
# much larger (roughly O(n^2) intermediate buffers vs flash-attn's fused,
# near-linear workspace), dwarfing the naive weights+KV estimate. Added
# --flash-attn on to match llama-server; re-verify with rocm-smi after
# deploy before trusting any of these numbers again. GPU_MAX_HW_QUEUES=1
# carried over from llama-server's comment above — same ROCm/ROCm#5706
# clock-pinning bug applies now that two HIP contexts (this +
# llama-server) share the card.
#
# --reasoning off is a no-cost safety net, not a confirmed-needed fix:
# ggml-org/llama.cpp#20809 (closed) documents some server builds
# misdetecting Qwen3-Instruct-2507 models as thinking models, routing
# tool-call output into reasoning_content instead of tool_calls — exactly
# the failure mode that ruled out the 27B model for this role in the
# first place. Whether the current image build still has it was never
# independently confirmed (see docs/research/fast-model-choice.md §4/§6).
# see docs/architecture.md#qwen-classifier
qwen-classifier:
image: ghcr.io/ggml-org/llama.cpp:server-rocm
container_name: qwen-classifier
@@ -108,6 +47,7 @@ services:
security_opt:
- seccomp=unconfined
ipc: host
# see docs/architecture.md#gpu-access
environment:
- GPU_MAX_HW_QUEUES=1
volumes:
@@ -131,16 +71,10 @@ services:
restart: unless-stopped
networks: [ai-stack]
# ponytail: one-off downloader, not a standing service — run via
# `docker compose --profile tools run --rm downloader`. Folded into
# scripts/update.sh, which runs this every time; the `test -f` guard is
# what makes that safe to re-run without re-downloading. Keeps the model
# files inside the named `models` volume instead of a host bind-mount.
# see docs/operations.md#model-downloads (also covers downloader-classifier)
downloader:
image: curlimages/curl:latest
profiles: ["tools"]
# ponytail: named volume is created root-owned; curl_user (uid 100) can't
# write into it otherwise, so run as root for this one-off job.
user: root
volumes:
- models:/models
@@ -152,8 +86,6 @@ services:
curl -L --fail --create-dirs -o /models/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${LLAMA_MODEL_FILE:-Qwen3.8-27B-UD-Q4_K_XL.gguf}
# Same pattern as downloader above, separate service so this one small
# file doesn't get re-checked/re-pulled by the big model's job.
downloader-classifier:
image: curlimages/curl:latest
profiles: ["tools"]
@@ -168,110 +100,7 @@ services:
curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
# Fetches the three Qwen-Image 2.1 (uncensored GGUF) files ComfyUI needs
# (diffusion model, text encoder, VAE) — same test -f guard pattern as
# downloader above. See docs/research/image-generation-model-choice.md and
# issue #42.
#
# GGUF weights go in models/unet (ComfyUI-GGUF's own convention), not
# models/diffusion_models (that was the old FP8-safetensors layout) — see
# github.com/city96/ComfyUI-GGUF. Text encoder/VAE stay safetensors, loaded
# by ComfyUI's normal CLIPLoader/VAELoader, so their folders are unchanged.
#
# ponytail: target paths otherwise assume ComfyUI's standard models/ layout
# under BASE_STORAGE_PATH (/storage) — same "not independently confirmed
# against the image's Dockerfile" caveat flagged on the comfyui service
# below. If ComfyUI doesn't pick these up, check its actual models root
# first.
downloader-comfyui:
image: curlimages/curl:latest
profiles: ["tools"]
user: root
volumes:
- comfyui-data:/storage
entrypoint: ["sh", "-c"]
command:
- >
mkdir -p /storage/models/unet /storage/models/text_encoders /storage/models/vae &&
(test -f /storage/models/unet/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen-image-2.1-UC-Q4_K_M.gguf} &&
echo "diffusion model already downloaded, skipping" ||
curl -L --fail --create-dirs -o /storage/models/unet/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen-image-2.1-UC-Q4_K_M.gguf}
https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF/resolve/main/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen-image-2.1-UC-Q4_K_M.gguf}) &&
(test -f /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen3vl_8b_int8_convrot.safetensors} &&
echo "text encoder already downloaded, skipping" ||
curl -L --fail --create-dirs -o /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen3vl_8b_int8_convrot.safetensors}
https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF/resolve/main/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen3vl_8b_int8_convrot.safetensors}) &&
(test -f /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_2.1_vae_bf16.safetensors} &&
echo "vae already downloaded, skipping" ||
curl -L --fail --create-dirs -o /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_2.1_vae_bf16.safetensors}
https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF/resolve/main/vae/${COMFYUI_VAE_FILE:-qwen_image_2.1_vae_bf16.safetensors})
# Local image generation — see issue #38 (wayfinder map). yurisasc's image
# is gfx1201-tuned specifically (R9700's arch), unlike the official/AMD
# ComfyUI image which doesn't pin RDNA4 support — see
# docs/research/image-generation-options.md.
#
# Model is Qwen-Image 2.1 GGUF (uncensored variant, see .env.example) —
# needs the ComfyUI-GGUF custom node (github.com/city96/ComfyUI-GGUF) for
# its "Unet Loader (GGUF)" node. Not confirmed pre-installed in this image;
# if missing, exec in and `git clone` it into custom_nodes, then pip
# install its requirements.txt and restart the container.
comfyui:
image: yurisasc/comfyui-rocm7.1:latest
container_name: comfyui
devices:
- /dev/kfd
- /dev/dri
# Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
# .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
group_add:
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
security_opt:
- seccomp=unconfined
ipc: host
environment:
- HSA_OVERRIDE_GFX_VERSION=12.0.1
- PYTORCH_ROCM_ARCH=gfx1201
volumes:
- comfyui-data:/storage
# yurisasc/comfyui-rocm7.1 has no entrypoint/CMD of its own (just drops
# into bash) and doesn't read PUID/PGID/VIDEO_GID/RENDER_GID or
# BASE_STORAGE_PATH — confirmed by inspecting the image directly, so
# this command is required, not optional, or the container exits 0
# immediately and restart-loops. Runs as root (image's only user);
# group_add above is what actually grants /dev/kfd + /dev/dri access.
#
# --base-directory fully replaces ComfyUI's custom_nodes/input/output/
# user/temp dirs with ones under /storage (confirmed in folder_paths.py)
# rather than adding to the image's baked-in ones — so first boot must
# seed /storage/custom_nodes from the image's copy (ComfyUI-Manager
# lives there) or it's gone, and main.py hard-crashes if custom_nodes
# doesn't exist yet at all.
entrypoint: ["sh", "-c"]
command:
- >
mkdir -p /storage/custom_nodes /storage/input /storage/output /storage/user &&
[ -z "$(ls -A /storage/custom_nodes)" ] && cp -r /workspace/ComfyUI/custom_nodes/. /storage/custom_nodes/ || true;
exec python main.py --listen 0.0.0.0 --port 8188 --base-directory /storage
#
# Published host port (unlike llama-server's ai-stack-only pattern):
# ComfyUI's own UI is meant to be reachable directly too, for a planned
# external nginx reverse-proxy route to comfy.home — not just through
# OmniRoute. Still also reachable at http://comfyui:8188 internally on
# ai-stack, which is the URL to register as OmniRoute's comfyui
# provider (dashboard or POST /api/providers, per docs/proxy-key-onboarding.md
# — same undocumented-in-repo manual flow already used for llama-server).
ports:
- "8138:8188"
restart: unless-stopped
networks: [ai-stack]
# Replaces litellm — see issue #31 (wayfinder map) for the full migration
# rationale/findings. No static config.yaml equivalent: provider routing
# (llama-server, searxng-search) is registered once through the dashboard
# or POST /api/providers after first boot, not checked into this repo —
# see docs/proxy-key-onboarding.md.
# see docs/architecture.md#omniroute; runtime config lives in its dashboard, see docs/gateway.md
omniroute:
image: diegosouzapw/omniroute:latest
container_name: omniroute
@@ -282,64 +111,28 @@ services:
- omniroute-data:/app/data
env_file: .env
environment:
# Split-port mode: dashboard and API are fully separate ports (unlike
# LiteLLM's single :4000 for both /v1 and /ui) — both published
# directly below, unlike the old :4000-only host mapping.
- API_HOST=0.0.0.0
- API_PORT=${OMNIROUTE_API_PORT:-20129}
- DASHBOARD_PORT=${OMNIROUTE_DASHBOARD_PORT:-20128}
# Required to register llama-server/searxng-search as providers —
# their base URLs are LAN/container-internal addresses, blocked by
# default (SSRF guard against public-provider spoofing).
# Security: relaxes OmniRoute's SSRF guard — see docs/gateway.md#provider-connections
- OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS=true
- OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS=true
# Required (production) per docs/reference/ENVIRONMENT.md — shared
# secret for the internal Codex Responses WebSocket bridge. Missed on
# first pass; docker-compose config validated fine without it, but
# the docs are explicit this one's required, not optional.
- OMNIROUTE_WS_BRIDGE_SECRET=${OMNIROUTE_WS_BRIDGE_SECRET}
# Default heap (1024MB) is dashboard-only sized per OmniRoute's own
# Docker guide — every client here is a coding CLI, which needs the
# larger figure the guide recommends. Paired with mem_limit below.
- OMNIROUTE_MEMORY_MB=8192
# Default 300000 (5 min) per OmniRoute's own docs, but this deployment
# had it dialed down elsewhere (dashboard) to ~95s — too tight for a
# contended local llama-server: large-context prefill under multiple
# concurrent slots can outrun that before the first SSE token arrives,
# so OmniRoute cancels a request that was actually still working (see
# LLAMA_PARALLEL above for the other half of this fix). Raised here so
# it's tracked in git instead of a dashboard-only setting.
# see docs/gateway.md#timeouts (both timeouts below)
- STREAM_IDLE_TIMEOUT_MS=${OMNIROUTE_STREAM_IDLE_TIMEOUT_MS:-180000}
# Different timer than STREAM_IDLE_TIMEOUT_MS above — that one only
# bounds gaps *between* SSE chunks once streaming has started.
# REQUEST_TIMEOUT_MS bounds the wait for the *first* non-ping SSE
# event, and it's what was still firing ("Stream produced no non-ping
# SSE event within 95000ms") the morning after the timeout above was
# raised — see docs/research/omniroute-non-ping-sse-stream-timeout.md.
# Default 600000 (10 min) per OmniRoute's own docs, but the effective
# deadline is remaining budget after retries/cooldowns eat into it, not
# a flat timer, so raised well past the default for headroom.
- REQUEST_TIMEOUT_MS=${OMNIROUTE_REQUEST_TIMEOUT_MS:-1800000}
# Same reasoning as litellm's extra_hosts entry below — ai-stack's bridge
# network can't resolve search.home on its own.
# see docs/gateway.md#web-search
extra_hosts:
- "search.home:${SEARXNG_LAN_IP}"
ports:
- "${OMNIROUTE_API_PORT:-20129}:${OMNIROUTE_API_PORT:-20129}"
- "${OMNIROUTE_DASHBOARD_PORT:-20128}:${OMNIROUTE_DASHBOARD_PORT:-20128}"
# 10+ GiB ceiling per OmniRoute's Docker guide, matching
# OMNIROUTE_MEMORY_MB=8192 above.
mem_limit: 10g
# SQLite WAL needs time to checkpoint back into the main DB file on
# shutdown — the Docker guide's --stop-timeout 40 equivalent.
# Don't shorten: SQLite needs it to checkpoint its WAL, or recent writes are lost — see docs/architecture.md#omniroute
stop_grace_period: 40s
restart: unless-stopped
networks: [ai-stack]
# ponytail: TCP-connect check, not an HTTP /healthz GET — the image has
# no python3/curl/wget (confirmed live, `which` found only node), and
# OmniRoute's own Docker guide already treats a bare TCP probe on this
# port as an acceptable liveness check, not just the HTTP one. Simpler
# and avoids depending on /healthz's exact path/response shape.
healthcheck:
test:
- CMD-SHELL
@@ -349,90 +142,14 @@ services:
retries: 3
start_period: 40s
lazytainer:
image: ghcr.io/vmorganp/lazytainer:master
container_name: lazytainer
# NOT network_mode: host — lazytainer identifies its own container by
# matching os.Hostname() against the Docker container-ID list
# (vmorganp/Lazytainer, configureFromLabels()); under host networking the
# container inherits the host's hostname instead of its own ID, so that
# match always fails and it panics with "Could not determine container ID
# of lazytainer" on every start. Host networking also can't see traffic
# to llama-server:8080 anyway — that port only exists on the ai-stack
# bridge network (no host port published, see issue #15 above). Joining
# ai-stack instead fixes both: hostname becomes the real container ID,
# and it's on the same network as the traffic it's watching.
networks: [ai-stack]
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
restart: unless-stopped
depends_on:
- llama-server
# RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
# directly like comfyui above, not gatewayed through omniroute (it isn't an
# LLM provider).
# see docs/architecture.md#qdrant, docs/network-access.md#qdrant
qdrant:
image: qdrant/qdrant:latest
container_name: qdrant
volumes:
- qdrant-data:/qdrant/storage
ports:
- "6333:6333"
restart: unless-stopped
networks: [ai-stack]
# RAG graph store, native vector index too (can absorb qdrant's job later
# if the two-DB split proves unnecessary — see wayfinder notes).
neo4j:
image: neo4j:5-community
container_name: neo4j
environment:
- NEO4J_AUTH=neo4j/${NEO4J_PASSWORD:?run scripts/update.sh first to resolve this}
volumes:
- neo4j-data:/data
ports:
- "7474:7474" # browser UI
- "7687:7687" # bolt
restart: unless-stopped
networks: [ai-stack]
# Chat + image-generation frontend. Talks to ComfyUI directly for images —
# NOT through omniroute: OmniRoute's built-in `comfyui` provider hardcodes
# a CheckpointLoaderSimple (single-file-checkpoint) workflow in
# open-sse/handlers/imageGeneration/providers/comfyUI.ts with no override,
# so it can't drive Qwen-Image 2.1's split unet/text-encoder/vae GGUF
# loader chain. Open WebUI's ComfyUI backend is fully generic instead —
# you upload the exact workflow_api.json exported from ComfyUI's own
# graph editor and map arbitrary node IDs to parameters.
#
# Chat backend still goes through omniroute (OpenAI-compatible), same
# gateway every other client uses — mint OPENWEBUI_OMNIROUTE_KEY by hand
# per docs/proxy-key-onboarding.md, same as any other workload.
#
# ponytail: only wires the base ComfyUI URL + chat backend via env vars.
# COMFYUI_WORKFLOW/COMFYUI_WORKFLOW_NODES env vars exist but are reported
# flaky upstream — set the actual workflow JSON by hand in Settings >
# Images in the web UI instead, once you've exported it from ComfyUI.
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
depends_on:
comfyui:
condition: service_started
omniroute:
condition: service_started
environment:
- OPENAI_API_BASE_URL=http://omniroute:${OMNIROUTE_API_PORT:-20129}/v1
- OPENAI_API_KEY=${OPENWEBUI_OMNIROUTE_KEY:-}
- ENABLE_IMAGE_GENERATION=True
- IMAGE_GENERATION_ENGINE=comfyui
- COMFYUI_BASE_URL=http://comfyui:8188
- WEBUI_AUTH=True
volumes:
- open-webui-data:/app/backend/data
ports:
- "8008:8080"
- "127.0.0.1:6333:6333"
restart: unless-stopped
networks: [ai-stack]
@@ -442,7 +159,4 @@ networks:
volumes:
models:
omniroute-data:
comfyui-data:
qdrant-data:
neo4j-data:
open-webui-data:
docs/agents/domain.md
+8
View File
@@ -28,6 +28,14 @@ When your output names a domain concept (in an issue title, a refactor proposal,
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
## New or conflicting terms
When you meet a term that isn't in `CONTEXT.md`:
- **A synonym of an existing term** (e.g. "proxy" for *gateway*): use the canonical term. There's nothing to edit or ask.
- **A genuinely new concept:** add it to `CONTEXT.md` in the same change, as the term, a one- or two-sentence definition and an _Avoid_ list, with no values or config. Name the new term in your final message so the user can veto it.
- **A conflict with an existing definition** (the code or the user uses a term differently than `CONTEXT.md` defines it): stop and ask the user. Redefining a term is their decision.
## Flag ADR conflicts
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
docs/agents/writing-docs.md
+23
View File
@@ -0,0 +1,23 @@
# Writing docs
How to write and update this repo's docs and code comments. The source is the truth: docs explain *why* and point at the source for *what*.
## Rules
1. **Source-of-truth line.** Every doc outside `docs/research/` and `docs/agents/` opens with a `> **Source of truth:** …` line. It names the files, services or keys the doc explains and says the source wins on disagreement. In the body, name keys and flags (`LLAMA_PARALLEL`, `--cache-reuse`) and leave their values in the source. When a doc and its source disagree, fix the doc in the same change.
2. **The README index is the doc map.** Every new doc gets a row in `README.md`'s Docs table.
3. **Code comments are pointers.** In `docker-compose.yml`, `.env.example` and `scripts/`, a comment is a one-line `# see docs/<file>.md#<section>` pointer, and the explanation goes in that doc section. Two things stay inline as one line, next to the pointer:
- a **warning** about data loss, security, or an edit that breaks the stack or a script
- on `.env.example` keys that `update.sh` fills in, the marker "auto-filled by update.sh — leave blank"
A `ponytail:` note becomes a pointer too, with its reasoning moved into the doc.
4. **A source change carries its doc.** When you change code that has a pointer, update the doc section it names in the same change.
5. **`docs/research/` is frozen history.** New research goes in a new file plus a row in [`docs/research/README.md`](../research/README.md). When a research doc goes stale, change its status in that index and leave the file itself untouched.
Terminology follows [`CONTEXT.md`](../../CONTEXT.md). For a term that isn't there, see [domain docs → New or conflicting terms](domain.md#new-or-conflicting-terms).
## Done when
- every pointer in the code you touched resolves to an existing heading
- every doc you touched opens with its source-of-truth line
- no doc you touched restates a value from its source
docs/architecture.md
+54
View File
@@ -0,0 +1,54 @@
# Architecture
> **Source of truth:** [`docker-compose.yml`](../docker-compose.yml) → each service below. This page explains *why*; flags, values and defaults live in the source. If they disagree, the source wins.
Terms as defined in [`CONTEXT.md`](../CONTEXT.md).
Every workload calls the **gateway** (`omniroute`) with its own API key. The gateway forwards each call over a **provider connection** to one of two **backends** on the one GPU: `llama-server` (main model) or `qwen-classifier` (classifier model). `qdrant` sits beside them on the same Docker network. It isn't a model, so it isn't behind the gateway.
## GPU access
Both backends share the one GPU, and both carry the same GPU settings in compose:
- **`group_add` uses numeric GIDs, not group names.** Docker resolves a *named* `group_add` entry against the container's own `/etc/group`, not the host's. That fails unpredictably when the image doesn't define the group, and it gets worse when several GPU services start at once. `scripts/update.sh` resolves the host's real GIDs into `HOST_VIDEO_GID` / `HOST_RENDER_GID` (see [operations → Host-resolved values](operations.md#host-resolved-values)).
- **`GPU_MAX_HW_QUEUES`** works around [ROCm/ROCm#5706](https://github.com/ROCm/ROCm/issues/5706): the card sits pinned at full load and boost clock whenever two HIP contexts touch it. With two backends running, there are always two.
Background: [`research/rocm-gpu-pin-and-render-group.md`](research/rocm-gpu-pin-and-render-group.md). Its service names are older than the current ones, but the findings still apply.
## llama-server
The main model's backend. It is always running. Lazytainer used to idle-stop it, but was removed because its idle detection could never fire behind the gateway ([`research/lazytainer-omniroute-idle-stop.md`](research/lazytainer-omniroute-idle-stop.md)).
- **Model file** (`LLAMA_MODEL_FILE`): the quant choice and its VRAM budget are in [`research/qwen3.8-27b-quant.md`](research/qwen3.8-27b-quant.md).
- **`LLAMA_GPU_LAYERS`**: the default puts every layer on the GPU. The model is dense, not Mixture-of-Experts, so this is the only RAM-offload knob. `--n-cpu-moe`, `--cpu-moe` and `--override-tensor "exps"` do nothing here. There is no SSD tier to enable either. llama.cpp mmaps the model file, so if GPU and RAM ever can't hold it, the OS pages the rest in from disk as a slow, implicit last resort. Explicit tiered offload is an [open llama.cpp request](https://github.com/ggml-org/llama.cpp/discussions/12507).
- **`LLAMA_CTX_SIZE`**: set to the model's own maximum (`max_position_embeddings`). It only fits beside the weights because the KV cache is quantized (`--cache-type-k/v`). An fp16 KV cache at that size would not fit the card. The numbers are in the quant research doc.
- **`LLAMA_PARALLEL`** is the backend's number of **slots**. It's kept low on purpose: more slots meant more contended prefill, which pushed requests past the gateway's timeouts ([gateway → Timeouts](gateway.md#timeouts)). Requests beyond the slot count wait in llama.cpp's own queue, which has no timeout. So the timeout that matters is the gateway's per-connection one.
- **Per-slot context = `LLAMA_CTX_SIZE / LLAMA_PARALLEL`.** llama.cpp divides the context across slots, and each request gets one slot's share. Every coding CLI's context-window setting must use this number, not the raw context size. Real sessions already fill a large share of it, so check that the per-slot figure stays comfortably above observed usage before lowering either value.
- **`--cache-reuse`** reuses cached KV for any matching prompt chunk instead of re-prefilling from scratch. That fixes the root cause behind the gateway's non-ping SSE timeout ([`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md)). It pairs with OmniRoute's prompt-cache affinity (a dashboard setting, on by default), which keeps a conversation on the same slot so that there is a matching prefix to reuse.
- **`--jinja`** enables the chat template that tool calling depends on. Without it, tool-use requests fail outright.
- **Tool-calling reliability** of this model family on llama.cpp has open upstream parser bugs: [`research/qwen3.8-27b-tool-calling.md`](research/qwen3.8-27b-tool-calling.md).
- **No host port**: see [network access → Backends](network-access.md#backends).
## qwen-classifier
The classifier model's backend. It serves qwen-code's Auto Mode classifier (`fastModel`), which judges every tool call before it runs.
- **Why a separate backend:** when the classifier was an alias of the main model, every classification queued behind whatever heavy generation already held the main backend's slots, sometimes for minutes.
- **Why GPU, not CPU:** a CPU-only instance was tried first. Real classification calls carry a conversation transcript, and they ran past the gateway's timeout and retry-looped. It now runs on the GPU with a partial layer count (`--n-gpu-layers`), which leaves VRAM headroom for the main model.
- **Context size:** qwen-code hard-caps the classifier transcript in its own source (`MAX_TRANSCRIPT_MESSAGES`, `MAX_HISTORICAL_ACTION_CHARS` in `packages/core/src/permissions/classifier-transcript.ts`). `--ctx-size` gives margin over that worst case, and nothing more. The backend has a single slot, so its whole context is per-slot.
- **VRAM:** measured usage came out well above the weights+KV estimate. The cause was a missing `--flash-attn`: without it, the unfused attention buffers grow roughly quadratically with context. Batch size was the wrong lever. Check with `rocm-smi` after any change to the context size or the model.
- **`--reasoning off`** is a no-cost safety net against [llama.cpp#20809](https://github.com/ggml-org/llama.cpp/issues/20809). Some builds misdetect Qwen3-Instruct-2507 as a thinking model and route tool calls into `reasoning_content`. Nobody has confirmed whether the current image still needs it.
- **Model choice:** the smallest Qwen3 with long native context and no RoPE scaling. See [`research/fast-model-choice.md`](research/fast-model-choice.md) and [`research/classifier-context-window-scaling.md`](research/classifier-context-window-scaling.md).
- Its provider connection needs specific gateway settings: see [gateway → Provider connections](gateway.md#provider-connections).
## omniroute
The gateway. Its runtime configuration (provider connections, API keys) lives in its own database and is managed through the dashboard, not in this repo: see [`gateway.md`](gateway.md).
- **`OMNIROUTE_MEMORY_MB` + `mem_limit`**: the default heap is sized for dashboard-only use. OmniRoute's Docker guide recommends more for coding-agent traffic, and `mem_limit` is the matching container ceiling.
- **`stop_grace_period`**: SQLite needs time on shutdown to checkpoint its WAL back into the main database file. If this is shortened, recent writes (keys, provider connections) can be lost.
- **Healthcheck is a TCP connect**, not an HTTP probe. The image has only `node` (no curl, wget or python), and OmniRoute's Docker guide accepts a TCP probe as a liveness check.
## qdrant
A vector store. It has no authentication, so its port is bound to the server's loopback interface only: see [network access → qdrant](network-access.md#qdrant).
docs/coding-cli-setup/claude-code.md
+5 -3
View File
@@ -2,12 +2,14 @@
[← back to overview](index.md)
> **Source of truth:** [Claude Code's docs](https://docs.claude.com/en/docs/claude-code/overview) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win.
Claude Code speaks the **Anthropic Messages API** — point it at the gateway's unified endpoint, not llama.cpp directly:
```bash
export ANTHROPIC_BASE_URL=http://<ai-box>:${OMNIROUTE_PORT:-4000}
export ANTHROPIC_API_KEY=<claude-code-cli virtual key>
export ANTHROPIC_BASE_URL=http://<server>:<OMNIROUTE_API_PORT>
export ANTHROPIC_API_KEY=<claude-code-cli API key>
claude
```
Requires llama.cpp's `--jinja` flag (already set in `docker-compose.yml`) — without it, tool-use requests fail outright.
Relies on the backend's `--jinja` flag ([architecture → llama-server](../architecture.md#llama-server)).
docs/coding-cli-setup/index.md
+17 -22
View File
@@ -1,34 +1,29 @@
# Pointing a coding-agent CLI at this stack
# Pointing a coding CLI at this stack
[← back to README](../../README.md)
This stack routes through the [AI gateway](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) (OmniRoute, see issue #31) rather than talking to llama.cpp directly — llama.cpp's own port is internal-only now (see `docker-compose.yml`). The gateway exposes:
> **Source of truth:** each CLI's own config docs (linked per page), plus [`docker-compose.yml`](../../docker-compose.yml) → `omniroute` for the gateway port. This page explains *how to wire them up*; ports and model settings live in those sources. If they disagree, the sources win.
- **OpenAI-compatible**: `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1`
- **Anthropic Messages API** (OmniRoute's own `/v1/messages` endpoint, translating to the OpenAI-compatible backend): `http://<ai-box>:${OMNIROUTE_PORT:-4000}`
Terms as defined in [`CONTEXT.md`](../../CONTEXT.md). Every coding CLI is a workload: it reaches the models through the gateway, never a backend directly ([network access](../network-access.md)). The gateway speaks two wire formats for the same models:
Both serve the same underlying model — `Qwen3.8-27B-UD-Q4_K_XL.gguf`, registered in the gateway (naming is yours to pick when adding the llama-cpp provider connection — these docs assume `qwen3.8-27b-local` for continuity) — behind whichever wire format the client speaks.
- **OpenAI-compatible:** `http://<server>:<OMNIROUTE_API_PORT>/v1`
- **Anthropic Messages:** `http://<server>:<OMNIROUTE_API_PORT>` (OmniRoute's `/v1/messages`)
`<ai-box>` is this machine's LAN address, or `proxy-ai.home` if your local DNS resolves that hostname directly to the box — see `docs/network-access.md`. If you're running a coding CLI from this machine itself, `localhost` works too.
`<server>` is the server's LAN address or `proxy-ai.home`, or `localhost` on the server itself. The main model's name is whatever it was registered as in the gateway. These docs use `qwen3.8-27b-local` ([gateway → Provider connections](../gateway.md#provider-connections)).
**Each CLI needs its own virtual key** — create one per docs/proxy-key-onboarding.md (omniroute's dashboard, `<workload>-<purpose>` naming, e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli`). No budget set by default. These are the machine's interactive/high-priority workloads per `docs/proxy-request-priority.md`.
**Each CLI needs its own API key**, labelled e.g. `claude-code-cli`, `kimi-cli`, `opencode-cli` ([gateway → API keys](../gateway.md#api-keys)).
> **Read this before relying on it for real work.** Qwen3.8-27B's tool-calling has **documented, open llama.cpp upstream bugs** (parser fails on text before `<tool_call>`, tool calls emitted as inert XML inside thinking blocks — see `docs/research/qwen3.8-27b-tool-calling.md`). Every CLI below inherits this risk identically, regardless of wire format. Don't trust it for unattended multi-step agentic work until you've run the smoke test in [issue #5](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) (and the proxy-specific smoke test in [issue #17](https://git.arthurerlich.de/haylan/LLM-Server/issues/17)).
**Context window:** wherever a CLI asks for the main model's context size, use the per-slot context, `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from the server's `.env`, not the raw context size ([architecture → llama-server](../architecture.md#llama-server)).
> **Tool-calling risk.** The main model's tool calling has open llama.cpp upstream parser bugs ([`research/qwen3.8-27b-tool-calling.md`](../research/qwen3.8-27b-tool-calling.md)). Every CLI below inherits that risk, whatever its wire format. Test it on your own workflow before relying on it for unattended multi-step agent work.
## Per-CLI setup
- [Claude Code CLI](claude-code.md)
- [Kimi CLI](kimi-cli.md)
- [OpenCode CLI](opencode.md)
- [Qwen Code CLI](qwen-code.md)
| CLI | Wire format | Config |
|---|---|---|
| [Claude Code](claude-code.md) | Anthropic Messages | `ANTHROPIC_BASE_URL` env var |
| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `config.toml` provider block |
| [OpenCode](opencode.md) | OpenAI Chat Completions | `opencode.json` provider block |
| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (main model + classifier model) | `~/.qwen/settings.json` `modelProviders.openai` |
## Summary
| CLI | Wire format | Endpoint | Config |
|---|---|---|---|
| [Claude Code](claude-code.md) | Anthropic Messages | `http://<ai-box>:${OMNIROUTE_PORT:-4000}` | `ANTHROPIC_BASE_URL` env var |
| [Kimi CLI](kimi-cli.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `config.toml` provider block |
| [OpenCode](opencode.md) | OpenAI Chat Completions | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `opencode.json` provider block |
| [Qwen Code](qwen-code.md) | OpenAI Chat Completions (2 models: chat + `fastModel`) | `http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1` | `~/.qwen/settings.json` `modelProviders.openai` |
Further reading: `docs/research/qwen3.8-27b-tool-calling.md`, `docs/proxy-key-onboarding.md`, `docs/research/omniroute-account-semaphore-timeout.md` (a connection that can only handle a few concurrent requests — like `llama-server` or `qwen-classifier` — hits a hardcoded 30s reject once more requests queue up than its `maxConcurrent`, unless configured around it).
If requests fail with a semaphore timeout under concurrent load, see [gateway → Timeouts](../gateway.md#timeouts).
docs/coding-cli-setup/kimi-cli.md
+4 -2
View File
@@ -2,13 +2,15 @@
[← back to overview](index.md)
> **Source of truth:** [Kimi CLI's repo](https://github.com/MoonshotAI/kimi-cli) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win.
Kimi CLI speaks plain **OpenAI Chat Completions**. Configure a provider block in its config file (`config.toml`):
```toml
[providers.openai]
type = "openai"
base_url = "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
api_key = "<kimi-cli virtual key>"
base_url = "http://<server>:<OMNIROUTE_API_PORT>/v1"
api_key = "<kimi-cli API key>"
```
If Kimi CLI's response parsing gets confused by Qwen's `<think>...</think>` reasoning tags, check its `reasoning_key` setting — it's configurable for non-standard local server responses.
docs/coding-cli-setup/opencode.md
+11 -9
View File
@@ -2,6 +2,8 @@
[← back to overview](index.md)
> **Source of truth:** [OpenCode's docs](https://opencode.ai/docs/) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win.
Confirmed project: **`anomalyco/opencode`** (renamed from `sst/opencode` — don't confuse with the unrelated `opencode-ai/opencode` Go TUI). Docs: https://opencode.ai/docs/
**Install**:
@@ -15,17 +17,17 @@ curl -fsSL https://opencode.ai/install | bash
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"aiproxy": {
"llm-server": {
"npm": "@ai-sdk/openai-compatible",
"name": "AI proxy (local)",
"name": "LLM-Server gateway",
"options": {
"baseURL": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"apiKey": "<opencode-cli virtual key>"
"baseURL": "http://<server>:<OMNIROUTE_API_PORT>/v1",
"apiKey": "<opencode-cli API key>"
},
"models": {
"qwen3.8-27b-local": {
"name": "Qwen3.8-27B",
"limit": { "context": 131072, "output": 8192 }
"limit": { "context": <LLAMA_CTX_SIZE / LLAMA_PARALLEL>, "output": 8192 }
}
}
}
@@ -33,13 +35,13 @@ curl -fsSL https://opencode.ai/install | bash
}
```
Set `limit.context` to the *per-slot* context this stack actually serves — `LLAMA_CTX_SIZE / LLAMA_PARALLEL` from `.env` (262144 / 2 = 131072 by default), not raw `LLAMA_CTX_SIZE` and not a value assumed from the model card: llama.cpp divides `--ctx-size` across concurrent slots, so each request only gets one slot's share. OpenCode uses this for its own context-management bookkeeping, not the server.
`limit.context` is the per-slot context ([overview](index.md)). OpenCode uses it for its own context bookkeeping; the server doesn't read it.
Select the model with `aiproxy/qwen3.8-27b-local`.
Select the model with `llm-server/qwen3.8-27b-local`.
**OpenCode-specific risks** (on top of the shared Qwen3.8-27B tool-calling risk — see [overview](index.md)):
- Requires llama.cpp's `--jinja` flag (already set) — without it, OpenCode's unconditional tool-call scaffolding gets a 500.
- Relies on the backend's `--jinja` flag ([architecture → llama-server](../architecture.md#llama-server)); without it, OpenCode's unconditional tool-call scaffolding gets a 500.
- [anomalyco/opencode#20669](https://github.com/anomalyco/opencode/issues/20669) (closed as "not planned" — a live, unfixed risk): OpenCode's `bash` tool crashes if the model omits the optional `description` field on a tool call; some local backends return `finish_reason: tool_calls` with an empty array, which can hang the agent loop instead of stopping cleanly.
- Thinking-mode handling (`options.reasoningEffort`) is undocumented for models that emit inline `<think>` tags rather than a native reasoning API field — expect no effect from that config on this model; untested.
Further reading: `docs/research/opencode-cli-setup.md`.
Further reading: [`research/opencode-cli-setup.md`](../research/opencode-cli-setup.md) (its base URL example predates the gateway), [`research/opencode-auto-compact.md`](../research/opencode-auto-compact.md).
docs/coding-cli-setup/qwen-code.md
+17 -20
View File
@@ -2,16 +2,13 @@
[← back to overview](index.md)
Qwen Code speaks plain **OpenAI Chat Completions**, and — unlike the other CLIs — needs *two* models: the main chat model, and a `fastModel` for Auto Mode's action classifier. Both are registered as separate providers in OmniRoute but reachable through the same gateway URL.
> **Source of truth:** [Qwen Code's repo](https://github.com/QwenLM/qwen-code) for config keys; the [overview](index.md) for the gateway endpoint and context size. If this page disagrees with them, they win.
Qwen Code speaks plain **OpenAI Chat Completions**. Unlike the other CLIs, it needs *two* models: the main model, and the classifier model as its `fastModel` for Auto Mode's action classifier. Each has its own provider connection in the gateway, and both are reached through the same gateway URL.
## Why a second model exists
Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's per-tool-call safety gate — it decides whether to auto-approve or block a shell command / tool call before it runs. It was originally aliased onto the main 27B model's own OmniRoute connection. That broke two ways in practice (see `docs/research/fast-model-choice.md` for the model research, and the issue-tracker history for the full incident):
- **Queued behind heavy work.** Every classification call competed for the main model's 2 GPU slots with whatever real generation was already running, so a classifier check could sit blocked for minutes.
- **CPU-only was tried first and was too slow.** Isolating the classifier onto its own CPU-only llama.cpp instance avoided the GPU queue entirely, but real classification calls (which can carry a non-trivial conversation transcript, not just the bare tool call) blew past OmniRoute's request timeout and retry-looped.
The fix: a dedicated, GPU-resident `qwen-classifier` service (`docker-compose.yml`) running a small model (`Qwen3-4B-Instruct-2507`) on its own **partial** GPU offload — enough layers on the R9700 to be fast, sized to leave real VRAM headroom next to the 27B model rather than trusting a naive weights+KV estimate (see that service's comment block in `docker-compose.yml` for the actual measured numbers and the two wrong turns — batch-size tuning, then flash-attn — before partial offload turned out to be the real lever).
Auto Mode's action classifier (`permissions.autoMode`) is qwen-code's safety gate for every tool call: it decides whether to auto-approve or block a shell command or tool call before it runs. It runs on its own backend, `qwen-classifier`. Why that backend exists and how it's sized: [architecture → qwen-classifier](../architecture.md#qwen-classifier).
## `~/.qwen/settings.json`
@@ -23,16 +20,16 @@ The fix: a dedicated, GPU-resident `qwen-classifier` service (`docker-compose.ym
"id": "<main-model-provider-id-in-omniroute>",
"name": "qwen3.8-27b-local",
"envKey": "OMNIROUTE_API_KEY",
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"generationConfig": { "contextWindowSize": 131072 }
"baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1",
"generationConfig": { "contextWindowSize": <LLAMA_CTX_SIZE / LLAMA_PARALLEL> }
},
{
"id": "<classifier-provider-id-in-omniroute>",
"name": "qwen3-4b-classifier",
"envKey": "OMNIROUTE_API_KEY",
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1",
"baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1",
"generationConfig": {
"contextWindowSize": 65536,
"contextWindowSize": <qwen-classifier --ctx-size>,
"extra_body": { "chat_template_kwargs": { "enable_thinking": false } }
}
}
@@ -41,21 +38,21 @@ The fix: a dedicated, GPU-resident `qwen-classifier` service (`docker-compose.ym
"security": { "auth": { "selectedType": "openai" } },
"model": {
"name": "<main-model-provider-id-in-omniroute>",
"baseUrl": "http://<ai-box>:${OMNIROUTE_PORT:-4000}/v1"
"baseUrl": "http://<server>:<OMNIROUTE_API_PORT>/v1"
},
"fastModel": "<classifier-provider-id-in-omniroute>"
}
```
- `envKey` names the environment variable Qwen Code reads the virtual key from — set `OMNIROUTE_API_KEY=<qwen-code-cli virtual key>` before launching. Both providers can share one virtual key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
- **`contextWindowSize` for the main model is per-slot, not `LLAMA_CTX_SIZE` itself** — llama.cpp divides `--ctx-size` across `LLAMA_PARALLEL` concurrent slots, and each request only gets one slot's share (same correction applies to OpenCode's `limit.context`). Compute it from `.env`: `LLAMA_CTX_SIZE / LLAMA_PARALLEL` = `262144 / 2` = **131072**.
- **The classifier's `contextWindowSize` (65536) is not per-slot math** — `qwen-classifier` runs `--parallel 1`, so its whole `--ctx-size` belongs to the one slot. 65536 isn't a guess either: qwen-code's own source hard-caps the classifier transcript (`MAX_TRANSCRIPT_MESSAGES=40`, `MAX_HISTORICAL_ACTION_CHARS=4000`/message in `packages/core/src/permissions/classifier-transcript.ts`) — worst case is ~40-50K tokens, so 65536 gives real margin without wasting VRAM the way the original 131072 (copied from the main model's entry, not an actual qwen-code requirement) would have.
- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see `fast-model-choice.md` §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
- `envKey` names the environment variable Qwen Code reads the API key from — set `OMNIROUTE_API_KEY=<qwen-code-cli API key>` before launching. Both providers can share one API key (as above); split it into two if you want separate usage tracking for chat vs. classifier calls.
- **The main model's `contextWindowSize` is the per-slot context**, not `LLAMA_CTX_SIZE` itself ([overview](index.md)).
- **The classifier's `contextWindowSize` is its whole `--ctx-size`.** It has a single slot, so there's no per-slot division ([architecture → qwen-classifier](../architecture.md#qwen-classifier)).
- `enable_thinking: false` on the classifier matters for parseability, though `Qwen3-4B-Instruct-2507` is already architecturally non-thinking (see [`research/fast-model-choice.md`](../research/fast-model-choice.md) §3) — this is belt-and-suspenders for any future fast-model swap that isn't.
- Qwen Code also recognizes `advisorModel`, `visionModel`, `compactionModel`, `imageModel` for other model roles — none are wired up in this stack; only `fastModel` is required.
## Web search via OmniRoute
Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search (README §"Web search") is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
Qwen Code's own built-in web search (`tools.webSearch.enabled`) has nothing to search with here — leave it `false`. Instead this stack's SearXNG-backed search ([gateway → Web search](../gateway.md#web-search)) is exposed through a thin stdio MCP wrapper around OmniRoute's `/v1/search` REST endpoint (that endpoint isn't itself MCP — OmniRoute's real MCP surface is admin-only/LOCAL_ONLY-gated). Save this as e.g. `~/.qwen/mcp-servers/omniroute-search/index.mjs` (needs `@modelcontextprotocol/sdk` and `zod`: `npm init -y && npm i @modelcontextprotocol/sdk zod` in that directory):
```js
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
@@ -104,13 +101,13 @@ Register it in `~/.qwen/settings.json`:
}
```
It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the virtual key needs search permission in OmniRoute, not just chat-completions.
It reuses the same `OMNIROUTE_API_KEY` env var as the model providers above — the API key needs search permission in OmniRoute, not just chat-completions.
**Non-interactive mode (`qwen -p ...`) needs this tool explicitly allow-listed.** MCP tools require interactive confirmation by default; `--approval-mode auto` alone doesn't bypass that for a non-interactive run — pass `--allowed-tools mcp__omniroute-search__search` (or `-y` for full YOLO) alongside `-p`, or the search call never reaches the classifier at all and silently no-ops. Confirmed live: without the allow-list, only the tool calls the CLI's non-interactive gate lets through end up as classifier requests.
## Auto Mode tuning
Auto Mode's action classifier calls the fast model above. Even on the dedicated GPU-resident instance, give it real timeout headroom rather than trusting OmniRoute's default — and since this stack is a single trusted local proxy, it's reasonable to pre-approve requests to it rather than confirm every call:
Auto Mode's action classifier calls the classifier model above. Even on its own GPU-resident backend, give it real timeout headroom rather than trusting OmniRoute's default. And since the gateway is a single trusted local endpoint, it's reasonable to pre-approve requests to it rather than confirm every call:
```json
{
@@ -125,6 +122,6 @@ Auto Mode's action classifier calls the fast model above. Even on the dedicated
`hints.allow` entries are free-text descriptions the classifier matches against, not exact strings — capped at 150 entries/200 chars each.
Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see `docs/network-access.md` for reaching the dashboard API). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.
Also set a generous per-connection timeout on the classifier's own OmniRoute provider connection (`providerSpecificData.timeoutMs`, dashboard or `PATCH /api/providers/{id}` — not a `.env` value, see [gateway → Dashboard](../gateway.md#dashboard)). 120000ms is comfortable for the current GPU-resident setup (real measured latency: well under a second for a short check, low seconds for the largest realistic transcript) — this isn't the 20-minute figure the main 27B connection needs, since the classifier isn't competing for a contended GPU slot the way the main model can.
Everything else in `~/.qwen/settings.json` (`hooks`, `security.auth`'s underlying tooling, editor prefs) is per-machine, not part of pointing at this stack — don't copy it wholesale between machines.
docs/gateway.md
+62
View File
@@ -0,0 +1,62 @@
# Gateway (OmniRoute)
> **Source of truth:** [`docker-compose.yml`](../docker-compose.yml) → `omniroute` service, and [`.env.example`](../.env.example) → OmniRoute section. Provider connections and API keys live in OmniRoute's own database, managed through its dashboard, not in this repo. This page explains *why* and *how*; values live in those sources. If they disagree, the source wins.
Terms as defined in [`CONTEXT.md`](../CONTEXT.md). How the gateway fits the rest of the stack: [`architecture.md`](architecture.md#omniroute).
## Dashboard
Browse to `http://<server>:<OMNIROUTE_DASHBOARD_PORT>` from the LAN. The dashboard isn't routed to the internet: see [network access → Gateway](network-access.md#gateway). For the first login, use `OMNIROUTE_INITIAL_PASSWORD` from the server's `.env`, then change the password in the dashboard.
## Provider connections
Each backend is registered once in the dashboard (Providers → add a **llama.cpp** provider) or via `POST /api/providers`. The base URL is `http://<service>:<container port>` on the stack network, using the service name and port from compose. None of this is checked into the repo. These docs assume the main model is registered as `qwen3.8-27b-local`.
- **`OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` / `OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`** have to be on, because backend URLs are private addresses. They relax OmniRoute's SSRF guard, which normally blocks registering a provider at a private or local address. Anyone with dashboard access can therefore point the gateway at any host on the stack network or the LAN.
- **Classifier connection:** turn OmniRoute's memory-tool injection off for it. Injected tools break the classifier's output ([`research/omniroute-memory-tool-injection-breaks-classifier.md`](research/omniroute-memory-tool-injection-breaks-classifier.md)). Also give it a generous per-connection timeout (see [Timeouts](#timeouts)).
- Background on registering local backends: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md).
## API keys
Every workload gets its own API key, so each has its own visible usage.
**Create**
1. [Open the dashboard](#dashboard) → "Keys" → "Create API key".
2. Label it `<workload>-<purpose>`, e.g. `paperless-ocr`, `claude-code-cli`. The dashboard lists keys by label, so the label list is the ledger. There's no separate tracking doc.
3. Copy the key value. OmniRoute shows it only once.
Keys can only be minted by hand for now. `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key, and that flow hasn't been verified against a live instance.
**Hand over:** put the key in that workload's own config or `.env`, never in this repo. A key used for [web search](#web-search) needs search permission, not just chat completions.
**Revoke** ("Keys" → delete) when the workload is retired or the key may have leaked, then remove it from the workload's config. There's no scheduled rotation.
## Timeouts
A slow request can be cut off at several layers. Each setting is named below, and its value lives where the list says:
- **`STREAM_IDLE_TIMEOUT_MS`** (`.env.example`): the longest allowed gap *between* SSE chunks once streaming has started.
- **`REQUEST_TIMEOUT_MS`** (`.env.example`): the wait budget for the *first* non-ping SSE event. Retries and cooldowns eat into it, so the effective deadline is whatever budget remains, not a flat timer. See [`research/omniroute-non-ping-sse-stream-timeout.md`](research/omniroute-non-ping-sse-stream-timeout.md).
- **Per-connection `providerSpecificData.timeoutMs`** (dashboard only, or `PATCH /api/providers/{id}`): bounds a request that is waiting behind the backend's busy slots.
- **Per-connection concurrency semaphore**: a hardcoded reject once more requests queue than the connection's `maxConcurrent`. It is not configurable, only avoidable: [`research/omniroute-account-semaphore-timeout.md`](research/omniroute-account-semaphore-timeout.md).
- **`OMNIROUTE_DIRECT_HEADERS_TIMEOUT_MS`** (not set in this repo): [`research/omniroute-direct-response-timeout-outage-2026-09-15.md`](research/omniroute-direct-response-timeout-outage-2026-09-15.md).
These are raised above OmniRoute's defaults because contended prefill on the main backend can take a long time before the first token ([architecture → llama-server](architecture.md#llama-server)).
## Web search
SearXNG runs elsewhere on the LAN as `search.home`. It's registered as a search provider connection and exposed through the gateway's `/v1/search` endpoint. The container can't resolve `search.home` (it's a LAN DNS name), so compose adds an `extra_hosts` entry from `SEARXNG_LAN_IP`, which `update.sh` resolves on the server ([operations → Host-resolved values](operations.md#host-resolved-values)).
- Wiring it into Qwen Code: [`coding-cli-setup/qwen-code.md`](coding-cli-setup/qwen-code.md#web-search-via-omniroute).
- Research: [`research/omniroute-qwen-websearch.md`](research/omniroute-qwen-websearch.md) (current setup), [`research/dashscope-websearch-selfhost-options.md`](research/dashscope-websearch-selfhost-options.md) (why DashScope isn't emulated).
## Upstream caution
OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md` and `PUBLIC_CREDS.md` in [its repo](https://github.com/diegosouzapw/OmniRoute)) describe shipped features for:
- evading AI-provider client detection
- intercepting HTTPS system-wide via a locally installed root CA
- hiding credentials from secret scanners
This stack's configuration uses none of them. They are a real characteristic of the upstream project, though, so weigh them before extending this integration.
docs/network-access.md
+19 -14
View File
@@ -1,24 +1,29 @@
# Network access: proxy-ai.home / proxy-ai.haylan.ch
# Network access
This stack has no chat UI — every client is a coding CLI reaching the AI gateway (OmniRoute). It doesn't run its own reverse proxy — it publishes the gateway's API port to the host and relies on the **existing Nginx Proxy Manager (NPM)** instance already fronting other self-hosted services on this network.
> **Source of truth:** [`docker-compose.yml`](../docker-compose.yml) → the `ports:` / `expose:` entries of each service. This page explains *who can reach what, and why*; port numbers live in the source. If they disagree, the source wins.
## llama.cpp's raw API stays LAN-only — deliberately
Terms as defined in [`CONTEXT.md`](../CONTEXT.md). The stack doesn't run its own reverse proxy. The existing **reverse proxy** (Nginx Proxy Manager, not part of this stack) publishes the gateway under hostnames.
The inference API (port `${LLAMA_PORT:-8080}`) is **not** registered in NPM and is **not** reachable externally. It has no authentication of its own — putting it on the public internet would mean an unauthenticated inference endpoint. Coding-agent CLIs (Claude Code, Kimi, OpenCode, Qwen Code — see `docs/coding-cli-setup/`) don't reach it directly at all now; they go through the gateway below, same as everything else.
| What | Reachable from |
|---|---|
| Gateway API | LAN, and the internet via the reverse proxy |
| Gateway dashboard | LAN only |
| Backends | Stack network only (through the gateway) |
| qdrant | The server itself only |
If you later want external CLI access too, that's a deliberate scope change — see the map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) before doing it, since it changes the security posture.
## Gateway
## The AI gateway (OmniRoute) — `proxy-ai.home` / `proxy-ai.haylan.ch`
- **API port (`OMNIROUTE_API_PORT`)**: published on the server's LAN address. Two reverse-proxy hosts point at it:
- `proxy-ai.home`: LAN only.
- `proxy-ai.haylan.ch`: internet, through the DMZ that already forwards to the reverse proxy, which also manages the TLS certificate.
As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) (migrated from LiteLLM), the gateway is OmniRoute:
Every call needs an [API key](gateway.md#api-keys), so the reverse proxy adds no auth of its own.
- **Dashboard port (`OMNIROUTE_DASHBOARD_PORT`)**: published on the LAN only. It's never registered in the reverse proxy and has no port-forward, so it isn't reachable from the internet. Keep it that way: the dashboard manages every API key.
- **`proxy-ai.home`** and **`proxy-ai.haylan.ch`** both point only at `${OMNIROUTE_PORT:-4000}` — the API port. Set up as two NPM Proxy Hosts pointing at this machine's LAN IP on that port; `proxy-ai.home` internal-only, `proxy-ai.haylan.ch` external via the DMZ already forwarding to NPM (let NPM issue/manage the TLS cert as usual).
- The **dashboard** (`${OMNIROUTE_DASHBOARD_PORT:-20128}`) is never registered in NPM at all, and `docker-compose.yml` never publishes that port to the host either — it manages every workload's keys, so it doesn't belong on the public internet, same reasoning as LiteLLM's old `/ui`. Unlike LiteLLM, OmniRoute's split-port mode means this is structural (no network route exists) rather than an NPM path-deny rule that has to be maintained and could be misconfigured. Reach the dashboard only from the host itself or over SSH port-forward.
## Backends
**Every gateway call already requires a valid API key** (Bearer token, see `docs/proxy-key-onboarding.md`), so no extra NPM-level auth is needed for the external hostname.
`llama-server` and `qwen-classifier` publish no host port (`expose` only). They have no authentication, so they're reachable only on the stack's Docker network, through the gateway. Making them directly reachable would put an unauthenticated inference endpoint on the network. That's a deliberate security change, not a config tweak.
## RAG knowledge graph (Neo4j) — `knowledge.proxy-ai.home`
## qdrant
Set up as an NPM Proxy Host pointing at this machine's LAN IP on Neo4j's Browser port (`7474`, see `docker-compose.yml`'s `neo4j` service, [PR #50](https://git.arthurerlich.de/haylan/LLM-Server/pulls/50)). Internal-only, same as `proxy-ai.home` — no DMZ/external route, this is admin/dev tooling, not a client-facing endpoint. Bolt (`7687`, the actual query protocol) isn't proxied through NPM at all — clients on the LAN reach it directly at `<this-machine>:7687`.
Qdrant's dashboard (`6333`) stays on its raw LAN IP/port for now — no hostname assigned yet.
Bound to the server's loopback interface, because qdrant has no authentication. From another machine, use an SSH port-forward to the server.
docs/operations.md
+42
View File
@@ -0,0 +1,42 @@
# Operations
> **Source of truth:** [`scripts/update.sh`](../scripts/update.sh) and [`.env.example`](../.env.example). This page explains *what the script does and why*; the steps and defaults live in the source. If they disagree, the source wins.
The stack runs on the **server**. After any change to the repo, run `./scripts/update.sh` there: it is the one command that brings the running stack in sync. It's safe to re-run at any time, and it only recreates what changed.
## Self-update
`git pull` runs before anything else. A script that updates itself isn't atomic against its own file changing mid-run. Pulling later once executed a mix of old and new script and compose content in a single pass. Keep the pull first.
## Env sync
`.env` is created from `.env.example` if it's missing. After that:
- **Keys with a real default in `.env.example`** are tracked config, synced every run. A key missing from `.env` is added. A key whose value matches is left alone. A key whose value **differs** is a conflict:
- Interactively, all conflicts appear on one screen ([gum](#gum)). Selected keys take the new value, and unselected keys keep the server's value.
- Without a TTY (cron, CI, a pipe), any conflict is an error unless `--force` is passed, which accepts every new value.
To change a default for good, change `.env.example`, not the server's `.env`.
- **Keys left blank in `.env.example`** ([secrets](#secrets) and [host-resolved values](#host-resolved-values)) are never synced. They're only filled when blank.
- Keys removed from `.env.example` stay in the server's `.env`: the script adds keys and never prunes them. They're harmless, and you can delete them by hand.
There's no rollback. If a bad config lands, `git revert` and re-run the script.
## Secrets
Random secrets are generated with `openssl` when blank and never overwritten after that. **`OMNIROUTE_STORAGE_ENCRYPTION_KEY` must never change after the first run**: it encrypts OmniRoute's whole database, and changing it makes the existing data unreadable. `OMNIROUTE_WS_BRIDGE_SECRET` is required by OmniRoute in production (its `docs/reference/ENVIRONMENT.md`). The per-workload API keys aren't here at all: see [gateway → API keys](gateway.md#api-keys).
## Host-resolved values
These are resolved on the server itself. Set them by hand only if resolution fails:
- `SEARXNG_LAN_IP`, from `search.home`: see [gateway → Web search](gateway.md#web-search).
- `HOST_VIDEO_GID` / `HOST_RENDER_GID`, from the host's `video` / `render` groups: see [architecture → GPU access](architecture.md#gpu-access).
## Gum
The conflict screen uses [gum](https://github.com/charmbracelet/gum), a single static binary. It's vendored as a release tarball under `scripts/vendor/`, because the server has no outbound internet access. Other architectures fall back to downloading it, and if that fails, to a plain-bash prompt. The extracted binary is cached under `.cache/gum/` (gitignored).
## Model downloads
Each model file has its own one-off downloader service in the `tools` compose profile. `update.sh` runs them every time. A `test -f` guard skips files that already exist, so re-runs don't re-download. Files live in the named `models` volume, not a host bind-mount. The downloaders run as root because the named volume is created root-owned. There's one service per model, so the small model's job never re-checks the large one.
docs/proxy-key-onboarding.md
-26
View File
@@ -1,26 +0,0 @@
# Onboarding a workload onto the AI gateway
How to issue a new per-workload API key against the OmniRoute gateway (see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) — the LiteLLM → OmniRoute migration; original gateway rationale in [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) / `docs/research/proxy-tool-choice.md`), so a new workload (a code-reviewer tool, Paperless-OCR, Gitea code review, etc.) gets its own key and its own visible usage/spend.
No workload in this stack itself needs a key right now — every client is external (a coding CLI, or another self-hosted service). There's no scripted mint yet either way: `POST /api/keys` needs a dashboard login session (`ManagementSessionAuth`), not a static bearer key like LiteLLM's old `/key/generate`, and that flow hasn't been verified against a live instance (see [issue #37](https://git.arthurerlich.de/haylan/LLM-Server/issues/37)). Create every key by hand for now, via the dashboard steps below.
## Create the key
1. Log into the omniroute dashboard. `DASHBOARD_PORT` (20128) is never published to the host (see `docker-compose.yml`'s `omniroute` service) — from the R9700 box itself, find the container's own address (`docker inspect -f '{{.NetworkSettings.Networks.ai_stack.IPAddress}}' omniroute`) and browse to `http://<that-ip>:20128` (the host can reach a container's bridge-network IP directly, published port or not). From elsewhere, SSH port-forward instead: `ssh -L 20128:<container-ip>:20128 <host>`, then browse `http://localhost:20128`.
2. "Keys" → "Create API key".
3. Label it `<workload>-<purpose>` — a short slug matching the workload, e.g. `paperless-ocr`, `gitea-code-review`, `claude-code-cli`. This label is the ledger: the dashboard lists keys by label, so there's no separate tracking doc to keep in sync.
4. Copy the key value shown — it's only shown once at creation, per OmniRoute's docs.
Once `POST /api/keys`'s session-auth flow is worked out (issue #37), the equivalent `curl` here can replace this manual step, the way `update.sh` used to automate LiteLLM's `/key/generate`.
## Hand it to the workload
Drop the key into that workload's own `.env` (or equivalent config) — never into this repo. Each workload's config is the source of truth for its own credential, same pattern as `.env.example` in this repo for the existing stack.
## Retiring or rotating a key
No scheduled rotation. Revoke the key by hand in the dashboard ("Keys" → delete) only when:
- the workload is retired, or
- the key is suspected leaked/compromised.
Then remove it from that workload's `.env`.
docs/research/README.md
+33
View File
@@ -0,0 +1,33 @@
# Research index
[← back to README](../../README.md)
Each file here is a **dated snapshot**: research or a decision record as it stood when it was written. The files are never edited afterwards, and their status lives only in this table. A *superseded* doc is kept as history. Read it for *why* a choice was made, never for how the stack works today (for that, see [`architecture.md`](../architecture.md) and the source it links).
| Doc | Status | Note |
|---|---|---|
| [500k-context-2-parallel-agents-gpu-roadmap.md](500k-context-2-parallel-agents-gpu-roadmap.md) | Background research | Hardware exploration for larger contexts. Not an active plan, and neither is [`../server-planing.md`](../server-planing.md), whatever this doc says about it |
| [classifier-context-window-scaling.md](classifier-context-window-scaling.md) | Current | Classifier model sizing decision |
| [colibri-ai-stack-evaluation.md](colibri-ai-stack-evaluation.md) | Current | Rejected option ("watch, don't integrate"). It lists ComfyUI, which has since been removed |
| [connecting-cloud-providers.md](connecting-cloud-providers.md) | Superseded by the OmniRoute migration | Written for LiteLLM. It mentions Open WebUI, which has been removed |
| [dashscope-websearch-selfhost-options.md](dashscope-websearch-selfhost-options.md) | Current | Why DashScope web search isn't emulated |
| [fast-model-choice.md](fast-model-choice.md) | Current (drift) | Classifier model choice. Its addendum records the service name (`qwen-classifier`) and quant drift |
| [langchain-pgvector-vs-litellm-pgvector.md](langchain-pgvector-vs-litellm-pgvector.md) | Superseded by the OmniRoute migration | LiteLLM-era RAG comparison |
| [lazytainer-omniroute-idle-stop.md](lazytainer-omniroute-idle-stop.md) | Superseded: Lazytainer and `switch-model.sh` removed | Root cause of why Lazytainer's idle-stop never fired behind the gateway, which is the reason it was removed |
| [litellm-knowledgebase.md](litellm-knowledgebase.md) | Superseded by the OmniRoute migration | LiteLLM vector-store feature. Its claim that Open WebUI is qdrant's only user no longer holds |
| [litellm-searxng-search.md](litellm-searxng-search.md) | Superseded by [omniroute-qwen-websearch.md](omniroute-qwen-websearch.md) | Still the origin of the `search.home` DNS gotcha ([gateway → Web search](../gateway.md#web-search)) |
| [omniroute-account-semaphore-timeout.md](omniroute-account-semaphore-timeout.md) | Current | Gateway timeout: per-connection semaphore |
| [omniroute-direct-response-timeout-outage-2026-09-15.md](omniroute-direct-response-timeout-outage-2026-09-15.md) | Current | Gateway timeout: direct response headers, plus the ROCm stall incident |
| [omniroute-memory-tool-injection-breaks-classifier.md](omniroute-memory-tool-injection-breaks-classifier.md) | Current | Memory tools must stay off for the classifier connection |
| [omniroute-non-ping-sse-stream-timeout.md](omniroute-non-ping-sse-stream-timeout.md) | Current | Gateway timeout: first non-ping SSE event, and why `--cache-reuse` is set |
| [omniroute-qwen-websearch.md](omniroute-qwen-websearch.md) | Current | Qwen Code routing and web search through the gateway |
| [opencode-auto-compact.md](opencode-auto-compact.md) | Current | OpenCode's client-side compaction |
| [opencode-cli-setup.md](opencode-cli-setup.md) | Current (drift) | The provider JSON shape is valid, but its base URL points at a backend directly. Use the gateway ([setup](../coding-cli-setup/opencode.md)) |
| [ponytail-audit-2026-09-09.md](ponytail-audit-2026-09-09.md) | Superseded (point-in-time audit) | Findings are fixed or folded into the docs cleanup |
| [proxy-request-priority.md](proxy-request-priority.md) | Superseded by the OmniRoute migration | Priority tiers via LiteLLM's scheduler, never built on OmniRoute |
| [proxy-shadow-pricing.md](proxy-shadow-pricing.md) | Superseded by the OmniRoute migration | Shadow cost was wired into LiteLLM config, and nothing implements it now |
| [proxy-tool-choice.md](proxy-tool-choice.md) | Superseded by the OmniRoute migration | Chose LiteLLM. It mentions Open WebUI, which has been removed |
| [qwen3.8-27b-quant.md](qwen3.8-27b-quant.md) | Current | Main model quant and VRAM budget |
| [qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) | Current | Tool-calling parser risk for the main model |
| [rocm-gpu-pin-and-render-group.md](rocm-gpu-pin-and-render-group.md) | Current (name drift) | Findings apply to `llama-server` + `qwen-classifier`. The service names in it are older |
| [voidllm-evaluation.md](voidllm-evaluation.md) | Superseded by the OmniRoute migration | VoidLLM vs. LiteLLM, both moot |
docs/research/image-generation-model-choice.md
-220
View File
@@ -1,220 +0,0 @@
# Research: which diffusion model to target with the full ~32GB R9700
**Question:** With Qwen/llama-server fully stopped (per issue #39's premise — see
map #38), ComfyUI has the whole ~32GB R9700 (ROCm/gfx1201) to itself instead
of the ~6GB left over during concurrent operation
(per [`image-generation-options.md`](image-generation-options.md)). Given that
headroom, should the build stay on FLUX.1-schnell, or move up to
FLUX.1-dev, SD3.5-large, Qwen-Image, HunyuanImage-3.0, or Krea-2?
**Answer, short version:** move up to **Qwen-Image at FP8 precision**
(`qwen_image_fp8_e4m3fn.safetensors` diffusion weights +
`qwen_2.5_vl_7b_fp8_scaled.safetensors` text encoder, ~25GB combined). It's
the only one of the five upgrade candidates with **direct, hardware-specific
evidence of running on this exact GPU architecture** (gfx1201/R9700) rather
than a generic "ComfyUI supports ROCm" inference, it carries the cleanest
license of the group (Apache-2.0, no revenue threshold, no non-commercial
clause), and its 20B MMDiT is a real capability step up from schnell's
distilled 12B (notably for text rendering and prompt adherence), while still
fitting with real margin inside 32GB.
## Why not just re-derive the schnell/dev/SDXL/SD3.5 findings
`docs/research/image-generation-options.md` already covers, with primary
sources: ComfyUI's ROCm/gfx1201 story (official AMD docs + RDNA4 blog post +
community gfx1201 Docker images), FLUX.1-schnell vs FLUX.1-dev vs SDXL vs
SD3.5 licenses, and FLUX GGUF VRAM figures at the ~6GB-headroom scale. None
of that is repeated here except where the ~32GB ceiling changes the
conclusion. This doc adds: FLUX.1-dev/SD3.5 at the *larger* headroom, plus
three models the prior doc didn't cover at all (Qwen-Image, HunyuanImage-3.0,
Krea-2).
## Candidate comparison
| Model | License (primary source) | Params | Stated/typical VRAM | ROCm/gfx1201 evidence |
|---|---|---|---|---|
| FLUX.1-schnell (current) | Apache-2.0 | 12B | GGUF Q4_K_S ~7GB (per prior doc) | Confirmed on gfx1201 (prior doc) |
| **Qwen-Image** | **Apache-2.0** | 20B (20.4B DiT + 8.3B Qwen2.5-VL text encoder) | fp8 ~16GB (diffusion) + ~9.4GB (fp8 text encoder) ≈ 25GB total; bf16 needs 24GB+ and "48GB+" per some quant write-ups | **Direct**: [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) ships a pre-validated "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow specifically for the R9700 AI Pro (gfx1201), on a ROCm 7 (TheRock nightlies) toolbox |
| FLUX.1-dev | [FLUX.1-dev Non-Commercial License](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) — non-commercial weights, outputs usable commercially | 12B | bf16 ~24GB; GGUF Q8 ~12-13GB | Same *family* evidence as schnell (gfx1201 Docker images target FLUX generally), but no R9700-specific FLUX.1-dev report found |
| SD3.5-large | [Stability Community License](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md) — free under $1M annual revenue | 8B | bf16 ~16GB; 4-bit NF4 fits small GPUs | AMD's own ComfyUI-ROCm doc lists an "SD3.5 Simple" template workflow (per prior doc) — vendor-blessed but not R9700-specific |
| HunyuanImage-3.0 | [tencent-hunyuan-community license](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0) | 80B total / 13B active (MoE, 64 experts) | Official repo: "≥ 3 × 80GB" VRAM for the base model, "≥ 8 × 80GB" for -Instruct; ~177GB at fp16 | **None, and actively contraindicated**: setup requires CUDA 12.8 + FlashAttention2/FlashInfer, no AMD/ROCm mention anywhere in the official repo |
| Krea-2 (Turbo) | [Krea 2 Community License](https://www.krea.ai/krea-2-licensing) — free under $1M annual revenue, homelab/personal explicitly covered | 12-13B DiT | No official VRAM figure; community reports (RTX hardware only) cite fp8 ~16GB, GGUF ~12GB | **None found** — released [June 22, 2026 per the HF model card](https://huggingface.co/krea/Krea-2-Turbo); ComfyUI added native support per [blog.comfy.org](https://blog.comfy.org/p/krea-2-open-source-models-are-now), and GGUF quants exist ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), but no R9700/gfx1201-specific report exists yet — too new for that evidence to have accumulated |
## Per-model detail
### Qwen-Image — recommended
- **License**: Apache-2.0, stated directly on the model card, no revenue
threshold, no non-commercial clause, no attribution/naming requirement.
The cleanest license of every model considered in this doc or the prior
one. Source: [Qwen/Qwen-Image on Hugging Face](https://huggingface.co/Qwen/Qwen-Image).
- **Architecture**: 20B-parameter MMDiT (Multimodal Diffusion Transformer)
combined with an 8.3B Qwen2.5-VL text encoder — notably larger and more
capable than FLUX.1-schnell's 12B distilled model, particularly for
multilingual text rendering and instruction-following, per the official
[QwenLM/Qwen-Image GitHub repo](https://github.com/QwenLM/Qwen-Image).
- **ComfyUI support**: native, not a wrapper/custom-node integration —
landed August 2025 per [ComfyUI Wiki's native-support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image),
with an official FP8 checkpoint (`qwen_image_fp8_e4m3fn.safetensors`) and
FP8-scaled text encoder (`qwen_2.5_vl_7b_fp8_scaled.safetensors`, 9.38GB)
published under [Comfy-Org/Qwen-Image_ComfyUI](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders).
- **VRAM at FP8**: the official Comfy-Org FP8 diffusion checkpoint plus FP8
text encoder land around ~25GB combined — comfortably inside the 32GB
ceiling with headroom for ComfyUI runtime/VAE overhead, versus 24GB+ (some
sources say 48GB+) for the unquantized bf16 model. Sources: community
VRAM write-ups aggregated via [Comfy-Org/ComfyUI issue #10852 ("Qwen-image in 24GB VRAM and 32GB RAM")](https://github.com/Comfy-Org/ComfyUI/issues/10852)
and the [official ComfyUI Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/) —
treat exact GB figures as secondary/community-sourced, consistent with how
the prior doc flagged FLUX's GGUF VRAM table.
- **Direct R9700/gfx1201 evidence (the deciding factor)**:
[kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) is a
Fedora-toolbox ROCm 7 (TheRock nightlies) environment built specifically
for the "AMD Radeon 9700 AI PRO (32GB)" and ships a pre-validated
"Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow
(plus a Qwen-Image-Edit 2511 FP8 workflow) as one of only four workflows
in the whole repo. This is the only model in this comparison with
card-architecture-specific (not just "ComfyUI supports ROCm generically")
validation — everything else relies on family-level or vendor-generic
ROCm claims. A related community discussion
([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
does flag that "Qwen image edit doesn't always work on AMD HIP/ROCm" in
some configurations — worth testing the specific workflow before assuming
zero friction, but this is a known-quantity, actively-discussed rough edge
rather than a documented hard blocker.
- **4-step Lightning LoRA**: the R9700-validated workflow pairs Qwen-Image
with a "Lightning" LoRA for 4-step inference — the same fast-inference
pattern FLUX.1-schnell uses, so switching models doesn't have to mean
giving up the short GPU-resident-time-per-image property the prior doc
called out as valuable for time-sliced use alongside llama-server (even
though this ticket's premise is llama-server being stopped, that pattern
still helps if the two services are ever run in an overlapping window).
### FLUX.1-dev — solid alternative, not the pick
- License permits non-commercial use of the weights; BFL's own license page
states generated outputs are separately usable commercially (already
covered in the prior doc; unchanged here). Source:
[FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).
- At bf16 (~24GB) or GGUF Q8 (~12-13GB), it fits the 32GB ceiling with room
to spare — a legitimate move up from schnell's distilled quality.
Source: [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
and community Q8 VRAM reports.
- Loses to Qwen-Image on two counts: license (non-commercial-weights clause
vs. Apache-2.0 — not a hard blocker for this homelab per the prior doc's
own reasoning, but strictly worse) and hardware evidence (FLUX's gfx1201
validation is at the *family* level — the prior doc's `yurisasc/comfyui-rocm-rdna4`
and `charlie12345/R9700AIProComfyUIPatch` sources are about running FLUX
models on this card generally, not a FLUX.1-dev-specific report the way
kyuz0's repo is Qwen-Image-specific).
### SD3.5-large — no longer the best use of the freed headroom
- License unchanged from the prior doc: Stability Community License, free
under $1M annual revenue (irrelevant threshold for this homelab). Source:
[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
- At 8B params (bf16 ~16GB), it's the smallest of the upgrade candidates —
which mattered when 6GB was the ceiling, but with 32GB available there's
no VRAM reason to pick the model the prior doc already flagged as "lower
fidelity than FLUX/SD3.5 by current standards" over Qwen-Image or
FLUX.1-dev. AMD's own ComfyUI-ROCm docs do list it as a first-party
example template ("SD3.5 Simple", per the prior doc), so it remains a fine
fallback if Qwen-Image's FP8 path hits the ROCm rough edge noted above.
### HunyuanImage-3.0 — ruled out
- Official repo states VRAM requirements of "≥ 3 × 80GB" for the base model
and "≥ 8 × 80GB" for HunyuanImage-3.0-Instruct — i.e. multi-GPU
datacenter-class NVIDIA clusters, not a single 32GB consumer/workstation
card at any precision. Source:
[Tencent-Hunyuan/HunyuanImage-3.0 GitHub repo](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0).
- Setup instructions require CUDA 12.8, PyTorch 2.8.0 built for CUDA, and
optionally FlashAttention2/FlashInfer for MoE routing speed — no AMD or
ROCm path is mentioned anywhere in the official repo. Even the
MoE-efficient "13B active" framing doesn't help here: the tooling itself
assumes an NVIDIA multi-GPU cluster, and the 80B total parameter set still
has to be resident somewhere.
- 32GB of headroom on one AMD card doesn't move this model into reach at
any precision considered here; it's excluded regardless of how much VRAM
frees up on this specific box.
### Krea-2 (Turbo) — promising, but unverified on this hardware
- Verified directly against primary sources per the ticket's instruction
(this is a June 2026 release, past most training cutoffs): the
[Hugging Face model card](https://huggingface.co/krea/Krea-2-Turbo) states
a release date of **June 22, 2026**, a 12-billion-parameter single-stream
diffusion transformer, `torch.bfloat16` as the reference precision, and
the **Krea 2 Community License**.
- License, per [krea.ai/krea-2-licensing](https://www.krea.ai/krea-2-licensing):
non-commercial (including explicitly personal/homelab) use is free;
commercial use is permitted royalty-free for entities under $1M
trailing-12-month revenue (same shape as SD3.5's and matching this
homelab's use case); content-filter and AI-disclosure obligations apply if
deployed publicly; derivative model names must start with "Krea".
- ComfyUI added native support for both open-weight checkpoints (Krea 2 Raw
and Krea 2 Turbo) per [blog.comfy.org's announcement](https://blog.comfy.org/p/krea-2-open-source-models-are-now),
and community GGUF quants already exist
([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), with
reports (RTX hardware only) of fp8 fitting 16GB and GGUF fitting 12GB.
- **No AMD/ROCm mention anywhere** in the model card, and no gfx1201/R9700
community report was found — unsurprising given the model is roughly
2.5 months old at the time of this research. Its architecture (a
standard-shaped DiT that ComfyUI loads through its normal diffusion-model
nodes, per the ComfyUI blog post) gives reasonable expectation it will run
on the same ROCm/PyTorch backend already proven for FLUX and Qwen-Image on
this card, but that's an inference, not a verified fact the way
kyuz0's Qwen-Image workflow is.
- **Not the pick today**, precisely because Qwen-Image already offers a
hardware-verified path at a comparable parameter count and VRAM budget.
Worth a follow-up research ticket once R9700/gfx1201-specific Krea-2
reports exist — the license and ComfyUI support are both already in
place, so the only open question is real-world ROCm behavior.
## Recommendation
**Qwen-Image, FP8 precision** (`qwen_image_fp8_e4m3fn.safetensors` +
`qwen_2.5_vl_7b_fp8_scaled.safetensors`, ~25GB combined), optionally paired
with the 4-step Lightning LoRA the R9700-specific validated workflow uses.
It wins on all three axes the ticket asked about:
1. **License**: Apache-2.0 — no restriction at all, strictly better than
every other candidate including the current FLUX.1-schnell pick.
2. **ROCm/gfx1201 compatibility**: the only candidate with a workflow
pre-validated specifically on this GPU architecture
([kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy)),
not just "ComfyUI supports ROCm in general."
3. **VRAM at the ~32GB ceiling**: ~25GB at FP8 leaves real margin for
ComfyUI runtime/VAE overhead, without needing the multi-step,
non-distilled FLUX.1-dev's full 24GB bf16 footprint or accepting
SD3.5's lower fidelity ceiling — and it's a genuine capability upgrade
over schnell (20B vs. 12B, non-distilled-quality text rendering) rather
than just a bigger file.
If the known Qwen-Image/ROCm edit-mode rough edge
([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
turns out to affect plain text-to-image generation too, SD3.5-large (AMD's
own first-party "SD3.5 Simple" ComfyUI-ROCm template workflow) is the
fallback, with FLUX.1-dev as a second option. HunyuanImage-3.0 is excluded
outright regardless of available VRAM (CUDA-only tooling, multi-GPU
datacenter VRAM floor). Krea-2 is worth revisiting once R9700-specific
field reports exist.
## Sources consulted
- [docs/research/image-generation-options.md](image-generation-options.md) (this repo — prior findings, not re-derived)
- [Qwen/Qwen-Image (Hugging Face)](https://huggingface.co/Qwen/Qwen-Image)
- [QwenLM/Qwen-Image (GitHub)](https://github.com/QwenLM/Qwen-Image)
- [ComfyUI Wiki — Qwen-Image native support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image)
- [Comfy-Org/Qwen-Image_ComfyUI (Hugging Face, FP8 checkpoints)](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders)
- [Comfy-Org/ComfyUI issue #10852 — Qwen-Image VRAM](https://github.com/Comfy-Org/ComfyUI/issues/10852)
- [ComfyUI official Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/)
- [kyuz0/amd-r9700-comfy (R9700-specific ROCm ComfyUI toolbox)](https://github.com/kyuz0/amd-r9700-comfy)
- [pollockjj/ComfyUI-MultiGPU discussion #133 (Qwen-Image-Edit ROCm rough edge)](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133)
- [black-forest-labs/FLUX.1-dev (Hugging Face) + LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev)
- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
- [stabilityai/stable-diffusion-3.5-large (Hugging Face) + LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)
- [Tencent-Hunyuan/HunyuanImage-3.0 (GitHub)](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0)
- [krea/Krea-2-Turbo (Hugging Face)](https://huggingface.co/krea/Krea-2-Turbo)
- [Krea 2 Community License Agreement (krea.ai)](https://www.krea.ai/krea-2-licensing)
- [blog.comfy.org — Krea 2 open-source models in ComfyUI](https://blog.comfy.org/p/krea-2-open-source-models-are-now)
- [molbal/krea2-gguf (Hugging Face)](https://huggingface.co/molbal/krea2-gguf)
docs/research/image-generation-options.md
-249
View File
@@ -1,249 +0,0 @@
# Research: adding local image generation to the stack
**Question:** What's the best way to add local image generation alongside
the existing Qwen3.8-27B / llama.cpp text stack, given a single AMD Radeon
R9700 (32GB VRAM, ROCm/gfx1201 — not CUDA), routed through the OmniRoute
gateway on the `ai-stack` Docker network?
**Answer, short version:** run **ComfyUI** (official AMD-blessed ROCm
Docker path exists, and OmniRoute already has a first-class `comfyui`
provider — no bespoke API wrapper needed) with **FLUX.1 [schnell]**
(Apache-2.0, 4-step, GGUF-quantizable) as the default model, falling back to
**SDXL** for anything schnell's distilled-step license/quality tradeoffs
don't suit. VRAM headroom against the current llama-server footprint is too
tight for both to be resident at once at any real image quality — plan for
**time-sliced use** (llama-server's existing lazytainer stop-on-idle pattern,
mirrored for the image-gen service, or a manual "stop one, start the other"
toggle), not concurrent operation.
## Current VRAM baseline (from this repo)
Per `docker-compose.yml` and `.env.example`, llama-server runs
`Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights) at `--ctx-size 262144` with
`--cache-type-k q8_0 --cache-type-v q8_0`, landing at **~25.6 GB** total
(weights + q8_0 KV cache), leaving **~6 GB** free on the 32GB card — this
matches the math already recorded in
[`docs/research/qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md). Per the auto-memory
note on this repo, real measured VRAM use has run closer to ~75% (~24 GB) in
practice versus the theoretical estimate, which doesn't change the
conclusion below but means the ~6 GB figure is closer to a ceiling than a
comfortable number.
**Implication:** 6 GB is not enough for any current-generation image model at
usable quality (see VRAM table below — even the smallest practical FLUX
quant wants ~7 GB alone, before ComfyUI's own runtime/VAE overhead). Running
image-gen *concurrently* with llama-server resident is not realistic on this
card. The two need to time-share the GPU, not split it.
## Backend evaluation (ROCm support, checked against primary sources)
### ComfyUI — recommended
- **Official AMD ROCm docs exist and are current.** AMD's own ROCm docs site
hosts a dedicated ComfyUI install guide with a prebuilt Docker image path
(recommended) or build-from-source, listing ROCm 7.2.0 and 7.1.0 as
supported versions, explicit `--device=/dev/kfd --device=/dev/dri
--group-add video` flags (same device-passthrough pattern this repo
already uses for llama-server), and template workflows including "SD3.5
Simple". Officially the guide only names AMD Instinct
MI355X/MI325X/MI300X (datacenter cards) as supported platforms.
Source: [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html).
- **The upstream ComfyUI README itself documents AMD support directly**,
including consumer cards: stable ROCm install via
`pip install torch torchvision torchaudio --index-url
https://download.pytorch.org/whl/rocm7.2`, plus an experimental Windows
build explicitly naming **RDNA 3 (RX 7000), RDNA 3.5 (Strix Halo), and
RDNA 4 (RX 9000 series)** — i.e. the same RDNA4 generation as the R9700 —
and `HSA_OVERRIDE_GFX_VERSION` workarounds for older/unlisted cards.
Source: [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI).
- **AMD has published a specific RDNA4/RX 9000 ComfyUI guide** (separate
from the Instinct-only install page above), confirming RDNA4 consumer
cards are an explicitly supported, first-party-documented target, not just
a community workaround.
Source: [ROCm blog — Getting Started with ComfyUI on AMD Radeon RX 9000 Series GPUs](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html).
- **gfx1201 (R9700's arch) specifically has active community Docker images**:
`yurisasc/comfyui-rocm-rdna4` targets ROCm 7.1 + PyTorch 2.9.1 with
`HSA_OVERRIDE_GFX_VERSION=12.0.1` / `PYTORCH_ROCM_ARCH=gfx1201` baked in,
and there's a published community patch specifically for R9700 AI Pro +
ComfyUI video-gen speedups, evidence the card is being run today, not just
theoretically compatible.
Sources: [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4),
[charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch).
- **Known gfx1201 caveat:** AMD's own TransformerEngine repo has an open
issue confirming gfx1201 is missing from the FP8 architecture table, so
FP8 kernels silently fall back to FP32 with ~50% throughput loss
(18-22 vs. 35-40 tok/s in the reporter's LLM benchmark) — not a
correctness blocker, but relevant if planning to use FP8-quantized image
models expecting native FP8 speed on this card; GGUF/Q-quants (see below)
avoid this path entirely since they dequantize to bf16/fp16, not fp8.
Source: [ROCm/TransformerEngine issue #520](https://github.com/ROCm/TransformerEngine/issues/520).
- **Actively maintained community Docker packaging** beyond AMD's own image:
`YanWenKun/ComfyUI-Docker` ships parallel `rocm` (PyTorch-build-based,
faster releases) and `rocm7` (AMD-build-based, more comprehensive)
variants, both targeting ROCm 7, with ~1000 commits of ongoing history —
a viable alternative to the official AMD image if it lags behind ComfyUI
releases.
Source: [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker).
### AUTOMATIC1111 / Forge — usable but a step down for this hardware
- ROCm support for A1111/Forge is real but community-patched, not
first-party. The upstream `lllyasviel/stable-diffusion-webui-forge` repo's
own discussion thread on AMD support points users to
`lshqqytiger/stable-diffusion-webui-amdgpu-forge`, a community fork
specifically maintained for AMD, "regarded as the go-to version" for
running FLUX-era models on AMD — i.e. the *mainline* Forge repo does not
claim ROCm support itself; you're expected to run a fork.
Source: [lllyasviel/stable-diffusion-webui-forge discussion #67](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67).
- No first-party AMD vendor documentation (unlike ComfyUI's AMD-authored
ROCm/RDNA4 blog posts above) was found for A1111/Forge specifically.
Given ComfyUI already has an AMD-blessed path plus a first-class OmniRoute
provider (below), there's no reason to take on a community fork's
maintenance risk instead.
### InvokeAI — usable but weaker AMD story for a new-generation card
- InvokeAI documents ROCm support but flags it as second-tier: "AMD GPUs
are only supported on Linux," and "support for newer AMD GPUs is spotty
... you may experience garbled images, black images, or long startup
delays." Its own install docs reference ROCm 5.4.2-era wheels, notably
older than the ROCm 7.x this stack's llama-server image already runs on
gfx1201.
Source: [InvokeAI installation docs (mauwii mirror)](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/).
- No OpenAI-compatible-images angle either — same drawback as A1111/Forge.
Not recommended as primary given ComfyUI's stronger, more current AMD
documentation trail.
## Model choice: FLUX.1 [schnell] vs FLUX.1 [dev] vs SDXL vs SD3.5
| Model | License | Params | Notes |
|---|---|---|---|
| **FLUX.1 [schnell]** | **Apache-2.0** — fully open, no commercial restriction | 12B | Distilled for 1-4 step inference (fast); Black Forest Labs' own model card states this license directly |
| FLUX.1 [dev] | [FLUX.1-dev Non-Commercial License](https://github.com/black-forest-labs/flux/blob/main/model_licenses/LICENSE-FLUX1-dev) | 12B | Non-commercial for the *model/weights*; generated *outputs* are explicitly usable commercially per BFL's license page. Higher quality than schnell (more steps, non-distilled) but the weights themselves can't be redistributed/used commercially |
| SDXL | CreativeML OpenRAIL++ (permissive, commercial-friendly) | ~3.5B | Older (2023), lower fidelity than FLUX/SD3.5 by current standards, but lowest VRAM footprint and best long-standing tooling maturity |
| SD3.5 (Large/Medium) | Stability AI Community License — free commercial use under $1M annual revenue, else enterprise license required | 8B / 2.5B | Free for this repo's non-commercial homelab use regardless; template already listed in AMD's own ComfyUI-ROCm doc ("SD3.5 Simple") as a first-party example workflow |
Sources: [black-forest-labs/flux model cards](https://github.com/black-forest-labs/flux/blob/main/model_cards/FLUX.1-dev.md),
[black-forest-labs/FLUX.1-schnell on Hugging Face](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
(license: apache-2.0), [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md),
[Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5),
[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
**Recommendation: FLUX.1 [schnell].** For a private homelab, license
enforcement isn't the deciding factor by itself, but schnell's Apache-2.0
status removes any future ambiguity if outputs or the setup are ever shared
or repurposed, and its whole design point — good quality in 1-4 sampling
steps — directly addresses the VRAM/time-slicing constraint below (less
GPU-resident time per image than a 20-50 step dev/SDXL/SD3.5 run).
Quantized via `city96/ComfyUI-GGUF` (an actively-referenced, community-
trusted quantization node — its GGUF Q-quants dequantize to bf16/fp16 at
runtime, sidestepping the gfx1201 FP8 dequant bug above entirely), FLUX fits
in a fraction of its fp16 footprint:
| Precision | Approx. VRAM (model only) |
|---|---|
| fp16 (baseline) | ~24 GB |
| fp8 | ~12 GB |
| GGUF Q5_K_S | ~12-15 GB (practical quality floor) |
| GGUF Q4_K_S | ~7 GB (quality starts degrading on hands/text below Q4) |
Source: aggregated VRAM figures from GGUF-quantization write-ups referencing
city96's FLUX GGUF conversions — treat as secondary/community sourced
(no single BFL-published VRAM table was found), consistent across multiple
independent sources.
[city96/ComfyUI-GGUF README](https://github.com/city96/ComfyUI-GGUF/blob/main/README.md),
[city96/FLUX.1-dev-gguf model card](https://huggingface.co/city96/FLUX.1-dev-gguf).
**Fallback pick: SDXL.** If schnell's distilled quality ceiling proves too
low for some use case, SDXL is the safer second choice over FLUX.1 [dev] or
SD3.5 specifically *because* of this card's tight headroom: it's the
smallest of the four by a wide margin, has the longest production track
record on ROCm of any of these models, and its OpenRAIL++ license carries no
revenue-threshold clause to track (unlike SD3.5's Community License) or
non-commercial weight restriction (unlike FLUX.1 [dev]).
## OpenAI-compatible API / OmniRoute integration
This is the best news in this research: **OmniRoute already ships a
first-class, built-in `comfyui` provider** — not a generic "point it at an
OpenAI base URL and hope" integration. Its provider reference documents it
explicitly: *"No API key required. Configure the local ComfyUI base URL
(default: http://localhost:8188)."* OmniRoute's own image-routing feature
set (`/v1/images/generations`, `/v1/images/edits`, `/v1/images/variations`,
with automatic provider fallback) is designed for exactly this pattern:
register ComfyUI as a backend, then any client already calling OmniRoute's
OpenAI-compatible images endpoints reaches it with no extra shim.
Source: [diegosouzapw/OmniRoute PROVIDER_REFERENCE.md](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md),
[diegosouzapw/OmniRoute repo description](https://github.com/diegosouzapw/OmniRoute).
This means **ComfyUI does not need an extra OpenAI-API wrapper project** —
the wrapper projects found during this research
([`ComfyUI-OpenAI-Compatible-API`](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API))
turned out to be a ComfyUI *custom node* for calling *outbound* to LLM APIs
from within a workflow (the reverse direction), not something this stack
needs — OmniRoute's own native ComfyUI provider is the actual integration
point, one layer up.
**Confidence note:** the provider-reference detail above was fetched via an
automated summarizer against the raw doc rather than manually re-verified
line-by-line; re-check `PROVIDER_REFERENCE.md`'s `comfyui` entry directly
before wiring this up, in case ComfyUI's own `/prompt` API (a
workflow-graph-shaped API, not a simple text-prompt-in/image-out call) needs
a specific default workflow JSON configured on the OmniRoute side to produce
a plain text-to-image call.
## Integration sketch (not a full compose — see caveats above)
- New service in `docker-compose.yml`, e.g. `comfyui`, image
`rocm/comfyui-rocm` (or `yurisasc/comfyui-rocm-rdna4` for a gfx1201-tuned
build) or built from AMD's own ROCm ComfyUI Dockerfile, same
`/dev/kfd` + `/dev/dri` + `group_add: [video, render]` device-passthrough
block already used for `llama-server`, joined to the same `ai-stack`
network so `omniroute` can reach it as `http://comfyui:8188` — no host
port needed (matches the existing llama-server pattern of no published
port, gateway-only access).
- Register it in OmniRoute's dashboard as a `comfyui` provider pointing at
that internal URL, same manual-registration pattern already used for
llama-server and searxng-search per `docs/proxy-key-onboarding.md`.
- **VRAM contention is the real design problem, not networking.** Given the
~6 GB headroom, the two services can't both sit GPU-resident.
Two workable patterns, in order of how well they fit what's already in
this repo:
1. **Mirror the existing lazytainer stop-on-idle pattern** already applied
to `llama-server` (`docker-compose.yml`'s `lazytainer.group.*` labels) —
add an equivalent idle-timeout group for `comfyui`, and rely on the two
services naturally not being hit at the same time for a single-user
homelab. This doesn't *guarantee* mutual exclusion (both could still be
woken concurrently and both try to fit in 6 GB free), so it's a
reasonable-effort fit, not a hard guarantee.
2. **Explicit mutual exclusion**: a small script/compose profile that
stops `llama-server` before starting `comfyui` (and vice versa) rather
than relying on lazytainer's independent idle timers — worth doing if
the reasonable-effort version above causes a visible OOM in practice.
Either way, this is a "pick one, then the other" story, not "run both."
- Given FLUX.1 [schnell]'s 1-4 step design, a cold-start-and-generate cycle
(wake ComfyUI from lazytainer sleep, generate, let it idle back down) is
a reasonably good fit for occasional image requests through the same
gateway that already does this for llama-server.
## Sources consulted
- [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html)
- [ROCm blog — ComfyUI on AMD Radeon RX 9000 Series (RDNA4)](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html)
- [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI)
- [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker)
- [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4)
- [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch)
- [ROCm/TransformerEngine issue #520 (gfx1201 FP8 fallback)](https://github.com/ROCm/TransformerEngine/issues/520)
- [lllyasviel/stable-diffusion-webui-forge discussion #67 (AMD support)](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67)
- [InvokeAI CUDA/ROCm install docs](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/)
- [black-forest-labs/flux GitHub repo + model cards](https://github.com/black-forest-labs/flux)
- [black-forest-labs/FLUX.1-schnell (Hugging Face, Apache-2.0)](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
- [black-forest-labs/FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)
- [Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5)
- [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)
- [city96/ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF)
- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) and its `PROVIDER_REFERENCE.md`
- [yeeyou/ComfyUI-OpenAI-Compatible-API](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API) (checked and ruled out — wrong direction)
- This repo: `docker-compose.yml`, `.env.example`, `docs/research/qwen3.8-27b-quant.md`
docs/proxy-request-priority.md → docs/research/proxy-request-priority.md
View File
File renamed without changes.
scripts/switch-model.sh
-50
View File
@@ -1,50 +0,0 @@
#!/usr/bin/env bash
# Swap GPU residency between llama-server (Qwen) and comfyui — they never
# run concurrently, VRAM doesn't fit both (see issue #38's map). Manual
# invocation only, no auto-switching.
#
# Bypasses lazytainer entirely and drives docker compose directly — its
# idle-stop can't be used for this. Root cause (see
# docs/research/lazytainer-omniroute-idle-stop.md, issue #40): lazytainer's
# packet-threshold detector is source-blind and can't tell OmniRoute's
# periodic health-check pings apart from real traffic on the same port, so
# it never reliably sleeps a service on its own. A scripted swap always
# knows which service should go up/down, so it doesn't need that heuristic.
#
# OmniRoute may show the just-stopped provider as errored/offline in its
# dashboard for up to CREDENTIAL_HEALTH_CHECK_INTERVAL (default 5 min) after
# a swap — cosmetic, not a functional problem (see the research doc above).
set -euo pipefail
cd "$(dirname "$0")/.."
usage() {
echo "Usage: $0 {qwen|comfyui}" >&2
echo " qwen - stop comfyui, start llama-server" >&2
echo " comfyui - stop llama-server, start comfyui" >&2
exit 1
}
[ $# -eq 1 ] || usage
case "$1" in
qwen)
from=comfyui
to=llama-server
;;
comfyui)
from=llama-server
to=comfyui
;;
*)
usage
;;
esac
echo "==> stopping $from"
docker compose stop "$from"
echo "==> starting $to"
docker compose up -d "$to"
echo "==> status"
docker compose ps
scripts/update.sh
+8 -68
View File
@@ -1,28 +1,6 @@
#!/usr/bin/env bash
# The one command to run after any change to this repo (compose file, .env,
# or a git pull) to bring the running stack in sync. Ensures secrets/keys
# exist, pulls, validates, rebuilds/re-pulls images, and recreates only what
# changed — safe to run any time, including with nothing to do.
#
# Tunable config values (LLAMA_*, ports, timeouts — anything with a real
# default in .env.example) are synced from .env.example every run. A value
# already matching is left alone silently. A value that DIFFERS from the
# server's current .env is a conflict: interactively, you're shown every
# conflict on one screen (via gum) and choose which to accept — unpicked
# keys keep the server's current value. Non-interactively (no TTY — cron,
# CI, piped), any conflict is a hard error unless --force is passed, which
# accepts every new value automatically. Secrets and host-resolved values
# (blank in .env.example — OMNIROUTE_*_SECRET/_KEY/_SALT/_PASSWORD,
# SEARXNG_LAN_IP, HOST_VIDEO_GID/RENDER_GID) are never
# touched by this — they keep going through set_if_blank as before.
#
# omniroute's own routing/provider config (llama-server, search) lives in
# its dashboard, not a checked-in file like the old litellm-config.yaml —
# see issue #31 and docs/proxy-key-onboarding.md.
#
# ponytail: no rollback/backup logic — this is a single-user homelab box,
# not a fleet. If a bad config lands, `git revert` + re-run is the recovery
# path, not this script.
# Brings the running stack in sync with this repo; safe to re-run any time.
# What it does and why: docs/operations.md
set -euo pipefail
cd "$(dirname "$0")/.."
@@ -34,25 +12,14 @@ for arg in "$@"; do
esac
done
# Must run before anything else touches a file this script itself reads
# (docker-compose.yml, .env.example, this script's own remaining lines) —
# a self-updating script isn't guaranteed atomic against its own file
# changing mid-run, so pulling later can execute a mix of old and new
# script/compose content in one pass. Bit us for real: old GID-resolution
# code ran, then this pulled in new var names docker-compose.yml now
# requires, and nothing re-ran the (now-current) resolution step for
# them — see issue #5's thread.
# Keep this first — see docs/operations.md#self-update
echo "==> git pull"
git pull --ff-only
[ -f .env ] || cp .env.example .env
echo "==> syncing tracked config values from .env.example"
# ponytail: gum (charmbracelet/gum) is a single static binary. Vendored as
# a release tarball under scripts/vendor/ (checked into git) for the R9700
# box, which has no outbound internet access — the download fallback below
# is only for other archs / when the vendored copy is missing or stale.
# Cached under .cache/gum/ (gitignored) so repeat runs don't re-extract.
# see docs/operations.md#gum
GUM_VERSION="0.14.5"
GUM_DIR="$(pwd)/.cache/gum"
GUM_BIN="$GUM_DIR/gum"
@@ -84,10 +51,7 @@ ensure_gum() {
[ -x "$GUM_BIN" ] && echo "$GUM_BIN" || echo ""
}
# Collect every key where .env.example has a real (non-blank) default:
# missing from .env -> just add it (no conflict, nothing to decide);
# present and identical -> leave alone silently; present and different ->
# a conflict to resolve below.
# see docs/operations.md#env-sync
conflict_keys=()
conflict_old=()
conflict_new=()
@@ -128,8 +92,6 @@ if [ "${#conflict_keys[@]}" -gt 0 ]; then
if [ -n "$gum_bin" ]; then
selected="$(printf '%s\n' "${labels[@]}" | "$gum_bin" choose --no-limit --selected "$(printf '%s\n' "${labels[@]}" | paste -sd,)" --header "Config differs from .env.example — selected keys take the new value, unselected keep the server's current value:")"
else
# ponytail: plain-bash fallback if gum couldn't be fetched (offline,
# unsupported arch) — same one-screen-of-conflicts idea, cruder UI.
echo "Config differs from .env.example. Enter space-separated numbers to KEEP the server's current value (all others take the new value), or press enter to take every new value:"
for i in "${!conflict_keys[@]}"; do
echo " $((i+1))) ${labels[$i]}"
@@ -155,9 +117,6 @@ if [ "${#conflict_keys[@]}" -gt 0 ]; then
fi
fi
# Handles all three cases: the KEY=value line is missing entirely (.env
# predates that var being added to .env.example — sed can't fix what isn't
# there, so this appends it), present but blank, or already set.
set_if_blank() {
local key="$1" value="$2"
if grep -qE "^${key}=.*[^[:space:]]" .env; then
@@ -172,10 +131,7 @@ set_if_blank() {
}
echo "==> filling in missing secrets"
# Random values — safe to re-run, never overwrites what's already set.
# OMNIROUTE_STORAGE_ENCRYPTION_KEY especially: never change it after first
# run, existing encrypted data becomes unreadable if you do (same caveat as
# LiteLLM's old LITELLM_SALT_KEY).
# see docs/operations.md#secrets
set_if_blank OMNIROUTE_INITIAL_PASSWORD "$(openssl rand -hex 16)"
set_if_blank OMNIROUTE_JWT_SECRET "$(openssl rand -base64 48)"
set_if_blank OMNIROUTE_API_KEY_SECRET "$(openssl rand -hex 32)"
@@ -183,12 +139,9 @@ set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
set_if_blank NEO4J_PASSWORD "$(openssl rand -hex 16)"
echo "==> resolving SEARXNG_LAN_IP"
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
# not from inside the omniroute container (see docs/research/litellm-searxng-search.md,
# still the relevant background even though omniroute replaced litellm — see issue #31).
# see docs/gateway.md#web-search
searxng_ip="$(getent hosts search.home 2>/dev/null | awk '{print $1}' | head -1)"
if [ -n "$searxng_ip" ]; then
set_if_blank SEARXNG_LAN_IP "$searxng_ip"
@@ -197,11 +150,7 @@ else
fi
echo "==> resolving host video/render GIDs (shared by every GPU service)"
# Numeric GIDs, not names, in docker-compose.yml's group_add: — Docker
# resolves a *named* group_add entry against the container's own /etc/group,
# not the host's, and fails unpredictably (worse with multiple GPU services
# starting concurrently and racing on the same lookup) — see
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5.
# see docs/architecture.md#gpu-access
video_gid="$(getent group video 2>/dev/null | cut -d: -f3)"
render_gid="$(getent group render 2>/dev/null | cut -d: -f3)"
if [ -n "$video_gid" ]; then
@@ -227,19 +176,10 @@ docker compose build --pull
echo "==> ensuring models are downloaded (skips already-present files)"
docker compose --profile tools run --rm downloader
docker compose --profile tools run --rm downloader-classifier
docker compose --profile tools run --rm downloader-comfyui
echo "==> bringing up omniroute"
docker compose up -d --wait omniroute
# ponytail: no scripted key-minting yet, unlike the old LiteLLM /key/generate
# flow — omniroute's POST /api/keys needs a dashboard login session
# (ManagementSessionAuth), not a static bearer key, and that flow hasn't
# been verified against a live instance (see issue #37). No in-stack
# workload needs a key right now (nothing left calls the gateway besides
# coding CLIs, which mint their own by hand per docs/proxy-key-onboarding.md)
# — revisit this script once that flow is automatable.
echo "==> recreating changed services"
docker compose up -d --remove-orphans