chore: remove image generation, Open WebUI, Lazytainer and neo4j
Drops the comfyui, downloader-comfyui, open-webui, lazytainer and neo4j services with their volumes, env keys and update.sh steps, plus scripts/switch-model.sh and the two image-generation research docs. Refs #55 (removal inventory #58, intent vs. source #63) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed
1 parent
8ae32102f9
commit
fd55ce6669
6 files changed
+4
-739
No files matched your search
+3
-34
@@ -48,10 +48,6 @@ LLAMA_PARALLEL=2
|
|||||||
# qwen-classifier service comment for the why and the VRAM/context math.
|
# qwen-classifier service comment for the why and the VRAM/context math.
|
||||||
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
|
LLAMA_CLASSIFIER_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf
|
||||||
|
|
||||||
# --- Lazytainer ---
|
|
||||||
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
|
||||||
LAZYTAINER_INACTIVE_TIMEOUT=900
|
|
||||||
|
|
||||||
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
|
# --- SearXNG web search (see docs/research/litellm-searxng-search.md) ---
|
||||||
# Resolved automatically by ./scripts/update.sh from search.home on this
|
# Resolved automatically by ./scripts/update.sh from search.home on this
|
||||||
# host — leave blank. Only set by hand if that resolution fails (e.g.
|
# host — leave blank. Only set by hand if that resolution fails (e.g.
|
||||||
@@ -100,9 +96,9 @@ OMNIROUTE_WS_BRIDGE_SECRET=
|
|||||||
# Mint them by hand in the dashboard, add a KEY=value line here per workload
|
# Mint them by hand in the dashboard, add a KEY=value line here per workload
|
||||||
# as you onboard one. See docs/proxy-key-onboarding.md.
|
# as you onboard one. See docs/proxy-key-onboarding.md.
|
||||||
|
|
||||||
# --- ComfyUI (local image generation, see issue #38 wayfinder map) ---
|
# --- GPU access (group_add GIDs) ---
|
||||||
# Shared by every GPU-touching service (llama-server, llama-server-fast,
|
# Shared by every GPU-touching service (llama-server, qwen-classifier)
|
||||||
# comfyui) for group_add: — resolved to real host GIDs by ./scripts/update.sh
|
# for group_add: — resolved to real host GIDs by ./scripts/update.sh
|
||||||
# rather than left as plain group names in docker-compose.yml, because Docker
|
# rather than left as plain group names in docker-compose.yml, because Docker
|
||||||
# resolves a *named* group_add entry against the container's own /etc/group,
|
# resolves a *named* group_add entry against the container's own /etc/group,
|
||||||
# not the host's, and fails unpredictably when the image doesn't define one
|
# not the host's, and fails unpredictably when the image doesn't define one
|
||||||
@@ -110,30 +106,3 @@ OMNIROUTE_WS_BRIDGE_SECRET=
|
|||||||
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
|
# docs/research/rocm-gpu-pin-and-render-group.md and issue #5). Leave blank.
|
||||||
HOST_VIDEO_GID=
|
HOST_VIDEO_GID=
|
||||||
HOST_RENDER_GID=
|
HOST_RENDER_GID=
|
||||||
|
|
||||||
# --- ComfyUI diffusion model (Qwen-Image 2.1, uncensored GGUF — see
|
|
||||||
# docs/research/image-generation-model-choice.md and issue #42) ---
|
|
||||||
# Three files, all from abenzerps/Qwen-Image-2.1-Uncensored-GGUF on
|
|
||||||
# HuggingFace: GGUF diffusion weights, safetensors text encoder, safetensors
|
|
||||||
# VAE. Q4_K_M is the repo's own recommended size/quality balance (~4.6GB);
|
|
||||||
# the int8 text encoder trades some quality for ~8GB less RAM than bf16.
|
|
||||||
#
|
|
||||||
# Requires the ComfyUI-GGUF custom node (github.com/city96/ComfyUI-GGUF) —
|
|
||||||
# NOT confirmed bundled in yurisasc/comfyui-rocm7.1. If the comfyui
|
|
||||||
# container doesn't show a "Unet Loader (GGUF)" node, `git clone` it into
|
|
||||||
# the image's custom_nodes dir and reinstall its requirements.
|
|
||||||
COMFYUI_DIFFUSION_MODEL_FILE=qwen-image-2.1-UC-Q4_K_M.gguf
|
|
||||||
COMFYUI_TEXT_ENCODER_FILE=qwen3vl_8b_int8_convrot.safetensors
|
|
||||||
COMFYUI_VAE_FILE=qwen_image_2.1_vae_bf16.safetensors
|
|
||||||
|
|
||||||
# --- Open WebUI (chat + ComfyUI image-gen frontend) ---
|
|
||||||
# Chat backend goes through omniroute like every other client — mint this
|
|
||||||
# key by hand in the omniroute dashboard per docs/proxy-key-onboarding.md
|
|
||||||
# (label it openwebui-chat), no scripted mint yet. Image generation talks
|
|
||||||
# to comfyui directly, no key needed there.
|
|
||||||
OPENWEBUI_OMNIROUTE_KEY=
|
|
||||||
|
|
||||||
# --- RAG databases (qdrant + neo4j, see wayfinder notes) ---
|
|
||||||
# No auth on qdrant (its default) — same trust boundary as llama-server:
|
|
||||||
# ai-stack is not exposed off-box. Random, filled in automatically:
|
|
||||||
NEO4J_PASSWORD=
|
|
||||||
+1
-184
@@ -47,13 +47,6 @@ services:
|
|||||||
- "8080"
|
- "8080"
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
networks: [ai-stack]
|
networks: [ai-stack]
|
||||||
labels:
|
|
||||||
# ponytail: idle-timeout tuning lives here, not in a separate lazytainer config file —
|
|
||||||
# one place to look. Raise LAZYTAINER_INACTIVE_TIMEOUT if 15 min proves too eager.
|
|
||||||
- "lazytainer.group.llamaserver.sleepMethod=stop"
|
|
||||||
- "lazytainer.group.llamaserver.ports=8080"
|
|
||||||
- "lazytainer.group.llamaserver.inactiveTimeout=${LAZYTAINER_INACTIVE_TIMEOUT:-900}"
|
|
||||||
- "lazytainer.group.llamaserver.minPacketThreshold=2"
|
|
||||||
|
|
||||||
# Dedicated backend for qwen-code's tool-call harmfulness classifier
|
# Dedicated backend for qwen-code's tool-call harmfulness classifier
|
||||||
# (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
|
# (fastModel in ~/.qwen/settings.json). Was aliased onto llama-server's own
|
||||||
@@ -168,105 +161,6 @@ services:
|
|||||||
curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
curl -L --fail --create-dirs -o /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||||
https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507-GGUF/resolve/main/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf}
|
||||||
|
|
||||||
# Fetches the three Qwen-Image 2.1 (uncensored GGUF) files ComfyUI needs
|
|
||||||
# (diffusion model, text encoder, VAE) — same test -f guard pattern as
|
|
||||||
# downloader above. See docs/research/image-generation-model-choice.md and
|
|
||||||
# issue #42.
|
|
||||||
#
|
|
||||||
# GGUF weights go in models/unet (ComfyUI-GGUF's own convention), not
|
|
||||||
# models/diffusion_models (that was the old FP8-safetensors layout) — see
|
|
||||||
# github.com/city96/ComfyUI-GGUF. Text encoder/VAE stay safetensors, loaded
|
|
||||||
# by ComfyUI's normal CLIPLoader/VAELoader, so their folders are unchanged.
|
|
||||||
#
|
|
||||||
# ponytail: target paths otherwise assume ComfyUI's standard models/ layout
|
|
||||||
# under BASE_STORAGE_PATH (/storage) — same "not independently confirmed
|
|
||||||
# against the image's Dockerfile" caveat flagged on the comfyui service
|
|
||||||
# below. If ComfyUI doesn't pick these up, check its actual models root
|
|
||||||
# first.
|
|
||||||
downloader-comfyui:
|
|
||||||
image: curlimages/curl:latest
|
|
||||||
profiles: ["tools"]
|
|
||||||
user: root
|
|
||||||
volumes:
|
|
||||||
- comfyui-data:/storage
|
|
||||||
entrypoint: ["sh", "-c"]
|
|
||||||
command:
|
|
||||||
- >
|
|
||||||
mkdir -p /storage/models/unet /storage/models/text_encoders /storage/models/vae &&
|
|
||||||
(test -f /storage/models/unet/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen-image-2.1-UC-Q4_K_M.gguf} &&
|
|
||||||
echo "diffusion model already downloaded, skipping" ||
|
|
||||||
curl -L --fail --create-dirs -o /storage/models/unet/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen-image-2.1-UC-Q4_K_M.gguf}
|
|
||||||
https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF/resolve/main/${COMFYUI_DIFFUSION_MODEL_FILE:-qwen-image-2.1-UC-Q4_K_M.gguf}) &&
|
|
||||||
(test -f /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen3vl_8b_int8_convrot.safetensors} &&
|
|
||||||
echo "text encoder already downloaded, skipping" ||
|
|
||||||
curl -L --fail --create-dirs -o /storage/models/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen3vl_8b_int8_convrot.safetensors}
|
|
||||||
https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF/resolve/main/text_encoders/${COMFYUI_TEXT_ENCODER_FILE:-qwen3vl_8b_int8_convrot.safetensors}) &&
|
|
||||||
(test -f /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_2.1_vae_bf16.safetensors} &&
|
|
||||||
echo "vae already downloaded, skipping" ||
|
|
||||||
curl -L --fail --create-dirs -o /storage/models/vae/${COMFYUI_VAE_FILE:-qwen_image_2.1_vae_bf16.safetensors}
|
|
||||||
https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF/resolve/main/vae/${COMFYUI_VAE_FILE:-qwen_image_2.1_vae_bf16.safetensors})
|
|
||||||
|
|
||||||
# Local image generation — see issue #38 (wayfinder map). yurisasc's image
|
|
||||||
# is gfx1201-tuned specifically (R9700's arch), unlike the official/AMD
|
|
||||||
# ComfyUI image which doesn't pin RDNA4 support — see
|
|
||||||
# docs/research/image-generation-options.md.
|
|
||||||
#
|
|
||||||
# Model is Qwen-Image 2.1 GGUF (uncensored variant, see .env.example) —
|
|
||||||
# needs the ComfyUI-GGUF custom node (github.com/city96/ComfyUI-GGUF) for
|
|
||||||
# its "Unet Loader (GGUF)" node. Not confirmed pre-installed in this image;
|
|
||||||
# if missing, exec in and `git clone` it into custom_nodes, then pip
|
|
||||||
# install its requirements.txt and restart the container.
|
|
||||||
comfyui:
|
|
||||||
image: yurisasc/comfyui-rocm7.1:latest
|
|
||||||
container_name: comfyui
|
|
||||||
devices:
|
|
||||||
- /dev/kfd
|
|
||||||
- /dev/dri
|
|
||||||
# Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
|
|
||||||
# .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
|
|
||||||
group_add:
|
|
||||||
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
|
|
||||||
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
|
|
||||||
security_opt:
|
|
||||||
- seccomp=unconfined
|
|
||||||
ipc: host
|
|
||||||
environment:
|
|
||||||
- HSA_OVERRIDE_GFX_VERSION=12.0.1
|
|
||||||
- PYTORCH_ROCM_ARCH=gfx1201
|
|
||||||
volumes:
|
|
||||||
- comfyui-data:/storage
|
|
||||||
# yurisasc/comfyui-rocm7.1 has no entrypoint/CMD of its own (just drops
|
|
||||||
# into bash) and doesn't read PUID/PGID/VIDEO_GID/RENDER_GID or
|
|
||||||
# BASE_STORAGE_PATH — confirmed by inspecting the image directly, so
|
|
||||||
# this command is required, not optional, or the container exits 0
|
|
||||||
# immediately and restart-loops. Runs as root (image's only user);
|
|
||||||
# group_add above is what actually grants /dev/kfd + /dev/dri access.
|
|
||||||
#
|
|
||||||
# --base-directory fully replaces ComfyUI's custom_nodes/input/output/
|
|
||||||
# user/temp dirs with ones under /storage (confirmed in folder_paths.py)
|
|
||||||
# rather than adding to the image's baked-in ones — so first boot must
|
|
||||||
# seed /storage/custom_nodes from the image's copy (ComfyUI-Manager
|
|
||||||
# lives there) or it's gone, and main.py hard-crashes if custom_nodes
|
|
||||||
# doesn't exist yet at all.
|
|
||||||
entrypoint: ["sh", "-c"]
|
|
||||||
command:
|
|
||||||
- >
|
|
||||||
mkdir -p /storage/custom_nodes /storage/input /storage/output /storage/user &&
|
|
||||||
[ -z "$(ls -A /storage/custom_nodes)" ] && cp -r /workspace/ComfyUI/custom_nodes/. /storage/custom_nodes/ || true;
|
|
||||||
exec python main.py --listen 0.0.0.0 --port 8188 --base-directory /storage
|
|
||||||
#
|
|
||||||
# Published host port (unlike llama-server's ai-stack-only pattern):
|
|
||||||
# ComfyUI's own UI is meant to be reachable directly too, for a planned
|
|
||||||
# external nginx reverse-proxy route to comfy.home — not just through
|
|
||||||
# OmniRoute. Still also reachable at http://comfyui:8188 internally on
|
|
||||||
# ai-stack, which is the URL to register as OmniRoute's comfyui
|
|
||||||
# provider (dashboard or POST /api/providers, per docs/proxy-key-onboarding.md
|
|
||||||
# — same undocumented-in-repo manual flow already used for llama-server).
|
|
||||||
ports:
|
|
||||||
- "8138:8188"
|
|
||||||
restart: unless-stopped
|
|
||||||
networks: [ai-stack]
|
|
||||||
|
|
||||||
# Replaces litellm — see issue #31 (wayfinder map) for the full migration
|
# Replaces litellm — see issue #31 (wayfinder map) for the full migration
|
||||||
# rationale/findings. No static config.yaml equivalent: provider routing
|
# rationale/findings. No static config.yaml equivalent: provider routing
|
||||||
# (llama-server, searxng-search) is registered once through the dashboard
|
# (llama-server, searxng-search) is registered once through the dashboard
|
||||||
@@ -349,28 +243,8 @@ services:
|
|||||||
retries: 3
|
retries: 3
|
||||||
start_period: 40s
|
start_period: 40s
|
||||||
|
|
||||||
lazytainer:
|
|
||||||
image: ghcr.io/vmorganp/lazytainer:master
|
|
||||||
container_name: lazytainer
|
|
||||||
# NOT network_mode: host — lazytainer identifies its own container by
|
|
||||||
# matching os.Hostname() against the Docker container-ID list
|
|
||||||
# (vmorganp/Lazytainer, configureFromLabels()); under host networking the
|
|
||||||
# container inherits the host's hostname instead of its own ID, so that
|
|
||||||
# match always fails and it panics with "Could not determine container ID
|
|
||||||
# of lazytainer" on every start. Host networking also can't see traffic
|
|
||||||
# to llama-server:8080 anyway — that port only exists on the ai-stack
|
|
||||||
# bridge network (no host port published, see issue #15 above). Joining
|
|
||||||
# ai-stack instead fixes both: hostname becomes the real container ID,
|
|
||||||
# and it's on the same network as the traffic it's watching.
|
|
||||||
networks: [ai-stack]
|
|
||||||
volumes:
|
|
||||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
|
||||||
restart: unless-stopped
|
|
||||||
depends_on:
|
|
||||||
- llama-server
|
|
||||||
|
|
||||||
# RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
|
# RAG vector store — see docs/agents/... (wayfinder). Dashboard UI published
|
||||||
# directly like comfyui above, not gatewayed through omniroute (it isn't an
|
# directly, not gatewayed through omniroute (it isn't an
|
||||||
# LLM provider).
|
# LLM provider).
|
||||||
qdrant:
|
qdrant:
|
||||||
image: qdrant/qdrant:latest
|
image: qdrant/qdrant:latest
|
||||||
@@ -382,67 +256,10 @@ services:
|
|||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
networks: [ai-stack]
|
networks: [ai-stack]
|
||||||
|
|
||||||
# RAG graph store, native vector index too (can absorb qdrant's job later
|
|
||||||
# if the two-DB split proves unnecessary — see wayfinder notes).
|
|
||||||
neo4j:
|
|
||||||
image: neo4j:5-community
|
|
||||||
container_name: neo4j
|
|
||||||
environment:
|
|
||||||
- NEO4J_AUTH=neo4j/${NEO4J_PASSWORD:?run scripts/update.sh first to resolve this}
|
|
||||||
volumes:
|
|
||||||
- neo4j-data:/data
|
|
||||||
ports:
|
|
||||||
- "7474:7474" # browser UI
|
|
||||||
- "7687:7687" # bolt
|
|
||||||
restart: unless-stopped
|
|
||||||
networks: [ai-stack]
|
|
||||||
|
|
||||||
# Chat + image-generation frontend. Talks to ComfyUI directly for images —
|
|
||||||
# NOT through omniroute: OmniRoute's built-in `comfyui` provider hardcodes
|
|
||||||
# a CheckpointLoaderSimple (single-file-checkpoint) workflow in
|
|
||||||
# open-sse/handlers/imageGeneration/providers/comfyUI.ts with no override,
|
|
||||||
# so it can't drive Qwen-Image 2.1's split unet/text-encoder/vae GGUF
|
|
||||||
# loader chain. Open WebUI's ComfyUI backend is fully generic instead —
|
|
||||||
# you upload the exact workflow_api.json exported from ComfyUI's own
|
|
||||||
# graph editor and map arbitrary node IDs to parameters.
|
|
||||||
#
|
|
||||||
# Chat backend still goes through omniroute (OpenAI-compatible), same
|
|
||||||
# gateway every other client uses — mint OPENWEBUI_OMNIROUTE_KEY by hand
|
|
||||||
# per docs/proxy-key-onboarding.md, same as any other workload.
|
|
||||||
#
|
|
||||||
# ponytail: only wires the base ComfyUI URL + chat backend via env vars.
|
|
||||||
# COMFYUI_WORKFLOW/COMFYUI_WORKFLOW_NODES env vars exist but are reported
|
|
||||||
# flaky upstream — set the actual workflow JSON by hand in Settings >
|
|
||||||
# Images in the web UI instead, once you've exported it from ComfyUI.
|
|
||||||
open-webui:
|
|
||||||
image: ghcr.io/open-webui/open-webui:main
|
|
||||||
container_name: open-webui
|
|
||||||
depends_on:
|
|
||||||
comfyui:
|
|
||||||
condition: service_started
|
|
||||||
omniroute:
|
|
||||||
condition: service_started
|
|
||||||
environment:
|
|
||||||
- OPENAI_API_BASE_URL=http://omniroute:${OMNIROUTE_API_PORT:-20129}/v1
|
|
||||||
- OPENAI_API_KEY=${OPENWEBUI_OMNIROUTE_KEY:-}
|
|
||||||
- ENABLE_IMAGE_GENERATION=True
|
|
||||||
- IMAGE_GENERATION_ENGINE=comfyui
|
|
||||||
- COMFYUI_BASE_URL=http://comfyui:8188
|
|
||||||
- WEBUI_AUTH=True
|
|
||||||
volumes:
|
|
||||||
- open-webui-data:/app/backend/data
|
|
||||||
ports:
|
|
||||||
- "8008:8080"
|
|
||||||
restart: unless-stopped
|
|
||||||
networks: [ai-stack]
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
ai-stack:
|
ai-stack:
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
models:
|
models:
|
||||||
omniroute-data:
|
omniroute-data:
|
||||||
comfyui-data:
|
|
||||||
qdrant-data:
|
qdrant-data:
|
||||||
neo4j-data:
|
|
||||||
open-webui-data:
|
|
||||||
@@ -1,220 +0,0 @@
|
|||||||
# Research: which diffusion model to target with the full ~32GB R9700
|
|
||||||
|
|
||||||
**Question:** With Qwen/llama-server fully stopped (per issue #39's premise — see
|
|
||||||
map #38), ComfyUI has the whole ~32GB R9700 (ROCm/gfx1201) to itself instead
|
|
||||||
of the ~6GB left over during concurrent operation
|
|
||||||
(per [`image-generation-options.md`](image-generation-options.md)). Given that
|
|
||||||
headroom, should the build stay on FLUX.1-schnell, or move up to
|
|
||||||
FLUX.1-dev, SD3.5-large, Qwen-Image, HunyuanImage-3.0, or Krea-2?
|
|
||||||
|
|
||||||
**Answer, short version:** move up to **Qwen-Image at FP8 precision**
|
|
||||||
(`qwen_image_fp8_e4m3fn.safetensors` diffusion weights +
|
|
||||||
`qwen_2.5_vl_7b_fp8_scaled.safetensors` text encoder, ~25GB combined). It's
|
|
||||||
the only one of the five upgrade candidates with **direct, hardware-specific
|
|
||||||
evidence of running on this exact GPU architecture** (gfx1201/R9700) rather
|
|
||||||
than a generic "ComfyUI supports ROCm" inference, it carries the cleanest
|
|
||||||
license of the group (Apache-2.0, no revenue threshold, no non-commercial
|
|
||||||
clause), and its 20B MMDiT is a real capability step up from schnell's
|
|
||||||
distilled 12B (notably for text rendering and prompt adherence), while still
|
|
||||||
fitting with real margin inside 32GB.
|
|
||||||
|
|
||||||
## Why not just re-derive the schnell/dev/SDXL/SD3.5 findings
|
|
||||||
|
|
||||||
`docs/research/image-generation-options.md` already covers, with primary
|
|
||||||
sources: ComfyUI's ROCm/gfx1201 story (official AMD docs + RDNA4 blog post +
|
|
||||||
community gfx1201 Docker images), FLUX.1-schnell vs FLUX.1-dev vs SDXL vs
|
|
||||||
SD3.5 licenses, and FLUX GGUF VRAM figures at the ~6GB-headroom scale. None
|
|
||||||
of that is repeated here except where the ~32GB ceiling changes the
|
|
||||||
conclusion. This doc adds: FLUX.1-dev/SD3.5 at the *larger* headroom, plus
|
|
||||||
three models the prior doc didn't cover at all (Qwen-Image, HunyuanImage-3.0,
|
|
||||||
Krea-2).
|
|
||||||
|
|
||||||
## Candidate comparison
|
|
||||||
|
|
||||||
| Model | License (primary source) | Params | Stated/typical VRAM | ROCm/gfx1201 evidence |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| FLUX.1-schnell (current) | Apache-2.0 | 12B | GGUF Q4_K_S ~7GB (per prior doc) | Confirmed on gfx1201 (prior doc) |
|
|
||||||
| **Qwen-Image** | **Apache-2.0** | 20B (20.4B DiT + 8.3B Qwen2.5-VL text encoder) | fp8 ~16GB (diffusion) + ~9.4GB (fp8 text encoder) ≈ 25GB total; bf16 needs 24GB+ and "48GB+" per some quant write-ups | **Direct**: [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) ships a pre-validated "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow specifically for the R9700 AI Pro (gfx1201), on a ROCm 7 (TheRock nightlies) toolbox |
|
|
||||||
| FLUX.1-dev | [FLUX.1-dev Non-Commercial License](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) — non-commercial weights, outputs usable commercially | 12B | bf16 ~24GB; GGUF Q8 ~12-13GB | Same *family* evidence as schnell (gfx1201 Docker images target FLUX generally), but no R9700-specific FLUX.1-dev report found |
|
|
||||||
| SD3.5-large | [Stability Community License](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md) — free under $1M annual revenue | 8B | bf16 ~16GB; 4-bit NF4 fits small GPUs | AMD's own ComfyUI-ROCm doc lists an "SD3.5 Simple" template workflow (per prior doc) — vendor-blessed but not R9700-specific |
|
|
||||||
| HunyuanImage-3.0 | [tencent-hunyuan-community license](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0) | 80B total / 13B active (MoE, 64 experts) | Official repo: "≥ 3 × 80GB" VRAM for the base model, "≥ 8 × 80GB" for -Instruct; ~177GB at fp16 | **None, and actively contraindicated**: setup requires CUDA 12.8 + FlashAttention2/FlashInfer, no AMD/ROCm mention anywhere in the official repo |
|
|
||||||
| Krea-2 (Turbo) | [Krea 2 Community License](https://www.krea.ai/krea-2-licensing) — free under $1M annual revenue, homelab/personal explicitly covered | 12-13B DiT | No official VRAM figure; community reports (RTX hardware only) cite fp8 ~16GB, GGUF ~12GB | **None found** — released [June 22, 2026 per the HF model card](https://huggingface.co/krea/Krea-2-Turbo); ComfyUI added native support per [blog.comfy.org](https://blog.comfy.org/p/krea-2-open-source-models-are-now), and GGUF quants exist ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), but no R9700/gfx1201-specific report exists yet — too new for that evidence to have accumulated |
|
|
||||||
|
|
||||||
## Per-model detail
|
|
||||||
|
|
||||||
### Qwen-Image — recommended
|
|
||||||
|
|
||||||
- **License**: Apache-2.0, stated directly on the model card, no revenue
|
|
||||||
threshold, no non-commercial clause, no attribution/naming requirement.
|
|
||||||
The cleanest license of every model considered in this doc or the prior
|
|
||||||
one. Source: [Qwen/Qwen-Image on Hugging Face](https://huggingface.co/Qwen/Qwen-Image).
|
|
||||||
- **Architecture**: 20B-parameter MMDiT (Multimodal Diffusion Transformer)
|
|
||||||
combined with an 8.3B Qwen2.5-VL text encoder — notably larger and more
|
|
||||||
capable than FLUX.1-schnell's 12B distilled model, particularly for
|
|
||||||
multilingual text rendering and instruction-following, per the official
|
|
||||||
[QwenLM/Qwen-Image GitHub repo](https://github.com/QwenLM/Qwen-Image).
|
|
||||||
- **ComfyUI support**: native, not a wrapper/custom-node integration —
|
|
||||||
landed August 2025 per [ComfyUI Wiki's native-support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image),
|
|
||||||
with an official FP8 checkpoint (`qwen_image_fp8_e4m3fn.safetensors`) and
|
|
||||||
FP8-scaled text encoder (`qwen_2.5_vl_7b_fp8_scaled.safetensors`, 9.38GB)
|
|
||||||
published under [Comfy-Org/Qwen-Image_ComfyUI](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders).
|
|
||||||
- **VRAM at FP8**: the official Comfy-Org FP8 diffusion checkpoint plus FP8
|
|
||||||
text encoder land around ~25GB combined — comfortably inside the 32GB
|
|
||||||
ceiling with headroom for ComfyUI runtime/VAE overhead, versus 24GB+ (some
|
|
||||||
sources say 48GB+) for the unquantized bf16 model. Sources: community
|
|
||||||
VRAM write-ups aggregated via [Comfy-Org/ComfyUI issue #10852 ("Qwen-image in 24GB VRAM and 32GB RAM")](https://github.com/Comfy-Org/ComfyUI/issues/10852)
|
|
||||||
and the [official ComfyUI Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/) —
|
|
||||||
treat exact GB figures as secondary/community-sourced, consistent with how
|
|
||||||
the prior doc flagged FLUX's GGUF VRAM table.
|
|
||||||
- **Direct R9700/gfx1201 evidence (the deciding factor)**:
|
|
||||||
[kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) is a
|
|
||||||
Fedora-toolbox ROCm 7 (TheRock nightlies) environment built specifically
|
|
||||||
for the "AMD Radeon 9700 AI PRO (32GB)" and ships a pre-validated
|
|
||||||
"Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow
|
|
||||||
(plus a Qwen-Image-Edit 2511 FP8 workflow) as one of only four workflows
|
|
||||||
in the whole repo. This is the only model in this comparison with
|
|
||||||
card-architecture-specific (not just "ComfyUI supports ROCm generically")
|
|
||||||
validation — everything else relies on family-level or vendor-generic
|
|
||||||
ROCm claims. A related community discussion
|
|
||||||
([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
|
|
||||||
does flag that "Qwen image edit doesn't always work on AMD HIP/ROCm" in
|
|
||||||
some configurations — worth testing the specific workflow before assuming
|
|
||||||
zero friction, but this is a known-quantity, actively-discussed rough edge
|
|
||||||
rather than a documented hard blocker.
|
|
||||||
- **4-step Lightning LoRA**: the R9700-validated workflow pairs Qwen-Image
|
|
||||||
with a "Lightning" LoRA for 4-step inference — the same fast-inference
|
|
||||||
pattern FLUX.1-schnell uses, so switching models doesn't have to mean
|
|
||||||
giving up the short GPU-resident-time-per-image property the prior doc
|
|
||||||
called out as valuable for time-sliced use alongside llama-server (even
|
|
||||||
though this ticket's premise is llama-server being stopped, that pattern
|
|
||||||
still helps if the two services are ever run in an overlapping window).
|
|
||||||
|
|
||||||
### FLUX.1-dev — solid alternative, not the pick
|
|
||||||
|
|
||||||
- License permits non-commercial use of the weights; BFL's own license page
|
|
||||||
states generated outputs are separately usable commercially (already
|
|
||||||
covered in the prior doc; unchanged here). Source:
|
|
||||||
[FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).
|
|
||||||
- At bf16 (~24GB) or GGUF Q8 (~12-13GB), it fits the 32GB ceiling with room
|
|
||||||
to spare — a legitimate move up from schnell's distilled quality.
|
|
||||||
Source: [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
|
|
||||||
and community Q8 VRAM reports.
|
|
||||||
- Loses to Qwen-Image on two counts: license (non-commercial-weights clause
|
|
||||||
vs. Apache-2.0 — not a hard blocker for this homelab per the prior doc's
|
|
||||||
own reasoning, but strictly worse) and hardware evidence (FLUX's gfx1201
|
|
||||||
validation is at the *family* level — the prior doc's `yurisasc/comfyui-rocm-rdna4`
|
|
||||||
and `charlie12345/R9700AIProComfyUIPatch` sources are about running FLUX
|
|
||||||
models on this card generally, not a FLUX.1-dev-specific report the way
|
|
||||||
kyuz0's repo is Qwen-Image-specific).
|
|
||||||
|
|
||||||
### SD3.5-large — no longer the best use of the freed headroom
|
|
||||||
|
|
||||||
- License unchanged from the prior doc: Stability Community License, free
|
|
||||||
under $1M annual revenue (irrelevant threshold for this homelab). Source:
|
|
||||||
[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
|
|
||||||
- At 8B params (bf16 ~16GB), it's the smallest of the upgrade candidates —
|
|
||||||
which mattered when 6GB was the ceiling, but with 32GB available there's
|
|
||||||
no VRAM reason to pick the model the prior doc already flagged as "lower
|
|
||||||
fidelity than FLUX/SD3.5 by current standards" over Qwen-Image or
|
|
||||||
FLUX.1-dev. AMD's own ComfyUI-ROCm docs do list it as a first-party
|
|
||||||
example template ("SD3.5 Simple", per the prior doc), so it remains a fine
|
|
||||||
fallback if Qwen-Image's FP8 path hits the ROCm rough edge noted above.
|
|
||||||
|
|
||||||
### HunyuanImage-3.0 — ruled out
|
|
||||||
|
|
||||||
- Official repo states VRAM requirements of "≥ 3 × 80GB" for the base model
|
|
||||||
and "≥ 8 × 80GB" for HunyuanImage-3.0-Instruct — i.e. multi-GPU
|
|
||||||
datacenter-class NVIDIA clusters, not a single 32GB consumer/workstation
|
|
||||||
card at any precision. Source:
|
|
||||||
[Tencent-Hunyuan/HunyuanImage-3.0 GitHub repo](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0).
|
|
||||||
- Setup instructions require CUDA 12.8, PyTorch 2.8.0 built for CUDA, and
|
|
||||||
optionally FlashAttention2/FlashInfer for MoE routing speed — no AMD or
|
|
||||||
ROCm path is mentioned anywhere in the official repo. Even the
|
|
||||||
MoE-efficient "13B active" framing doesn't help here: the tooling itself
|
|
||||||
assumes an NVIDIA multi-GPU cluster, and the 80B total parameter set still
|
|
||||||
has to be resident somewhere.
|
|
||||||
- 32GB of headroom on one AMD card doesn't move this model into reach at
|
|
||||||
any precision considered here; it's excluded regardless of how much VRAM
|
|
||||||
frees up on this specific box.
|
|
||||||
|
|
||||||
### Krea-2 (Turbo) — promising, but unverified on this hardware
|
|
||||||
|
|
||||||
- Verified directly against primary sources per the ticket's instruction
|
|
||||||
(this is a June 2026 release, past most training cutoffs): the
|
|
||||||
[Hugging Face model card](https://huggingface.co/krea/Krea-2-Turbo) states
|
|
||||||
a release date of **June 22, 2026**, a 12-billion-parameter single-stream
|
|
||||||
diffusion transformer, `torch.bfloat16` as the reference precision, and
|
|
||||||
the **Krea 2 Community License**.
|
|
||||||
- License, per [krea.ai/krea-2-licensing](https://www.krea.ai/krea-2-licensing):
|
|
||||||
non-commercial (including explicitly personal/homelab) use is free;
|
|
||||||
commercial use is permitted royalty-free for entities under $1M
|
|
||||||
trailing-12-month revenue (same shape as SD3.5's and matching this
|
|
||||||
homelab's use case); content-filter and AI-disclosure obligations apply if
|
|
||||||
deployed publicly; derivative model names must start with "Krea".
|
|
||||||
- ComfyUI added native support for both open-weight checkpoints (Krea 2 Raw
|
|
||||||
and Krea 2 Turbo) per [blog.comfy.org's announcement](https://blog.comfy.org/p/krea-2-open-source-models-are-now),
|
|
||||||
and community GGUF quants already exist
|
|
||||||
([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), with
|
|
||||||
reports (RTX hardware only) of fp8 fitting 16GB and GGUF fitting 12GB.
|
|
||||||
- **No AMD/ROCm mention anywhere** in the model card, and no gfx1201/R9700
|
|
||||||
community report was found — unsurprising given the model is roughly
|
|
||||||
2.5 months old at the time of this research. Its architecture (a
|
|
||||||
standard-shaped DiT that ComfyUI loads through its normal diffusion-model
|
|
||||||
nodes, per the ComfyUI blog post) gives reasonable expectation it will run
|
|
||||||
on the same ROCm/PyTorch backend already proven for FLUX and Qwen-Image on
|
|
||||||
this card, but that's an inference, not a verified fact the way
|
|
||||||
kyuz0's Qwen-Image workflow is.
|
|
||||||
- **Not the pick today**, precisely because Qwen-Image already offers a
|
|
||||||
hardware-verified path at a comparable parameter count and VRAM budget.
|
|
||||||
Worth a follow-up research ticket once R9700/gfx1201-specific Krea-2
|
|
||||||
reports exist — the license and ComfyUI support are both already in
|
|
||||||
place, so the only open question is real-world ROCm behavior.
|
|
||||||
|
|
||||||
## Recommendation
|
|
||||||
|
|
||||||
**Qwen-Image, FP8 precision** (`qwen_image_fp8_e4m3fn.safetensors` +
|
|
||||||
`qwen_2.5_vl_7b_fp8_scaled.safetensors`, ~25GB combined), optionally paired
|
|
||||||
with the 4-step Lightning LoRA the R9700-specific validated workflow uses.
|
|
||||||
It wins on all three axes the ticket asked about:
|
|
||||||
|
|
||||||
1. **License**: Apache-2.0 — no restriction at all, strictly better than
|
|
||||||
every other candidate including the current FLUX.1-schnell pick.
|
|
||||||
2. **ROCm/gfx1201 compatibility**: the only candidate with a workflow
|
|
||||||
pre-validated specifically on this GPU architecture
|
|
||||||
([kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy)),
|
|
||||||
not just "ComfyUI supports ROCm in general."
|
|
||||||
3. **VRAM at the ~32GB ceiling**: ~25GB at FP8 leaves real margin for
|
|
||||||
ComfyUI runtime/VAE overhead, without needing the multi-step,
|
|
||||||
non-distilled FLUX.1-dev's full 24GB bf16 footprint or accepting
|
|
||||||
SD3.5's lower fidelity ceiling — and it's a genuine capability upgrade
|
|
||||||
over schnell (20B vs. 12B, non-distilled-quality text rendering) rather
|
|
||||||
than just a bigger file.
|
|
||||||
|
|
||||||
If the known Qwen-Image/ROCm edit-mode rough edge
|
|
||||||
([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
|
|
||||||
turns out to affect plain text-to-image generation too, SD3.5-large (AMD's
|
|
||||||
own first-party "SD3.5 Simple" ComfyUI-ROCm template workflow) is the
|
|
||||||
fallback, with FLUX.1-dev as a second option. HunyuanImage-3.0 is excluded
|
|
||||||
outright regardless of available VRAM (CUDA-only tooling, multi-GPU
|
|
||||||
datacenter VRAM floor). Krea-2 is worth revisiting once R9700-specific
|
|
||||||
field reports exist.
|
|
||||||
|
|
||||||
## Sources consulted
|
|
||||||
|
|
||||||
- [docs/research/image-generation-options.md](image-generation-options.md) (this repo — prior findings, not re-derived)
|
|
||||||
- [Qwen/Qwen-Image (Hugging Face)](https://huggingface.co/Qwen/Qwen-Image)
|
|
||||||
- [QwenLM/Qwen-Image (GitHub)](https://github.com/QwenLM/Qwen-Image)
|
|
||||||
- [ComfyUI Wiki — Qwen-Image native support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image)
|
|
||||||
- [Comfy-Org/Qwen-Image_ComfyUI (Hugging Face, FP8 checkpoints)](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders)
|
|
||||||
- [Comfy-Org/ComfyUI issue #10852 — Qwen-Image VRAM](https://github.com/Comfy-Org/ComfyUI/issues/10852)
|
|
||||||
- [ComfyUI official Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/)
|
|
||||||
- [kyuz0/amd-r9700-comfy (R9700-specific ROCm ComfyUI toolbox)](https://github.com/kyuz0/amd-r9700-comfy)
|
|
||||||
- [pollockjj/ComfyUI-MultiGPU discussion #133 (Qwen-Image-Edit ROCm rough edge)](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133)
|
|
||||||
- [black-forest-labs/FLUX.1-dev (Hugging Face) + LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev)
|
|
||||||
- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
|
|
||||||
- [stabilityai/stable-diffusion-3.5-large (Hugging Face) + LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)
|
|
||||||
- [Tencent-Hunyuan/HunyuanImage-3.0 (GitHub)](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0)
|
|
||||||
- [krea/Krea-2-Turbo (Hugging Face)](https://huggingface.co/krea/Krea-2-Turbo)
|
|
||||||
- [Krea 2 Community License Agreement (krea.ai)](https://www.krea.ai/krea-2-licensing)
|
|
||||||
- [blog.comfy.org — Krea 2 open-source models in ComfyUI](https://blog.comfy.org/p/krea-2-open-source-models-are-now)
|
|
||||||
- [molbal/krea2-gguf (Hugging Face)](https://huggingface.co/molbal/krea2-gguf)
|
|
||||||
@@ -1,249 +0,0 @@
|
|||||||
# Research: adding local image generation to the stack
|
|
||||||
|
|
||||||
**Question:** What's the best way to add local image generation alongside
|
|
||||||
the existing Qwen3.8-27B / llama.cpp text stack, given a single AMD Radeon
|
|
||||||
R9700 (32GB VRAM, ROCm/gfx1201 — not CUDA), routed through the OmniRoute
|
|
||||||
gateway on the `ai-stack` Docker network?
|
|
||||||
|
|
||||||
**Answer, short version:** run **ComfyUI** (official AMD-blessed ROCm
|
|
||||||
Docker path exists, and OmniRoute already has a first-class `comfyui`
|
|
||||||
provider — no bespoke API wrapper needed) with **FLUX.1 [schnell]**
|
|
||||||
(Apache-2.0, 4-step, GGUF-quantizable) as the default model, falling back to
|
|
||||||
**SDXL** for anything schnell's distilled-step license/quality tradeoffs
|
|
||||||
don't suit. VRAM headroom against the current llama-server footprint is too
|
|
||||||
tight for both to be resident at once at any real image quality — plan for
|
|
||||||
**time-sliced use** (llama-server's existing lazytainer stop-on-idle pattern,
|
|
||||||
mirrored for the image-gen service, or a manual "stop one, start the other"
|
|
||||||
toggle), not concurrent operation.
|
|
||||||
|
|
||||||
## Current VRAM baseline (from this repo)
|
|
||||||
|
|
||||||
Per `docker-compose.yml` and `.env.example`, llama-server runs
|
|
||||||
`Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights) at `--ctx-size 262144` with
|
|
||||||
`--cache-type-k q8_0 --cache-type-v q8_0`, landing at **~25.6 GB** total
|
|
||||||
(weights + q8_0 KV cache), leaving **~6 GB** free on the 32GB card — this
|
|
||||||
matches the math already recorded in
|
|
||||||
[`docs/research/qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md). Per the auto-memory
|
|
||||||
note on this repo, real measured VRAM use has run closer to ~75% (~24 GB) in
|
|
||||||
practice versus the theoretical estimate, which doesn't change the
|
|
||||||
conclusion below but means the ~6 GB figure is closer to a ceiling than a
|
|
||||||
comfortable number.
|
|
||||||
|
|
||||||
**Implication:** 6 GB is not enough for any current-generation image model at
|
|
||||||
usable quality (see VRAM table below — even the smallest practical FLUX
|
|
||||||
quant wants ~7 GB alone, before ComfyUI's own runtime/VAE overhead). Running
|
|
||||||
image-gen *concurrently* with llama-server resident is not realistic on this
|
|
||||||
card. The two need to time-share the GPU, not split it.
|
|
||||||
|
|
||||||
## Backend evaluation (ROCm support, checked against primary sources)
|
|
||||||
|
|
||||||
### ComfyUI — recommended
|
|
||||||
|
|
||||||
- **Official AMD ROCm docs exist and are current.** AMD's own ROCm docs site
|
|
||||||
hosts a dedicated ComfyUI install guide with a prebuilt Docker image path
|
|
||||||
(recommended) or build-from-source, listing ROCm 7.2.0 and 7.1.0 as
|
|
||||||
supported versions, explicit `--device=/dev/kfd --device=/dev/dri
|
|
||||||
--group-add video` flags (same device-passthrough pattern this repo
|
|
||||||
already uses for llama-server), and template workflows including "SD3.5
|
|
||||||
Simple". Officially the guide only names AMD Instinct
|
|
||||||
MI355X/MI325X/MI300X (datacenter cards) as supported platforms.
|
|
||||||
Source: [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html).
|
|
||||||
- **The upstream ComfyUI README itself documents AMD support directly**,
|
|
||||||
including consumer cards: stable ROCm install via
|
|
||||||
`pip install torch torchvision torchaudio --index-url
|
|
||||||
https://download.pytorch.org/whl/rocm7.2`, plus an experimental Windows
|
|
||||||
build explicitly naming **RDNA 3 (RX 7000), RDNA 3.5 (Strix Halo), and
|
|
||||||
RDNA 4 (RX 9000 series)** — i.e. the same RDNA4 generation as the R9700 —
|
|
||||||
and `HSA_OVERRIDE_GFX_VERSION` workarounds for older/unlisted cards.
|
|
||||||
Source: [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI).
|
|
||||||
- **AMD has published a specific RDNA4/RX 9000 ComfyUI guide** (separate
|
|
||||||
from the Instinct-only install page above), confirming RDNA4 consumer
|
|
||||||
cards are an explicitly supported, first-party-documented target, not just
|
|
||||||
a community workaround.
|
|
||||||
Source: [ROCm blog — Getting Started with ComfyUI on AMD Radeon RX 9000 Series GPUs](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html).
|
|
||||||
- **gfx1201 (R9700's arch) specifically has active community Docker images**:
|
|
||||||
`yurisasc/comfyui-rocm-rdna4` targets ROCm 7.1 + PyTorch 2.9.1 with
|
|
||||||
`HSA_OVERRIDE_GFX_VERSION=12.0.1` / `PYTORCH_ROCM_ARCH=gfx1201` baked in,
|
|
||||||
and there's a published community patch specifically for R9700 AI Pro +
|
|
||||||
ComfyUI video-gen speedups, evidence the card is being run today, not just
|
|
||||||
theoretically compatible.
|
|
||||||
Sources: [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4),
|
|
||||||
[charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch).
|
|
||||||
- **Known gfx1201 caveat:** AMD's own TransformerEngine repo has an open
|
|
||||||
issue confirming gfx1201 is missing from the FP8 architecture table, so
|
|
||||||
FP8 kernels silently fall back to FP32 with ~50% throughput loss
|
|
||||||
(18-22 vs. 35-40 tok/s in the reporter's LLM benchmark) — not a
|
|
||||||
correctness blocker, but relevant if planning to use FP8-quantized image
|
|
||||||
models expecting native FP8 speed on this card; GGUF/Q-quants (see below)
|
|
||||||
avoid this path entirely since they dequantize to bf16/fp16, not fp8.
|
|
||||||
Source: [ROCm/TransformerEngine issue #520](https://github.com/ROCm/TransformerEngine/issues/520).
|
|
||||||
- **Actively maintained community Docker packaging** beyond AMD's own image:
|
|
||||||
`YanWenKun/ComfyUI-Docker` ships parallel `rocm` (PyTorch-build-based,
|
|
||||||
faster releases) and `rocm7` (AMD-build-based, more comprehensive)
|
|
||||||
variants, both targeting ROCm 7, with ~1000 commits of ongoing history —
|
|
||||||
a viable alternative to the official AMD image if it lags behind ComfyUI
|
|
||||||
releases.
|
|
||||||
Source: [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker).
|
|
||||||
|
|
||||||
### AUTOMATIC1111 / Forge — usable but a step down for this hardware
|
|
||||||
|
|
||||||
- ROCm support for A1111/Forge is real but community-patched, not
|
|
||||||
first-party. The upstream `lllyasviel/stable-diffusion-webui-forge` repo's
|
|
||||||
own discussion thread on AMD support points users to
|
|
||||||
`lshqqytiger/stable-diffusion-webui-amdgpu-forge`, a community fork
|
|
||||||
specifically maintained for AMD, "regarded as the go-to version" for
|
|
||||||
running FLUX-era models on AMD — i.e. the *mainline* Forge repo does not
|
|
||||||
claim ROCm support itself; you're expected to run a fork.
|
|
||||||
Source: [lllyasviel/stable-diffusion-webui-forge discussion #67](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67).
|
|
||||||
- No first-party AMD vendor documentation (unlike ComfyUI's AMD-authored
|
|
||||||
ROCm/RDNA4 blog posts above) was found for A1111/Forge specifically.
|
|
||||||
Given ComfyUI already has an AMD-blessed path plus a first-class OmniRoute
|
|
||||||
provider (below), there's no reason to take on a community fork's
|
|
||||||
maintenance risk instead.
|
|
||||||
|
|
||||||
### InvokeAI — usable but weaker AMD story for a new-generation card
|
|
||||||
|
|
||||||
- InvokeAI documents ROCm support but flags it as second-tier: "AMD GPUs
|
|
||||||
are only supported on Linux," and "support for newer AMD GPUs is spotty
|
|
||||||
... you may experience garbled images, black images, or long startup
|
|
||||||
delays." Its own install docs reference ROCm 5.4.2-era wheels, notably
|
|
||||||
older than the ROCm 7.x this stack's llama-server image already runs on
|
|
||||||
gfx1201.
|
|
||||||
Source: [InvokeAI installation docs (mauwii mirror)](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/).
|
|
||||||
- No OpenAI-compatible-images angle either — same drawback as A1111/Forge.
|
|
||||||
Not recommended as primary given ComfyUI's stronger, more current AMD
|
|
||||||
documentation trail.
|
|
||||||
|
|
||||||
## Model choice: FLUX.1 [schnell] vs FLUX.1 [dev] vs SDXL vs SD3.5
|
|
||||||
|
|
||||||
| Model | License | Params | Notes |
|
|
||||||
|---|---|---|---|
|
|
||||||
| **FLUX.1 [schnell]** | **Apache-2.0** — fully open, no commercial restriction | 12B | Distilled for 1-4 step inference (fast); Black Forest Labs' own model card states this license directly |
|
|
||||||
| FLUX.1 [dev] | [FLUX.1-dev Non-Commercial License](https://github.com/black-forest-labs/flux/blob/main/model_licenses/LICENSE-FLUX1-dev) | 12B | Non-commercial for the *model/weights*; generated *outputs* are explicitly usable commercially per BFL's license page. Higher quality than schnell (more steps, non-distilled) but the weights themselves can't be redistributed/used commercially |
|
|
||||||
| SDXL | CreativeML OpenRAIL++ (permissive, commercial-friendly) | ~3.5B | Older (2023), lower fidelity than FLUX/SD3.5 by current standards, but lowest VRAM footprint and best long-standing tooling maturity |
|
|
||||||
| SD3.5 (Large/Medium) | Stability AI Community License — free commercial use under $1M annual revenue, else enterprise license required | 8B / 2.5B | Free for this repo's non-commercial homelab use regardless; template already listed in AMD's own ComfyUI-ROCm doc ("SD3.5 Simple") as a first-party example workflow |
|
|
||||||
|
|
||||||
Sources: [black-forest-labs/flux model cards](https://github.com/black-forest-labs/flux/blob/main/model_cards/FLUX.1-dev.md),
|
|
||||||
[black-forest-labs/FLUX.1-schnell on Hugging Face](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
|
|
||||||
(license: apache-2.0), [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md),
|
|
||||||
[Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5),
|
|
||||||
[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
|
|
||||||
|
|
||||||
**Recommendation: FLUX.1 [schnell].** For a private homelab, license
|
|
||||||
enforcement isn't the deciding factor by itself, but schnell's Apache-2.0
|
|
||||||
status removes any future ambiguity if outputs or the setup are ever shared
|
|
||||||
or repurposed, and its whole design point — good quality in 1-4 sampling
|
|
||||||
steps — directly addresses the VRAM/time-slicing constraint below (less
|
|
||||||
GPU-resident time per image than a 20-50 step dev/SDXL/SD3.5 run).
|
|
||||||
Quantized via `city96/ComfyUI-GGUF` (an actively-referenced, community-
|
|
||||||
trusted quantization node — its GGUF Q-quants dequantize to bf16/fp16 at
|
|
||||||
runtime, sidestepping the gfx1201 FP8 dequant bug above entirely), FLUX fits
|
|
||||||
in a fraction of its fp16 footprint:
|
|
||||||
|
|
||||||
| Precision | Approx. VRAM (model only) |
|
|
||||||
|---|---|
|
|
||||||
| fp16 (baseline) | ~24 GB |
|
|
||||||
| fp8 | ~12 GB |
|
|
||||||
| GGUF Q5_K_S | ~12-15 GB (practical quality floor) |
|
|
||||||
| GGUF Q4_K_S | ~7 GB (quality starts degrading on hands/text below Q4) |
|
|
||||||
|
|
||||||
Source: aggregated VRAM figures from GGUF-quantization write-ups referencing
|
|
||||||
city96's FLUX GGUF conversions — treat as secondary/community sourced
|
|
||||||
(no single BFL-published VRAM table was found), consistent across multiple
|
|
||||||
independent sources.
|
|
||||||
[city96/ComfyUI-GGUF README](https://github.com/city96/ComfyUI-GGUF/blob/main/README.md),
|
|
||||||
[city96/FLUX.1-dev-gguf model card](https://huggingface.co/city96/FLUX.1-dev-gguf).
|
|
||||||
|
|
||||||
**Fallback pick: SDXL.** If schnell's distilled quality ceiling proves too
|
|
||||||
low for some use case, SDXL is the safer second choice over FLUX.1 [dev] or
|
|
||||||
SD3.5 specifically *because* of this card's tight headroom: it's the
|
|
||||||
smallest of the four by a wide margin, has the longest production track
|
|
||||||
record on ROCm of any of these models, and its OpenRAIL++ license carries no
|
|
||||||
revenue-threshold clause to track (unlike SD3.5's Community License) or
|
|
||||||
non-commercial weight restriction (unlike FLUX.1 [dev]).
|
|
||||||
|
|
||||||
## OpenAI-compatible API / OmniRoute integration
|
|
||||||
|
|
||||||
This is the best news in this research: **OmniRoute already ships a
|
|
||||||
first-class, built-in `comfyui` provider** — not a generic "point it at an
|
|
||||||
OpenAI base URL and hope" integration. Its provider reference documents it
|
|
||||||
explicitly: *"No API key required. Configure the local ComfyUI base URL
|
|
||||||
(default: http://localhost:8188)."* OmniRoute's own image-routing feature
|
|
||||||
set (`/v1/images/generations`, `/v1/images/edits`, `/v1/images/variations`,
|
|
||||||
with automatic provider fallback) is designed for exactly this pattern:
|
|
||||||
register ComfyUI as a backend, then any client already calling OmniRoute's
|
|
||||||
OpenAI-compatible images endpoints reaches it with no extra shim.
|
|
||||||
Source: [diegosouzapw/OmniRoute PROVIDER_REFERENCE.md](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md),
|
|
||||||
[diegosouzapw/OmniRoute repo description](https://github.com/diegosouzapw/OmniRoute).
|
|
||||||
|
|
||||||
This means **ComfyUI does not need an extra OpenAI-API wrapper project** —
|
|
||||||
the wrapper projects found during this research
|
|
||||||
([`ComfyUI-OpenAI-Compatible-API`](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API))
|
|
||||||
turned out to be a ComfyUI *custom node* for calling *outbound* to LLM APIs
|
|
||||||
from within a workflow (the reverse direction), not something this stack
|
|
||||||
needs — OmniRoute's own native ComfyUI provider is the actual integration
|
|
||||||
point, one layer up.
|
|
||||||
|
|
||||||
**Confidence note:** the provider-reference detail above was fetched via an
|
|
||||||
automated summarizer against the raw doc rather than manually re-verified
|
|
||||||
line-by-line; re-check `PROVIDER_REFERENCE.md`'s `comfyui` entry directly
|
|
||||||
before wiring this up, in case ComfyUI's own `/prompt` API (a
|
|
||||||
workflow-graph-shaped API, not a simple text-prompt-in/image-out call) needs
|
|
||||||
a specific default workflow JSON configured on the OmniRoute side to produce
|
|
||||||
a plain text-to-image call.
|
|
||||||
|
|
||||||
## Integration sketch (not a full compose — see caveats above)
|
|
||||||
|
|
||||||
- New service in `docker-compose.yml`, e.g. `comfyui`, image
|
|
||||||
`rocm/comfyui-rocm` (or `yurisasc/comfyui-rocm-rdna4` for a gfx1201-tuned
|
|
||||||
build) or built from AMD's own ROCm ComfyUI Dockerfile, same
|
|
||||||
`/dev/kfd` + `/dev/dri` + `group_add: [video, render]` device-passthrough
|
|
||||||
block already used for `llama-server`, joined to the same `ai-stack`
|
|
||||||
network so `omniroute` can reach it as `http://comfyui:8188` — no host
|
|
||||||
port needed (matches the existing llama-server pattern of no published
|
|
||||||
port, gateway-only access).
|
|
||||||
- Register it in OmniRoute's dashboard as a `comfyui` provider pointing at
|
|
||||||
that internal URL, same manual-registration pattern already used for
|
|
||||||
llama-server and searxng-search per `docs/proxy-key-onboarding.md`.
|
|
||||||
- **VRAM contention is the real design problem, not networking.** Given the
|
|
||||||
~6 GB headroom, the two services can't both sit GPU-resident.
|
|
||||||
Two workable patterns, in order of how well they fit what's already in
|
|
||||||
this repo:
|
|
||||||
1. **Mirror the existing lazytainer stop-on-idle pattern** already applied
|
|
||||||
to `llama-server` (`docker-compose.yml`'s `lazytainer.group.*` labels) —
|
|
||||||
add an equivalent idle-timeout group for `comfyui`, and rely on the two
|
|
||||||
services naturally not being hit at the same time for a single-user
|
|
||||||
homelab. This doesn't *guarantee* mutual exclusion (both could still be
|
|
||||||
woken concurrently and both try to fit in 6 GB free), so it's a
|
|
||||||
reasonable-effort fit, not a hard guarantee.
|
|
||||||
2. **Explicit mutual exclusion**: a small script/compose profile that
|
|
||||||
stops `llama-server` before starting `comfyui` (and vice versa) rather
|
|
||||||
than relying on lazytainer's independent idle timers — worth doing if
|
|
||||||
the reasonable-effort version above causes a visible OOM in practice.
|
|
||||||
Either way, this is a "pick one, then the other" story, not "run both."
|
|
||||||
- Given FLUX.1 [schnell]'s 1-4 step design, a cold-start-and-generate cycle
|
|
||||||
(wake ComfyUI from lazytainer sleep, generate, let it idle back down) is
|
|
||||||
a reasonably good fit for occasional image requests through the same
|
|
||||||
gateway that already does this for llama-server.
|
|
||||||
|
|
||||||
## Sources consulted
|
|
||||||
|
|
||||||
- [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html)
|
|
||||||
- [ROCm blog — ComfyUI on AMD Radeon RX 9000 Series (RDNA4)](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html)
|
|
||||||
- [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI)
|
|
||||||
- [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker)
|
|
||||||
- [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4)
|
|
||||||
- [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch)
|
|
||||||
- [ROCm/TransformerEngine issue #520 (gfx1201 FP8 fallback)](https://github.com/ROCm/TransformerEngine/issues/520)
|
|
||||||
- [lllyasviel/stable-diffusion-webui-forge discussion #67 (AMD support)](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67)
|
|
||||||
- [InvokeAI CUDA/ROCm install docs](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/)
|
|
||||||
- [black-forest-labs/flux GitHub repo + model cards](https://github.com/black-forest-labs/flux)
|
|
||||||
- [black-forest-labs/FLUX.1-schnell (Hugging Face, Apache-2.0)](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
|
|
||||||
- [black-forest-labs/FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)
|
|
||||||
- [Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5)
|
|
||||||
- [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)
|
|
||||||
- [city96/ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF)
|
|
||||||
- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
|
|
||||||
- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) and its `PROVIDER_REFERENCE.md`
|
|
||||||
- [yeeyou/ComfyUI-OpenAI-Compatible-API](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API) (checked and ruled out — wrong direction)
|
|
||||||
- This repo: `docker-compose.yml`, `.env.example`, `docs/research/qwen3.8-27b-quant.md`
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Swap GPU residency between llama-server (Qwen) and comfyui — they never
|
|
||||||
# run concurrently, VRAM doesn't fit both (see issue #38's map). Manual
|
|
||||||
# invocation only, no auto-switching.
|
|
||||||
#
|
|
||||||
# Bypasses lazytainer entirely and drives docker compose directly — its
|
|
||||||
# idle-stop can't be used for this. Root cause (see
|
|
||||||
# docs/research/lazytainer-omniroute-idle-stop.md, issue #40): lazytainer's
|
|
||||||
# packet-threshold detector is source-blind and can't tell OmniRoute's
|
|
||||||
# periodic health-check pings apart from real traffic on the same port, so
|
|
||||||
# it never reliably sleeps a service on its own. A scripted swap always
|
|
||||||
# knows which service should go up/down, so it doesn't need that heuristic.
|
|
||||||
#
|
|
||||||
# OmniRoute may show the just-stopped provider as errored/offline in its
|
|
||||||
# dashboard for up to CREDENTIAL_HEALTH_CHECK_INTERVAL (default 5 min) after
|
|
||||||
# a swap — cosmetic, not a functional problem (see the research doc above).
|
|
||||||
set -euo pipefail
|
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
echo "Usage: $0 {qwen|comfyui}" >&2
|
|
||||||
echo " qwen - stop comfyui, start llama-server" >&2
|
|
||||||
echo " comfyui - stop llama-server, start comfyui" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
[ $# -eq 1 ] || usage
|
|
||||||
|
|
||||||
case "$1" in
|
|
||||||
qwen)
|
|
||||||
from=comfyui
|
|
||||||
to=llama-server
|
|
||||||
;;
|
|
||||||
comfyui)
|
|
||||||
from=llama-server
|
|
||||||
to=comfyui
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
usage
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
echo "==> stopping $from"
|
|
||||||
docker compose stop "$from"
|
|
||||||
|
|
||||||
echo "==> starting $to"
|
|
||||||
docker compose up -d "$to"
|
|
||||||
|
|
||||||
echo "==> status"
|
|
||||||
docker compose ps
|
|
||||||
@@ -183,7 +183,6 @@ set_if_blank OMNIROUTE_STORAGE_ENCRYPTION_KEY "$(openssl rand -hex 32)"
|
|||||||
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
|
set_if_blank OMNIROUTE_MACHINE_ID_SALT "$(openssl rand -hex 16)"
|
||||||
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
|
set_if_blank OMNIROUTE_CLI_SALT "$(openssl rand -hex 16)"
|
||||||
set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
|
set_if_blank OMNIROUTE_WS_BRIDGE_SECRET "$(openssl rand -hex 32)"
|
||||||
set_if_blank NEO4J_PASSWORD "$(openssl rand -hex 16)"
|
|
||||||
|
|
||||||
echo "==> resolving SEARXNG_LAN_IP"
|
echo "==> resolving SEARXNG_LAN_IP"
|
||||||
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
|
# search.home is a LAN mDNS/local-DNS name — resolvable from this host, just
|
||||||
@@ -227,7 +226,6 @@ docker compose build --pull
|
|||||||
echo "==> ensuring models are downloaded (skips already-present files)"
|
echo "==> ensuring models are downloaded (skips already-present files)"
|
||||||
docker compose --profile tools run --rm downloader
|
docker compose --profile tools run --rm downloader
|
||||||
docker compose --profile tools run --rm downloader-classifier
|
docker compose --profile tools run --rm downloader-classifier
|
||||||
docker compose --profile tools run --rm downloader-comfyui
|
|
||||||
|
|
||||||
echo "==> bringing up omniroute"
|
echo "==> bringing up omniroute"
|
||||||
docker compose up -d --wait omniroute
|
docker compose up -d --wait omniroute
|
||||||
|
|||||||
Reference in new issue
Block a user