haylan/LLM-Server
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

chore: remove image generation, Open WebUI, Lazytainer and neo4j

Browse Source 
Drops the comfyui, downloader-comfyui, open-webui, lazytainer and neo4j
services with their volumes, env keys and update.sh steps, plus
scripts/switch-model.sh and the two image-generation research docs.

Refs #55 (removal inventory #58, intent vs. source #63)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-09-24 21:47:12 +02:00
1 parent 8ae32102f9
commit fd55ce6669
6 files changed
+4 -739
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/research/image-generation-model-choice.md
-220
View File
@@ -1,220 +0,0 @@
# Research: which diffusion model to target with the full ~32GB R9700
**Question:** With Qwen/llama-server fully stopped (per issue #39's premise — see
map #38), ComfyUI has the whole ~32GB R9700 (ROCm/gfx1201) to itself instead
of the ~6GB left over during concurrent operation
(per [`image-generation-options.md`](image-generation-options.md)). Given that
headroom, should the build stay on FLUX.1-schnell, or move up to
FLUX.1-dev, SD3.5-large, Qwen-Image, HunyuanImage-3.0, or Krea-2?
**Answer, short version:** move up to **Qwen-Image at FP8 precision**
(`qwen_image_fp8_e4m3fn.safetensors` diffusion weights +
`qwen_2.5_vl_7b_fp8_scaled.safetensors` text encoder, ~25GB combined). It's
the only one of the five upgrade candidates with **direct, hardware-specific
evidence of running on this exact GPU architecture** (gfx1201/R9700) rather
than a generic "ComfyUI supports ROCm" inference, it carries the cleanest
license of the group (Apache-2.0, no revenue threshold, no non-commercial
clause), and its 20B MMDiT is a real capability step up from schnell's
distilled 12B (notably for text rendering and prompt adherence), while still
fitting with real margin inside 32GB.
## Why not just re-derive the schnell/dev/SDXL/SD3.5 findings
`docs/research/image-generation-options.md` already covers, with primary
sources: ComfyUI's ROCm/gfx1201 story (official AMD docs + RDNA4 blog post +
community gfx1201 Docker images), FLUX.1-schnell vs FLUX.1-dev vs SDXL vs
SD3.5 licenses, and FLUX GGUF VRAM figures at the ~6GB-headroom scale. None
of that is repeated here except where the ~32GB ceiling changes the
conclusion. This doc adds: FLUX.1-dev/SD3.5 at the *larger* headroom, plus
three models the prior doc didn't cover at all (Qwen-Image, HunyuanImage-3.0,
Krea-2).
## Candidate comparison
| Model | License (primary source) | Params | Stated/typical VRAM | ROCm/gfx1201 evidence |
|---|---|---|---|---|
| FLUX.1-schnell (current) | Apache-2.0 | 12B | GGUF Q4_K_S ~7GB (per prior doc) | Confirmed on gfx1201 (prior doc) |
| **Qwen-Image** | **Apache-2.0** | 20B (20.4B DiT + 8.3B Qwen2.5-VL text encoder) | fp8 ~16GB (diffusion) + ~9.4GB (fp8 text encoder) ≈ 25GB total; bf16 needs 24GB+ and "48GB+" per some quant write-ups | **Direct**: [kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) ships a pre-validated "Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow specifically for the R9700 AI Pro (gfx1201), on a ROCm 7 (TheRock nightlies) toolbox |
| FLUX.1-dev | [FLUX.1-dev Non-Commercial License](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) — non-commercial weights, outputs usable commercially | 12B | bf16 ~24GB; GGUF Q8 ~12-13GB | Same *family* evidence as schnell (gfx1201 Docker images target FLUX generally), but no R9700-specific FLUX.1-dev report found |
| SD3.5-large | [Stability Community License](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md) — free under $1M annual revenue | 8B | bf16 ~16GB; 4-bit NF4 fits small GPUs | AMD's own ComfyUI-ROCm doc lists an "SD3.5 Simple" template workflow (per prior doc) — vendor-blessed but not R9700-specific |
| HunyuanImage-3.0 | [tencent-hunyuan-community license](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0) | 80B total / 13B active (MoE, 64 experts) | Official repo: "≥ 3 × 80GB" VRAM for the base model, "≥ 8 × 80GB" for -Instruct; ~177GB at fp16 | **None, and actively contraindicated**: setup requires CUDA 12.8 + FlashAttention2/FlashInfer, no AMD/ROCm mention anywhere in the official repo |
| Krea-2 (Turbo) | [Krea 2 Community License](https://www.krea.ai/krea-2-licensing) — free under $1M annual revenue, homelab/personal explicitly covered | 12-13B DiT | No official VRAM figure; community reports (RTX hardware only) cite fp8 ~16GB, GGUF ~12GB | **None found** — released [June 22, 2026 per the HF model card](https://huggingface.co/krea/Krea-2-Turbo); ComfyUI added native support per [blog.comfy.org](https://blog.comfy.org/p/krea-2-open-source-models-are-now), and GGUF quants exist ([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), but no R9700/gfx1201-specific report exists yet — too new for that evidence to have accumulated |
## Per-model detail
### Qwen-Image — recommended
- **License**: Apache-2.0, stated directly on the model card, no revenue
threshold, no non-commercial clause, no attribution/naming requirement.
The cleanest license of every model considered in this doc or the prior
one. Source: [Qwen/Qwen-Image on Hugging Face](https://huggingface.co/Qwen/Qwen-Image).
- **Architecture**: 20B-parameter MMDiT (Multimodal Diffusion Transformer)
combined with an 8.3B Qwen2.5-VL text encoder — notably larger and more
capable than FLUX.1-schnell's 12B distilled model, particularly for
multilingual text rendering and instruction-following, per the official
[QwenLM/Qwen-Image GitHub repo](https://github.com/QwenLM/Qwen-Image).
- **ComfyUI support**: native, not a wrapper/custom-node integration —
landed August 2025 per [ComfyUI Wiki's native-support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image),
with an official FP8 checkpoint (`qwen_image_fp8_e4m3fn.safetensors`) and
FP8-scaled text encoder (`qwen_2.5_vl_7b_fp8_scaled.safetensors`, 9.38GB)
published under [Comfy-Org/Qwen-Image_ComfyUI](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders).
- **VRAM at FP8**: the official Comfy-Org FP8 diffusion checkpoint plus FP8
text encoder land around ~25GB combined — comfortably inside the 32GB
ceiling with headroom for ComfyUI runtime/VAE overhead, versus 24GB+ (some
sources say 48GB+) for the unquantized bf16 model. Sources: community
VRAM write-ups aggregated via [Comfy-Org/ComfyUI issue #10852 ("Qwen-image in 24GB VRAM and 32GB RAM")](https://github.com/Comfy-Org/ComfyUI/issues/10852)
and the [official ComfyUI Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/) —
treat exact GB figures as secondary/community-sourced, consistent with how
the prior doc flagged FLUX's GGUF VRAM table.
- **Direct R9700/gfx1201 evidence (the deciding factor)**:
[kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy) is a
Fedora-toolbox ROCm 7 (TheRock nightlies) environment built specifically
for the "AMD Radeon 9700 AI PRO (32GB)" and ships a pre-validated
"Qwen Image 2512 (FP8) & Lightning LoRA (4 steps)" ComfyUI workflow
(plus a Qwen-Image-Edit 2511 FP8 workflow) as one of only four workflows
in the whole repo. This is the only model in this comparison with
card-architecture-specific (not just "ComfyUI supports ROCm generically")
validation — everything else relies on family-level or vendor-generic
ROCm claims. A related community discussion
([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
does flag that "Qwen image edit doesn't always work on AMD HIP/ROCm" in
some configurations — worth testing the specific workflow before assuming
zero friction, but this is a known-quantity, actively-discussed rough edge
rather than a documented hard blocker.
- **4-step Lightning LoRA**: the R9700-validated workflow pairs Qwen-Image
with a "Lightning" LoRA for 4-step inference — the same fast-inference
pattern FLUX.1-schnell uses, so switching models doesn't have to mean
giving up the short GPU-resident-time-per-image property the prior doc
called out as valuable for time-sliced use alongside llama-server (even
though this ticket's premise is llama-server being stopped, that pattern
still helps if the two services are ever run in an overlapping window).
### FLUX.1-dev — solid alternative, not the pick
- License permits non-commercial use of the weights; BFL's own license page
states generated outputs are separately usable commercially (already
covered in the prior doc; unchanged here). Source:
[FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md).
- At bf16 (~24GB) or GGUF Q8 (~12-13GB), it fits the 32GB ceiling with room
to spare — a legitimate move up from schnell's distilled quality.
Source: [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
and community Q8 VRAM reports.
- Loses to Qwen-Image on two counts: license (non-commercial-weights clause
vs. Apache-2.0 — not a hard blocker for this homelab per the prior doc's
own reasoning, but strictly worse) and hardware evidence (FLUX's gfx1201
validation is at the *family* level — the prior doc's `yurisasc/comfyui-rocm-rdna4`
and `charlie12345/R9700AIProComfyUIPatch` sources are about running FLUX
models on this card generally, not a FLUX.1-dev-specific report the way
kyuz0's repo is Qwen-Image-specific).
### SD3.5-large — no longer the best use of the freed headroom
- License unchanged from the prior doc: Stability Community License, free
under $1M annual revenue (irrelevant threshold for this homelab). Source:
[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
- At 8B params (bf16 ~16GB), it's the smallest of the upgrade candidates —
which mattered when 6GB was the ceiling, but with 32GB available there's
no VRAM reason to pick the model the prior doc already flagged as "lower
fidelity than FLUX/SD3.5 by current standards" over Qwen-Image or
FLUX.1-dev. AMD's own ComfyUI-ROCm docs do list it as a first-party
example template ("SD3.5 Simple", per the prior doc), so it remains a fine
fallback if Qwen-Image's FP8 path hits the ROCm rough edge noted above.
### HunyuanImage-3.0 — ruled out
- Official repo states VRAM requirements of "≥ 3 × 80GB" for the base model
and "≥ 8 × 80GB" for HunyuanImage-3.0-Instruct — i.e. multi-GPU
datacenter-class NVIDIA clusters, not a single 32GB consumer/workstation
card at any precision. Source:
[Tencent-Hunyuan/HunyuanImage-3.0 GitHub repo](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0).
- Setup instructions require CUDA 12.8, PyTorch 2.8.0 built for CUDA, and
optionally FlashAttention2/FlashInfer for MoE routing speed — no AMD or
ROCm path is mentioned anywhere in the official repo. Even the
MoE-efficient "13B active" framing doesn't help here: the tooling itself
assumes an NVIDIA multi-GPU cluster, and the 80B total parameter set still
has to be resident somewhere.
- 32GB of headroom on one AMD card doesn't move this model into reach at
any precision considered here; it's excluded regardless of how much VRAM
frees up on this specific box.
### Krea-2 (Turbo) — promising, but unverified on this hardware
- Verified directly against primary sources per the ticket's instruction
(this is a June 2026 release, past most training cutoffs): the
[Hugging Face model card](https://huggingface.co/krea/Krea-2-Turbo) states
a release date of **June 22, 2026**, a 12-billion-parameter single-stream
diffusion transformer, `torch.bfloat16` as the reference precision, and
the **Krea 2 Community License**.
- License, per [krea.ai/krea-2-licensing](https://www.krea.ai/krea-2-licensing):
non-commercial (including explicitly personal/homelab) use is free;
commercial use is permitted royalty-free for entities under $1M
trailing-12-month revenue (same shape as SD3.5's and matching this
homelab's use case); content-filter and AI-disclosure obligations apply if
deployed publicly; derivative model names must start with "Krea".
- ComfyUI added native support for both open-weight checkpoints (Krea 2 Raw
and Krea 2 Turbo) per [blog.comfy.org's announcement](https://blog.comfy.org/p/krea-2-open-source-models-are-now),
and community GGUF quants already exist
([molbal/krea2-gguf](https://huggingface.co/molbal/krea2-gguf)), with
reports (RTX hardware only) of fp8 fitting 16GB and GGUF fitting 12GB.
- **No AMD/ROCm mention anywhere** in the model card, and no gfx1201/R9700
community report was found — unsurprising given the model is roughly
2.5 months old at the time of this research. Its architecture (a
standard-shaped DiT that ComfyUI loads through its normal diffusion-model
nodes, per the ComfyUI blog post) gives reasonable expectation it will run
on the same ROCm/PyTorch backend already proven for FLUX and Qwen-Image on
this card, but that's an inference, not a verified fact the way
kyuz0's Qwen-Image workflow is.
- **Not the pick today**, precisely because Qwen-Image already offers a
hardware-verified path at a comparable parameter count and VRAM budget.
Worth a follow-up research ticket once R9700/gfx1201-specific Krea-2
reports exist — the license and ComfyUI support are both already in
place, so the only open question is real-world ROCm behavior.
## Recommendation
**Qwen-Image, FP8 precision** (`qwen_image_fp8_e4m3fn.safetensors` +
`qwen_2.5_vl_7b_fp8_scaled.safetensors`, ~25GB combined), optionally paired
with the 4-step Lightning LoRA the R9700-specific validated workflow uses.
It wins on all three axes the ticket asked about:
1. **License**: Apache-2.0 — no restriction at all, strictly better than
every other candidate including the current FLUX.1-schnell pick.
2. **ROCm/gfx1201 compatibility**: the only candidate with a workflow
pre-validated specifically on this GPU architecture
([kyuz0/amd-r9700-comfy](https://github.com/kyuz0/amd-r9700-comfy)),
not just "ComfyUI supports ROCm in general."
3. **VRAM at the ~32GB ceiling**: ~25GB at FP8 leaves real margin for
ComfyUI runtime/VAE overhead, without needing the multi-step,
non-distilled FLUX.1-dev's full 24GB bf16 footprint or accepting
SD3.5's lower fidelity ceiling — and it's a genuine capability upgrade
over schnell (20B vs. 12B, non-distilled-quality text rendering) rather
than just a bigger file.
If the known Qwen-Image/ROCm edit-mode rough edge
([pollockjj/ComfyUI-MultiGPU #133](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133))
turns out to affect plain text-to-image generation too, SD3.5-large (AMD's
own first-party "SD3.5 Simple" ComfyUI-ROCm template workflow) is the
fallback, with FLUX.1-dev as a second option. HunyuanImage-3.0 is excluded
outright regardless of available VRAM (CUDA-only tooling, multi-GPU
datacenter VRAM floor). Krea-2 is worth revisiting once R9700-specific
field reports exist.
## Sources consulted
- [docs/research/image-generation-options.md](image-generation-options.md) (this repo — prior findings, not re-derived)
- [Qwen/Qwen-Image (Hugging Face)](https://huggingface.co/Qwen/Qwen-Image)
- [QwenLM/Qwen-Image (GitHub)](https://github.com/QwenLM/Qwen-Image)
- [ComfyUI Wiki — Qwen-Image native support announcement](https://comfyui-wiki.com/en/news/2025-08-05-qwen-image)
- [Comfy-Org/Qwen-Image_ComfyUI (Hugging Face, FP8 checkpoints)](https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders)
- [Comfy-Org/ComfyUI issue #10852 — Qwen-Image VRAM](https://github.com/Comfy-Org/ComfyUI/issues/10852)
- [ComfyUI official Qwen-Image example](https://comfyanonymous.github.io/ComfyUI_examples/qwen_image/)
- [kyuz0/amd-r9700-comfy (R9700-specific ROCm ComfyUI toolbox)](https://github.com/kyuz0/amd-r9700-comfy)
- [pollockjj/ComfyUI-MultiGPU discussion #133 (Qwen-Image-Edit ROCm rough edge)](https://github.com/pollockjj/ComfyUI-MultiGPU/discussions/133)
- [black-forest-labs/FLUX.1-dev (Hugging Face) + LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev)
- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
- [stabilityai/stable-diffusion-3.5-large (Hugging Face) + LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)
- [Tencent-Hunyuan/HunyuanImage-3.0 (GitHub)](https://github.com/Tencent-Hunyuan/HunyuanImage-3.0)
- [krea/Krea-2-Turbo (Hugging Face)](https://huggingface.co/krea/Krea-2-Turbo)
- [Krea 2 Community License Agreement (krea.ai)](https://www.krea.ai/krea-2-licensing)
- [blog.comfy.org — Krea 2 open-source models in ComfyUI](https://blog.comfy.org/p/krea-2-open-source-models-are-now)
- [molbal/krea2-gguf (Hugging Face)](https://huggingface.co/molbal/krea2-gguf)
docs/research/image-generation-options.md
-249
View File
@@ -1,249 +0,0 @@
# Research: adding local image generation to the stack
**Question:** What's the best way to add local image generation alongside
the existing Qwen3.8-27B / llama.cpp text stack, given a single AMD Radeon
R9700 (32GB VRAM, ROCm/gfx1201 — not CUDA), routed through the OmniRoute
gateway on the `ai-stack` Docker network?
**Answer, short version:** run **ComfyUI** (official AMD-blessed ROCm
Docker path exists, and OmniRoute already has a first-class `comfyui`
provider — no bespoke API wrapper needed) with **FLUX.1 [schnell]**
(Apache-2.0, 4-step, GGUF-quantizable) as the default model, falling back to
**SDXL** for anything schnell's distilled-step license/quality tradeoffs
don't suit. VRAM headroom against the current llama-server footprint is too
tight for both to be resident at once at any real image quality — plan for
**time-sliced use** (llama-server's existing lazytainer stop-on-idle pattern,
mirrored for the image-gen service, or a manual "stop one, start the other"
toggle), not concurrent operation.
## Current VRAM baseline (from this repo)
Per `docker-compose.yml` and `.env.example`, llama-server runs
`Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6 GB weights) at `--ctx-size 262144` with
`--cache-type-k q8_0 --cache-type-v q8_0`, landing at **~25.6 GB** total
(weights + q8_0 KV cache), leaving **~6 GB** free on the 32GB card — this
matches the math already recorded in
[`docs/research/qwen3.8-27b-quant.md`](qwen3.8-27b-quant.md). Per the auto-memory
note on this repo, real measured VRAM use has run closer to ~75% (~24 GB) in
practice versus the theoretical estimate, which doesn't change the
conclusion below but means the ~6 GB figure is closer to a ceiling than a
comfortable number.
**Implication:** 6 GB is not enough for any current-generation image model at
usable quality (see VRAM table below — even the smallest practical FLUX
quant wants ~7 GB alone, before ComfyUI's own runtime/VAE overhead). Running
image-gen *concurrently* with llama-server resident is not realistic on this
card. The two need to time-share the GPU, not split it.
## Backend evaluation (ROCm support, checked against primary sources)
### ComfyUI — recommended
- **Official AMD ROCm docs exist and are current.** AMD's own ROCm docs site
hosts a dedicated ComfyUI install guide with a prebuilt Docker image path
(recommended) or build-from-source, listing ROCm 7.2.0 and 7.1.0 as
supported versions, explicit `--device=/dev/kfd --device=/dev/dri
--group-add video` flags (same device-passthrough pattern this repo
already uses for llama-server), and template workflows including "SD3.5
Simple". Officially the guide only names AMD Instinct
MI355X/MI325X/MI300X (datacenter cards) as supported platforms.
Source: [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html).
- **The upstream ComfyUI README itself documents AMD support directly**,
including consumer cards: stable ROCm install via
`pip install torch torchvision torchaudio --index-url
https://download.pytorch.org/whl/rocm7.2`, plus an experimental Windows
build explicitly naming **RDNA 3 (RX 7000), RDNA 3.5 (Strix Halo), and
RDNA 4 (RX 9000 series)** — i.e. the same RDNA4 generation as the R9700 —
and `HSA_OVERRIDE_GFX_VERSION` workarounds for older/unlisted cards.
Source: [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI).
- **AMD has published a specific RDNA4/RX 9000 ComfyUI guide** (separate
from the Instinct-only install page above), confirming RDNA4 consumer
cards are an explicitly supported, first-party-documented target, not just
a community workaround.
Source: [ROCm blog — Getting Started with ComfyUI on AMD Radeon RX 9000 Series GPUs](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html).
- **gfx1201 (R9700's arch) specifically has active community Docker images**:
`yurisasc/comfyui-rocm-rdna4` targets ROCm 7.1 + PyTorch 2.9.1 with
`HSA_OVERRIDE_GFX_VERSION=12.0.1` / `PYTORCH_ROCM_ARCH=gfx1201` baked in,
and there's a published community patch specifically for R9700 AI Pro +
ComfyUI video-gen speedups, evidence the card is being run today, not just
theoretically compatible.
Sources: [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4),
[charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch).
- **Known gfx1201 caveat:** AMD's own TransformerEngine repo has an open
issue confirming gfx1201 is missing from the FP8 architecture table, so
FP8 kernels silently fall back to FP32 with ~50% throughput loss
(18-22 vs. 35-40 tok/s in the reporter's LLM benchmark) — not a
correctness blocker, but relevant if planning to use FP8-quantized image
models expecting native FP8 speed on this card; GGUF/Q-quants (see below)
avoid this path entirely since they dequantize to bf16/fp16, not fp8.
Source: [ROCm/TransformerEngine issue #520](https://github.com/ROCm/TransformerEngine/issues/520).
- **Actively maintained community Docker packaging** beyond AMD's own image:
`YanWenKun/ComfyUI-Docker` ships parallel `rocm` (PyTorch-build-based,
faster releases) and `rocm7` (AMD-build-based, more comprehensive)
variants, both targeting ROCm 7, with ~1000 commits of ongoing history —
a viable alternative to the official AMD image if it lags behind ComfyUI
releases.
Source: [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker).
### AUTOMATIC1111 / Forge — usable but a step down for this hardware
- ROCm support for A1111/Forge is real but community-patched, not
first-party. The upstream `lllyasviel/stable-diffusion-webui-forge` repo's
own discussion thread on AMD support points users to
`lshqqytiger/stable-diffusion-webui-amdgpu-forge`, a community fork
specifically maintained for AMD, "regarded as the go-to version" for
running FLUX-era models on AMD — i.e. the *mainline* Forge repo does not
claim ROCm support itself; you're expected to run a fork.
Source: [lllyasviel/stable-diffusion-webui-forge discussion #67](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67).
- No first-party AMD vendor documentation (unlike ComfyUI's AMD-authored
ROCm/RDNA4 blog posts above) was found for A1111/Forge specifically.
Given ComfyUI already has an AMD-blessed path plus a first-class OmniRoute
provider (below), there's no reason to take on a community fork's
maintenance risk instead.
### InvokeAI — usable but weaker AMD story for a new-generation card
- InvokeAI documents ROCm support but flags it as second-tier: "AMD GPUs
are only supported on Linux," and "support for newer AMD GPUs is spotty
... you may experience garbled images, black images, or long startup
delays." Its own install docs reference ROCm 5.4.2-era wheels, notably
older than the ROCm 7.x this stack's llama-server image already runs on
gfx1201.
Source: [InvokeAI installation docs (mauwii mirror)](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/).
- No OpenAI-compatible-images angle either — same drawback as A1111/Forge.
Not recommended as primary given ComfyUI's stronger, more current AMD
documentation trail.
## Model choice: FLUX.1 [schnell] vs FLUX.1 [dev] vs SDXL vs SD3.5
| Model | License | Params | Notes |
|---|---|---|---|
| **FLUX.1 [schnell]** | **Apache-2.0** — fully open, no commercial restriction | 12B | Distilled for 1-4 step inference (fast); Black Forest Labs' own model card states this license directly |
| FLUX.1 [dev] | [FLUX.1-dev Non-Commercial License](https://github.com/black-forest-labs/flux/blob/main/model_licenses/LICENSE-FLUX1-dev) | 12B | Non-commercial for the *model/weights*; generated *outputs* are explicitly usable commercially per BFL's license page. Higher quality than schnell (more steps, non-distilled) but the weights themselves can't be redistributed/used commercially |
| SDXL | CreativeML OpenRAIL++ (permissive, commercial-friendly) | ~3.5B | Older (2023), lower fidelity than FLUX/SD3.5 by current standards, but lowest VRAM footprint and best long-standing tooling maturity |
| SD3.5 (Large/Medium) | Stability AI Community License — free commercial use under $1M annual revenue, else enterprise license required | 8B / 2.5B | Free for this repo's non-commercial homelab use regardless; template already listed in AMD's own ComfyUI-ROCm doc ("SD3.5 Simple") as a first-party example workflow |
Sources: [black-forest-labs/flux model cards](https://github.com/black-forest-labs/flux/blob/main/model_cards/FLUX.1-dev.md),
[black-forest-labs/FLUX.1-schnell on Hugging Face](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
(license: apache-2.0), [FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md),
[Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5),
[stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md).
**Recommendation: FLUX.1 [schnell].** For a private homelab, license
enforcement isn't the deciding factor by itself, but schnell's Apache-2.0
status removes any future ambiguity if outputs or the setup are ever shared
or repurposed, and its whole design point — good quality in 1-4 sampling
steps — directly addresses the VRAM/time-slicing constraint below (less
GPU-resident time per image than a 20-50 step dev/SDXL/SD3.5 run).
Quantized via `city96/ComfyUI-GGUF` (an actively-referenced, community-
trusted quantization node — its GGUF Q-quants dequantize to bf16/fp16 at
runtime, sidestepping the gfx1201 FP8 dequant bug above entirely), FLUX fits
in a fraction of its fp16 footprint:
| Precision | Approx. VRAM (model only) |
|---|---|
| fp16 (baseline) | ~24 GB |
| fp8 | ~12 GB |
| GGUF Q5_K_S | ~12-15 GB (practical quality floor) |
| GGUF Q4_K_S | ~7 GB (quality starts degrading on hands/text below Q4) |
Source: aggregated VRAM figures from GGUF-quantization write-ups referencing
city96's FLUX GGUF conversions — treat as secondary/community sourced
(no single BFL-published VRAM table was found), consistent across multiple
independent sources.
[city96/ComfyUI-GGUF README](https://github.com/city96/ComfyUI-GGUF/blob/main/README.md),
[city96/FLUX.1-dev-gguf model card](https://huggingface.co/city96/FLUX.1-dev-gguf).
**Fallback pick: SDXL.** If schnell's distilled quality ceiling proves too
low for some use case, SDXL is the safer second choice over FLUX.1 [dev] or
SD3.5 specifically *because* of this card's tight headroom: it's the
smallest of the four by a wide margin, has the longest production track
record on ROCm of any of these models, and its OpenRAIL++ license carries no
revenue-threshold clause to track (unlike SD3.5's Community License) or
non-commercial weight restriction (unlike FLUX.1 [dev]).
## OpenAI-compatible API / OmniRoute integration
This is the best news in this research: **OmniRoute already ships a
first-class, built-in `comfyui` provider** — not a generic "point it at an
OpenAI base URL and hope" integration. Its provider reference documents it
explicitly: *"No API key required. Configure the local ComfyUI base URL
(default: http://localhost:8188)."* OmniRoute's own image-routing feature
set (`/v1/images/generations`, `/v1/images/edits`, `/v1/images/variations`,
with automatic provider fallback) is designed for exactly this pattern:
register ComfyUI as a backend, then any client already calling OmniRoute's
OpenAI-compatible images endpoints reaches it with no extra shim.
Source: [diegosouzapw/OmniRoute PROVIDER_REFERENCE.md](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/PROVIDER_REFERENCE.md),
[diegosouzapw/OmniRoute repo description](https://github.com/diegosouzapw/OmniRoute).
This means **ComfyUI does not need an extra OpenAI-API wrapper project** —
the wrapper projects found during this research
([`ComfyUI-OpenAI-Compatible-API`](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API))
turned out to be a ComfyUI *custom node* for calling *outbound* to LLM APIs
from within a workflow (the reverse direction), not something this stack
needs — OmniRoute's own native ComfyUI provider is the actual integration
point, one layer up.
**Confidence note:** the provider-reference detail above was fetched via an
automated summarizer against the raw doc rather than manually re-verified
line-by-line; re-check `PROVIDER_REFERENCE.md`'s `comfyui` entry directly
before wiring this up, in case ComfyUI's own `/prompt` API (a
workflow-graph-shaped API, not a simple text-prompt-in/image-out call) needs
a specific default workflow JSON configured on the OmniRoute side to produce
a plain text-to-image call.
## Integration sketch (not a full compose — see caveats above)
- New service in `docker-compose.yml`, e.g. `comfyui`, image
`rocm/comfyui-rocm` (or `yurisasc/comfyui-rocm-rdna4` for a gfx1201-tuned
build) or built from AMD's own ROCm ComfyUI Dockerfile, same
`/dev/kfd` + `/dev/dri` + `group_add: [video, render]` device-passthrough
block already used for `llama-server`, joined to the same `ai-stack`
network so `omniroute` can reach it as `http://comfyui:8188` — no host
port needed (matches the existing llama-server pattern of no published
port, gateway-only access).
- Register it in OmniRoute's dashboard as a `comfyui` provider pointing at
that internal URL, same manual-registration pattern already used for
llama-server and searxng-search per `docs/proxy-key-onboarding.md`.
- **VRAM contention is the real design problem, not networking.** Given the
~6 GB headroom, the two services can't both sit GPU-resident.
Two workable patterns, in order of how well they fit what's already in
this repo:
1. **Mirror the existing lazytainer stop-on-idle pattern** already applied
to `llama-server` (`docker-compose.yml`'s `lazytainer.group.*` labels) —
add an equivalent idle-timeout group for `comfyui`, and rely on the two
services naturally not being hit at the same time for a single-user
homelab. This doesn't *guarantee* mutual exclusion (both could still be
woken concurrently and both try to fit in 6 GB free), so it's a
reasonable-effort fit, not a hard guarantee.
2. **Explicit mutual exclusion**: a small script/compose profile that
stops `llama-server` before starting `comfyui` (and vice versa) rather
than relying on lazytainer's independent idle timers — worth doing if
the reasonable-effort version above causes a visible OOM in practice.
Either way, this is a "pick one, then the other" story, not "run both."
- Given FLUX.1 [schnell]'s 1-4 step design, a cold-start-and-generate cycle
(wake ComfyUI from lazytainer sleep, generate, let it idle back down) is
a reasonably good fit for occasional image requests through the same
gateway that already does this for llama-server.
## Sources consulted
- [ROCm docs — ComfyUI on ROCm installation](https://rocm.docs.amd.com/projects/comfyui/en/docs-26.04/install/comfyui-install.html)
- [ROCm blog — ComfyUI on AMD Radeon RX 9000 Series (RDNA4)](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html)
- [comfyanonymous/ComfyUI README](https://github.com/comfyanonymous/ComfyUI)
- [YanWenKun/ComfyUI-Docker](https://github.com/YanWenKun/ComfyUI-Docker)
- [yurisasc/comfyui-rocm-rdna4](https://github.com/yurisasc/comfyui-rocm-rdna4)
- [charlie12345/R9700AIProComfyUIPatch](https://github.com/charlie12345/R9700AIProComfyUIPatch)
- [ROCm/TransformerEngine issue #520 (gfx1201 FP8 fallback)](https://github.com/ROCm/TransformerEngine/issues/520)
- [lllyasviel/stable-diffusion-webui-forge discussion #67 (AMD support)](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/67)
- [InvokeAI CUDA/ROCm install docs](https://mauwii.github.io/InvokeAI/installation/030_INSTALL_CUDA_AND_ROCM/)
- [black-forest-labs/flux GitHub repo + model cards](https://github.com/black-forest-labs/flux)
- [black-forest-labs/FLUX.1-schnell (Hugging Face, Apache-2.0)](https://huggingface.co/black-forest-labs/FLUX.1-schnell)
- [black-forest-labs/FLUX.1-dev LICENSE.md](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)
- [Stability AI — Introducing Stable Diffusion 3.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5)
- [stabilityai/stable-diffusion-3.5-large LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md)
- [city96/ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF)
- [city96/FLUX.1-dev-gguf](https://huggingface.co/city96/FLUX.1-dev-gguf)
- [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) and its `PROVIDER_REFERENCE.md`
- [yeeyou/ComfyUI-OpenAI-Compatible-API](https://github.com/yeeyou/ComfyUI-OpenAI-Compatible-API) (checked and ruled out — wrong direction)
- This repo: `docker-compose.yml`, `.env.example`, `docs/research/qwen3.8-27b-quant.md`