Fix GPU pinned at 100% with two containers, flaky render group #47

2026-09-06T19:33:53Z

haylan commented

2026-09-06 19:33:53 +00:00

Two real-hardware findings from issue #5, researched and fixed together — see docs/research/rocm-gpu-pin-and-render-group.md for full sourcing.

GPU pinned at 100% with both llama.cpp containers up

Confirmed on real hardware (see #5's comment thread): either llama-server or llama-server-fast alone is fine (3% GPU, low power) — it's specifically two concurrent HIP contexts that pins the R9700 at 100%/boost-clock. This is ROCm/ROCm#5706 (an MES firmware bug, AMD-confirmed), and GPU_MAX_HW_QUEUES=1 is a validated workaround on the exact image this stack uses. Added to both containers' environment:.

"unable to find group render: no matching entries in group file"

Confirmed new since the second GPU service was added — not hardware flakiness, a known Docker bug (docker/cli#4714): group_add by name resolves against the container's /etc/group, not the host's, and multiple GPU services starting concurrently race on that lookup. Switched group_add on all three GPU services (llama-server, llama-server-fast, comfyui) from plain names to resolved numeric GIDs. scripts/update.sh's existing comfyui-only GID resolution is generalized (COMFYUI_VIDEO_GID/COMFYUI_RENDER_GID → shared HOST_VIDEO_GID/HOST_RENDER_GID) to cover all three.

docker compose config -q validated both ways: fails fast with a clear message if the GIDs aren't resolved yet, passes once scripts/update.sh has run.

Not verified by this PR — needs the real box:

Confirm rocm-smi actually drops to normal idle with both containers up and GPU_MAX_HW_QUEUES=1 set (the two-concurrent-container case is an extrapolation from a single-process-validated fix, not directly reproduced — see the research doc's caveat)
Confirm the render-group error doesn't recur across a few docker compose up cycles

Refs #5

🤖 Generated with Claude Code

https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

Two real-hardware findings from issue #5, researched and fixed together — see `docs/research/rocm-gpu-pin-and-render-group.md` for full sourcing. **GPU pinned at 100% with both llama.cpp containers up** Confirmed on real hardware (see #5's comment thread): either `llama-server` or `llama-server-fast` alone is fine (3% GPU, low power) — it's specifically two concurrent HIP contexts that pins the R9700 at 100%/boost-clock. This is `ROCm/ROCm#5706` (an MES firmware bug, AMD-confirmed), and `GPU_MAX_HW_QUEUES=1` is a validated workaround on the exact image this stack uses. Added to both containers' `environment:`. **"unable to find group render: no matching entries in group file"** Confirmed new since the second GPU service was added — not hardware flakiness, a known Docker bug (`docker/cli#4714`): `group_add` by name resolves against the *container's* `/etc/group`, not the host's, and multiple GPU services starting concurrently race on that lookup. Switched `group_add` on all three GPU services (`llama-server`, `llama-server-fast`, `comfyui`) from plain names to resolved numeric GIDs. `scripts/update.sh`'s existing comfyui-only GID resolution is generalized (`COMFYUI_VIDEO_GID`/`COMFYUI_RENDER_GID` → shared `HOST_VIDEO_GID`/`HOST_RENDER_GID`) to cover all three. `docker compose config -q` validated both ways: fails fast with a clear message if the GIDs aren't resolved yet, passes once `scripts/update.sh` has run. **Not verified by this PR** — needs the real box: - Confirm `rocm-smi` actually drops to normal idle with both containers up and `GPU_MAX_HW_QUEUES=1` set (the two-concurrent-container case is an extrapolation from a single-process-validated fix, not directly reproduced — see the research doc's caveat) - Confirm the render-group error doesn't recur across a few `docker compose up` cycles Refs #5 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

haylan added 1 commit 2026-09-06 19:33:53 +00:00

fix: GPU pinned at 100% with two llama.cpp containers, flaky render group 75033dacd7

Two real-hardware findings from issue #5, both researched and fixed
together (docs/research/rocm-gpu-pin-and-render-group.md):

- GPU_MAX_HW_QUEUES=1 on llama-server and llama-server-fast. Confirmed
  on real hardware: either container alone is fine (3% GPU, low
  power), only two concurrent HIP contexts pin the R9700 at 100%/
  boost-clock (ROCm/ROCm#5706, an MES firmware bug). The env var is
  validated on the exact image this stack uses, per-process by design
  — applying it to both containers is the correct scope.

- group_add switched from plain names (video/render) to resolved
  numeric GIDs (HOST_VIDEO_GID/HOST_RENDER_GID) on all three GPU
  services. The "unable to find group render: no matching entries in
  group file" error confirmed new since the second GPU service was
  added is a known Docker bug (docker/cli#4714): group_add by name
  resolves against the container's own /etc/group, not the host's,
  and multiple GPU services starting concurrently race on that lookup.
  Numeric GIDs skip resolution entirely. scripts/update.sh's existing
  comfyui-only GID resolution is generalized to resolve these once for
  all three services.

docker compose config -q validated (fails fast with a clear error if
the GIDs aren't resolved yet, passes once they are).

Refs #5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx