fix: GPU pinned at 100% with two llama.cpp containers, flaky render group
Two real-hardware findings from issue #5, both researched and fixed together (docs/research/rocm-gpu-pin-and-render-group.md): - GPU_MAX_HW_QUEUES=1 on llama-server and llama-server-fast. Confirmed on real hardware: either container alone is fine (3% GPU, low power), only two concurrent HIP contexts pin the R9700 at 100%/ boost-clock (ROCm/ROCm#5706, an MES firmware bug). The env var is validated on the exact image this stack uses, per-process by design — applying it to both containers is the correct scope. - group_add switched from plain names (video/render) to resolved numeric GIDs (HOST_VIDEO_GID/HOST_RENDER_GID) on all three GPU services. The "unable to find group render: no matching entries in group file" error confirmed new since the second GPU service was added is a known Docker bug (docker/cli#4714): group_add by name resolves against the container's own /etc/group, not the host's, and multiple GPU services starting concurrently race on that lookup. Numeric GIDs skip resolution entirely. scripts/update.sh's existing comfyui-only GID resolution is generalized to resolve these once for all three services. docker compose config -q validated (fails fast with a clear error if the GIDs aren't resolved yet, passes once they are). Refs #5 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
This commit is contained in:
+25
-12
@@ -5,12 +5,20 @@ services:
|
||||
devices:
|
||||
- /dev/kfd
|
||||
- /dev/dri
|
||||
# Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
|
||||
# .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
|
||||
group_add:
|
||||
- video
|
||||
- render
|
||||
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
|
||||
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
|
||||
security_opt:
|
||||
- seccomp=unconfined
|
||||
ipc: host
|
||||
# Caps this process's HIP hardware-queue allocation — works around
|
||||
# ROCm/ROCm#5706 (GPU pinned at 100%/boost-clock whenever two
|
||||
# concurrent HIP contexts touch this card, confirmed on real hardware
|
||||
# against llama-server-fast below). See the research doc above.
|
||||
environment:
|
||||
- GPU_MAX_HW_QUEUES=1
|
||||
volumes:
|
||||
- models:/models
|
||||
command: >
|
||||
@@ -46,11 +54,14 @@ services:
|
||||
- /dev/kfd
|
||||
- /dev/dri
|
||||
group_add:
|
||||
- video
|
||||
- render
|
||||
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
|
||||
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
|
||||
security_opt:
|
||||
- seccomp=unconfined
|
||||
ipc: host
|
||||
# See llama-server's identical setting above — same fix, same bug.
|
||||
environment:
|
||||
- GPU_MAX_HW_QUEUES=1
|
||||
volumes:
|
||||
- models:/models
|
||||
command: >
|
||||
@@ -159,23 +170,25 @@ services:
|
||||
devices:
|
||||
- /dev/kfd
|
||||
- /dev/dri
|
||||
# Numeric GIDs, not names — see HOST_VIDEO_GID/HOST_RENDER_GID in
|
||||
# .env.example and docs/research/rocm-gpu-pin-and-render-group.md.
|
||||
group_add:
|
||||
- video
|
||||
- render
|
||||
- "${HOST_VIDEO_GID:?run scripts/update.sh first to resolve this}"
|
||||
- "${HOST_RENDER_GID:?run scripts/update.sh first to resolve this}"
|
||||
security_opt:
|
||||
- seccomp=unconfined
|
||||
ipc: host
|
||||
environment:
|
||||
- HSA_OVERRIDE_GFX_VERSION=12.0.1
|
||||
- PYTORCH_ROCM_ARCH=gfx1201
|
||||
# This image manages GPU-group access via GID env vars rather than
|
||||
# relying solely on group_add above (its own README asks for both) —
|
||||
# scripts/update.sh resolves these from the host, same pattern as
|
||||
# SEARXNG_LAN_IP.
|
||||
# This image also wants GID env vars directly (its own README asks
|
||||
# for both these and group_add above) — same HOST_VIDEO_GID/
|
||||
# HOST_RENDER_GID resolved by scripts/update.sh, shared with
|
||||
# llama-server/llama-server-fast now instead of comfyui-only vars.
|
||||
- PUID=${COMFYUI_PUID}
|
||||
- PGID=${COMFYUI_PGID}
|
||||
- VIDEO_GID=${COMFYUI_VIDEO_GID}
|
||||
- RENDER_GID=${COMFYUI_RENDER_GID}
|
||||
- VIDEO_GID=${HOST_VIDEO_GID}
|
||||
- RENDER_GID=${HOST_RENDER_GID}
|
||||
- BASE_STORAGE_PATH=/storage
|
||||
volumes:
|
||||
- comfyui-data:/storage
|
||||
|
||||
Reference in New Issue
Block a user