LLM-Server/docs/research/rocm-gpu-pin-and-render-group.md

Research: GPU pinned at 100% with two concurrent llama.cpp containers, and the intermittent "render" group startup error

Question: After adding llama-server-fast (#44), real-hardware testing on the R9700 showed rocm-smi pinned at 100% GPU / ~73-101W whenever both llama-server and llama-server-fast run concurrently, dropping to 3% / ~25-60W the moment either one alone is stopped. Separately, docker compose up intermittently failed with Error response from daemon: unable to find group render: no matching entries in group file — confirmed new since the second GPU service was added. See issue #5's comment thread for the raw rocm-smi/free -h output this doc is diagnosing.

GPU pin: root cause and fix

Confirmed via #5's own data: either container alone is fine (3% GPU, low power). The pin only appears with two concurrent HIP-context-holding processes on the same GPU. This matches ROCm/ROCm#5706 (already flagged as a risk in map #1) — full comment thread confirms:

• Root cause: an AMD MES (Micro Engine Scheduler) firmware bug triggered by HIP hardware-queue creation, pinning the GPU at boost clock the moment ROCm initializes a queue. Not llama.cpp-specific — reproduced with vLLM and bare PyTorch ROCm too. Source: ROCm/ROCm#5706 (tcgu-amd, AMD engineer, confirms MES firmware root cause; closed as "fixed" in March, but a report as recent as May 24 shows it recurring even on patched firmware/kernel).
• Validated workaround: GPU_MAX_HW_QUEUES=1 as a container env var. One report ran a controlled before/after on the exact image this stack uses (ghcr.io/ggml-org/llama.cpp:server-rocm, R9700/gfx1201): baseline 100% GPU / 95W → with the var set, 3% GPU / 22W, VRAM unchanged. Source: same thread, interconnectedMe's comment.
• Semantics (why this should apply to our two-container case, not just the single-process case tested above): GPU_MAX_HW_QUEUES is a per-process HIP runtime setting — it caps how many HSA/hardware queues that process's HIP runtime allocates, default higher (over-subscription is what causes the penalty). Source: AMD ROCm workload-optimization docs. Since it's per-process, setting it on each container independently is the correct scope — it should reduce total concurrent hardware-queue creation across both processes, which is the trigger condition MES chokes on. Caveat: no primary source explicitly tested two concurrent containers both set to GPU_MAX_HW_QUEUES=1 — this is a well-grounded extrapolation from confirmed per-process semantics and the same root-cause mechanism, not a directly-reproduced fix for our exact topology. Verify with rocm-smi after applying, both containers up.

"unable to find group render" — a real Docker bug, not flaky hardware

This is a known, documented Docker limitation, not something specific to this stack: group_add by name requires Docker to resolve that name against the container's own /etc/group file — if the image doesn't define a render entry there (common for minimal/slim base images), resolution fails. Source: docker/cli#4714 ("docker run --group-add by name doesn't add group from host as documented") and docker/compose#7277 (same "no matching entries in group file" error).

Confirms why it's now intermittent rather than always-broken: this repo's docker-compose.yml uses group_add: [video, render] (plain names) on three GPU services now (llama-server, llama-server-fast, comfyui). Docker Compose starts containers concurrently, and each does its own name-resolution lookup independently — with only one GPU service before #44, the resolution almost always won its race; with two (soon three, once comfyui's downloader/model land per #46) the odds of losing that race and hitting the unresolved-name path go up. This is consistent with the user's own observation that it's new since the second GPU service.

Fix, already precedented in this repo: scripts/update.sh already resolves the host's real video/render numeric GIDs for the comfyui service (COMFYUI_VIDEO_GID/COMFYUI_RENDER_GID, passed as app-level env vars) — but comfyui's own group_add: still uses plain names too, so it isn't actually protected by that either. The correct fix per the Docker issues above: use the resolved numeric GIDs in group_add: itself (Compose accepts numeric strings directly), not names, on all three GPU services. Numeric GIDs skip the name-resolution step entirely, eliminating both the flakiness and the race.

Recommendation

1. Add GPU_MAX_HW_QUEUES=1 to both llama-server and llama-server-fast's environment: blocks. Verify with rocm-smi after redeploy, both containers up — this is the one part of this doc that's extrapolated rather than directly reproduced, so real confirmation matters here.
2. Resolve host video/render GIDs once (generalize the existing COMFYUI_VIDEO_GID/COMFYUI_RENDER_GID pattern in scripts/update.sh to shared HOST_VIDEO_GID/HOST_RENDER_GID vars), and switch group_add: on all three GPU services (llama-server, llama-server-fast, comfyui) from [video, render] (names) to the resolved numeric GIDs. Removes the race entirely rather than reducing its odds.

Sources

• ROCm/ROCm#5706 — full comment thread
• AMD ROCm — MI300/MI350 workload optimization docs (GPU_MAX_HW_QUEUES)
• docker/cli#4714 — group_add by name doesn't work as documented
• docker/compose#7277 — "no matching entries in group file"
• This repo's issue #5 (real-hardware rocm-smi/free -h evidence this doc diagnoses)