haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Verify the stack on the real Radeon R9700 box (GPU passthrough + Lazytainer idle-stop) #5

New Issue
Closed
opened 2026-08-24 10:09:57 +00:00 by haylan · 3 comments
haylan commented 2026-08-24 10:09:57 +00:00
Owner
Copy Link
Copy Source

Part of #1
Blocked by: #4

Question

On the actual Ryzen 9 5950X / R9700 machine: bring the stack up via docker compose up, confirm ROCm sees the GPU inside the llama.cpp container, confirm inference works end-to-end (Open WebUI chat + Anthropic-shim request), then confirm Lazytainer correctly stops the idle container after the configured timeout and cleanly restarts it on the next request without leaking VRAM or hitting the known ROCm 100%-idle-GPU bug (ROCm/ROCm#5706) — check GPU utilization before/after a stop/start cycle. Report pass/fail and any config fixes needed; if the ROCm/HIP backend misbehaves, note whether switching to llama.cpp's Vulkan backend resolves it (see the map's Not-yet-specified section).

Part of #1 Blocked by: #4 ## Question On the actual Ryzen 9 5950X / R9700 machine: bring the stack up via `docker compose up`, confirm ROCm sees the GPU inside the llama.cpp container, confirm inference works end-to-end (Open WebUI chat + Anthropic-shim request), then confirm Lazytainer correctly stops the idle container after the configured timeout and cleanly restarts it on the next request without leaking VRAM or hitting the known ROCm 100%-idle-GPU bug (ROCm/ROCm#5706) — check GPU utilization before/after a stop/start cycle. Report pass/fail and any config fixes needed; if the ROCm/HIP backend misbehaves, note whether switching to llama.cpp's Vulkan backend resolves it (see the map's Not-yet-specified section).
haylan added the wayfinder:task label 2026-08-24 10:09:57 +00:00
haylan self-assigned this 2026-08-24 15:58:51 +00:00
haylan commented 2026-09-06 19:17:21 +00:00
Author
Owner
Copy Link
Copy Source

Confirmed on real hardware. rocm-smi after normal use (llama-server + llama-server-fast both up):

VRAM% 98%   GPU% 100%   SCLK 3402Mhz   Power 101.0W   Temp 81.0°C
WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status
  • VRAM 98%: expected, matches the known math (27B ~25.6GB + fast model ~5GB ≈ 30.6/32GB, see #44). Not a leak.
  • System RAM: fine — free -h available: 26Gi/31Gi, the used/buff/cache split is just llama.cpp's mmap'd model files in page cache, not a leak.
  • GPU 100% + "low-power state" warning simultaneously: this is ROCm/ROCm#5706, exactly as flagged in #1's Notes — GPU stuck pinned at boost clock (3402MHz) while actually idle, a monitoring/power-gating bug, not real compute load (power draw 101W, well under the 300W cap). Confirmed present; two always-resident llama.cpp/ROCm processes (since #44) may make it more likely to trigger than the original single-instance design, though it was already an open risk before that.

Not yet done: the idle-stop cycle test this ticket also asks for (Lazytainer stop/start, check VRAM doesn't leak across a cycle) — moot for llama-server specifically per #40's finding (lazytainer's idle-stop doesn't reliably fire on it anyway, see docs/research/lazytainer-omniroute-idle-stop.md). Also stale: this ticket's "Open WebUI chat" mention predates OmniRoute replacing Open WebUI as the client surface (see #31) — verification should now mean a CLI request through OmniRoute, not Open WebUI.

Next candidate step (unstarted, needs a decision): try llama.cpp's Vulkan backend instead of ROCm/HIP, per the map's Not-yet-specified section — nobody has tested whether that avoids this bug on this card.

Confirmed on real hardware. rocm-smi after normal use (llama-server + llama-server-fast both up): ``` VRAM% 98% GPU% 100% SCLK 3402Mhz Power 101.0W Temp 81.0°C WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status ``` - **VRAM 98%**: expected, matches the known math (27B ~25.6GB + fast model ~5GB ≈ 30.6/32GB, see #44). Not a leak. - **System RAM**: fine — `free -h` `available: 26Gi/31Gi`, the `used`/`buff/cache` split is just llama.cpp's mmap'd model files in page cache, not a leak. - **GPU 100% + "low-power state" warning simultaneously**: this is ROCm/ROCm#5706, exactly as flagged in #1's Notes — GPU stuck pinned at boost clock (3402MHz) while actually idle, a monitoring/power-gating bug, not real compute load (power draw 101W, well under the 300W cap). Confirmed present; two always-resident llama.cpp/ROCm processes (since #44) may make it more likely to trigger than the original single-instance design, though it was already an open risk before that. Not yet done: the idle-stop cycle test this ticket also asks for (Lazytainer stop/start, check VRAM doesn't leak across a cycle) — moot for llama-server specifically per #40's finding (lazytainer's idle-stop doesn't reliably fire on it anyway, see docs/research/lazytainer-omniroute-idle-stop.md). Also stale: this ticket's "Open WebUI chat" mention predates OmniRoute replacing Open WebUI as the client surface (see #31) — verification should now mean a CLI request through OmniRoute, not Open WebUI. Next candidate step (unstarted, needs a decision): try llama.cpp's Vulkan backend instead of ROCm/HIP, per the map's Not-yet-specified section — nobody has tested whether that avoids this bug on this card.
haylan commented 2026-09-06 20:27:21 +00:00
Author
Owner
Copy Link
Copy Source

Confirmed on the real server (#49, merged): the interactive gum-based config sync in scripts/update.sh works as designed — the conflict screen appeared for the stale LLAMA_FAST_PARALLEL=2 vs git's 1, selecting it applied the fix, and the Auto Mode classifier (qwen-code, llama-server-fast) works now. gum's release asset naming matched what ensure_gum() expected — no fix needed there.

Outstanding from this thread's earlier GPU-pin findings (#47, also merged): GPU_MAX_HW_QUEUES=1 and the numeric-GID group_add fix — still worth a follow-up rocm-smi check with both llama-server and llama-server-fast up, if not already done.

Confirmed on the real server (#49, merged): the interactive gum-based config sync in scripts/update.sh works as designed — the conflict screen appeared for the stale LLAMA_FAST_PARALLEL=2 vs git's 1, selecting it applied the fix, and the Auto Mode classifier (qwen-code, llama-server-fast) works now. gum's release asset naming matched what ensure_gum() expected — no fix needed there. Outstanding from this thread's earlier GPU-pin findings (#47, also merged): GPU_MAX_HW_QUEUES=1 and the numeric-GID group_add fix — still worth a follow-up rocm-smi check with both llama-server and llama-server-fast up, if not already done.
haylan commented 2026-09-06 20:30:06 +00:00
Author
Owner
Copy Link
Copy Source

Confirmed: GPU_MAX_HW_QUEUES=1 + the numeric-GID group_add fix (#47) resolved the 100%-GPU-pin with both llama-server and llama-server-fast up. Combined with the earlier classifier-context and gum-sync confirmations above, real-hardware verification for this stack is complete: GPU passthrough works, ROCm sees the card, inference works end-to-end (via OmniRoute, not Open WebUI — that premise is stale, see earlier comment), and the known-risk GPU pin bug is fixed and confirmed rather than just theorized. The original Lazytainer idle-stop cycle test is moot by design (#40: it doesn't reliably fire on llama-server regardless; switch-model.sh bypasses it intentionally) rather than untested. Closing.

Confirmed: GPU_MAX_HW_QUEUES=1 + the numeric-GID group_add fix (#47) resolved the 100%-GPU-pin with both llama-server and llama-server-fast up. Combined with the earlier classifier-context and gum-sync confirmations above, real-hardware verification for this stack is complete: GPU passthrough works, ROCm sees the card, inference works end-to-end (via OmniRoute, not Open WebUI — that premise is stale, see earlier comment), and the known-risk GPU pin bug is fixed and confirmed rather than just theorized. The original Lazytainer idle-stop cycle test is moot by design (#40: it doesn't reliably fire on llama-server regardless; switch-model.sh bypasses it intentionally) rather than untested. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:task
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Depends on
#4 Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer)
haylan/LLM-Server
Reference: haylan/LLM-Server#5