From 76043e2c6f04063bb62cc132eadb9c3fbfe12863 Mon Sep 17 00:00:00 2001 From: ArthurErlich Date: Wed, 9 Sep 2026 18:10:39 +0200 Subject: [PATCH] tune(llm): partial GPU offload for qwen-classifier, real headroom MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full offload (999 layers) left only ~768MB free VRAM regardless of batch-size/flash-attn tuning — that gap tracks roughly fixed regardless of those knobs, most likely ROCm's own per-process HIP context overhead (same ROCm#5706 quirk already noted for two HIP contexts sharing this card, see llama-server's GPU_MAX_HW_QUEUES comment above). Dropping to 28/36 layers on GPU (6 layers + their KV on CPU) trades a slice of speed for real freed VRAM — still >80% of layers on GPU, nowhere near CPU-only's unusable latency. Verifying live before locking this number in. Co-Authored-By: Claude Sonnet 5 --- docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker-compose.yml b/docker-compose.yml index 705ece5..87bc738 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -99,7 +99,7 @@ services: -m /models/${LLAMA_CLASSIFIER_MODEL_FILE:-Qwen3-4B-Instruct-2507-UD-Q4_K_XL.gguf} --host 0.0.0.0 --port 8080 - --n-gpu-layers 999 + --n-gpu-layers 28 --ctx-size 65536 --parallel 1 --batch-size 512