Local AI inference stack: llama.cpp + Qwen3.8-27B on Radeon R9700 #1

New Issue

2026-08-24T10:09:29Z

haylan commented

2026-08-24 10:09:29 +00:00

Destination

A finished, working docker-compose stack (with a .env file) for local AI inference on this machine: llama.cpp serving Qwen3.8-27B (Unsloth-style dynamic 4-bit GGUF) over ROCm on the AMD Radeon AI PRO R9700, fronted by Open WebUI (login-protected) with RAG + Memory backed by a standalone Qdrant service, Lazytainer auto-suspending the inference container after idle, all persistent state in named Docker volumes — plus documentation for pointing Claude Code CLI, Kimi CLI, and OpenCode CLI at the local endpoint as a fallback coding backend when Claude Code tokens run out.

Notes

Hardware: AMD Ryzen 9 5950X, 64GB RAM, AMD Radeon AI PRO R9700 (32GB VRAM, RDNA4/gfx1201, ROCm 7.2+ officially supported). Single machine, LAN-only, single user (no multi-tenant/auth-hardening concerns beyond basic login).

Locked decisions (settled during charting, not open tickets):

Backend: llama.cpp (ROCm/HIP, target gfx1201), not vLLM — more mature ROCm/RDNA4 path today, and has a native Anthropic Messages API shim (/v1/messages) that Claude Code CLI needs.
Model: Qwen3.8-27B — dense, hybrid linear-attention architecture (VLM-capable), Apache 2.0, 262K native context. Chosen explicitly by the user over the better-documented/purpose-built Qwen3-Coder-30B-A3B-Instruct (MoE, agentic-coding-tuned). Known risk, accepted knowingly — now sharpened by research: Qwen3.8-27B's tool-calling reliability is not just unverified but has dated, open upstream llama.cpp bugs against its Qwen3.5 lineage (parser fails on text preceding <tool_call>, tool calls emitted as inert XML inside thinking blocks). The fix path may need an llama.cpp parser change, not just a template swap. #6 (docs) and #5 (real-box verification) must include a live multi-turn tool-use smoke test before treating this as a working Claude Code fallback. Details in docs/research/qwen3.8-27b-tool-calling.md.
Quantization: confirmed — unsloth/Qwen3.8-27B-GGUF, file Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6GB). Fits comfortably: ~19.6GB weights+KV at 32K context, ~25.6GB at 128K (thanks to the model's hybrid attention — only 16/64 layers are full-attention/KV-caching, the rest are fixed-state Gated DeltaNet layers, so KV cache grows far cheaper than a plain-attention 27B). Optional mmproj-BF16.gguf (931MB) only needed if image/video input is ever used.
Vector DB: standalone Qdrant as its own compose service — not Open WebUI's bundled Chroma — deliberately decoupled to leave room for a possible future multi-model "AI gateway" (see Out of scope).
Memory: both Open WebUI's Memory feature (facts about the user/projects) and RAG (document retrieval), both backed by the Qdrant service.
Auth: Open WebUI login/password enabled (first signup becomes admin — no env-seeded account, see #4's resolution).
Network access: Open WebUI reachable as ai.home (LAN) and ai.haylan.ch (external, via an existing DMZ) through the existing Nginx Proxy Manager instance already fronting other self-hosted services on this network — no reverse proxy added to this stack. llama.cpp's raw inference API stays LAN-only, not registered in NPM (it has no auth of its own). See docs/network-access.md.
Idle-suspend: Lazytainer fronting the inference container, 15 min idle timeout. Known open risk: an open ROCm bug (ROCm/ROCm#5706) pins the R9700 at 100% GPU usage after idle on llama.cpp's HIP backend (the Vulkan backend doesn't have this bug) — may interact badly with Lazytainer's activity-based stop/start. Needs empirical verification on real hardware, not an assumption.
Storage: named Docker volumes (not host bind-mounts) for models and all service data.
Model acquisition: an automated download script (run manually, not baked into docker compose up), once the exact model file/quant is confirmed.
Coding-agent wiring: Claude Code CLI via ANTHROPIC_BASE_URL pointed at llama.cpp's native Anthropic Messages API shim; Kimi CLI via its OpenAI-compatible base_url config (config.toml) pointed at llama.cpp's /v1/chat/completions; OpenCode CLI — confirmed as anomalyco/opencode (renamed from sst/opencode; distinct from the unrelated opencode-ai/opencode Go TUI), installed via curl -fsSL https://opencode.ai/install | bash, configured via opencode.json with a custom @ai-sdk/openai-compatible provider pointed at http://localhost:8080/v1. Requires llama.cpp's --jinja flag (already set in docker-compose.yml) or OpenCode's unconditional tool-calling scaffolding 500s. All three need local-usage documentation as a deliverable (#6).

Execution note: this effort carries execution into the map (per the destination above) — tickets can end in actual compose files, scripts, and docs committed to the repo, not only decisions recorded here.

Skills to consult per ticket: /research for fact-finding tickets; /grilling + /domain-modeling if a new open decision surfaces; /prototype if a UI/behavior question comes up (unlikely for this effort).

Tickets

Decisions so far

Add llama-server-fast: small non-thinking classifier/fast model for qwen-code Auto Mode — confirmed working end-to-end on the server: deployed, registered in OmniRoute as qwen3-4b-fast, wired into qwen-code's fastModel, Auto Mode classifier functioning after the fixes in #47/#49.
Verify the stack on the real Radeon R9700 box (GPU passthrough + Lazytainer idle-stop) — confirmed: GPU passthrough/ROCm works, inference works end-to-end via OmniRoute (Open WebUI premise was stale). Surfaced and fixed three real bugs along the way: the ROCm/ROCm#5706 100%-GPU-pin (GPU_MAX_HW_QUEUES=1, #47), a Docker group_add-by-name race (numeric GIDs, #47), and llama-server-fast per-slot context exhaustion breaking qwen-code Auto Mode (#49). Lazytainer idle-stop cycle test is moot by design (#40) rather than untested. See docs/research/rocm-gpu-pin-and-render-group.md.
How is ai.home / ai.haylan.ch reverse-proxied and TLS-terminated? — resolved via the user's existing NPM instance, no new proxy needed; llama.cpp API kept LAN-only. Docs corrected (they wrongly said localhost). See docs/network-access.md.
Write local-usage docs for Claude Code CLI, Kimi CLI, and OpenCode CLI against the local endpoint — done: docs/coding-cli-setup.md committed on main, covers config for all three CLIs and leads with the shared tool-calling risk warning.
Research OpenCode CLI installation and local-endpoint setup — confirmed project identity (anomalyco/opencode, ex-sst/opencode), install command, and opencode.json provider config. Flagged: OpenCode has its own documented brittleness against local OpenAI-compatible backends (anomalyco/opencode#1890, #20669 — empty tool_calls hangs, missing-description crashes) independent of the Qwen3.8-27B llama.cpp parser risk already noted from #3. See docs/research/opencode-cli-setup.md.
Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer) — done: docker-compose.yml, .env.example, scripts/download-model.sh, README.md committed on main. Config validates; untested on real hardware (that's #5).
Confirm quantization availability for Qwen3.8-27B — confirmed: unsloth/Qwen3.8-27B-GGUF / Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6GB), fits 32K-128K context comfortably on 32GB VRAM. See docs/research/qwen3.8-27b-quant.md on branch research/qwen3.8-27b-quant.
Confirm Qwen3.8-27B tool-calling compatibility with llama.cpp's Anthropic Messages API shim — not confirmed compatible; Qwen3.8-27B's Qwen3.5-lineage has open/partially-fixed llama.cpp tool-call parser bugs (llama.cpp#21158, #20837, #20260) distinct from the Qwen3-Coder issue — a live smoke test (multi-turn tool use, thinking enabled) is needed before relying on it unattended. See docs/research/qwen3.8-27b-tool-calling.md on branch research/qwen3.8-27b-tool-calling.

Not yet specified

Whether the ROCm 100%-idle-GPU bug forces a switch from llama.cpp's HIP/ROCm backend to its Vulkan backend for this deployment — can't phrase the fix sharply until real-hardware verification happens.
Whether vLLM becomes a fallback backend if llama.cpp's ROCm/HIP path proves broken in practice on this exact card — contingent on the same verification.

Out of scope

Multi-model "AI gateway" / routing service — the user noted Qdrant's standalone-service architecture leaves room for this later, but building an actual gateway is not part of this stack. Not ticketed.

## Destination A finished, working docker-compose stack (with a .env file) for local AI inference on this machine: **llama.cpp** serving **Qwen3.8-27B** (Unsloth-style dynamic 4-bit GGUF) over ROCm on the **AMD Radeon AI PRO R9700**, fronted by **Open WebUI** (login-protected) with RAG + Memory backed by a standalone **Qdrant** service, **Lazytainer** auto-suspending the inference container after idle, all persistent state in named Docker volumes — plus documentation for pointing **Claude Code CLI**, **Kimi CLI**, and **OpenCode CLI** at the local endpoint as a fallback coding backend when Claude Code tokens run out. ## Notes **Hardware**: AMD Ryzen 9 5950X, 64GB RAM, AMD Radeon AI PRO R9700 (32GB VRAM, RDNA4/gfx1201, ROCm 7.2+ officially supported). Single machine, LAN-only, single user (no multi-tenant/auth-hardening concerns beyond basic login). **Locked decisions** (settled during charting, not open tickets): - **Backend**: llama.cpp (ROCm/HIP, target gfx1201), not vLLM — more mature ROCm/RDNA4 path today, and has a native Anthropic Messages API shim (/v1/messages) that Claude Code CLI needs. - **Model**: Qwen3.8-27B — dense, hybrid linear-attention architecture (VLM-capable), Apache 2.0, 262K native context. Chosen explicitly by the user over the better-documented/purpose-built Qwen3-Coder-30B-A3B-Instruct (MoE, agentic-coding-tuned). **Known risk, accepted knowingly — now sharpened by research**: Qwen3.8-27B's tool-calling reliability is not just unverified but has *dated, open upstream llama.cpp bugs* against its Qwen3.5 lineage (parser fails on text preceding `<tool_call>`, tool calls emitted as inert XML inside thinking blocks). The fix path may need an llama.cpp parser change, not just a template swap. #6 (docs) and #5 (real-box verification) must include a live multi-turn tool-use smoke test before treating this as a working Claude Code fallback. Details in `docs/research/qwen3.8-27b-tool-calling.md`. - **Quantization**: **confirmed** — `unsloth/Qwen3.8-27B-GGUF`, file `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6GB). Fits comfortably: ~19.6GB weights+KV at 32K context, ~25.6GB at 128K (thanks to the model's hybrid attention — only 16/64 layers are full-attention/KV-caching, the rest are fixed-state Gated DeltaNet layers, so KV cache grows far cheaper than a plain-attention 27B). Optional `mmproj-BF16.gguf` (931MB) only needed if image/video input is ever used. - **Vector DB**: standalone **Qdrant** as its own compose service — not Open WebUI's bundled Chroma — deliberately decoupled to leave room for a possible future multi-model "AI gateway" (see Out of scope). - **Memory**: both Open WebUI's **Memory** feature (facts about the user/projects) and **RAG** (document retrieval), both backed by the Qdrant service. - **Auth**: Open WebUI login/password enabled (first signup becomes admin — no env-seeded account, see #4's resolution). - **Network access**: Open WebUI reachable as `ai.home` (LAN) and `ai.haylan.ch` (external, via an existing DMZ) through the **existing Nginx Proxy Manager** instance already fronting other self-hosted services on this network — no reverse proxy added to this stack. llama.cpp's raw inference API stays **LAN-only, not registered in NPM** (it has no auth of its own). See `docs/network-access.md`. - **Idle-suspend**: Lazytainer fronting the inference container, **15 min** idle timeout. **Known open risk**: an open ROCm bug (ROCm/ROCm#5706) pins the R9700 at 100% GPU usage after idle on llama.cpp's HIP backend (the Vulkan backend doesn't have this bug) — may interact badly with Lazytainer's activity-based stop/start. Needs empirical verification on real hardware, not an assumption. - **Storage**: named Docker volumes (not host bind-mounts) for models and all service data. - **Model acquisition**: an automated download script (run manually, not baked into docker compose up), once the exact model file/quant is confirmed. - **Coding-agent wiring**: Claude Code CLI via ANTHROPIC_BASE_URL pointed at llama.cpp's native Anthropic Messages API shim; Kimi CLI via its OpenAI-compatible base_url config (config.toml) pointed at llama.cpp's /v1/chat/completions; **OpenCode CLI** — confirmed as `anomalyco/opencode` (renamed from `sst/opencode`; distinct from the unrelated `opencode-ai/opencode` Go TUI), installed via `curl -fsSL https://opencode.ai/install | bash`, configured via `opencode.json` with a custom `@ai-sdk/openai-compatible` provider pointed at `http://localhost:8080/v1`. Requires llama.cpp's `--jinja` flag (already set in docker-compose.yml) or OpenCode's unconditional tool-calling scaffolding 500s. All three need local-usage documentation as a deliverable (#6). **Execution note**: this effort carries execution into the map (per the destination above) — tickets can end in actual compose files, scripts, and docs committed to the repo, not only decisions recorded here. **Skills to consult per ticket**: /research for fact-finding tickets; /grilling + /domain-modeling if a new open decision surfaces; /prototype if a UI/behavior question comes up (unlikely for this effort). ## Tickets - [x] [Confirm quantization availability for Qwen3.8-27B](https://git.arthurerlich.de/haylan/LLM-Server/issues/2) - [x] [Confirm Qwen3.8-27B tool-calling compatibility with llama.cpp's Anthropic Messages API shim](https://git.arthurerlich.de/haylan/LLM-Server/issues/3) - [x] [Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer)](https://git.arthurerlich.de/haylan/LLM-Server/issues/4) - [x] [Verify the stack on the real Radeon R9700 box (GPU passthrough + Lazytainer idle-stop)](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) - [x] [Research OpenCode CLI installation and local-endpoint setup](https://git.arthurerlich.de/haylan/LLM-Server/issues/7) - [x] [Write local-usage docs for Claude Code CLI, Kimi CLI, and OpenCode CLI against the local endpoint](https://git.arthurerlich.de/haylan/LLM-Server/issues/6) - [x] [How is ai.home / ai.haylan.ch reverse-proxied and TLS-terminated?](https://git.arthurerlich.de/haylan/LLM-Server/issues/8) - [x] [Add llama-server-fast: small non-thinking classifier/fast model for qwen-code Auto Mode](https://git.arthurerlich.de/haylan/LLM-Server/issues/44) ## Decisions so far - [Add llama-server-fast: small non-thinking classifier/fast model for qwen-code Auto Mode](https://git.arthurerlich.de/haylan/LLM-Server/issues/44) — confirmed working end-to-end on the server: deployed, registered in OmniRoute as qwen3-4b-fast, wired into qwen-code's fastModel, Auto Mode classifier functioning after the fixes in #47/#49. - [Verify the stack on the real Radeon R9700 box (GPU passthrough + Lazytainer idle-stop)](https://git.arthurerlich.de/haylan/LLM-Server/issues/5) — confirmed: GPU passthrough/ROCm works, inference works end-to-end via OmniRoute (Open WebUI premise was stale). Surfaced and fixed three real bugs along the way: the ROCm/ROCm#5706 100%-GPU-pin (GPU_MAX_HW_QUEUES=1, #47), a Docker group_add-by-name race (numeric GIDs, #47), and llama-server-fast per-slot context exhaustion breaking qwen-code Auto Mode (#49). Lazytainer idle-stop cycle test is moot by design (#40) rather than untested. See docs/research/rocm-gpu-pin-and-render-group.md. - [How is ai.home / ai.haylan.ch reverse-proxied and TLS-terminated?](https://git.arthurerlich.de/haylan/LLM-Server/issues/8) — resolved via the user's existing NPM instance, no new proxy needed; llama.cpp API kept LAN-only. Docs corrected (they wrongly said `localhost`). See `docs/network-access.md`. - [Write local-usage docs for Claude Code CLI, Kimi CLI, and OpenCode CLI against the local endpoint](https://git.arthurerlich.de/haylan/LLM-Server/issues/6) — done: `docs/coding-cli-setup.md` committed on `main`, covers config for all three CLIs and leads with the shared tool-calling risk warning. - [Research OpenCode CLI installation and local-endpoint setup](https://git.arthurerlich.de/haylan/LLM-Server/issues/7) — confirmed project identity (`anomalyco/opencode`, ex-`sst/opencode`), install command, and `opencode.json` provider config. Flagged: OpenCode has its own documented brittleness against local OpenAI-compatible backends (anomalyco/opencode#1890, #20669 — empty tool_calls hangs, missing-description crashes) independent of the Qwen3.8-27B llama.cpp parser risk already noted from #3. See `docs/research/opencode-cli-setup.md`. - [Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer)](https://git.arthurerlich.de/haylan/LLM-Server/issues/4) — done: `docker-compose.yml`, `.env.example`, `scripts/download-model.sh`, `README.md` committed on `main`. Config validates; untested on real hardware (that's #5). - [Confirm quantization availability for Qwen3.8-27B](https://git.arthurerlich.de/haylan/LLM-Server/issues/2) — confirmed: `unsloth/Qwen3.8-27B-GGUF` / `Qwen3.8-27B-UD-Q4_K_XL.gguf` (17.6GB), fits 32K-128K context comfortably on 32GB VRAM. See `docs/research/qwen3.8-27b-quant.md` on branch `research/qwen3.8-27b-quant`. - [Confirm Qwen3.8-27B tool-calling compatibility with llama.cpp's Anthropic Messages API shim](https://git.arthurerlich.de/haylan/LLM-Server/issues/3) — **not confirmed compatible**; Qwen3.8-27B's Qwen3.5-lineage has open/partially-fixed llama.cpp tool-call parser bugs (llama.cpp#21158, #20837, #20260) distinct from the Qwen3-Coder issue — a live smoke test (multi-turn tool use, thinking enabled) is needed before relying on it unattended. See `docs/research/qwen3.8-27b-tool-calling.md` on branch `research/qwen3.8-27b-tool-calling`. ## Not yet specified - Whether the ROCm 100%-idle-GPU bug forces a switch from llama.cpp's HIP/ROCm backend to its Vulkan backend for this deployment — can't phrase the fix sharply until real-hardware verification happens. - Whether vLLM becomes a fallback backend if llama.cpp's ROCm/HIP path proves broken in practice on this exact card — contingent on the same verification. ## Out of scope - **Multi-model "AI gateway" / routing service** — the user noted Qdrant's standalone-service architecture leaves room for this later, but building an actual gateway is not part of this stack. Not ticketed.

haylan added the wayfinder:map label 2026-08-24 10:09:29 +00:00