haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Author the docker-compose stack (llama.cpp + Open WebUI + Qdrant + Lazytainer) #4

New Issue
Closed
opened 2026-08-24 10:09:57 +00:00 by haylan · 1 comment
haylan commented 2026-08-24 10:09:57 +00:00
Owner
Copy Link
Copy Source

Part of #1
Blocked by: #2, #3

Question

Author docker-compose.yml, .env.example, and a model-download helper script implementing the locked decisions on the map (#1): llama.cpp (ROCm/HIP, gfx1201) serving the confirmed Qwen3.8-27B quant from #2 with its Anthropic Messages API shim enabled per #3's findings; Open WebUI (login-enabled, RAG+Memory pointed at Qdrant); a standalone Qdrant service; Lazytainer fronting the llama.cpp container with a 15 min idle timeout; named Docker volumes for all model/service data (no host bind-mounts); ROCm GPU device passthrough (/dev/kfd, /dev/dri) wired correctly for the container. This is the task that turns the map's decisions into a working stack in the repo.

Part of #1 Blocked by: #2, #3 ## Question Author `docker-compose.yml`, `.env.example`, and a model-download helper script implementing the locked decisions on the map (#1): llama.cpp (ROCm/HIP, gfx1201) serving the confirmed Qwen3.8-27B quant from #2 with its Anthropic Messages API shim enabled per #3's findings; Open WebUI (login-enabled, RAG+Memory pointed at Qdrant); a standalone Qdrant service; Lazytainer fronting the llama.cpp container with a 15 min idle timeout; named Docker volumes for all model/service data (no host bind-mounts); ROCm GPU device passthrough (/dev/kfd, /dev/dri) wired correctly for the container. This is the task that turns the map's decisions into a working stack in the repo.
haylan added the wayfinder:task label 2026-08-24 10:09:57 +00:00
haylan added a new dependency 2026-08-24 10:10:27 +00:00
#2 Confirm quantization availability for Qwen3.8-27B
haylan self-assigned this 2026-08-24 10:58:41 +00:00
haylan commented 2026-08-24 11:01:37 +00:00
Author
Owner
Copy Link
Copy Source

Resolved. Authored docker-compose.yml, .env.example, scripts/download-model.sh, and a top-level README.md quickstart — committed on main (f0e4730).

  • llama-server: ghcr.io/ggml-org/llama.cpp:server-rocm (gfx1201-supported), ROCm device passthrough (/dev/kfd, /dev/dri, video/render groups), serving Qwen3.8-27B-UD-Q4_K_XL.gguf from #2. Port 8080 published to the host for direct Claude Code CLI / Kimi CLI access, alongside Open WebUI's internal use.
  • open-webui: WEBUI_AUTH=True, wired to llama-server's OpenAI-compatible endpoint, VECTOR_DB=qdrant + QDRANT_URI. Note: Open WebUI has no env var to seed an admin account — the first signup becomes admin, documented in the README.
  • qdrant: standalone service, own named volume, per the map's decision.
  • lazytainer: labels on llama-server (sleepMethod=stop, 15 min inactiveTimeout, configurable via .env).
  • downloader: one-off compose profile (--profile tools run downloader) pulling the confirmed GGUF straight into the models named volume — no host bind-mount, no extra host tooling needed. Wrapped by scripts/download-model.sh.
  • All persistent state in named volumes (models, qdrant-data, openwebui-data) — no host bind-mounts, per the map's decision.

docker compose config validates. Not yet tested against real hardware — that's #5.

Resolved. Authored `docker-compose.yml`, `.env.example`, `scripts/download-model.sh`, and a top-level `README.md` quickstart — committed on `main` (f0e4730). - **llama-server**: `ghcr.io/ggml-org/llama.cpp:server-rocm` (gfx1201-supported), ROCm device passthrough (`/dev/kfd`, `/dev/dri`, `video`/`render` groups), serving `Qwen3.8-27B-UD-Q4_K_XL.gguf` from #2. Port 8080 published to the host for direct Claude Code CLI / Kimi CLI access, alongside Open WebUI's internal use. - **open-webui**: `WEBUI_AUTH=True`, wired to llama-server's OpenAI-compatible endpoint, `VECTOR_DB=qdrant` + `QDRANT_URI`. Note: Open WebUI has no env var to seed an admin account — the first signup becomes admin, documented in the README. - **qdrant**: standalone service, own named volume, per the map's decision. - **lazytainer**: labels on `llama-server` (`sleepMethod=stop`, 15 min `inactiveTimeout`, configurable via `.env`). - **downloader**: one-off compose profile (`--profile tools run downloader`) pulling the confirmed GGUF straight into the `models` named volume — no host bind-mount, no extra host tooling needed. Wrapped by `scripts/download-model.sh`. - All persistent state in named volumes (`models`, `qdrant-data`, `openwebui-data`) — no host bind-mounts, per the map's decision. `docker compose config` validates. **Not yet tested against real hardware** — that's #5.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:task
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Blocks
#5 Verify the stack on the real Radeon R9700 box (GPU passthrough + Lazytainer idle-stop)
haylan/LLM-Server
#6 Write local-usage docs for Claude Code CLI, Kimi CLI, and OpenCode CLI against the local endpoint
haylan/LLM-Server
Depends on
#2 Confirm quantization availability for Qwen3.8-27B
haylan/LLM-Server
#3 Confirm Qwen3.8-27B tool-calling compatibility with llama.cpp's Anthropic Messages API shim
haylan/LLM-Server
Reference: haylan/LLM-Server#4