QWEN.md

Agent instructions for working in this repo (issue tracker, domain docs, mandatory deploy flow) live in CLAUDE.md — read it first.

Project Overview

Local AI inference stack for a single AMD Radeon AI PRO R9700 (32GB VRAM, gfx1201/ROCm) homelab box. Not a code project — it's a Docker Compose deployment plus operational docs and scripts. The stack:

llama.cpp (ROCm) serves Qwen3.8-27B (Qwen3.8-27B-UD-Q4_K_XL.gguf, fully GPU-resident, 262K context with q8_0 KV cache). Internal-only: no published host port, no auth of its own.
OmniRoute (AI gateway, replaced LiteLLM in issue #31) fronts everything: per-workload API keys, usage tracking, SearXNG-backed web search. Split ports: API ${OMNIROUTE_API_PORT:-20129}, dashboard ${OMNIROUTE_DASHBOARD_PORT:-20128} (dashboard is host/LAN-only, never published externally).
ComfyUI (yurisasc's ROCm image, gfx1201-tuned) for local image generation (Qwen-Image FP8). Shares the GPU with llama-server — never runs concurrently with it; use ./scripts/switch-model.sh.
Lazytainer auto-suspends llama-server after idle (15 min default). Note: its packet-threshold detector can't reliably distinguish OmniRoute's health pings from real traffic (issue #40) — the scripted swap in switch-model.sh exists because of this.
RAG stores: Qdrant (vector, 6333) + Neo4j (graph, 7474/7687).
All services live on the ai-stack docker network. External clients reach the gateway via proxy-ai.home / proxy-ai.haylan.ch (Nginx Proxy Manager); see docs/network-access.md.

Coding CLIs (Claude Code, Kimi, OpenCode, Qwen Code) point at the gateway, never at llama-server directly — see docs/coding-cli-setup/index.md.

Known risk: Qwen3.8-27B tool-calling against llama.cpp's Anthropic shim has open upstream parser bugs — see docs/research/qwen3.8-27b-tool-calling.md. Don't trust it for unattended agentic work until smoke-tested (issues #5, #17).

Key Files

File	Purpose
`docker-compose.yml`	The whole stack. Comments in it are load-bearing (ROCm GID workarounds, GPU_MAX_HW_QUEUES, timeout rationale) — read before editing.
`.env.example`	Defaults for every tunable. Secrets/host-resolved values are blank and auto-filled by `update.sh`.
`scripts/update.sh`	The one command to run after any repo change on the server. Creates `.env`, fills blank secrets (openssl), resolves `SEARXNG_LAN_IP`/GIDs, syncs tunables from `.env.example` (conflicts are interactive or hard errors non-interactively), downloads missing model files, pulls/builds, recreates only what changed. Idempotent.
`scripts/switch-model.sh {qwen\|comfyui}`	Manual GPU-residency swap between llama-server and comfyui.
`docs/proxy-key-onboarding.md`	Minting per-workload API keys (manual, dashboard-only — no scripted flow yet, issue #37).
`docs/coding-cli-setup/`	Per-CLI endpoint/wire-format/config recipes.
`docs/research/`	Research trail behind every major decision (model choice, quant, ROCm quirks, gateway selection). Read the relevant doc before re-litigating a decision.
`docs/agents/issue-tracker.md`	Gitea/`tea` CLI conventions for this repo's issue tracking.
`docs/agents/domain.md`	How to consume `CONTEXT.md` + `docs/adr/` (both may not exist yet — proceed silently if absent).

Working in This Repo

Deploying changes — mandatory flow

The running stack lives on a separate box (the R9700 server), not wherever this repo is edited. After any change to docker-compose.yml, .env.example, or a scripts/ file:

Commit and push.
Run ./scripts/update.sh on the server to apply it.
If this session has no shell access to the server, say so explicitly and tell the user to run it — never describe a change as done without step 2.

Issue tracker

Issues live as Gitea issues on git.arthurerlich.de (repo haylan/LLM-Server). Use the tea CLI (already authenticated) — conventions in docs/agents/issue-tracker.md. Large efforts are tracked via wayfinder map issues (wayfinder:map label) with child tickets and native dependency blocking.

Conventions

ponytail: comments mark deliberate simplifications with their known ceiling and upgrade path (e.g. lazytainer timeout tuning lives in compose labels, not a separate config file; no rollback logic in update.sh — git revert + re-run is recovery). Keep them when editing nearby code; they encode "why this looks like a shortcut".
Secrets stay blank in .env.example and are filled by update.sh via set_if_blank — never hardcode or commit real secrets. OMNIROUTE_STORAGE_ENCRYPTION_KEY and the like must never change after first run (encrypted data becomes unreadable).
GPU group access is numeric GIDs (HOST_VIDEO_GID/HOST_RENDER_GID), resolved by update.sh — don't switch group_add to named groups (Docker resolves names against the container's /etc/group, not the host's; see docs/research/rocm-gpu-pin-and-render-group.md).
One-off downloaders (downloader* services) use test -f guards so re-runs skip existing files; they run as root because the named volume is root-owned.
Comments in this repo are unusually dense and explanatory — that's the house style. When changing behavior, update the comment explaining why, not just the what.
Tunables with real defaults live in .env.example; update.sh syncs them into the server's .env every run. A divergent server value is a conflict, not a silent overwrite.

@CLAUDE.md

5.5 KiB Raw Blame History