haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Fix/omniroute pr agent timeout #54

Merged
haylan merged 2 commits from fix/omniroute-pr-agent-timeout into main 2026-09-09 14:29:13 +00:00
Conversation 0 Commits 2 Files Changed 5
+2 -59
2 changed files with 2 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit b3a64fe4b5 - Show all commits
AGENTS.md
+1 -2
View File
@@ -1,2 +1 @@
# AGENTS.md
Agent instructions for this repo live in [CLAUDE.md](./CLAUDE.md) — read it before working here.
@CLAUDE.md
QWEN.md
-56
View File
@@ -1,57 +1 @@
# QWEN.md
Agent instructions for working in this repo (issue tracker, domain docs, mandatory deploy flow) live in [CLAUDE.md](./CLAUDE.md) — read it first.
## Project Overview
Local AI inference stack for a single AMD Radeon AI PRO R9700 (32GB VRAM, gfx1201/ROCm) homelab box. Not a code project — it's a **Docker Compose deployment** plus operational docs and scripts. The stack:
- **llama.cpp (ROCm)** serves Qwen3.8-27B (`Qwen3.8-27B-UD-Q4_K_XL.gguf`, fully GPU-resident, 262K context with q8_0 KV cache). Internal-only: no published host port, no auth of its own.
- **OmniRoute** (AI gateway, replaced LiteLLM in issue #31) fronts everything: per-workload API keys, usage tracking, SearXNG-backed web search. Split ports: API `${OMNIROUTE_API_PORT:-20129}`, dashboard `${OMNIROUTE_DASHBOARD_PORT:-20128}` (dashboard is host/LAN-only, never published externally).
- **ComfyUI** (yurisasc's ROCm image, gfx1201-tuned) for local image generation (Qwen-Image FP8). Shares the GPU with llama-server — **never runs concurrently** with it; use `./scripts/switch-model.sh`.
- **Lazytainer** auto-suspends llama-server after idle (15 min default). Note: its packet-threshold detector can't reliably distinguish OmniRoute's health pings from real traffic (issue #40) — the scripted swap in `switch-model.sh` exists because of this.
- **RAG stores**: Qdrant (vector, 6333) + Neo4j (graph, 7474/7687).
- All services live on the `ai-stack` docker network. External clients reach the gateway via `proxy-ai.home` / `proxy-ai.haylan.ch` (Nginx Proxy Manager); see `docs/network-access.md`.
Coding CLIs (Claude Code, Kimi, OpenCode, Qwen Code) point at the gateway, never at llama-server directly — see `docs/coding-cli-setup/index.md`.
**Known risk**: Qwen3.8-27B tool-calling against llama.cpp's Anthropic shim has open upstream parser bugs — see `docs/research/qwen3.8-27b-tool-calling.md`. Don't trust it for unattended agentic work until smoke-tested (issues #5, #17).
## Key Files
| File | Purpose |
|---|---|
| `docker-compose.yml` | The whole stack. Comments in it are load-bearing (ROCm GID workarounds, GPU_MAX_HW_QUEUES, timeout rationale) — read before editing. |
| `.env.example` | Defaults for every tunable. Secrets/host-resolved values are blank and auto-filled by `update.sh`. |
| `scripts/update.sh` | **The one command to run after any repo change** on the server. Creates `.env`, fills blank secrets (openssl), resolves `SEARXNG_LAN_IP`/GIDs, syncs tunables from `.env.example` (conflicts are interactive or hard errors non-interactively), downloads missing model files, pulls/builds, recreates only what changed. Idempotent. |
| `scripts/switch-model.sh {qwen\|comfyui}` | Manual GPU-residency swap between llama-server and comfyui. |
| `docs/proxy-key-onboarding.md` | Minting per-workload API keys (manual, dashboard-only — no scripted flow yet, issue #37). |
| `docs/coding-cli-setup/` | Per-CLI endpoint/wire-format/config recipes. |
| `docs/research/` | Research trail behind every major decision (model choice, quant, ROCm quirks, gateway selection). Read the relevant doc before re-litigating a decision. |
| `docs/agents/issue-tracker.md` | Gitea/`tea` CLI conventions for this repo's issue tracking. |
| `docs/agents/domain.md` | How to consume `CONTEXT.md` + `docs/adr/` (both may not exist yet — proceed silently if absent). |
## Working in This Repo
### Deploying changes — mandatory flow
The running stack lives on a **separate box** (the R9700 server), not wherever this repo is edited. After any change to `docker-compose.yml`, `.env.example`, or a `scripts/` file:
1. Commit and push.
2. Run `./scripts/update.sh` **on the server** to apply it.
3. If this session has no shell access to the server, say so explicitly and tell the user to run it — never describe a change as done without step 2.
### Issue tracker
Issues live as Gitea issues on `git.arthurerlich.de` (repo `haylan/LLM-Server`). Use the **`tea` CLI** (already authenticated) — conventions in `docs/agents/issue-tracker.md`. Large efforts are tracked via wayfinder map issues (`wayfinder:map` label) with child tickets and native dependency blocking.
### Conventions
- **`ponytail:` comments** mark deliberate simplifications with their known ceiling and upgrade path (e.g. lazytainer timeout tuning lives in compose labels, not a separate config file; no rollback logic in `update.sh` — `git revert` + re-run is recovery). Keep them when editing nearby code; they encode "why this looks like a shortcut".
- **Secrets stay blank in `.env.example`** and are filled by `update.sh` via `set_if_blank` — never hardcode or commit real secrets. `OMNIROUTE_STORAGE_ENCRYPTION_KEY` and the like must never change after first run (encrypted data becomes unreadable).
- **GPU group access is numeric GIDs** (`HOST_VIDEO_GID`/`HOST_RENDER_GID`), resolved by `update.sh` — don't switch `group_add` to named groups (Docker resolves names against the container's `/etc/group`, not the host's; see `docs/research/rocm-gpu-pin-and-render-group.md`).
- **One-off downloaders** (`downloader*` services) use `test -f` guards so re-runs skip existing files; they run as root because the named volume is root-owned.
- Comments in this repo are unusually dense and explanatory — that's the house style. When changing behavior, update the comment explaining *why*, not just the *what*.
- Tunables with real defaults live in `.env.example`; `update.sh` syncs them into the server's `.env` every run. A divergent server value is a conflict, not a silent overwrite.
@CLAUDE.md