haylanandClaude Bot
8144286946
Replace the per-model downloader services with a single downloader that reads file/url pairs from models.yml. Downloads now go to a .part file first, so an interrupted run can't leave a truncated model that later runs skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LLM-Server
Source of truth:
docker-compose.yml(what runs) andscripts/update.sh(how it's deployed). The docs explain why; if they disagree with the source, the source wins.
A self-hosted inference stack on one AMD GPU server. A llama.cpp backend serving the main model sits behind an OmniRoute gateway. Every workload, mostly coding CLIs, reaches it through the gateway with its own API key. Terms are defined in CONTEXT.md.
Quickstart
On the server:
./scripts/update.sh
It creates and syncs .env, generates secrets, downloads the models, and brings the stack up. It's safe to re-run at any time. What it does and why: docs/operations.md.
Docs
| Doc | What it covers |
|---|---|
CONTEXT.md |
Glossary: the terms every doc uses |
docs/architecture.md |
Each service, why it exists, and why it's configured the way it is |
docs/gateway.md |
OmniRoute: dashboard, provider connections, API keys, timeouts, web search |
docs/network-access.md |
Who can reach which port, and why |
docs/operations.md |
What update.sh does: env sync, secrets, downloads; speed bench |
docs/coding-cli-setup/ |
Pointing Claude Code, Kimi CLI, OpenCode and Qwen Code at the gateway |
docs/research/ |
Dated research and decision records, each marked current or superseded |
docs/server-planing.md |
Background research: a 4× GPU rack build. Not an active plan |
docs/agents/ |
Instructions for coding agents: issue tracker, domain docs, writing docs |