haylanandClaude Bot
17747694bd
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
30 lines
1.8 KiB
Markdown
30 lines
1.8 KiB
Markdown
# LLM-Server
|
||
|
||
> **Source of truth:** [`docker-compose.yml`](docker-compose.yml) (what runs) and [`scripts/update.sh`](scripts/update.sh) (how it's deployed). The docs explain *why*; if they disagree with the source, the source wins.
|
||
|
||
A self-hosted inference stack on one AMD GPU server. A llama.cpp **backend** serving the main model sits behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches it through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
|
||
|
||
## Quickstart
|
||
|
||
On the server:
|
||
|
||
```bash
|
||
./scripts/update.sh
|
||
```
|
||
|
||
It creates and syncs `.env`, generates secrets, downloads the models, and brings the stack up. It's safe to re-run at any time. What it does and why: [`docs/operations.md`](docs/operations.md).
|
||
|
||
## Docs
|
||
|
||
| Doc | What it covers |
|
||
|---|---|
|
||
| [`CONTEXT.md`](CONTEXT.md) | Glossary: the terms every doc uses |
|
||
| [`docs/architecture.md`](docs/architecture.md) | Each service, why it exists, and why it's configured the way it is |
|
||
| [`docs/gateway.md`](docs/gateway.md) | OmniRoute: dashboard, provider connections, API keys, timeouts, web search |
|
||
| [`docs/network-access.md`](docs/network-access.md) | Who can reach which port, and why |
|
||
| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads |
|
||
| [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md) | Pointing Claude Code, Kimi CLI, OpenCode and Qwen Code at the gateway |
|
||
| [`docs/research/`](docs/research/README.md) | Dated research and decision records, each marked current or superseded |
|
||
| [`docs/server-planing.md`](docs/server-planing.md) | Background research: a 4× GPU rack build. Not an active plan |
|
||
| [`docs/agents/`](docs/agents/) | Instructions for coding agents: [issue tracker](docs/agents/issue-tracker.md), [domain docs](docs/agents/domain.md), [writing docs](docs/agents/writing-docs.md) |
|