docs: restructure around source-of-truth pages

- CONTEXT.md glossary
- README reduced to intro, quickstart and doc index
- new architecture.md, gateway.md (from proxy-key-onboarding.md),
  operations.md; network-access.md rewritten
- coding-cli-setup: stale OMNIROUTE_PORT:-4000 endpoint, terms and
  duplicated per-slot/classifier explanations fixed
- research/README.md index; proxy-request-priority.md moved there as
  superseded

Every doc names keys instead of restating values and links its source.

Refs #55 (#56, #57, #59, #60)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-09-24 21:51:01 +02:00
1 parent 072eeb78c2
commit a0e5e9704b
14 files changed
+340 -119

No files matched your search

+17 -23
View File
@@ -1,35 +1,29 @@
# LLM-Server
Local AI inference stack: llama.cpp (ROCm) serving Qwen3.8-27B on an AMD Radeon AI PRO R9700, fronted by the OmniRoute AI gateway, with Lazytainer auto-suspending the inference container when idle.
> **Source of truth:** [`docker-compose.yml`](docker-compose.yml) (what runs) and [`scripts/update.sh`](scripts/update.sh) (how it's deployed). The docs explain *why*; if they disagree with the source, the source wins.
See the wayfinder map ([issue #1](https://git.arthurerlich.de/haylan/LLM-Server/issues/1)) for the full architecture rationale and open questions.
A self-hosted inference stack on one AMD GPU server. Two llama.cpp **backends**, the main model and the classifier model, sit behind an OmniRoute **gateway**. Every **workload**, mostly coding CLIs, reaches them through the gateway with its own **API key**. Terms are defined in [`CONTEXT.md`](CONTEXT.md).
## Quickstart
On the server:
```bash
./scripts/update.sh
```
`update.sh` creates `.env` from `.env.example` if missing, fills in every random secret it can generate itself (via `openssl`, `SEARXNG_LAN_IP` resolved from `search.home` on this host), downloads the model GGUF into the `models` volume if it's not there yet, then pulls/builds/brings up the whole stack. Safe to re-run any time — it only fills in what's still blank, skips the model if already downloaded, and only recreates what changed.
It creates and syncs `.env`, generates secrets, downloads the models, and brings the stack up. It's safe to re-run at any time. What it does and why: [`docs/operations.md`](docs/operations.md).
llama.cpp's own API is internal-only — everything routes through the AI gateway below.
## Docs
Pointing Claude Code CLI, Kimi CLI, OpenCode CLI, or Qwen Code CLI at the local endpoint: see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md).
**Known risk**: Qwen3.8-27B's tool-calling reliability against llama.cpp's Anthropic shim is not yet verified (open upstream parser bugs against its model lineage) — see `docs/research/qwen3.8-27b-tool-calling.md`.
## AI gateway (OmniRoute)
An [AI gateway/proxy](https://git.arthurerlich.de/haylan/LLM-Server/issues/9) fronts llama.cpp: per-workload API keys and usage tracking. As of [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31) this is [OmniRoute](https://github.com/diegosouzapw/OmniRoute), replacing the original LiteLLM setup. `./scripts/update.sh` handles most of OmniRoute's secrets (see `.env.example`); per-workload API keys still need minting by hand in the dashboard — see [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
- Gateway API: `http://<this-machine>:${OMNIROUTE_PORT:-4000}/v1` locally, or `proxy-ai.home` / `proxy-ai.haylan.ch` once routed through NPM — see [`docs/network-access.md`](docs/network-access.md).
- Dashboard (key/provider management): LAN/host-only, never published to the internet — see `docs/network-access.md`.
- Issuing a key for a new workload: [`docs/proxy-key-onboarding.md`](docs/proxy-key-onboarding.md).
Coding CLIs (see [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md)) route through the gateway — llama-server has no published host port. **Not yet verified**: none of this has been smoke-tested on real hardware yet — see [issue #31](https://git.arthurerlich.de/haylan/LLM-Server/issues/31)'s tickets for the open items (provider registration, per-workload key minting).
**Note on this choice**: OmniRoute's own docs (`docs/security/STEALTH_GUIDE.md`, `MITM-TPROXY-DECRYPT.md`, `PUBLIC_CREDS.md` in its repo) describe shipped features for evading AI-provider client detection, system-wide HTTPS interception via a locally-installed root CA, and hiding credentials from secret scanners. None of that is used by this stack's configuration, but it's a real characteristic of the upstream project — see issue #31's Notes for the full research trail before extending this integration further.
### Web search
The gateway also fronts SearXNG-backed web search — see `docs/research/litellm-searxng-search.md` for the original research (still applicable — same standalone-endpoint pattern, see issue #31's #35).
| Doc | What it covers |
|---|---|
| [`CONTEXT.md`](CONTEXT.md) | Glossary: the terms every doc uses |
| [`docs/architecture.md`](docs/architecture.md) | Each service, why it exists, and why it's configured the way it is |
| [`docs/gateway.md`](docs/gateway.md) | OmniRoute: dashboard, provider connections, API keys, timeouts, web search |
| [`docs/network-access.md`](docs/network-access.md) | Who can reach which port, and why |
| [`docs/operations.md`](docs/operations.md) | What `update.sh` does: env sync, secrets, downloads |
| [`docs/coding-cli-setup/`](docs/coding-cli-setup/index.md) | Pointing Claude Code, Kimi CLI, OpenCode and Qwen Code at the gateway |
| [`docs/research/`](docs/research/README.md) | Dated research and decision records, each marked current or superseded |
| [`docs/server-planing.md`](docs/server-planing.md) | Background research: a 4× GPU rack build. Not an active plan |
| [`docs/agents/`](docs/agents/) | Instructions for coding agents: [issue tracker](docs/agents/issue-tracker.md), [domain docs](docs/agents/domain.md), [writing docs](docs/agents/writing-docs.md) |