haylanandClaude Bot
17747694bd
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
60 lines
1.8 KiB
Markdown
60 lines
1.8 KiB
Markdown
# LLM-Server
|
|
|
|
A self-hosted inference stack: local models served on one GPU machine, reached by other tools through a single authenticated entry point. This file is a glossary only — for how things are configured, follow the links in the docs to the source.
|
|
|
|
## Language
|
|
|
|
### Machine
|
|
|
|
**Server**:
|
|
The one machine that runs the stack.
|
|
_Avoid_: AI box, R9700 box, this machine
|
|
|
|
**Host**:
|
|
The Docker host, as opposed to a container — only used for that distinction.
|
|
_Avoid_: using it as a synonym for Server
|
|
|
|
### Serving models
|
|
|
|
**Backend**:
|
|
A model server process that serves one model to the gateway (`llama-server`).
|
|
_Avoid_: provider, model server
|
|
|
|
**Slot**:
|
|
One concurrent request lane inside a backend; a backend handles as many requests at once as it has slots.
|
|
_Avoid_: parallel request, lane
|
|
|
|
**Main model**:
|
|
The large model that does the actual work for workloads.
|
|
_Avoid_: chat model
|
|
|
|
**Draft model**:
|
|
The small speculative-decoding drafter that runs beside the main model inside `llama-server` (DFlash2).
|
|
_Avoid_: classifier model (the separate classifier backend was removed)
|
|
|
|
### Reaching the models
|
|
|
|
**Gateway**:
|
|
The single entry point every model call goes through (OmniRoute).
|
|
_Avoid_: proxy, AI proxy
|
|
|
|
**Provider connection**:
|
|
The gateway's registration of one backend.
|
|
_Avoid_: provider (alone), connection (alone)
|
|
|
|
**API key**:
|
|
A credential issued by the gateway to exactly one workload.
|
|
_Avoid_: virtual key, workload key
|
|
|
|
**Workload**:
|
|
Anything that calls the gateway with its own API key.
|
|
_Avoid_: client, consumer
|
|
|
|
**Coding CLI**:
|
|
A workload that is an interactive coding agent (Claude Code, Kimi CLI, OpenCode, Qwen Code).
|
|
_Avoid_: harness
|
|
|
|
**Reverse proxy**:
|
|
The existing Nginx Proxy Manager instance that publishes the gateway under hostnames; not part of this stack.
|
|
_Avoid_: proxy (alone)
|