haylanandClaude Bot
17747694bd
Replace MTP with draft-dflash and a separate DFlash2 drafter GGUF. Remove the qwen-classifier backend to free VRAM (GPU was at 100%). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1.8 KiB
LLM-Server
A self-hosted inference stack: local models served on one GPU machine, reached by other tools through a single authenticated entry point. This file is a glossary only — for how things are configured, follow the links in the docs to the source.
Language
Machine
Server: The one machine that runs the stack. Avoid: AI box, R9700 box, this machine
Host: The Docker host, as opposed to a container — only used for that distinction. Avoid: using it as a synonym for Server
Serving models
Backend:
A model server process that serves one model to the gateway (llama-server).
Avoid: provider, model server
Slot: One concurrent request lane inside a backend; a backend handles as many requests at once as it has slots. Avoid: parallel request, lane
Main model: The large model that does the actual work for workloads. Avoid: chat model
Draft model:
The small speculative-decoding drafter that runs beside the main model inside llama-server (DFlash2).
Avoid: classifier model (the separate classifier backend was removed)
Reaching the models
Gateway: The single entry point every model call goes through (OmniRoute). Avoid: proxy, AI proxy
Provider connection: The gateway's registration of one backend. Avoid: provider (alone), connection (alone)
API key: A credential issued by the gateway to exactly one workload. Avoid: virtual key, workload key
Workload: Anything that calls the gateway with its own API key. Avoid: client, consumer
Coding CLI: A workload that is an interactive coding agent (Claude Code, Kimi CLI, OpenCode, Qwen Code). Avoid: harness
Reverse proxy: The existing Nginx Proxy Manager instance that publishes the gateway under hostnames; not part of this stack. Avoid: proxy (alone)