haylanandClaude Bot
a0e5e9704b
- CONTEXT.md glossary - README reduced to intro, quickstart and doc index - new architecture.md, gateway.md (from proxy-key-onboarding.md), operations.md; network-access.md rewritten - coding-cli-setup: stale OMNIROUTE_PORT:-4000 endpoint, terms and duplicated per-slot/classifier explanations fixed - research/README.md index; proxy-request-priority.md moved there as superseded Every doc names keys instead of restating values and links its source. Refs #55 (#56, #57, #59, #60) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
60 lines
1.8 KiB
Markdown
60 lines
1.8 KiB
Markdown
# LLM-Server
|
|
|
|
A self-hosted inference stack: local models served on one GPU machine, reached by other tools through a single authenticated entry point. This file is a glossary only — for how things are configured, follow the links in the docs to the source.
|
|
|
|
## Language
|
|
|
|
### Machine
|
|
|
|
**Server**:
|
|
The one machine that runs the stack.
|
|
_Avoid_: AI box, R9700 box, this machine
|
|
|
|
**Host**:
|
|
The Docker host, as opposed to a container — only used for that distinction.
|
|
_Avoid_: using it as a synonym for Server
|
|
|
|
### Serving models
|
|
|
|
**Backend**:
|
|
A model server process that serves one model to the gateway (`llama-server`, `qwen-classifier`).
|
|
_Avoid_: provider, model server
|
|
|
|
**Slot**:
|
|
One concurrent request lane inside a backend; a backend handles as many requests at once as it has slots.
|
|
_Avoid_: parallel request, lane
|
|
|
|
**Main model**:
|
|
The large model that does the actual work for workloads.
|
|
_Avoid_: chat model
|
|
|
|
**Classifier model**:
|
|
The small model that judges a coding CLI's tool calls before they run.
|
|
_Avoid_: fast model (`fastModel` only when quoting qwen-code's config key)
|
|
|
|
### Reaching the models
|
|
|
|
**Gateway**:
|
|
The single entry point every model call goes through (OmniRoute).
|
|
_Avoid_: proxy, AI proxy
|
|
|
|
**Provider connection**:
|
|
The gateway's registration of one backend.
|
|
_Avoid_: provider (alone), connection (alone)
|
|
|
|
**API key**:
|
|
A credential issued by the gateway to exactly one workload.
|
|
_Avoid_: virtual key, workload key
|
|
|
|
**Workload**:
|
|
Anything that calls the gateway with its own API key.
|
|
_Avoid_: client, consumer
|
|
|
|
**Coding CLI**:
|
|
A workload that is an interactive coding agent (Claude Code, Kimi CLI, OpenCode, Qwen Code).
|
|
_Avoid_: harness
|
|
|
|
**Reverse proxy**:
|
|
The existing Nginx Proxy Manager instance that publishes the gateway under hostnames; not part of this stack.
|
|
_Avoid_: proxy (alone)
|