Add llama-server-fast: small non-thinking classifier/fast model #45

2026-09-06T18:27:27Z

haylan commented

2026-09-06 18:27:27 +00:00

Adds a second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for use as qwen-code CLI's Auto Mode classifier fastModel.

Model choice researched in docs/research/fast-model-choice.md: architecturally non-thinking (unlike Qwen3-1.7B/0.6B, which need a per-call toggle), --reasoning off added defensively per a known (closed) llama.cpp misdetection bug against this model.

docker-compose.yml: llama-server-fast + downloader-fast services, omniroute depends_on updated
.env.example: LLAMA_FAST_* vars
scripts/update.sh: runs the new downloader profile

docker compose config -q validated clean.

Closes #44, part of #1.

Not done by this PR (manual, per this repo's own OmniRoute docs — dashboard/API only):

Run ./scripts/update.sh on the R9700 box
Register llama-server-fast as an OmniRoute provider (http://llama-server-fast:8080/v1)
Point qwen-code's fastModel at the resulting model id

🤖 Generated with Claude Code

https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

Adds a second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for use as qwen-code CLI's Auto Mode classifier `fastModel`. Model choice researched in `docs/research/fast-model-choice.md`: architecturally non-thinking (unlike Qwen3-1.7B/0.6B, which need a per-call toggle), `--reasoning off` added defensively per a known (closed) llama.cpp misdetection bug against this model. - `docker-compose.yml`: `llama-server-fast` + `downloader-fast` services, `omniroute` depends_on updated - `.env.example`: `LLAMA_FAST_*` vars - `scripts/update.sh`: runs the new downloader profile `docker compose config -q` validated clean. Closes #44, part of #1. **Not done by this PR** (manual, per this repo's own OmniRoute docs — dashboard/API only): - Run `./scripts/update.sh` on the R9700 box - Register `llama-server-fast` as an OmniRoute provider (`http://llama-server-fast:8080/v1`) - Point qwen-code's `fastModel` at the resulting model id 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

haylan added 1 commit 2026-09-06 18:27:27 +00:00

feat: add llama-server-fast, a small non-thinking classifier model d984c10835

Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507,
Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for
use as qwen-code CLI's Auto Mode classifier fastModel. Model choice
researched in docs/research/fast-model-choice.md: architecturally
non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added
defensively per a known (closed) llama.cpp misdetection bug.

- docker-compose.yml: llama-server-fast + downloader-fast services,
  omniroute depends_on updated
- .env.example: LLAMA_FAST_* vars
- scripts/update.sh: runs the new downloader profile

Refs #44

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

haylan merged commit 1932981f09 into main

2026-09-06 18:39:31 +00:00

haylan deleted branch add-fast-model

2026-09-06 18:39:31 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: haylan/LLM-Server#45