haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

Add llama-server-fast: small non-thinking classifier/fast model for qwen-code Auto Mode #44

New Issue
Closed
opened 2026-09-06 18:17:27 +00:00 by haylan · 1 comment
haylan commented 2026-09-06 18:17:27 +00:00
Owner
Copy Link
Copy Source

Part of #1

Add a second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance on the same R9700, register it as a backend in OmniRoute, and point qwen-code CLI's fastModel at it.

Model choice already researched and decided — see docs/research/fast-model-choice.md (Qwen3-4B-Instruct-2507, Q8_0, unsloth GGUF, --reasoning off needed).

What the agent can do directly

  • New llama-server-fast service in docker-compose.yml (own lazytainer group, ai-stack-only, --reasoning off, --ctx-size 8192)
  • New downloader-fast service (test -f guard, same pattern as existing downloader)
  • New LLAMA_FAST_* vars in .env.example
  • scripts/update.sh: run the new downloader profile
  • omniroute service: add llama-server-fast to depends_on
  • Commit + push to a branch, open a PR

Manual checklist (human — not scriptable per this repo's own OmniRoute docs)

  • Run ./scripts/update.sh on the R9700 box itself (no shell access to it from the dev session)
  • Register llama-server-fast as an OmniRoute provider (dashboard or POST /api/providers), base URL http://llama-server-fast:8080/v1 — same manual flow used for the original llama-server provider
  • Give it a model id/alias in OmniRoute
  • Update ~/.qwen/settings.json's fastModel to that model id (agent can do this part once the id is known)
Part of #1 Add a second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance on the same R9700, register it as a backend in OmniRoute, and point qwen-code CLI's `fastModel` at it. Model choice already researched and decided — see docs/research/fast-model-choice.md (Qwen3-4B-Instruct-2507, Q8_0, unsloth GGUF, `--reasoning off` needed). ## What the agent can do directly - [ ] New `llama-server-fast` service in docker-compose.yml (own lazytainer group, ai-stack-only, --reasoning off, --ctx-size 8192) - [ ] New `downloader-fast` service (test -f guard, same pattern as existing downloader) - [ ] New `LLAMA_FAST_*` vars in .env.example - [ ] scripts/update.sh: run the new downloader profile - [ ] omniroute service: add llama-server-fast to depends_on - [ ] Commit + push to a branch, open a PR ## Manual checklist (human — not scriptable per this repo's own OmniRoute docs) - [ ] Run ./scripts/update.sh on the R9700 box itself (no shell access to it from the dev session) - [ ] Register llama-server-fast as an OmniRoute provider (dashboard or POST /api/providers), base URL http://llama-server-fast:8080/v1 — same manual flow used for the original llama-server provider - [ ] Give it a model id/alias in OmniRoute - [ ] Update ~/.qwen/settings.json's fastModel to that model id (agent can do this part once the id is known)
haylan added the wayfinder:task label 2026-09-06 18:17:27 +00:00
haylan self-assigned this 2026-09-06 18:17:32 +00:00
haylan commented 2026-09-06 20:34:29 +00:00
Author
Owner
Copy Link
Copy Source

Confirmed working end-to-end on the real server: llama-server-fast deployed, registered in OmniRoute (qwen3-4b-fast), wired into qwen-code's fastModel, and the Auto Mode classifier now works (after fixing the context-size bug in #49 and the GPU-pin/group_add bugs in #47). Closing.

Confirmed working end-to-end on the real server: llama-server-fast deployed, registered in OmniRoute (qwen3-4b-fast), wired into qwen-code's fastModel, and the Auto Mode classifier now works (after fixing the context-size bug in #49 and the GPU-pin/group_add bugs in #47). Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels wayfinder:task
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees haylan
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: haylan/LLM-Server#44