haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: remove llama-server-fast (Qwen3-4B classifier model)

Browse Source 
Drops the second always-resident llama.cpp instance, its downloader,
and the omniroute depends_on entry. Also strips the now-dead
LLAMA_FAST_* block from .env.example and the stale VRAM-budget comment
in scripts/switch-model.sh that assumed this service was always up.

Note: this was qwen-code's Auto Mode Stage 1 classifier (fastModel) —
see docs/research/fast-model-choice.md and issue #44. Auto Mode will
lose that classifier until/unless it's reconfigured to route
elsewhere or fall back to prompt-only classification.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DjhxWirQepKFEQj1huXNJR
This commit is contained in:
haylan
2026-09-07 19:33:50 +02:00
co-authored by Claude-Bot
parent 5b7548dc7c
commit 5d6a17fd9b
3 changed files with 4 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
-28
View File
@@ -105,34 +105,6 @@ COMFYUI_PGID=
HOST_VIDEO_GID=
HOST_RENDER_GID=
# --- llama.cpp / fast model (second, always-resident instance — see
# docs/research/fast-model-choice.md and issue #44) ---
# Qwen3-4B-Instruct-2507: architecturally non-thinking (never emits
# <think> blocks, unlike Qwen3-1.7B/0.6B which need a per-call toggle) —
# picked specifically so it stays fast enough for qwen-code's Auto Mode
# classifier (Stage 1 wants ~300ms). Same publisher (unsloth) as the main
# model for consistency.
LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
# is dense too.
LLAMA_FAST_GPU_LAYERS=999
# --ctx-size is the TOTAL across every LLAMA_FAST_PARALLEL slot, not per
# request — same halving already called out for the main model above.
# Was PARALLEL=2, silently halving this to 4096/slot — too small: a real
# classifier call (hints + environment + recent tool-call history) hit
# "exceeds the available context size (4096 tokens)" in practice, which
# qwen-code surfaces as "Auto Mode couldn't classify this action
# (Classifier stage 1 unavailable)" — see issue #5. Fixed by dropping to
# a single slot instead of raising ctx-size (no extra VRAM, and this
# service doesn't need concurrent classifier calls the way the main
# model needs concurrent chat sessions) — the full 8192 now goes to the
# one slot. If hints.allow/softDeny/hardDeny ever approach their
# 50-entries-each ceiling, raise LLAMA_FAST_CTX_SIZE instead — qwen-code
# caps those at 200 chars x 150 entries plus 40,000 chars of
# historical-action context, which can exceed 8192 tokens worst-case.
LLAMA_FAST_CTX_SIZE=8192
LLAMA_FAST_PARALLEL=1
# --- ComfyUI diffusion model (Qwen-Image, FP8 — see docs/research/
# image-generation-model-choice.md and issue #42) ---
# Three files: diffusion weights, text encoder, VAE — all from the official