feat: add llama-server-fast, a small non-thinking classifier model
Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for use as qwen-code CLI's Auto Mode classifier fastModel. Model choice researched in docs/research/fast-model-choice.md: architecturally non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added defensively per a known (closed) llama.cpp misdetection bug. - docker-compose.yml: llama-server-fast + downloader-fast services, omniroute depends_on updated - .env.example: LLAMA_FAST_* vars - scripts/update.sh: runs the new downloader profile Refs #44 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
This commit is contained in:
@@ -96,3 +96,20 @@ COMFYUI_PUID=
|
||||
COMFYUI_PGID=
|
||||
COMFYUI_VIDEO_GID=
|
||||
COMFYUI_RENDER_GID=
|
||||
|
||||
# --- llama.cpp / fast model (second, always-resident instance — see
|
||||
# docs/research/fast-model-choice.md and issue #44) ---
|
||||
# Qwen3-4B-Instruct-2507: architecturally non-thinking (never emits
|
||||
# <think> blocks, unlike Qwen3-1.7B/0.6B which need a per-call toggle) —
|
||||
# picked specifically so it stays fast enough for qwen-code's Auto Mode
|
||||
# classifier (Stage 1 wants ~300ms). Same publisher (unsloth) as the main
|
||||
# model for consistency.
|
||||
LLAMA_FAST_MODEL_FILE=Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf
|
||||
# Same reasoning as LLAMA_GPU_LAYERS above — full GPU offload, this model
|
||||
# is dense too.
|
||||
LLAMA_FAST_GPU_LAYERS=999
|
||||
# Classifier transcripts are truncated/bounded by qwen-code itself (see its
|
||||
# own Auto Mode docs) — no need for anywhere near the 27B's huge context.
|
||||
# 8192 keeps this instance's KV cache negligible.
|
||||
LLAMA_FAST_CTX_SIZE=8192
|
||||
LLAMA_FAST_PARALLEL=2
|
||||
|
||||
Reference in New Issue
Block a user