feat: add llama-server-fast, a small non-thinking classifier model
Second, always-resident llama.cpp instance (Qwen3-4B-Instruct-2507, Q8_0 GGUF, ~5GB VRAM) alongside the existing Qwen3.8-27B instance, for use as qwen-code CLI's Auto Mode classifier fastModel. Model choice researched in docs/research/fast-model-choice.md: architecturally non-thinking (unlike Qwen3-1.7B/0.6B), --reasoning off added defensively per a known (closed) llama.cpp misdetection bug. - docker-compose.yml: llama-server-fast + downloader-fast services, omniroute depends_on updated - .env.example: LLAMA_FAST_* vars - scripts/update.sh: runs the new downloader profile Refs #44 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx
This commit is contained in:
@@ -88,6 +88,7 @@ docker compose build --pull
|
||||
|
||||
echo "==> ensuring models are downloaded (skips already-present files)"
|
||||
docker compose --profile tools run --rm downloader
|
||||
docker compose --profile tools run --rm downloader-fast
|
||||
|
||||
echo "==> bringing up omniroute"
|
||||
docker compose up -d --wait omniroute
|
||||
|
||||
Reference in New Issue
Block a user