fix: llama-server-fast context-size exhaustion breaking Auto Mode classifier

Real failure: "Auto Mode couldn't classify this action (Classifier
stage 1 unavailable)". Reproduced directly against the server:

  {"error":{"message":"[400]: request (6186 tokens) exceeds the
  available context size (4096 tokens)"...

LLAMA_FAST_CTX_SIZE=8192 is the TOTAL across every LLAMA_FAST_PARALLEL
slot, not per-request — the main model's own .env.example comment
already calls this out, missed it when llama-server-fast was set up
(#44). With PARALLEL=2 that's 4096/slot, too small for a real
classifier call (hints + environment + recent tool-call history).

Fixed by dropping to a single slot (LLAMA_FAST_PARALLEL=1) rather than
raising ctx-size — this service doesn't need concurrent classifier
calls the way the main model needs concurrent chat sessions, so this
costs no extra VRAM. The full 8192 now goes to the one slot.

docker compose config -q validated.

Refs #5

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrnMEdzeQzqZE5soVEXPCx

This commit is contained in:

haylan

2026-09-06 22:06:59 +02:00

co-authored by

Claude-Bot

parent 63938e95c9

commit 52a92f6508

2 changed files with 16 additions and 5 deletions

									
										docker-compose.yml
									
		+1
		-1
	
												View File
												
				@@ -70,7 +70,7 @@ services:

				      --port 8080

				      --n-gpu-layers ${LLAMA_FAST_GPU_LAYERS:-999}

				      --ctx-size ${LLAMA_FAST_CTX_SIZE:-8192}

				      --parallel ${LLAMA_FAST_PARALLEL:-2}

				      --parallel ${LLAMA_FAST_PARALLEL:-1}

				      --flash-attn on

				      --cache-type-k q8_0

				      --cache-type-v q8_0