fix(llama-server): cap concurrent slots at 2 to curb prefill contention
Default --parallel of 4 let concurrent subagent requests split GPU compute, pushing large-context prefill past OmniRoute's stream-idle timeout and triggering cancel-on-both-sides. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -32,6 +32,15 @@ LLAMA_GPU_LAYERS=999
|
||||
# to spare) or drop back to 131072 (128K), the last known-good value
|
||||
# (~25.6GB total, ~6GB headroom) — see docs/research/qwen3.8-27b-quant.md.
|
||||
LLAMA_CTX_SIZE=262144
|
||||
# Concurrent request slots. Was implicitly 4 (llama.cpp's compiled-in
|
||||
# default) with no flag set — under concurrent subagent fan-out, 4 requests
|
||||
# split the same GPU compute, so a large-context prefill can queue behind
|
||||
# others long enough to blow past OmniRoute's stream-idle timeout, which then
|
||||
# cancels the request (see issue-tracker notes on the timeout/cancel loop).
|
||||
# Dropped to 2 so each slot gets more compute and finishes prefill sooner;
|
||||
# raise back toward 4 if throughput (not latency) becomes the bottleneck
|
||||
# instead.
|
||||
LLAMA_PARALLEL=2
|
||||
|
||||
# --- Lazytainer ---
|
||||
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
||||
|
||||
Reference in New Issue
Block a user