fix(llama-server): cap concurrent slots at 2 to curb prefill contention

Default --parallel of 4 let concurrent subagent requests split GPU compute, pushing large-context prefill past OmniRoute's stream-idle timeout and triggering cancel-on-both-sides. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-05 11:26:30 +02:00
parent ae812cd9e0
commit 23e90fe8fb
2 changed files with 10 additions and 0 deletions
@@ -19,6 +19,7 @@ services:
      --port 8080
      --n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
      --ctx-size ${LLAMA_CTX_SIZE:-131072}
+      --parallel ${LLAMA_PARALLEL:-2}
      --jinja
    # No published host port: llama-server is reached only via the omniroute
    # gateway on the ai-stack docker network now — see issue #15. Its