perf(llama-server): enable flash attention
Cuts prefill time with no accuracy cost, per markaicode.com's llama.cpp timeout writeup — directly helps the prefill-vs-stream-idle contention issue fixed in the prior two commits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,7 @@ services:
|
||||
--n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
|
||||
--ctx-size ${LLAMA_CTX_SIZE:-131072}
|
||||
--parallel ${LLAMA_PARALLEL:-2}
|
||||
--flash-attn on
|
||||
--jinja
|
||||
# No published host port: llama-server is reached only via the omniroute
|
||||
# gateway on the ai-stack docker network now — see issue #15. Its
|
||||
|
||||
Reference in New Issue
Block a user