haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

perf(llama-server): enable flash attention

Browse Source 
Cuts prefill time with no accuracy cost, per markaicode.com's
llama.cpp timeout writeup — directly helps the prefill-vs-stream-idle
contention issue fixed in the prior two commits.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
haylan
2026-09-05 11:32:17 +02:00
co-authored by Claude-Bot
parent 633292b291
commit 5767f548c3
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+1
View File
@@ -20,6 +20,7 @@ services:
--n-gpu-layers ${LLAMA_GPU_LAYERS:-999} --n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
--ctx-size ${LLAMA_CTX_SIZE:-131072} --ctx-size ${LLAMA_CTX_SIZE:-131072}
--parallel ${LLAMA_PARALLEL:-2} --parallel ${LLAMA_PARALLEL:-2}
--flash-attn on
--jinja --jinja
# No published host port: llama-server is reached only via the omniroute # No published host port: llama-server is reached only via the omniroute
# gateway on the ai-stack docker network now — see issue #15. Its # gateway on the ai-stack docker network now — see issue #15. Its