chore(llm): univyed token size for agent usage
This commit is contained in:
1 parent
b74c0e0ccc
commit
1e3ad169c0
1 file changed
+1
@@ -23,6 +23,7 @@ services:
|
||||
--n-gpu-layers ${LLAMA_GPU_LAYERS:-999}
|
||||
--ctx-size ${LLAMA_CTX_SIZE:-262144}
|
||||
--parallel ${LLAMA_PARALLEL:-2}
|
||||
--kv-unified
|
||||
--flash-attn on
|
||||
--cache-type-k q8_0
|
||||
--cache-type-v q8_0
|
||||
|
||||
Reference in new issue
Block a user