feat(ctx): larger context size
This commit is contained in:
+11
-6
@@ -21,12 +21,17 @@ LLAMA_MODEL_FILE=Qwen3.8-27B-UD-Q4_K_XL.gguf
|
||||
# offload has been an open llama.cpp feature request since 2025 (still
|
||||
# unimplemented): https://github.com/ggml-org/llama.cpp/discussions/12507
|
||||
LLAMA_GPU_LAYERS=999
|
||||
# 131072 (128K): ~25.6GB (17.6GB weights + ~8GB KV cache) on the 32GB
|
||||
# R9700, ~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full
|
||||
# table (64K only used ~19.6GB/~12GB headroom, but real usage was burning
|
||||
# through 64K fast). If headroom gets tight, quantize the KV cache instead
|
||||
# of dropping context: --cache-type-k/v q8_0 roughly halves it.
|
||||
LLAMA_CTX_SIZE=131072
|
||||
# 262144 = this model's true max (max_position_embeddings in Qwen/Qwen3.8-27B's
|
||||
# config.json) — the largest --ctx-size llama.cpp will even accept for it.
|
||||
# fp16 KV cache at full context is ~16GB (262,144 x 64KiB/token), on top of
|
||||
# 17.6GB weights = ~33.6GB, which does NOT fit the 32GB R9700 — expect
|
||||
# llama-server to fail to allocate and refuse to start at this setting.
|
||||
# Chosen anyway per explicit request to run at the model's real ceiling
|
||||
# rather than a VRAM-safe value. To actually get a working max: quantize the
|
||||
# KV cache (--cache-type-k/v q8_0, roughly halves it to ~8GB, fits with room
|
||||
# to spare) or drop back to 131072 (128K), the last known-good value
|
||||
# (~25.6GB total, ~6GB headroom) — see docs/research/qwen3.8-27b-quant.md.
|
||||
LLAMA_CTX_SIZE=262144
|
||||
|
||||
# --- Lazytainer ---
|
||||
# Seconds of inactivity before llama-server is stopped. 900 = 15 min.
|
||||
|
||||
Reference in New Issue
Block a user