feat(llama.cpp): raise default context to 128K, document RAM/SSD offload knobs #30

Merged

haylan merged 1 commits from ctx-size-128k into main

2026-09-03 04:44:34 +00:00

Author	SHA1	Message	Date
haylanandClaude-Bot	c5864beec9	feat(llama.cpp): raise default context to 128K, document RAM/SSD offload knobs LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700, ~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table. Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/ --cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that there's no explicit SSD offload tier to enable — llama.cpp's default mmap already falls back to disk implicitly if GPU+RAM run out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m	2026-09-03 06:35:48 +02:00

Author

SHA1

Message

Date

haylanandClaude-Bot

c5864beec9

feat(llama.cpp): raise default context to 128K, document RAM/SSD offload knobs

LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K
fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700,
~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table.

Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense
model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/
--cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that
there's no explicit SSD offload tier to enable — llama.cpp's default mmap
already falls back to disk implicitly if GPU+RAM run out.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m

2026-09-03 06:35:48 +02:00