haylan/LLM-Server
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llama.cpp): raise default context to 128K, document RAM/SSD offload knobs #30

Merged
haylan merged 1 commits from ctx-size-128k into main 2026-09-03 04:44:34 +00:00
Conversation 0 Commits 1 Files Changed 2
+21 -5
haylan commented 2026-09-03 04:44:27 +00:00
Owner
Copy Link
Copy Source

LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K
fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700,
~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table.

Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense
model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/
--cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that
there's no explicit SSD offload tier to enable — llama.cpp's default mmap
already falls back to disk implicitly if GPU+RAM run out.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m

LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700, ~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table. Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/ --cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that there's no explicit SSD offload tier to enable — llama.cpp's default mmap already falls back to disk implicitly if GPU+RAM run out. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
haylan added 1 commit 2026-09-03 04:44:28 +00:00
feat(llama.cpp): raise default context to 128K, document RAM/SSD offload knobs c5864beec9 
LLAMA_CTX_SIZE default 65536 -> 131072: real usage was burning through 64K
fast. ~25.6GB VRAM (17.6GB weights + ~8GB KV cache) on the 32GB R9700,
~6GB headroom — see docs/research/qwen3.8-27b-quant.md for the full table.

Also documents LLAMA_GPU_LAYERS as the RAM-offload knob for this dense
model (no separate RAM-offload flag exists in llama.cpp, and --n-cpu-moe/
--cpu-moe/--override-tensor "exps" are MoE-only, no-ops here), and that
there's no explicit SSD offload tier to enable — llama.cpp's default mmap
already falls back to disk implicitly if GPU+RAM run out.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCAUsjGNSoJTtK8hyLKg5m
haylan merged commit 6b06d6001f into main 2026-09-03 04:44:34 +00:00
haylan deleted branch ctx-size-128k 2026-09-03 04:44:34 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
wayfinder:grilling
Wayfinder grilling ticket
 wayfinder:map
Wayfinder map issue
 wayfinder:prototype
Wayfinder prototype ticket
 wayfinder:research
Wayfinder research ticket
 wayfinder:task
Wayfinder task ticket
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
haylan (Arthur Erlich)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: haylan/LLM-Server#30