feat(llm): raise prompt cache RAM budget via --cache-ram

Log shows 253 prompt-cache evictions and 146 full ~42k re-prefills at the default budget.
Add LLAMA_CACHE_RAM and the throughput-tuning research note.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
haylanandClaude Bot committed 2026-10-20 21:05:27 +02:00
1 parent 433ebc6597
commit 408ed3e586
5 files changed
+144

No files matched your search

+1
View File
@@ -28,6 +28,7 @@ Each file here is a **dated snapshot**: research or a decision record as it stoo
| [proxy-shadow-pricing.md](proxy-shadow-pricing.md) | Superseded by the OmniRoute migration | Shadow cost was wired into LiteLLM config, and nothing implements it now |
| [proxy-tool-choice.md](proxy-tool-choice.md) | Superseded by the OmniRoute migration | Chose LiteLLM. It mentions Open WebUI, which has been removed |
| [qwen3.8-27b-quant.md](qwen3.8-27b-quant.md) | Current | Main model quant and VRAM budget |
| [qwen3.8-27b-throughput-tuning.md](qwen3.8-27b-throughput-tuning.md) | Current | Tokens/s tuning from the server log, DDR RAM as a cache tier, comparison with five outside R9700 configs. Proposals only, nothing applied |
| [qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) | Current | Tool-calling parser risk for the main model |
| [rocm-gpu-pin-and-render-group.md](rocm-gpu-pin-and-render-group.md) | Current (name drift) | Findings apply to `llama-server` + `qwen-classifier`. The service names in it are older |
| [voidllm-evaluation.md](voidllm-evaluation.md) | Superseded by the OmniRoute migration | VoidLLM vs. LiteLLM, both moot |