feat(llm): raise prompt cache RAM budget via --cache-ram
Log shows 253 prompt-cache evictions and 146 full ~42k re-prefills at the default budget. Add LLAMA_CACHE_RAM and the throughput-tuning research note. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
433ebc6597
commit
408ed3e586
5 files changed
+144
No files matched your search
@@ -28,6 +28,7 @@ Each file here is a **dated snapshot**: research or a decision record as it stoo
|
||||
| [proxy-shadow-pricing.md](proxy-shadow-pricing.md) | Superseded by the OmniRoute migration | Shadow cost was wired into LiteLLM config, and nothing implements it now |
|
||||
| [proxy-tool-choice.md](proxy-tool-choice.md) | Superseded by the OmniRoute migration | Chose LiteLLM. It mentions Open WebUI, which has been removed |
|
||||
| [qwen3.8-27b-quant.md](qwen3.8-27b-quant.md) | Current | Main model quant and VRAM budget |
|
||||
| [qwen3.8-27b-throughput-tuning.md](qwen3.8-27b-throughput-tuning.md) | Current | Tokens/s tuning from the server log, DDR RAM as a cache tier, comparison with five outside R9700 configs. Proposals only, nothing applied |
|
||||
| [qwen3.8-27b-tool-calling.md](qwen3.8-27b-tool-calling.md) | Current | Tool-calling parser risk for the main model |
|
||||
| [rocm-gpu-pin-and-render-group.md](rocm-gpu-pin-and-render-group.md) | Current (name drift) | Findings apply to `llama-server` + `qwen-classifier`. The service names in it are older |
|
||||
| [voidllm-evaluation.md](voidllm-evaluation.md) | Superseded by the OmniRoute migration | VoidLLM vs. LiteLLM, both moot |
|
||||
Reference in new issue
Block a user