Reference cloud model/pricing for the shadow-cost estimate #11

New Issue

2026-08-25T04:39:54Z

haylan commented

2026-08-25 04:39:54 +00:00

Part of #9

Question

To show "what this local usage would have cost on a cloud API" (per #9's Destination), which reference cloud model/API pricing should the shadow-cost estimate use, and how should that mapping work given the tool chosen in #10 (its custom-pricing mechanism, and whatever token/usage metadata it captures per request)?

Consider: a fixed reference model (e.g. current Claude pricing, since that's what the user already tracks day-to-day) vs. a pricing table covering a few comparable tiers; how token counts from a local model map onto another model's pricing units; where the confirmed rate gets configured in the chosen tool.

Part of #9 ## Question To show "what this local usage would have cost on a cloud API" (per #9's Destination), which reference cloud model/API pricing should the shadow-cost estimate use, and how should that mapping work given the tool chosen in #10 (its custom-pricing mechanism, and whatever token/usage metadata it captures per request)? Consider: a fixed reference model (e.g. current Claude pricing, since that's what the user already tracks day-to-day) vs. a pricing table covering a few comparable tiers; how token counts from a local model map onto another model's pricing units; where the confirmed rate gets configured in the chosen tool.

haylan added the wayfinder:research label 2026-08-25 04:39:54 +00:00

haylan referenced this issue

2026-08-25 04:40:05 +00:00

Author the docker-compose service for the chosen proxy #14

haylan added a new dependency 2026-08-25 04:40:13 +00:00

#10 Which self-hosted AI gateway/proxy tool fits this effort's needs?

haylan added a new dependency 2026-08-25 04:40:14 +00:00

#14 Author the docker-compose service for the chosen proxy

haylan referenced this issue

2026-08-25 04:40:25 +00:00

AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack #9

haylan referenced this issue

2026-08-25 04:45:40 +00:00

Which self-hosted AI gateway/proxy tool fits this effort's needs? #10

haylan self-assigned this 2026-08-25 04:47:40 +00:00

haylan closed this issue

2026-08-25 04:49:50 +00:00

haylan commented

2026-08-25 04:50:14 +00:00

Recommendation: Claude Sonnet 5, fixed single reference. $2/MTok input, $10/MTok output (current standard pricing per Anthropic's pricing page, checked 2026-08-25) — matches what the user already tracks day-to-day and what this repo's own coding-cli-setup.md points coding CLIs at, and is the closer size-match to the locally hosted Qwen3.8-27B than Opus tier. Rejected a multi-tier pricing table as unneeded bookkeeping for a for-fun shadow estimate.

LiteLLM config shape (confirms/extends #10's mechanism): a model_info block on the local model's model_list entry in config.yaml:

model_info:
  input_cost_per_token: 0.000002
  output_cost_per_token: 0.00001

Input/output are distinguished, overrides are per-model (LiteLLM's built-in cost map for other providers is untouched), and the resulting spend surfaces via /key/info, /team/info//user/info, the Admin UI Usage dashboard, and kwargs["response_cost"] per call — all through LiteLLM's normal completion_cost() path, no extra tracking code.

Token mapping: LiteLLM prices whatever token counts llama.cpp reports in its own usage.prompt_tokens/completion_tokens — no re-tokenization against Claude's tokenizer. Treating that count as directly billable at Claude's rate is a fine approximation for a for-fun shadow estimate, not real accounting.

Staleness: local model pricing lives only in this repo's explicit config (LiteLLM's built-in default pricing table doesn't apply to unlisted local models), so risk is a one-line config edit if Anthropic changes Sonnet 5's price — recommend a source-URL+date comment in the config for #14.

Full findings + sources: docs/research/proxy-shadow-pricing.md on branch research/proxy-shadow-pricing.

**Recommendation: Claude Sonnet 5, fixed single reference.** $2/MTok input, $10/MTok output (current standard pricing per Anthropic's pricing page, checked 2026-08-25) — matches what the user already tracks day-to-day and what this repo's own coding-cli-setup.md points coding CLIs at, and is the closer size-match to the locally hosted Qwen3.8-27B than Opus tier. Rejected a multi-tier pricing table as unneeded bookkeeping for a for-fun shadow estimate. **LiteLLM config shape** (confirms/extends #10's mechanism): a `model_info` block on the local model's `model_list` entry in `config.yaml`: ```yaml model_info: input_cost_per_token: 0.000002 output_cost_per_token: 0.00001 ``` Input/output are distinguished, overrides are per-model (LiteLLM's built-in cost map for other providers is untouched), and the resulting spend surfaces via `/key/info`, `/team/info`/`/user/info`, the Admin UI Usage dashboard, and `kwargs["response_cost"]` per call — all through LiteLLM's normal `completion_cost()` path, no extra tracking code. **Token mapping**: LiteLLM prices whatever token counts llama.cpp reports in its own `usage.prompt_tokens`/`completion_tokens` — no re-tokenization against Claude's tokenizer. Treating that count as directly billable at Claude's rate is a fine approximation for a for-fun shadow estimate, not real accounting. **Staleness**: local model pricing lives only in this repo's explicit config (LiteLLM's built-in default pricing table doesn't apply to unlisted local models), so risk is a one-line config edit if Anthropic changes Sonnet 5's price — recommend a source-URL+date comment in the config for #14. Full findings + sources: [docs/research/proxy-shadow-pricing.md](https://git.arthurerlich.de/haylan/LLM-Server/src/branch/research/proxy-shadow-pricing/docs/research/proxy-shadow-pricing.md) on branch `research/proxy-shadow-pricing`.

haylan referenced this issue from a commit

2026-08-25 05:11:17 +00:00

Author LiteLLM AI proxy service (resolves #14)

haylan referenced this issue

2026-08-25 05:11:23 +00:00

Author the docker-compose service for the chosen proxy #14

Sign in to join this conversation.