LLM-Server/docs/research/proxy-shadow-pricing.md

# Research: Reference cloud model/pricing for the shadow-cost estimate

**Question:** Which reference cloud model/API pricing should the shadow-cost
estimate use (per #9's Destination), and how does that rate get wired into
LiteLLM (per #10's tool choice) to compute "what this local usage would have
cost on a real cloud API"?

**Answer:** Use a single fixed reference — **Claude Sonnet 5**, at its
current published API price, hardcoded into one `model_info` block in
LiteLLM's `config.yaml`. Skip a multi-tier pricing table.

## Reference model: single fixed price, not a tier table

Two options were weighed, per #11's framing:

**Option A — single fixed reference (recommended).** Pick one current Claude
model and price the local model against it always.

**Option B — small pricing table across a couple of tiers** (e.g. price the
same local usage against both a Haiku-tier and a Sonnet-tier rate
simultaneously, showing a range).

Recommendation: **Option A**, using **Claude Sonnet 5** — the model this
Claude Code CLI session itself runs on, and the model this repo's own
`docs/coding-cli-setup.md` documents pointing coding CLIs at this stack's
local Qwen3.8-27B model. Reasoning:

- **Matches what the user already tracks.** Map #9's Notes says the user
  already tracks Claude pricing day-to-day. Sonnet is the tier actually used
  for coding-CLI work against Claude directly (and is what this session is
  running as), not a hypothetical comparison tier — a single number tied to
  "what I'd have paid on the tool I actually use" is more meaningful than an
  abstract low/high band.
- **Simple to maintain.** One `model_info` block, one number to update when
  Anthropic changes Sonnet pricing (rare — pricing on this page has been
  stable for months at a time; see staleness section below), versus a table
  that needs every row kept current. This is explicitly "for fun" (map #9's
  Destination/Notes), not real accounting — a table adds bookkeeping
  overhead the use case doesn't need.
- **Local model's size is Sonnet-comparable, not flagship-comparable.** The
  locally hosted model is `Qwen3.8-27B` (`docs/coding-cli-setup.md`) — a
  ~27B-class model. Pricing it against Anthropic's flagship (Opus tier,
  $5/$25 per MTok) would overstate the shadow cost for what's actually a
  mid-tier local model; pricing it against Sonnet ($2/$10 per MTok, the
  mid-tier) is the closer-fitting comparison, and it's also the tier this
  repo already documents pointing coding CLIs at when using the *real*
  Claude API (as opposed to the local shim) is desired.
- A tier table would matter if the goal were "estimate real cloud spend
  across scenarios," but map #9 explicitly frames this as a shadow/for-fun
  estimate with no real external routing wired in — one clear number serves
  that better than a range that needs interpreting.

**Current price** (as of 2026-08-25, per Anthropic's official pricing page):

| Model | Input | Output |
|---|---|---|
| **Claude Sonnet 5** | **$2.00 / MTok** ($0.000002/token) | **$10.00 / MTok** ($0.00001/token) |

Source: [Claude Platform docs — Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
(Model pricing table). Note: this $2/$10 rate was originally introductory
pricing through 2026-08-31 with a scheduled increase to $3/$15 on
2026-09-01; Anthropic's pricing page (fetched today) states that increase
"will not occur" and $2/$10 is now the standard price — so this is a stable
number, not a rate about to change out from under the config.

Ignore prompt-caching, batch, and tool-use-overhead pricing modifiers for
this shadow estimate — llama.cpp's local usage has none of those API
features, so there's nothing to map them onto; base input/output token
pricing is the only thing that has a clean local-usage analogue.

## LiteLLM config shape: `model_info` custom pricing

Confirmed against LiteLLM's docs (same mechanism #10 already found; this
plugs the confirmed rate straight in). Add a `model_info` block to the local
model's entry in `config.yaml`:

```yaml
model_list:
  - model_name: qwen3.8-27b-local          # or whatever this stack names it
    litellm_params:
      model: openai/qwen3.8-27b            # or the provider shim used to reach llama.cpp
      api_base: http://llama-server:8080/v1
    model_info:
      input_cost_per_token: 0.000002       # $2 / 1,000,000 — Claude Sonnet 5 input rate
      output_cost_per_token: 0.00001       # $10 / 1,000,000 — Claude Sonnet 5 output rate
```

Confirmed details:

- **Exact keys**: `model_info.input_cost_per_token` and
  `model_info.output_cost_per_token`, both plain decimal USD-per-token
  floats. LiteLLM also supports `input_cost_per_second` (time-based, e.g.
  SageMaker-style billing) and `input_cost_per_character` /
  `input_cost_per_image` / `input_cost_per_audio_token` /
  `input_cost_per_video_per_second` for other modalities — none needed here
  since this is a plain text chat model priced token-for-token.
- **Input vs. output distinguished**: yes — separate keys, matching how
  Claude's own pricing (and llama.cpp's own `usage.prompt_tokens` /
  `usage.completion_tokens` split) is already input/output-separated.
- **Per-model override**: yes — `model_info` is set per entry in
  `model_list`, so only the local model entry needs it; LiteLLM's own
  built-in cost map for 100+ known providers is untouched for any other
  model added to the proxy later.
- **Where the resulting cost surfaces**: computed via LiteLLM's internal
  `completion_cost()` function (the same path used for every provider,
  built-in or custom-priced) on every `/chat/completions` /
  `/v1/messages` call. It surfaces in:
  - **Per-key spend**: `/key/info` returns a `spend` field (cumulative USD)
    per virtual key — this is the per-workload number map #9 wants.
  - **Admin UI dashboard** (`/ui`, confirmed present in #10's research):
    the Usage tab visualizes spend by key/team, sourced from the same
    spend ledger.
  - **Spend logs**: written to LiteLLM's Postgres-backed
    `LiteLLM_SpendLogs` / verification-token table, queryable via
    `/team/info` and `/user/info` for aggregation.
  - **Per-call logging object**: `kwargs["response_cost"]` on each
    completion call, for anyone hooking custom logging/callbacks later.

Source: [LiteLLM — Custom Pricing docs](https://docs.litellm.ai/docs/proxy/custom_pricing),
[LiteLLM — Virtual Keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys)
(`/key/info` spend field example), [LiteLLM — completion_cost /
model_prices_and_context_window.json](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
(the built-in cost map that `model_info` overrides for a given entry).

## Token-count mapping: token-for-token is fine here

Claude and Qwen3.8-27B use different tokenizers, so the *same text* produces
different token counts on each — a genuinely rigorous "what would this
exact conversation have cost on Claude" would need to re-tokenize the local
conversation text with Claude's tokenizer and price *that* count, not
llama.cpp's own token count.

LiteLLM does not do this re-tokenization for a custom-priced model: its
`completion_cost()` multiplies the token counts reported in the backend's
own `usage.prompt_tokens` / `usage.completion_tokens` response by whatever
`input_cost_per_token` / `output_cost_per_token` is configured for that
`model_list` entry — it does not re-count tokens against a different
provider's tokenizer for cost purposes (LiteLLM's own token-counting
utilities, e.g. `token_counter()`, exist as a fallback for providers that
don't return usage at all, not as a re-tokenization step for cost
calculation on a provider that does).

**This is fine, and doesn't need fixing**, given map #9's own framing: this
is explicitly a shadow/for-fun estimate, not real accounting. Treating
llama.cpp's reported token count as if it were "Claude tokens" and applying
Claude's per-token rate directly is a reasonable, cheap approximation —
token counts between modern tokenizers for English text are typically
within a similar order of magnitude (roughly comparable, not identical), so
the estimate is order-of-magnitude meaningful ("this conversation would
have cost about $X on Claude") without claiming precision it doesn't have.
Building actual re-tokenization against Claude's tokenizer purely to feed a
for-fun number would be effort disproportionate to the destination. If this
ever needs to be exact, the fix is a small conversion factor applied at
config time (e.g. inflate the configured per-token rate by a fudge factor
to roughly account for tokenizer differences) — not worth doing now.

One tokenizer wrinkle worth noting for future-proofing, not action:
Anthropic's pricing page notes Claude 4.7-and-later models (which includes
Sonnet 5) use a newer tokenizer producing "approximately 30% more tokens for
the same text" than earlier Claude models. This doesn't change the
recommendation (Sonnet 5 is still the right reference), it's just a reminder
that "tokens" are already an approximate, provider-specific unit even within
Anthropic's own model lineup — reinforcing that treating llama.cpp's token
count as directly billable at Claude's rate is consistent with how loosely
"a token" is already defined across models, not a special-case shortcut
being taken here.

## Config staleness risk

LiteLLM ships a built-in default pricing table
([`model_prices_and_context_window.json`](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json))
covering 100+ known providers/models, refreshed via the LiteLLM project's
own releases — that table is what could silently drift out of date for any
model relying on it. **This doesn't apply to the local model here**: because
the local llama.cpp model isn't a real named provider model, its pricing is
only ever set via the explicit `model_info` block in this repo's own
`config.yaml`, which LiteLLM never overwrites or auto-refreshes — the only
way the shadow-cost number goes stale is if Anthropic changes Sonnet 5's
published price and nobody updates the two numbers in this repo's config.

Given Anthropic's pricing page explicitly states the Sonnet 5 rate is now
locked in as the standard price (the previously-scheduled 2026-09-01
increase was cancelled), staleness risk here is low and infrequent — but
not zero, since Anthropic can still change prices with future model
launches or repricing. Practical mitigation for implementation (#14): put
the reference price in `config.yaml` with a comment noting the source URL
and date it was last checked, so a future price change is a one-line
`config.yaml` edit plus a comment-date bump — no code change, no migration.
No need for anything more automated (e.g. scraping Anthropic's pricing page
at startup) — that's more machinery than a for-fun estimate warrants.

## Bottom line for #14 (compose/config authoring)

- Use **Claude Sonnet 5** as the fixed shadow-pricing reference:
  `input_cost_per_token: 0.000002`, `output_cost_per_token: 0.00001` in the
  local model's `model_info` block in LiteLLM's `config.yaml`.
- No pricing table, no per-request tier selection — one rate, one config
  block, matches map #9's "for fun not real accounting" framing and the
  user's existing day-to-day use of Claude Sonnet for coding-CLI work.
- Token counts come straight from llama.cpp's own reported
  `usage.prompt_tokens`/`completion_tokens` via LiteLLM's normal
  `completion_cost()` path — no re-tokenization against Claude's tokenizer,
  which is an acceptable approximation for a shadow estimate.
- Resulting spend is visible per-key via `/key/info`, aggregated via
  `/team/info`/`/user/info`, and in the Admin UI's Usage dashboard — no
  extra tracking code needed, this is the same spend-tracking path LiteLLM
  uses for any other provider.
- Staleness risk is low (Sonnet 5's rate is currently locked as standard
  pricing) and, if it ever drifts, is a one-line `config.yaml` edit — worth
  a source-URL-and-date comment in the config, nothing more elaborate.