Merge remote-tracking branch 'origin/research/voidllm-evaluation'

Answers whether the $20/mo Claude Pro subscription can be used as a
LiteLLM backend instead of a pay-per-token Anthropic API key. Finding:
no — Pro/Max OAuth credentials are contractually restricted to Claude
Code / native Anthropic apps (Consumer ToS + Usage Policy, actively
enforced server-side since March 2026). Documents the supported path:
a separate Anthropic Console API key as a model_list entry, same shape
as the existing llama.cpp entry. Same pattern applies to other
providers (ChatGPT Plus, Gemini subscriptions).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs/research/connecting-cloud-providers.md

@@ -0,0 +1,149 @@
+# Connecting real cloud LLM providers (Anthropic Claude et al.) behind the LiteLLM gateway
+
+**Date:** 2026-08-26
+**Scope:** Can the existing $20/mo Claude Pro subscription be used as a backend behind our
+self-hosted LiteLLM gateway (`docker-compose.yml`, `litellm-config.yaml`) instead of, or alongside,
+llama.cpp? If not, what's the concrete path to add real Anthropic (and other) cloud models?
+
+## 1. Claude Pro subscription quota vs. Anthropic API — is it the same thing?
+
+**No. They are two separate products with separate billing, and Anthropic's own documentation and
+Terms of Service explicitly prohibit routing Pro/Max subscription credentials through third-party
+tools like a self-hosted gateway.**
+
+### 1a. Pro plan does not include API access
+
+Anthropic's help center is explicit:
+
+> "The Pro plan does not include API usage through the Claude Console. If you're interested in both
+> enhanced Claude features and the Claude API, you'll need to set up Console access to pay for API
+> usage separately."
+— https://support.claude.com/en/articles/8325606-what-is-the-pro-plan
+
+Pro/Max are flat-fee subscriptions to *products* (claude.ai web/desktop/mobile chat, and Claude Code)
+with rolling usage limits. The Anthropic API (Console/`platform.claude.com`) is a metered,
+pay-per-token product with its own separate billing account. There is no documented way to
+authenticate a non-Anthropic tool against Pro/Max quota "as if" it were an API key — no such
+integration exists.
+
+### 1b. How Claude Code itself authenticates, and why that path is closed off to other tools
+
+When you run `claude` and log in via `/login` with a claude.ai account, Claude Code obtains an
+**OAuth token scoped to the subscription** (standard OAuth 2.0 authorization-code flow against
+Anthropic's consent page). Anthropic also documents `claude setup-token`, which mints a **long-lived
+(1-year) OAuth token** (`CLAUDE_CODE_OAUTH_TOKEN`, prefixed `sk-ant-oat01-...`) explicitly *for use
+outside interactive login* — e.g. CI. This is real, officially documented, subscription-backed
+credential material that is technically exportable as an environment variable.
+— https://code.claude.com/docs/en/authentication
+
+**However, Anthropic's official Legal & Compliance page for Claude Code states this is authorized for
+Claude Code (and other native Anthropic apps) only, not for arbitrary third-party tools:**
+
+> "**OAuth authentication** is intended exclusively for purchasers of Claude Free, Pro, Max, Team, and
+> Enterprise subscription plans and is designed to support ordinary use of Claude Code and other
+> native Anthropic applications."
+>
+> "Anthropic does not permit third-party developers to offer Claude.ai login into their own
+> applications, or to route requests through Free, Pro, or Max plan credentials on behalf of their
+> users. Moreover, developers may not collect, store, or intermediate Claude.ai credentials or session
+> tokens — sign-in to a Claude account must complete through Anthropic's own flow."
+>
+> "Anthropic reserves the right to take measures to enforce these restrictions and may do so without
+> prior notice."
+— https://code.claude.com/docs/en/legal-and-compliance ("Authentication and credential use" section)
+
+That page cites the governing documents directly:
+- Consumer Terms of Service (Free/Pro/Max): https://www.anthropic.com/legal/consumer-terms
+- Commercial Terms of Service (Team/Enterprise/API): https://www.anthropic.com/legal/commercial-terms
+- Anthropic Usage Policy: https://www.anthropic.com/legal/aup
+
+**This is not just a ToS technicality — it's actively enforced.** In March 2026 Anthropic used
+server-side enforcement (in addition to legal action) to block third-party "harnesses" (e.g.
+OpenClaw and similar tools) from routing traffic through Claude subscription OAuth credentials,
+specifically citing that such traffic bypasses the telemetry/behavior the Claude Code harness
+provides. Coverage: https://www.theregister.com/software/2026/02/20/anthropic-clarifies-ban-on-third-party-tool-access-to-claude/
+(secondary source; the primary enforcement basis is the Legal & Compliance page above, which is
+current as fetched today).
+
+**Community finding (flagged as unofficial/against ToS):** technically, `claude setup-token`'s OAuth
+token *can* be handed to LiteLLM (some LiteLLM community docs / discussions describe pointing
+`ANTHROPIC_AUTH_TOKEN`/`CLAUDE_CODE_OAUTH_TOKEN` at a proxy, e.g.
+https://docs.litellm.ai/docs/tutorials/claude_code_max_subscription and
+https://github.com/BerriAI/litellm/discussions/30827). This works at the protocol level because the
+token is a bearer credential like any other. **It is exactly the pattern Anthropic's compliance page
+above says is not permitted** ("route requests through Free, Pro, or Max plan credentials on behalf
+of \[other] users" / outside "ordinary use of Claude Code and other native Anthropic applications").
+Doing this for **personal, single-user use** through the unmodified `claude` binary (e.g. just running
+`claude` itself pointed at your own gateway) is different from what's prohibited — the prohibition
+targets *intermediating* the subscription credential through a third-party tool/product on behalf of
+requests that aren't Claude Code itself. Routing arbitrary LiteLLM/OpenWebUI traffic through a
+subscription OAuth token extracted from `claude setup-token` falls squarely in the prohibited
+category. **Do not build this into the gateway.**
+
+### 1c. Bottom line on subscription reuse
+
+No documented, ToS-compliant way exists to point LiteLLM (or any third-party gateway) at Claude
+Pro/Max quota instead of a real API key. The $20/mo subscription is for claude.ai and Claude Code
+usage only.
+
+## 2. The real path: an Anthropic API key (Console, pay-per-token)
+
+This is the supported way to add Claude models to LiteLLM, and it's the same config shape already
+used for llama.cpp in `litellm-config.yaml`.
+
+LiteLLM's Anthropic provider docs (https://docs.litellm.ai/docs/providers/anthropic) give this
+`model_list` shape:
+
+```yaml
+model_list:
+  - model_name: claude-sonnet-5   # whatever alias you want callers to use
+    litellm_params:
+      model: anthropic/claude-sonnet-5    # "anthropic/<model-id>" tells LiteLLM which provider
+      api_key: os.environ/ANTHROPIC_API_KEY
+```
+
+Practical steps:
+1. Create a Console account / org at https://platform.claude.com (separate from claude.ai login),
+   generate an API key there.
+2. Add it to `.env` as `ANTHROPIC_API_KEY=sk-ant-api03-...` and reference it via
+   `os.environ/ANTHROPIC_API_KEY` in `litellm-config.yaml`, following the existing pattern used for
+   `LITELLM_MASTER_KEY`/`LITELLM_SALT_KEY` in `docker-compose.yml`.
+3. Add a `model_list` entry per Claude model you want exposed (e.g. `claude-sonnet-5`,
+   `claude-haiku-4-5`), each with its own `input_cost_per_token`/`output_cost_per_token` if you want
+   LiteLLM's cost tracking to reflect real spend (today `litellm-config.yaml`'s cost fields are a
+   shadow estimate against Sonnet 5 pricing for the *local* model — real Anthropic model entries
+   should carry the model's *actual* published rate from
+   https://platform.claude.com/docs/en/about-claude/pricing).
+4. **This is billed separately from, and in addition to, the $20/mo subscription** — pay-per-token,
+   metered on the Console account, no relationship to Pro/Max usage limits.
+
+No other config changes are implied — the router/`general_settings` blocks already in place don't
+need to change for a second provider; LiteLLM routes per-`model_name` based on the `litellm_params`
+each entry declares.
+
+## 3. Other providers — same general pattern
+
+Every major consumer AI subscription (OpenAI's ChatGPT Plus, and by the same logic Google's Gemini
+subscription tiers) works identically to Anthropic's: **the consumer chat subscription and the
+pay-per-token developer API are separate products with separate billing, and the subscription does
+not unlock API access.** Confirmed for OpenAI: "ChatGPT Plus does not include API access... no API
+key in the box, no monthly API credits, and no discount on per-token prices. The OpenAI API is a
+separate product with its own billing." There's no equivalent OAuth-token-reuse loophole documented
+for OpenAI/Google either — this isn't an Anthropic-specific restriction, it's the standard shape of
+how these companies segment consumer vs. developer products. Adding OpenAI, Gemini, or any other
+cloud model to LiteLLM means the same recipe as Anthropic: get a real pay-per-token API key from that
+provider's own developer console and add a `model_list` entry with `litellm_params.model` set to
+that provider's LiteLLM prefix (`openai/...`, `gemini/...`, etc.) — see
+https://docs.litellm.ai/docs/providers for the full prefix list.
+
+## 4. Recommended next step
+
+- Do **not** attempt to feed the Claude Pro subscription's OAuth token into LiteLLM/Open WebUI — it's
+  against Anthropic's Consumer Terms of Service and Usage Policy, and Anthropic has shown it will
+  enforce this server-side without notice (per the March 2026 crackdown on third-party harnesses).
+- If real cloud Claude models behind the gateway are wanted, get a **separate Anthropic Console API
+  key** (pay-per-token, its own bill, independent of the $20/mo subscription) and add it as a
+  `model_list` entry per §2 above. Same recipe for any other provider (§3).
+- Keep using the existing $20/mo Claude Pro subscription only for what it's licensed for: the claude.ai
+  web app and the Claude Code CLI itself (this session included) — not as a backend behind the
+  self-hosted gateway.

docs/research/proxy-shadow-pricing.md

@@ -0,0 +1,210 @@
+# Research: Reference cloud model/pricing for the shadow-cost estimate
+
+**Question:** Which reference cloud model/API pricing should the shadow-cost
+estimate use (per #9's Destination), and how does that rate get wired into
+LiteLLM (per #10's tool choice) to compute "what this local usage would have
+cost on a real cloud API"?
+
+**Answer:** Use a single fixed reference — **Claude Sonnet 5**, at its
+current published API price, hardcoded into one `model_info` block in
+LiteLLM's `config.yaml`. Skip a multi-tier pricing table.
+
+## Reference model: single fixed price, not a tier table
+
+Two options were weighed, per #11's framing:
+
+**Option A — single fixed reference (recommended).** Pick one current Claude
+model and price the local model against it always.
+
+**Option B — small pricing table across a couple of tiers** (e.g. price the
+same local usage against both a Haiku-tier and a Sonnet-tier rate
+simultaneously, showing a range).
+
+Recommendation: **Option A**, using **Claude Sonnet 5** — the model this
+Claude Code CLI session itself runs on, and the model this repo's own
+`docs/coding-cli-setup.md` documents pointing coding CLIs at this stack's
+local Qwen3.8-27B model. Reasoning:
+
+- **Matches what the user already tracks.** Map #9's Notes says the user
+  already tracks Claude pricing day-to-day. Sonnet is the tier actually used
+  for coding-CLI work against Claude directly (and is what this session is
+  running as), not a hypothetical comparison tier — a single number tied to
+  "what I'd have paid on the tool I actually use" is more meaningful than an
+  abstract low/high band.
+- **Simple to maintain.** One `model_info` block, one number to update when
+  Anthropic changes Sonnet pricing (rare — pricing on this page has been
+  stable for months at a time; see staleness section below), versus a table
+  that needs every row kept current. This is explicitly "for fun" (map #9's
+  Destination/Notes), not real accounting — a table adds bookkeeping
+  overhead the use case doesn't need.
+- **Local model's size is Sonnet-comparable, not flagship-comparable.** The
+  locally hosted model is `Qwen3.8-27B` (`docs/coding-cli-setup.md`) — a
+  ~27B-class model. Pricing it against Anthropic's flagship (Opus tier,
+  $5/$25 per MTok) would overstate the shadow cost for what's actually a
+  mid-tier local model; pricing it against Sonnet ($2/$10 per MTok, the
+  mid-tier) is the closer-fitting comparison, and it's also the tier this
+  repo already documents pointing coding CLIs at when using the *real*
+  Claude API (as opposed to the local shim) is desired.
+- A tier table would matter if the goal were "estimate real cloud spend
+  across scenarios," but map #9 explicitly frames this as a shadow/for-fun
+  estimate with no real external routing wired in — one clear number serves
+  that better than a range that needs interpreting.
+
+**Current price** (as of 2026-08-25, per Anthropic's official pricing page):
+
+| Model | Input | Output |
+|---|---|---|
+| **Claude Sonnet 5** | **$2.00 / MTok** ($0.000002/token) | **$10.00 / MTok** ($0.00001/token) |
+
+Source: [Claude Platform docs — Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
+(Model pricing table). Note: this $2/$10 rate was originally introductory
+pricing through 2026-08-31 with a scheduled increase to $3/$15 on
+2026-09-01; Anthropic's pricing page (fetched today) states that increase
+"will not occur" and $2/$10 is now the standard price — so this is a stable
+number, not a rate about to change out from under the config.
+
+Ignore prompt-caching, batch, and tool-use-overhead pricing modifiers for
+this shadow estimate — llama.cpp's local usage has none of those API
+features, so there's nothing to map them onto; base input/output token
+pricing is the only thing that has a clean local-usage analogue.
+
+## LiteLLM config shape: `model_info` custom pricing
+
+Confirmed against LiteLLM's docs (same mechanism #10 already found; this
+plugs the confirmed rate straight in). Add a `model_info` block to the local
+model's entry in `config.yaml`:
+
+```yaml
+model_list:
+  - model_name: qwen3.8-27b-local          # or whatever this stack names it
+    litellm_params:
+      model: openai/qwen3.8-27b            # or the provider shim used to reach llama.cpp
+      api_base: http://llama-server:8080/v1
+    model_info:
+      input_cost_per_token: 0.000002       # $2 / 1,000,000 — Claude Sonnet 5 input rate
+      output_cost_per_token: 0.00001       # $10 / 1,000,000 — Claude Sonnet 5 output rate
+```
+
+Confirmed details:
+
+- **Exact keys**: `model_info.input_cost_per_token` and
+  `model_info.output_cost_per_token`, both plain decimal USD-per-token
+  floats. LiteLLM also supports `input_cost_per_second` (time-based, e.g.
+  SageMaker-style billing) and `input_cost_per_character` /
+  `input_cost_per_image` / `input_cost_per_audio_token` /
+  `input_cost_per_video_per_second` for other modalities — none needed here
+  since this is a plain text chat model priced token-for-token.
+- **Input vs. output distinguished**: yes — separate keys, matching how
+  Claude's own pricing (and llama.cpp's own `usage.prompt_tokens` /
+  `usage.completion_tokens` split) is already input/output-separated.
+- **Per-model override**: yes — `model_info` is set per entry in
+  `model_list`, so only the local model entry needs it; LiteLLM's own
+  built-in cost map for 100+ known providers is untouched for any other
+  model added to the proxy later.
+- **Where the resulting cost surfaces**: computed via LiteLLM's internal
+  `completion_cost()` function (the same path used for every provider,
+  built-in or custom-priced) on every `/chat/completions` /
+  `/v1/messages` call. It surfaces in:
+  - **Per-key spend**: `/key/info` returns a `spend` field (cumulative USD)
+    per virtual key — this is the per-workload number map #9 wants.
+  - **Admin UI dashboard** (`/ui`, confirmed present in #10's research):
+    the Usage tab visualizes spend by key/team, sourced from the same
+    spend ledger.
+  - **Spend logs**: written to LiteLLM's Postgres-backed
+    `LiteLLM_SpendLogs` / verification-token table, queryable via
+    `/team/info` and `/user/info` for aggregation.
+  - **Per-call logging object**: `kwargs["response_cost"]` on each
+    completion call, for anyone hooking custom logging/callbacks later.
+
+Source: [LiteLLM — Custom Pricing docs](https://docs.litellm.ai/docs/proxy/custom_pricing),
+[LiteLLM — Virtual Keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys)
+(`/key/info` spend field example), [LiteLLM — completion_cost /
+model_prices_and_context_window.json](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
+(the built-in cost map that `model_info` overrides for a given entry).
+
+## Token-count mapping: token-for-token is fine here
+
+Claude and Qwen3.8-27B use different tokenizers, so the *same text* produces
+different token counts on each — a genuinely rigorous "what would this
+exact conversation have cost on Claude" would need to re-tokenize the local
+conversation text with Claude's tokenizer and price *that* count, not
+llama.cpp's own token count.
+
+LiteLLM does not do this re-tokenization for a custom-priced model: its
+`completion_cost()` multiplies the token counts reported in the backend's
+own `usage.prompt_tokens` / `usage.completion_tokens` response by whatever
+`input_cost_per_token` / `output_cost_per_token` is configured for that
+`model_list` entry — it does not re-count tokens against a different
+provider's tokenizer for cost purposes (LiteLLM's own token-counting
+utilities, e.g. `token_counter()`, exist as a fallback for providers that
+don't return usage at all, not as a re-tokenization step for cost
+calculation on a provider that does).
+
+**This is fine, and doesn't need fixing**, given map #9's own framing: this
+is explicitly a shadow/for-fun estimate, not real accounting. Treating
+llama.cpp's reported token count as if it were "Claude tokens" and applying
+Claude's per-token rate directly is a reasonable, cheap approximation —
+token counts between modern tokenizers for English text are typically
+within a similar order of magnitude (roughly comparable, not identical), so
+the estimate is order-of-magnitude meaningful ("this conversation would
+have cost about $X on Claude") without claiming precision it doesn't have.
+Building actual re-tokenization against Claude's tokenizer purely to feed a
+for-fun number would be effort disproportionate to the destination. If this
+ever needs to be exact, the fix is a small conversion factor applied at
+config time (e.g. inflate the configured per-token rate by a fudge factor
+to roughly account for tokenizer differences) — not worth doing now.
+
+One tokenizer wrinkle worth noting for future-proofing, not action:
+Anthropic's pricing page notes Claude 4.7-and-later models (which includes
+Sonnet 5) use a newer tokenizer producing "approximately 30% more tokens for
+the same text" than earlier Claude models. This doesn't change the
+recommendation (Sonnet 5 is still the right reference), it's just a reminder
+that "tokens" are already an approximate, provider-specific unit even within
+Anthropic's own model lineup — reinforcing that treating llama.cpp's token
+count as directly billable at Claude's rate is consistent with how loosely
+"a token" is already defined across models, not a special-case shortcut
+being taken here.
+
+## Config staleness risk
+
+LiteLLM ships a built-in default pricing table
+([`model_prices_and_context_window.json`](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json))
+covering 100+ known providers/models, refreshed via the LiteLLM project's
+own releases — that table is what could silently drift out of date for any
+model relying on it. **This doesn't apply to the local model here**: because
+the local llama.cpp model isn't a real named provider model, its pricing is
+only ever set via the explicit `model_info` block in this repo's own
+`config.yaml`, which LiteLLM never overwrites or auto-refreshes — the only
+way the shadow-cost number goes stale is if Anthropic changes Sonnet 5's
+published price and nobody updates the two numbers in this repo's config.
+
+Given Anthropic's pricing page explicitly states the Sonnet 5 rate is now
+locked in as the standard price (the previously-scheduled 2026-09-01
+increase was cancelled), staleness risk here is low and infrequent — but
+not zero, since Anthropic can still change prices with future model
+launches or repricing. Practical mitigation for implementation (#14): put
+the reference price in `config.yaml` with a comment noting the source URL
+and date it was last checked, so a future price change is a one-line
+`config.yaml` edit plus a comment-date bump — no code change, no migration.
+No need for anything more automated (e.g. scraping Anthropic's pricing page
+at startup) — that's more machinery than a for-fun estimate warrants.
+
+## Bottom line for #14 (compose/config authoring)
+
+- Use **Claude Sonnet 5** as the fixed shadow-pricing reference:
+  `input_cost_per_token: 0.000002`, `output_cost_per_token: 0.00001` in the
+  local model's `model_info` block in LiteLLM's `config.yaml`.
+- No pricing table, no per-request tier selection — one rate, one config
+  block, matches map #9's "for fun not real accounting" framing and the
+  user's existing day-to-day use of Claude Sonnet for coding-CLI work.
+- Token counts come straight from llama.cpp's own reported
+  `usage.prompt_tokens`/`completion_tokens` via LiteLLM's normal
+  `completion_cost()` path — no re-tokenization against Claude's tokenizer,
+  which is an acceptable approximation for a shadow estimate.
+- Resulting spend is visible per-key via `/key/info`, aggregated via
+  `/team/info`/`/user/info`, and in the Admin UI's Usage dashboard — no
+  extra tracking code needed, this is the same spend-tracking path LiteLLM
+  uses for any other provider.
+- Staleness risk is low (Sonnet 5's rate is currently locked as standard
+  pricing) and, if it ever drifts, is a one-line `config.yaml` edit — worth
+  a source-URL-and-date comment in the config, nothing more elaborate.

docs/research/proxy-tool-choice.md

@@ -0,0 +1,241 @@
+# Research: Which self-hosted AI gateway/proxy tool fits this effort's needs?
+
+**Question:** Which self-hosted AI gateway/proxy tool should front llama.cpp,
+given the requirements in issue #10 (Anthropic/OpenAI-compatible routing,
+per-workload virtual keys with separate usage views, a spend dashboard,
+custom cost-per-token pricing for a local model, docker-compose
+self-hosting alongside the existing stack, room to add backends later, and
+a plus for native queue/priority support relevant to #16)?
+
+**Answer: LiteLLM proxy.** It is the only candidate that meets every
+hard requirement out of the box, self-hosted, in its free/MIT tier. Portkey's
+open-source gateway is disqualified on the dashboard/virtual-key/budget
+requirement (those are cloud-only). Helicone is a weaker fit (maintenance
+mode, unclear virtual-key/custom-pricing story, feature-reduced self-host
+build, observability-first rather than budget/gateway-first). A hand-rolled
+nginx+script layer would mean re-building LiteLLM's virtual-key store, spend
+DB, dashboard, and Anthropic↔OpenAI translation from scratch — a maintenance
+trap, not a shortcut.
+
+## Candidate: LiteLLM proxy
+
+**License / project health:** MIT-licensed, with a separate `enterprise/`
+subdirectory under its own license for a small set of add-on features (SSO,
+audit logs, guaranteed-capacity priority reservation — see below). Widely
+deployed, 100+ provider integrations.
+Source: [BerriAI/litellm LICENSE](https://raw.githubusercontent.com/BerriAI/litellm/main/LICENSE),
+[BerriAI/litellm GitHub repo](https://github.com/BerriAI/litellm).
+
+**Anthropic/OpenAI-compatible routing:** LiteLLM proxy exposes a unified
+`/v1/messages` endpoint that accepts Anthropic-format requests and
+translates them to whatever backend format the target model needs (and
+translates the response back), so Anthropic-format clients (coding CLIs)
+and OpenAI-format clients (Open WebUI) can both hit the same proxy against
+the same `model_list` entry pointing at llama.cpp's OpenAI-compatible
+`/v1/chat/completions`. This means llama.cpp's own native `/v1/messages`
+shim doesn't strictly need to be reached directly through the proxy — LiteLLM
+does its own Anthropic↔OpenAI translation in front of llama.cpp's OpenAI
+endpoint, which is one plausible wiring; routing straight through to
+llama.cpp's native shim as a passthrough is a second option worth checking
+at implementation time (issue #14/#15 territory, not this ticket).
+Source: [LiteLLM /v1/messages unified endpoint docs](https://docs.litellm.ai/docs/anthropic_unified/),
+[LiteLLM — Claude Code with non-Anthropic models](https://docs.litellm.ai/docs/tutorials/claude_non_anthropic_models).
+
+**Virtual keys / per-workload accounts:** `/key/generate` issues a virtual
+key with its own `max_budget`, `budget_duration`, and `tpm_limit`/`rpm_limit`.
+Keys can be owned by a `user_id` or a `team_id`, so each workload (Open
+WebUI, a coding CLI, Gitea code review, Paperless OCR, etc.) gets its own
+key with its own budget and its own spend record, queryable via
+`/key/info` and aggregated per team via `/team/info`. Spend is written to
+the `LiteLLM_VerificationTokenTable` and computed via LiteLLM's own
+`completion_cost()` on every call.
+Source: [LiteLLM — Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys).
+
+**Spend dashboard:** The Admin UI (`/ui`) ships in the open-source build —
+key/model management plus a Usage tab showing spend tracked per key/team.
+Enterprise adds SSO/SAML, audit logs, and a more advanced UI on top, but
+core spend-by-key visualization is not gated.
+Source: [LiteLLM — Proxy UI docs](https://docs.litellm.ai/docs/proxy/ui),
+cross-checked via [LiteLLM GitHub repo description](https://github.com/BerriAI/litellm)
+("100+ LLM integrations, budgets, rate limits, logging, and virtual keys" in
+the free edition).
+
+**Custom cost-per-token pricing (shadow cloud-cost estimate):** Add a
+`model_info` block per model in `config.yaml`:
+
+```yaml
+model_list:
+  - model_name: my-local-model
+    litellm_params:
+      model: openai/local-model   # or whatever provider shim fits llama.cpp
+      api_base: http://llama-server:8080/v1
+    model_info:
+      input_cost_per_token: 0.000001
+      output_cost_per_token: 0.000002
+```
+
+This is exactly the mechanism #11 (reference cloud pricing) needs: once #11
+picks a reference cloud model/price, its per-token rate goes straight into
+this block and LiteLLM computes "what this local usage would have cost"
+using its normal `completion_cost()` path — no separate cost-tracking code
+needed.
+Source: [LiteLLM — Custom pricing docs](https://docs.litellm.ai/docs/proxy/custom_pricing).
+
+**Docker-compose deployment:** The documented quickstart is a two-service
+compose (LiteLLM gateway + Postgres, Postgres storing keys/models/spend
+logs); Redis is optional and only needed for multi-instance state (rate
+limiting, cross-instance priority queueing) — a single-instance deployment
+alongside the existing llama.cpp/Open WebUI/Qdrant/Lazytainer stack doesn't
+require it. `LITELLM_SALT_KEY` must be set to something real before
+production use (it encrypts stored provider keys); the documented example
+otherwise uses `sk-1234` as a placeholder master key that must be replaced.
+A YAML-only, no-database mode exists but drops budget enforcement — not
+useful here since budgets/spend-per-key are a hard requirement.
+Source: [LiteLLM — Docker Quick Start](https://docs.litellm.ai/docs/proxy/docker_quick_start).
+
+**Room for more backends later:** LiteLLM's whole design point is a
+`model_list` of arbitrary provider entries behind one routing layer (100+
+providers documented) — adding a second/third backend later is a config
+edit, not an architecture change.
+Source: [BerriAI/litellm GitHub repo](https://github.com/BerriAI/litellm).
+
+**Queuing/priority (relevant to #16, not decided here):** LiteLLM has an
+open-source (beta) request-prioritization scheduler: callers pass a
+`priority` field (lower number = higher priority) and a Router-level queue
+polls until a slot opens; multi-instance deployments need Redis to share
+queue state. This is a real, if beta-quality, building block for #16's
+priority queue and would mean #16 doesn't need a separate queuing
+component in front of the proxy. However:
+- The stricter **`priority_reservation`** feature (hard-reserving a % of
+  TPM/RPM capacity per priority tier — not just soft-prioritizing) is
+  gated behind the enterprise license.
+- The plain `priority` parameter had a real reported bug (leaking the
+  `priority` field into the provider request, breaking calls to some
+  providers) that maintainers closed as "not planned" rather than fixed —
+  worth a smoke test against llama.cpp specifically before #16 relies on
+  it, and worth treating the whole feature as "beta, verify before
+  depending on it" rather than a settled capability.
+Sources: [LiteLLM — Request Prioritization (scheduler) docs](https://docs.litellm.ai/docs/scheduler),
+[BerriAI/litellm issue #7144 — "Priority feature is broken"](https://github.com/BerriAI/litellm/issues/7144)
+(closed not-planned),
+[BerriAI/litellm issue #6867 — scheduler polling bug](https://github.com/BerriAI/litellm/issues/6867),
+[BerriAI/litellm issue #13405 — feature request for priority-based request
+handling via API keys, i.e. the simpler ergonomic form isn't fully built
+yet either](https://github.com/BerriAI/litellm/issues/13405).
+
+## Candidate: Portkey — disqualified
+
+Portkey's core gateway (`portkey-ai/gateway`) went fully open-source
+(Apache 2.0) and is self-hostable via Docker with routing, fallbacks,
+retries, load balancing, and a local logging console. But per the repo's
+own docs, **usage analytics/cost tracking, budget enforcement, and the
+spend dashboard are explicitly listed as "available in hosted and
+enterprise versions"** — i.e. they require Portkey's cloud control plane,
+not the self-hosted gateway alone. That fails this ticket's hard
+requirement for a self-hosted spend dashboard and per-key usage views, so
+Portkey is out regardless of its otherwise-solid routing feature set.
+Source: [Portkey-AI/gateway GitHub repo](https://github.com/portkey-ai/gateway).
+
+## Candidate: Helicone — weaker fit
+
+Helicone is self-hostable via a documented docker-compose stack (web
+dashboard, a combined API+proxy service, Postgres, ClickHouse, MinIO for
+S3-compatible storage), Apache-2.0, so it can be run independent of
+Helicone's own cloud. But:
+- **Project status:** Mintlify acquired Helicone (2026-03-03) and the
+  product is reported to be in maintenance mode; the repo is still active
+  and Apache-2.0 so self-hosting isn't blocked, but it's a weaker bet for
+  a component meant to grow (more backends, more workloads) over time.
+- **Feature parity gap:** the self-hosted docs explicitly note other
+  providers (Vertex AI, Bedrock, Azure OpenAI) aren't supported in the
+  self-hosted build the way they are in the cloud version — a signal the
+  self-host path is the less-maintained one.
+- **Virtual keys / custom pricing:** not clearly documented for the
+  self-hosted build in the pages checked — Helicone's primary framing is
+  LLM *observability* (logging, tracing, cost dashboards computed from
+  known-provider pricing tables) rather than a virtual-key issuing/budget
+  gateway; injecting a custom per-token price for an unlisted local model
+  isn't documented the way LiteLLM's `model_info.input_cost_per_token` is.
+- **Format translation:** self-host docs show separate `oai/` and
+  `anthropic/` proxy paths rather than a documented single endpoint that
+  translates Anthropic-format calls to an OpenAI-format backend the way
+  LiteLLM's unified `/v1/messages` does.
+
+Sources: [Helicone — Docker Compose self-deploy docs](https://docs.helicone.ai/getting-started/self-deploy-docker),
+[Helicone — self-hosting launch announcement](https://www.helicone.ai/blog/self-hosting-launch),
+[Helicone/helicone GitHub repo](https://github.com/helicone/helicone).
+**Confidence note:** these self-host feature-gap claims come from
+docs-page summaries rather than a hands-on deployment; if Helicone is
+ever reconsidered, verify virtual-key/custom-pricing support directly
+against a running self-hosted instance rather than trusting this summary
+alone.
+
+## Candidate: hand-rolled nginx + scripts — rejected as a maintenance trap
+
+A thin reverse proxy plus custom scripts could technically satisfy each
+bullet in isolation (issue an API key = generate a token and check it in
+an nginx `map`/Lua script; track spend = write to a DB on each request;
+dashboard = a small custom UI; custom pricing = a config file the script
+reads; Anthropic↔OpenAI translation = hand-written request/response
+transformers). But that's rebuilding LiteLLM's virtual-key store, spend
+ledger, dashboard, and format-translation layer from scratch, in a
+piece this repo would then own and maintain indefinitely, against a
+target (LiteLLM) that already does all of it, is MIT-licensed, and is
+a straightforward docker-compose service. Not adopted.
+
+## Risks / gaps to carry into later tickets
+
+1. **Lazytainer idle-suspend interaction (flagged in #9's "Not yet
+   specified").** Lazytainer decides to stop `llama-server` based on
+   network packet activity on its published port
+   (`lazytainer.group.llamaserver.minPacketThreshold` /
+   `inactiveTimeout` in this repo's `docker-compose.yml`). If LiteLLM
+   proxy performs periodic background health checks against configured
+   models (a common gateway behavior), that traffic could look like
+   real usage to Lazytainer and prevent it from ever idling the
+   container down. This needs to be checked against LiteLLM's actual
+   health-check config (there are documented options to disable/tune
+   background health checks) once the compose service is authored in
+   #14, and verified on real hardware per #9's "Not yet specified" note
+   on Lazytainer + multi-workload proxy interaction.
+2. **llama.cpp's native `/v1/messages` shim vs. LiteLLM's own Anthropic
+   translation.** Two wiring options exist (LiteLLM translates
+   Anthropic→OpenAI itself and hits llama.cpp's OpenAI endpoint; or
+   LiteLLM passes Anthropic-format requests straight through to
+   llama.cpp's own native shim). This ticket confirms both are
+   plausible per LiteLLM's docs but doesn't pick one — that's
+   implementation detail for #14/#15, and should be smoke-tested against
+   the actual coding-CLI flows in `docs/coding-cli-setup.md` once wired.
+3. **Priority queueing for #16 is a real but beta/rough-edged LiteLLM
+   feature**, with at least one reported-and-declined bug in the exact
+   `priority` mechanism, and the stronger reserved-capacity variant is
+   enterprise-gated. #16 should treat LiteLLM's scheduler as a
+   starting point to validate hands-on, not an assumed solved problem —
+   if it doesn't hold up under test, #16 may need a lightweight queuing
+   shim in front of the proxy (e.g. a small request-queue sidecar) rather
+   than reworking the whole gateway choice.
+4. **Redis need is deferred, not eliminated.** A single-instance LiteLLM
+   deployment (the right size for this effort) doesn't need Redis for
+   virtual keys/spend/dashboard, but the priority scheduler's
+   multi-instance behavior and rate-limit sharing do use it — if #16
+   ends up needing Redis-backed prioritization even in a single-instance
+   deployment, add a `redis` service to the compose file at that point;
+   no need to provision it speculatively now.
+
+## Bottom line for the wayfinder map
+
+- Adopt **LiteLLM proxy** (MIT-licensed, `BerriAI/litellm`) as the
+  gateway/proxy in front of llama.cpp for this effort.
+- It meets every hard requirement in #10 in its free/OSS build: virtual
+  keys with per-key spend, a self-hosted Admin UI spend dashboard,
+  config-driven custom per-token pricing (feeds #11 directly), a
+  documented two-service docker-compose deployment, and an
+  arbitrary-provider `model_list` that keeps future backends a config
+  change away.
+- Its beta priority/queueing feature is a promising but unproven fit for
+  #16 — validate it hands-on rather than assuming it's settled.
+- Portkey's self-hosted OSS gateway is disqualified (no self-hosted
+  dashboard/budgets). Helicone is a workable but weaker fallback
+  (maintenance-mode signal, feature-reduced self-host build, less clearly
+  documented virtual-key/custom-pricing support) if LiteLLM turns out to
+  be a poor fit during implementation.

docs/research/voidllm-evaluation.md

@@ -0,0 +1,133 @@
+# Evaluation: VoidLLM as a replacement for LiteLLM proxy
+
+**Question:** Does [voidmind-io/voidllm](https://github.com/voidmind-io/voidllm)
+(the user asked us to look at it) beat the already-chosen tool (LiteLLM proxy,
+see [`docs/research/proxy-tool-choice.md`](https://git.arthurerlich.de/haylan/LLM-Server/raw/branch/research/proxy-tool-choice/docs/research/proxy-tool-choice.md)
+on branch `research/proxy-tool-choice`, and [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10))
+against this effort's requirements ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9))?
+
+**Headline: the repo is real and is a genuine, functioning AI gateway/proxy** —
+not a placeholder, not something unrelated to the name. It is young
+(created March 2026), effectively a one-person project, and it has a
+documented, explicit gap that disqualifies it for this repo's actual usage
+pattern: it cannot proxy the coding CLIs' LLM traffic at all.
+
+**Answer: stay on LiteLLM.** VoidLLM fails one hard requirement outright
+(coding-CLI routing) and has no equivalent to LiteLLM's priority-queue
+building block. It matches or is comparable on virtual keys, dashboard, and
+custom pricing, but that isn't enough to justify a switch, let alone the
+migration cost of re-doing #12–#15's completed work.
+
+## Does the repo exist and is it what it claims to be?
+
+Yes on both counts, confirmed directly against the GitHub API and repo
+content (not a blog post or secondhand summary):
+
+- Repo: `voidmind-io/voidllm`, public, not a fork, not archived. Description:
+  "Privacy-first LLM proxy and AI gateway - load balancing, multi-provider
+  routing, API key management, usage tracking, rate limiting. Self-hosted.
+  Zero knowledge of your prompts." Created 2026-03-17, last pushed
+  2026-08-25 (same week as this evaluation). Language: Go, 129 stars, 15
+  forks, 28 open issues.
+  Source: `https://api.github.com/repos/voidmind-io/voidllm` (fetched
+  directly).
+- README confirms it is exactly what the description says: a self-hosted
+  proxy sitting in front of OpenAI/Anthropic/Azure/Ollama/vLLM/custom
+  providers, with virtual API keys, RBAC (org/team/user/key), rate limits,
+  token budgets, a web dashboard (usage, keys, playground), and an MCP
+  gateway feature.
+  Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/README.md`.
+- It has real release artifacts (Linux/Windows/macOS binaries), a Helm
+  chart, CI/codecov/Go-report-card/OpenSSF-scorecard/Snyk badges, and a
+  documented `docs/` tree with real content behind every link checked
+  (providers, load balancing, API reference) — not stub pages.
+  Source: same README; `docs/models/providers.md`, `docs/api/overview.md`,
+  `docs/models/load-balancing.md`, `docs/index.md`, all fetched from
+  `raw.githubusercontent.com/voidmind-io/voidllm/main/`.
+
+**Maturity/health caveat:** this is a small, young project. Contributors
+per the GitHub API: `christianromeni` (151 commits — the sole real author),
+`dependabot[bot]` (36, automated), and two accounts with 1 commit each
+(`martinsotirov`, `SAY-5`). Effectively a solo maintainer, ~5 months old.
+The README itself discloses "This project was built with significant
+assistance from AI (Claude by Anthropic)." None of this makes it fake, but
+it is a materially less-established project than LiteLLM (widely deployed,
+100+ integrations, multi-year history) and carries the usual small-project
+risks: bus factor, slower security response, less community troubleshooting
+history.
+Source: `https://api.github.com/repos/voidmind-io/voidllm/contributors`.
+
+**License:** Business Source License 1.1, not OSI open source. Self-hosting
+for internal/production use is explicitly and unconditionally permitted
+("regardless of the number of instances, users, or volume of traffic");
+the restriction is only on reselling it as a competing hosted/managed
+service. Converts to Apache 2.0 four years after each release. Fine for
+this repo's private homelab use, but a step down from LiteLLM's plain MIT.
+Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/LICENSE`.
+
+It also has a paid tier structure (Pro €49/mo, Enterprise €149/mo, one-time
+"Founding Member" €999) gating cross-org analytics, SSO/OIDC, audit logs,
+OpenTelemetry, and Redis-backed multi-instance state behind payment. The
+features this evaluation needs (virtual keys, per-key usage, dashboard,
+custom pricing, docker-compose deploy) are all listed under the free
+Community tier, so the paywall doesn't block this repo's use case — but it's
+a different project shape than LiteLLM's free/MIT-with-optional-enterprise-
+addon model.
+Source: README "Features" table.
+
+## Requirement-by-requirement
+
+| Requirement | LiteLLM (current) | VoidLLM |
+|---|---|---|
+| OpenAI-compatible routing | Yes | Yes — `/v1/chat/completions`, embeddings, images, audio, streaming |
+| Anthropic-compatible / unified Anthropic Messages endpoint | Yes — native `/v1/messages` unified endpoint accepts Anthropic-format requests, translates to any backend | **No.** No `/v1/messages` or any Anthropic-shaped *inbound* endpoint exists. VoidLLM only accepts OpenAI-format requests and can translate *outbound* to an Anthropic-format upstream (`provider: anthropic` in config) — the reverse direction of what's needed |
+| Coding CLIs routed through the proxy | Yes — Claude Code, Kimi, OpenCode all point at LiteLLM today (issue #15, `docs/coding-cli-setup.md`) | **No — explicitly unsupported.** `docs/models/providers.md`: *"Claude Code talks directly to Anthropic's API for LLM access - you can't route its LLM requests through VoidLLM."* VoidLLM can only be added as an MCP server to Claude Code, not as its LLM backend |
+| Per-workload virtual keys with separate usage views | Yes | Yes — `vl_uk_`/`vl_tk_`/`vl_sa_`/`vl_sk_` key types, org→team→user→key RBAC hierarchy, per-key and per-team usage (`GET /api/v1/usage/me`, `GET /api/v1/orgs/:org_id/usage`) |
+| Usage/spend dashboard (not logs-only) | Yes — Admin UI `/ui`, Usage tab, free tier | Yes — Web UI with dashboard/usage/keys/playground screens, listed as Community (free) tier |
+| Custom cost-per-token pricing for local model | Yes — `model_info.input_cost_per_token`/`output_cost_per_token` in `config.yaml` (already wired in this repo's `litellm-config.yaml` against Claude Sonnet 5's published rate) | Yes, equivalent mechanism — per-model `pricing.input_per_1m`/`output_per_1m` in `voidllm.yaml` |
+| docker-compose self-hostable alongside existing stack | Yes — already running (`litellm` + `litellm-db` services in `docker-compose.yml`) | Yes — documented `docker-compose up` quick start, single Go binary, SQLite by default or Postgres |
+| Native request queuing/priority | Beta, real but flaky — scheduler with a `priority` field, known bug (leaks into provider request, closed not-planned); needs smoke test (issue #17) | **Not found.** No queuing/priority-scheduling doc page exists in VoidLLM's docs index. The only "priority" concept is a *load-balancing* strategy (which upstream **deployment** to prefer/fail over to) — not request-level queue ordering for concurrent callers hitting one backend. Rate limiting is reject-on-429, not queue-and-wait. Concretely weaker than even LiteLLM's beta scheduler for this repo's actual need (one local GPU, interactive vs. batch tiers) |
+| Room to add more LLM backends later | Yes — `model_list` of arbitrary provider entries, 100+ providers | Yes — 6 built-in provider types (OpenAI, Anthropic, Azure, Ollama, vLLM, custom-OpenAI-compatible), multi-deployment load balancing/failover per model |
+| Project health/maturity | MIT, multi-year, widely deployed, 100+ integrations | Real project, ~5 months old, effectively solo-maintained, BSL 1.1, 129 stars |
+
+Sources for the VoidLLM column: `README.md`, `docs/models/providers.md`,
+`docs/api/overview.md`, `docs/models/load-balancing.md`, `docs/index.md`
+(all `raw.githubusercontent.com/voidmind-io/voidllm/main/...`, fetched
+directly during this evaluation). LiteLLM column sourced from
+`docs/research/proxy-tool-choice.md` on branch `research/proxy-tool-choice`
+and this repo's live `litellm-config.yaml` / `docker-compose.yml`.
+
+## Why this disqualifies VoidLLM here
+
+Two failures, not one, and they hit the requirements list at its hardest
+points:
+
+1. **Coding-CLI routing is a hard requirement this repo already depends on.**
+   Issue #15 migrated Claude Code, Kimi, and OpenCode to route through the
+   proxy (`docs/coding-cli-setup.md`), and issue #9's destination explicitly
+   lists coding CLIs as one of the gateway's fronted consumers. VoidLLM's own
+   docs say plainly that Claude Code's LLM traffic cannot go through it.
+   Even setting Claude Code aside, VoidLLM has no inbound Anthropic
+   Messages-shaped endpoint at all — any Anthropic-format client (present or
+   future) is unsupported, only OpenAI-format inbound is. LiteLLM's
+   `/v1/messages` unified endpoint is a direct, working answer to this same
+   need today.
+2. **No request-priority/queuing story**, which issue #16 already settled on
+   using LiteLLM's beta scheduler for. VoidLLM has nothing documented in
+   this space beyond reject-on-limit rate limiting and load-balancer
+   deployment ordering. Switching would mean giving up even LiteLLM's shaky
+   beta feature for nothing.
+
+On top of both dealbreakers, VoidLLM is a much younger, single-maintainer
+project against an already-integrated, working LiteLLM deployment (#14/#15
+done, only the scheduler smoke test in #17 outstanding). There's no
+requirement VoidLLM meets that LiteLLM doesn't already meet as well or
+better, so there's no upside to weigh against the migration cost and the
+two outright gaps.
+
+## Recommendation
+
+**Stick with LiteLLM.** Do not switch. VoidLLM is worth a second look in the
+future only if it adds an Anthropic-format inbound endpoint (making
+coding-CLI routing possible) and a real request-queuing/priority mechanism —
+neither exists today.