Which self-hosted AI gateway/proxy tool fits this effort's needs? #10

New Issue

2026-08-25T04:39:41Z

haylan commented

2026-08-25 04:39:41 +00:00

Part of #9

Question

Which self-hosted AI gateway/proxy tool should front llama.cpp for this effort? It needs, at minimum:

OpenAI- and/or Anthropic-compatible routing (so Open WebUI, coding CLIs, and llama.cpp's existing /v1/messages shim keep working through it)
per-workload virtual API keys/accounts, each with its own usage view
usage/spend tracking with a dashboard (not just logs)
support for injecting custom cost-per-token pricing for a local model (needed for the shadow cloud-cost estimate — see #9's Destination)
self-hostable via docker-compose, fits alongside the existing stack (docker-compose.yml, named volumes, Lazytainer idle-suspend on the llama.cpp container)
room to add more backends later without a rebuild (per #9's Notes: real external routing isn't built now, but shouldn't be architecturally foreclosed)

Compare candidates (LiteLLM proxy is a plausible leading option per #9's Notes, but confirm rather than assume) against alternatives (e.g. Portkey, Helicone, a lighter custom option) and recommend one, with the reasoning and any gaps/risks (e.g. interaction with Lazytainer's idle-stop, compatibility with llama.cpp's Anthropic Messages shim).

Part of #9 ## Question Which self-hosted AI gateway/proxy tool should front llama.cpp for this effort? It needs, at minimum: - OpenAI- and/or Anthropic-compatible routing (so Open WebUI, coding CLIs, and llama.cpp's existing /v1/messages shim keep working through it) - per-workload virtual API keys/accounts, each with its own usage view - usage/spend tracking with a dashboard (not just logs) - support for injecting **custom cost-per-token pricing** for a local model (needed for the shadow cloud-cost estimate — see #9's Destination) - self-hostable via docker-compose, fits alongside the existing stack (docker-compose.yml, named volumes, Lazytainer idle-suspend on the llama.cpp container) - room to add more backends later without a rebuild (per #9's Notes: real external routing isn't built now, but shouldn't be architecturally foreclosed) Compare candidates (LiteLLM proxy is a plausible leading option per #9's Notes, but confirm rather than assume) against alternatives (e.g. Portkey, Helicone, a lighter custom option) and recommend one, with the reasoning and any gaps/risks (e.g. interaction with Lazytainer's idle-stop, compatibility with llama.cpp's Anthropic Messages shim).

haylan added the wayfinder:research label 2026-08-25 04:39:41 +00:00

haylan referenced this issue

2026-08-25 04:39:54 +00:00

Reference cloud model/pricing for the shadow-cost estimate #11

haylan referenced this issue

2026-08-25 04:39:55 +00:00

How should per-workload API keys/accounts be provisioned, rotated, and documented? #12

haylan referenced this issue

2026-08-25 04:40:05 +00:00

Author the docker-compose service for the chosen proxy #14

haylan added a new dependency 2026-08-25 04:40:13 +00:00

#11 Reference cloud model/pricing for the shadow-cost estimate

haylan added a new dependency 2026-08-25 04:40:13 +00:00

#12 How should per-workload API keys/accounts be provisioned, rotated, and documented?

haylan added a new dependency 2026-08-25 04:40:14 +00:00

#14 Author the docker-compose service for the chosen proxy

haylan referenced this issue

2026-08-25 04:40:25 +00:00

AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack #9

haylan referenced this issue

2026-08-25 04:41:13 +00:00

How should the proxy queue and prioritize concurrent requests across workloads? #16

haylan added a new dependency 2026-08-25 04:41:14 +00:00

#16 How should the proxy queue and prioritize concurrent requests across workloads?

haylan closed this issue

2026-08-25 04:45:24 +00:00

haylan commented

2026-08-25 04:45:40 +00:00

Recommendation: LiteLLM proxy (MIT-licensed, BerriAI/litellm).

It's the only self-hostable candidate meeting all hard requirements out of the box: virtual keys with per-key/per-team spend (/key/generate, /key/info), a self-hosted Admin UI spend dashboard included in the free build, config-driven custom per-token pricing (model_info.input_cost_per_token/output_cost_per_token — feeds #11 directly), a unified /v1/messages endpoint that translates Anthropic-format calls to an OpenAI-format backend (so it fronts both Open WebUI and Anthropic-format coding CLIs against llama.cpp), a documented 2-service docker-compose deploy (gateway + Postgres, Redis optional), and an arbitrary-provider model_list so more backends are a config edit later.

Alternatives checked and rejected:

Portkey: self-hosted OSS gateway (Apache 2.0) explicitly excludes spend dashboard/budgets/virtual-key usage tracking — those are cloud/enterprise-only per its own docs. Fails the dashboard requirement outright.
Helicone: self-hostable but now in Mintlify-acquisition maintenance mode, self-host build has a documented feature gap vs. cloud, and virtual-key/custom-pricing support for a local model isn't clearly documented — observability-first, not gateway-first.
Hand-rolled nginx+scripts: would mean rebuilding LiteLLM's key store, spend ledger, dashboard, and Anthropic↔OpenAI translation from scratch — a maintenance trap, not a shortcut.

Risks/gaps carried forward (see doc for detail): LiteLLM's background health-checks could keep Lazytainer's idle-suspend from ever triggering (verify/tune in #14); two plausible wirings exist for llama.cpp's native /v1/messages shim vs. LiteLLM's own translation (pick in #14/#15); LiteLLM's priority/queueing scheduler is real but beta with at least one declined bug report, and its stronger reserved-capacity variant is enterprise-gated — treat as a starting point for #16 to validate hands-on, not a solved problem.

Full findings + sources: docs/research/proxy-tool-choice.md on branch research/proxy-tool-choice

**Recommendation: LiteLLM proxy** (MIT-licensed, BerriAI/litellm). It's the only self-hostable candidate meeting all hard requirements out of the box: virtual keys with per-key/per-team spend (`/key/generate`, `/key/info`), a self-hosted Admin UI spend dashboard included in the free build, config-driven custom per-token pricing (`model_info.input_cost_per_token`/`output_cost_per_token` — feeds #11 directly), a unified `/v1/messages` endpoint that translates Anthropic-format calls to an OpenAI-format backend (so it fronts both Open WebUI and Anthropic-format coding CLIs against llama.cpp), a documented 2-service docker-compose deploy (gateway + Postgres, Redis optional), and an arbitrary-provider `model_list` so more backends are a config edit later. **Alternatives checked and rejected:** - **Portkey**: self-hosted OSS gateway (Apache 2.0) explicitly excludes spend dashboard/budgets/virtual-key usage tracking — those are cloud/enterprise-only per its own docs. Fails the dashboard requirement outright. - **Helicone**: self-hostable but now in Mintlify-acquisition maintenance mode, self-host build has a documented feature gap vs. cloud, and virtual-key/custom-pricing support for a local model isn't clearly documented — observability-first, not gateway-first. - **Hand-rolled nginx+scripts**: would mean rebuilding LiteLLM's key store, spend ledger, dashboard, and Anthropic↔OpenAI translation from scratch — a maintenance trap, not a shortcut. **Risks/gaps carried forward** (see doc for detail): LiteLLM's background health-checks could keep Lazytainer's idle-suspend from ever triggering (verify/tune in #14); two plausible wirings exist for llama.cpp's native /v1/messages shim vs. LiteLLM's own translation (pick in #14/#15); LiteLLM's priority/queueing scheduler is real but beta with at least one declined bug report, and its stronger reserved-capacity variant is enterprise-gated — treat as a starting point for #16 to validate hands-on, not a solved problem. Full findings + sources: [docs/research/proxy-tool-choice.md on branch research/proxy-tool-choice](https://git.arthurerlich.de/haylan/LLM-Server/src/branch/research/proxy-tool-choice/docs/research/proxy-tool-choice.md)

haylan referenced this issue

2026-08-25 04:50:14 +00:00

Reference cloud model/pricing for the shadow-cost estimate #11

Sign in to join this conversation.