AI gateway/proxy: routing, per-workload keys, and cloud-cost estimation for the local LLM stack #9

New Issue

2026-08-25T04:39:30Z

haylan commented

2026-08-25 04:39:30 +00:00

Destination

An AI gateway/proxy service, fronting llama.cpp (and reachable by Open WebUI, coding CLIs, and future LLM-shaped workloads like Gitea code review and Paperless OCR/tagging), that: (a) issues per-workload API keys/accounts, (b) tracks usage per key and estimates what that usage would have cost on a reference cloud API (reference model TBD by research), and (c) is architected to route to additional LLM backends later even though only local llama.cpp is wired now — deployed as real docker-compose config + docs alongside the existing stack.

Notes

Prior context: builds on Local AI inference stack: llama.cpp + Qwen3.8-27B on Radeon R9700, which explicitly deferred "multi-model AI gateway / routing" as out of scope. This map picks that up as its own effort. Relevant existing docs: docker-compose.yml, docs/network-access.md, docs/coding-cli-setup.md.

Standing decisions from grilling (settled during charting, not open tickets):

Destination shape: in-place change — tickets end in real docker-compose/config/docs committed to the repo, same as map #1, not just decisions.
Breadth: routing + cost tracking + per-workload keys are one gateway, not separable phases.
Integration point: the proxy sits in front of everything — Open WebUI, coding CLIs, and every new workload — not just new workloads. Open WebUI's current direct connection to llama.cpp gets migrated to go through the proxy.
Backend scope now vs. later: only local llama.cpp is actually routed to. No real paid external provider is wired in. Instead: usage against local llama.cpp is priced against a reference cloud API's published pricing purely to show "what this would have cost" — a shadow estimate, not real spend. Real external routing stays architecturally possible (tool choice should support adding a backend later) but isn't built now.
Network exposure: LAN + external, mirroring Open WebUI's ai.home / ai.haylan.ch pattern (existing NPM instance) — not LAN-only like llama.cpp's raw API.
Dashboard: a UI dashboard for spend-by-key over time is wanted (most gateway tools ship this), not just API/log queries.
Concurrency: only one local model/GPU serves every workload through this proxy. A priority queue (interactive usage ahead of batch jobs) is wanted, not plain FIFO — mechanics resolved by How should the proxy queue and prioritize concurrent requests across workloads?.
Workloads: two concrete-ish targets named (a code-reviewer tool, Paperless OCR/tagging) plus a longer wishlist surfaced during grilling — general coding CLIs, home automation/misc scripts, Gitea code review, Paperless OCR quality via docling, an AI watermark-removal service, Nextcloud Memories face-recognition speedup. None of these consumer integrations are scoped/ordered yet (see Not yet specified) — this map builds the gateway's capability to issue them keys, not every integration.
Tool choice: open decision, not pre-picked. LiteLLM proxy is a plausible fit (OpenAI/Anthropic-compatible routing, virtual keys, spend tracking, custom pricing) but must be confirmed against alternatives, not assumed.

Skills to consult per ticket: /research for tool-choice and pricing-reference tickets; /grilling + /domain-modeling if a new open decision surfaces.

Execution note: this effort carries execution into the map (per the destination above) — tickets can end in actual compose files, scripts, and docs committed to the repo, not only decisions recorded here.

Tickets

[ ] Verify the AI proxy stack on the real R9700 box (LiteLLM scheduler smoke test) (#17)
[x] Which self-hosted AI gateway/proxy tool fits this effort's needs? (#10)
[x] Reference cloud model/pricing for the shadow-cost estimate (#11)
[x] How should per-workload API keys/accounts be provisioned, rotated, and documented? (#12)
[x] Network/hostname plan for exposing the proxy (LAN + external) (#13)
[x] How should the proxy queue and prioritize concurrent requests across workloads? (#16)
[x] Author the docker-compose service for the chosen proxy (#14)
[x] Migrate Open WebUI (and existing consumers) to route through the new proxy (#15)

Decisions so far

Tool choice settled: Which self-hosted AI gateway/proxy tool fits this effort's needs? (#10) — LiteLLM proxy, confirmed against its docs (virtual keys with per-key spend, self-hosted Admin UI dashboard, config-driven custom per-token pricing, unified Anthropic/OpenAI routing, 2-service docker-compose deploy) and against alternatives (Portkey's self-hosted OSS gateway lacks a dashboard/budgets; Helicone is in maintenance mode and weaker on virtual keys/custom pricing); see docs/research/proxy-tool-choice.md on branch research/proxy-tool-choice.
Reference pricing settled: Reference cloud model/pricing for the shadow-cost estimate (#11) — Claude Sonnet 5 as a single fixed reference ($2/MTok input, $10/MTok output), not a multi-tier table, wired via a model_info.input_cost_per_token/output_cost_per_token block on the local model's LiteLLM config.yaml entry; local llama.cpp token counts are priced as-is (no re-tokenization against Claude's tokenizer), which is an acceptable approximation for a for-fun shadow estimate; see docs/research/proxy-shadow-pricing.md on branch research/proxy-shadow-pricing.
Key provisioning settled: How should per-workload API keys/accounts be provisioned, rotated, and documented? (#12) — keys created by hand in LiteLLM's Admin UI, named as a workload-purpose slug (e.g. paperless-ocr); no budget/rate limit by default (for-fun estimate, not resource protection); dropped into each workload's own .env, never committed here; the Admin UI's key list is the ledger (no separate tracking doc); revocation is manual, only on retirement/compromise. Documented in docs/proxy-key-onboarding.md on main.
Network plan settled: Network/hostname plan for exposing the proxy (LAN + external) (#13) — proxy.ai.home (LAN) / proxy.ai.haylan.ch (external), new NPM Proxy Hosts mirroring Open WebUI's pattern. LiteLLM's own virtual-key auth is sufficient for external exposure, no extra NPM auth layer. Admin UI (/ui) stays LAN-only — proxy.ai.haylan.ch needs an explicit NPM rule denying /ui since it shares a port with the API. Documented in docs/network-access.md on main.
Queueing/priority settled: How should the proxy queue and prioritize concurrent requests across workloads? (#16) — two tiers (interactive: Open WebUI/coding CLIs/Gitea code review vs. batch: Paperless OCR/Nextcloud Memories/watermark removal), via LiteLLM's built-in (beta) scheduler as the default plan, smoke-test required in #14 before relying on it, fallback shim only if that fails. Queued-too-long requests time out rather than waiting indefinitely. Confirmed no new Lazytainer risk. Documented in docs/proxy-request-priority.md on main.
Compose authoring done: Author the docker-compose service for the chosen proxy (#14) — litellm + litellm-db services added to docker-compose.yml, litellm-config.yaml wiring #11's shadow-cost pricing and #16's priority-scheduling stub, required env vars in .env.example. Untested on real hardware (see #17). Committed on main.
Migration done: Migrate Open WebUI (and existing consumers) to route through the new proxy (#15) — Open WebUI and all three coding CLIs (Claude Code/Kimi/OpenCode) now point at litellm instead of llama-server directly, each with its own virtual key; llama-server's host port publish is dropped (internal-only). docs/coding-cli-setup.md updated. Committed on main.

Not yet specified

Which of the many named future workloads (Gitea code review, Paperless-OCR/docling, AI watermark removal, Nextcloud Memories speedup, general coding CLIs, home automation) actually get provisioned accounts first, and in what order — only a wishlist so far, not a plan.
Whether/how a real paid external provider ever gets wired in for actual routing (not just shadow pricing) — deferred, no trigger condition defined yet.
Whether GPU idle-suspend (Lazytainer, from map #1) behaves correctly once multiple concurrent workloads hit the proxy instead of just Open WebUI — needs the proxy to exist before this is testable.

Out of scope

GPU scheduling/access for non-LLM workloads (COLMAP photogrammetry/splatting) — raised during grilling, but it's a GPU-contention problem, not an AI-routing problem. Not ticketed; would be its own effort if it ever needs actual AI (not just raw GPU) support.

## Destination An AI gateway/proxy service, fronting llama.cpp (and reachable by Open WebUI, coding CLIs, and future LLM-shaped workloads like Gitea code review and Paperless OCR/tagging), that: (a) issues per-workload API keys/accounts, (b) tracks usage per key and estimates what that usage would have cost on a reference cloud API (reference model TBD by research), and (c) is architected to route to additional LLM backends later even though only local llama.cpp is wired now — deployed as real docker-compose config + docs alongside the existing stack. ## Notes **Prior context**: builds on [Local AI inference stack: llama.cpp + Qwen3.8-27B on Radeon R9700](https://git.arthurerlich.de/haylan/LLM-Server/issues/1), which explicitly deferred "multi-model AI gateway / routing" as out of scope. This map picks that up as its own effort. Relevant existing docs: docker-compose.yml, docs/network-access.md, docs/coding-cli-setup.md. **Standing decisions from grilling** (settled during charting, not open tickets): - **Destination shape**: in-place change — tickets end in real docker-compose/config/docs committed to the repo, same as map #1, not just decisions. - **Breadth**: routing + cost tracking + per-workload keys are one gateway, not separable phases. - **Integration point**: the proxy sits in front of *everything* — Open WebUI, coding CLIs, and every new workload — not just new workloads. Open WebUI's current direct connection to llama.cpp gets migrated to go through the proxy. - **Backend scope now vs. later**: only local llama.cpp is actually routed to. No real paid external provider is wired in. Instead: usage against local llama.cpp is priced against a **reference cloud API's published pricing** purely to show "what this would have cost" — a shadow estimate, not real spend. Real external routing stays architecturally possible (tool choice should support adding a backend later) but isn't built now. - **Network exposure**: LAN + external, mirroring Open WebUI's ai.home / ai.haylan.ch pattern (existing NPM instance) — not LAN-only like llama.cpp's raw API. - **Dashboard**: a UI dashboard for spend-by-key over time is wanted (most gateway tools ship this), not just API/log queries. - **Concurrency**: only one local model/GPU serves every workload through this proxy. A **priority queue** (interactive usage ahead of batch jobs) is wanted, not plain FIFO — mechanics resolved by [How should the proxy queue and prioritize concurrent requests across workloads?](https://git.arthurerlich.de/haylan/LLM-Server/issues/16). - **Workloads**: two concrete-ish targets named (a code-reviewer tool, Paperless OCR/tagging) plus a longer wishlist surfaced during grilling — general coding CLIs, home automation/misc scripts, Gitea code review, Paperless OCR quality via docling, an AI watermark-removal service, Nextcloud Memories face-recognition speedup. None of these consumer integrations are scoped/ordered yet (see Not yet specified) — this map builds the gateway's *capability* to issue them keys, not every integration. - **Tool choice**: open decision, not pre-picked. LiteLLM proxy is a plausible fit (OpenAI/Anthropic-compatible routing, virtual keys, spend tracking, custom pricing) but must be confirmed against alternatives, not assumed. **Skills to consult per ticket**: /research for tool-choice and pricing-reference tickets; /grilling + /domain-modeling if a new open decision surfaces. **Execution note**: this effort carries execution into the map (per the destination above) — tickets can end in actual compose files, scripts, and docs committed to the repo, not only decisions recorded here. ## Tickets [ ] Verify the AI proxy stack on the real R9700 box (LiteLLM scheduler smoke test) (#17) [x] Which self-hosted AI gateway/proxy tool fits this effort's needs? (#10) [x] Reference cloud model/pricing for the shadow-cost estimate (#11) [x] How should per-workload API keys/accounts be provisioned, rotated, and documented? (#12) [x] Network/hostname plan for exposing the proxy (LAN + external) (#13) [x] How should the proxy queue and prioritize concurrent requests across workloads? (#16) [x] Author the docker-compose service for the chosen proxy (#14) [x] Migrate Open WebUI (and existing consumers) to route through the new proxy (#15) ## Decisions so far - **Tool choice settled**: [Which self-hosted AI gateway/proxy tool fits this effort's needs? (#10)](https://git.arthurerlich.de/haylan/LLM-Server/issues/10) — **LiteLLM proxy**, confirmed against its docs (virtual keys with per-key spend, self-hosted Admin UI dashboard, config-driven custom per-token pricing, unified Anthropic/OpenAI routing, 2-service docker-compose deploy) and against alternatives (Portkey's self-hosted OSS gateway lacks a dashboard/budgets; Helicone is in maintenance mode and weaker on virtual keys/custom pricing); see docs/research/proxy-tool-choice.md on branch research/proxy-tool-choice. - **Reference pricing settled**: [Reference cloud model/pricing for the shadow-cost estimate (#11)](https://git.arthurerlich.de/haylan/LLM-Server/issues/11) — **Claude Sonnet 5** as a single fixed reference ($2/MTok input, $10/MTok output), not a multi-tier table, wired via a `model_info.input_cost_per_token`/`output_cost_per_token` block on the local model's LiteLLM `config.yaml` entry; local llama.cpp token counts are priced as-is (no re-tokenization against Claude's tokenizer), which is an acceptable approximation for a for-fun shadow estimate; see docs/research/proxy-shadow-pricing.md on branch research/proxy-shadow-pricing. - **Key provisioning settled**: [How should per-workload API keys/accounts be provisioned, rotated, and documented? (#12)](https://git.arthurerlich.de/haylan/LLM-Server/issues/12) — keys created by hand in LiteLLM's Admin UI, named as a workload-purpose slug (e.g. `paperless-ocr`); no budget/rate limit by default (for-fun estimate, not resource protection); dropped into each workload's own .env, never committed here; the Admin UI's key list is the ledger (no separate tracking doc); revocation is manual, only on retirement/compromise. Documented in docs/proxy-key-onboarding.md on main. - **Network plan settled**: [Network/hostname plan for exposing the proxy (LAN + external) (#13)](https://git.arthurerlich.de/haylan/LLM-Server/issues/13) — `proxy.ai.home` (LAN) / `proxy.ai.haylan.ch` (external), new NPM Proxy Hosts mirroring Open WebUI's pattern. LiteLLM's own virtual-key auth is sufficient for external exposure, no extra NPM auth layer. Admin UI (`/ui`) stays LAN-only — `proxy.ai.haylan.ch` needs an explicit NPM rule denying `/ui` since it shares a port with the API. Documented in docs/network-access.md on main. - **Queueing/priority settled**: [How should the proxy queue and prioritize concurrent requests across workloads? (#16)](https://git.arthurerlich.de/haylan/LLM-Server/issues/16) — two tiers (interactive: Open WebUI/coding CLIs/Gitea code review vs. batch: Paperless OCR/Nextcloud Memories/watermark removal), via LiteLLM's built-in (beta) scheduler as the default plan, smoke-test required in #14 before relying on it, fallback shim only if that fails. Queued-too-long requests time out rather than waiting indefinitely. Confirmed no new Lazytainer risk. Documented in docs/proxy-request-priority.md on main. - **Compose authoring done**: [Author the docker-compose service for the chosen proxy (#14)](https://git.arthurerlich.de/haylan/LLM-Server/issues/14) — `litellm` + `litellm-db` services added to docker-compose.yml, `litellm-config.yaml` wiring #11's shadow-cost pricing and #16's priority-scheduling stub, required env vars in .env.example. Untested on real hardware (see #17). Committed on main. - **Migration done**: [Migrate Open WebUI (and existing consumers) to route through the new proxy (#15)](https://git.arthurerlich.de/haylan/LLM-Server/issues/15) — Open WebUI and all three coding CLIs (Claude Code/Kimi/OpenCode) now point at litellm instead of llama-server directly, each with its own virtual key; llama-server's host port publish is dropped (internal-only). docs/coding-cli-setup.md updated. Committed on main. ## Not yet specified - Which of the many named future workloads (Gitea code review, Paperless-OCR/docling, AI watermark removal, Nextcloud Memories speedup, general coding CLIs, home automation) actually get provisioned accounts first, and in what order — only a wishlist so far, not a plan. - Whether/how a *real* paid external provider ever gets wired in for actual routing (not just shadow pricing) — deferred, no trigger condition defined yet. - Whether GPU idle-suspend (Lazytainer, from map #1) behaves correctly once multiple concurrent workloads hit the proxy instead of just Open WebUI — needs the proxy to exist before this is testable. ## Out of scope - **GPU scheduling/access for non-LLM workloads** (COLMAP photogrammetry/splatting) — raised during grilling, but it's a GPU-contention problem, not an AI-routing problem. Not ticketed; would be its own effort if it ever needs actual AI (not just raw GPU) support.

haylan added the wayfinder:map label 2026-08-25 04:39:30 +00:00

haylan referenced this issue

2026-08-25 04:39:41 +00:00

Which self-hosted AI gateway/proxy tool fits this effort's needs? #10

haylan referenced this issue

2026-08-25 04:39:54 +00:00

Reference cloud model/pricing for the shadow-cost estimate #11

haylan referenced this issue

2026-08-25 04:39:55 +00:00

How should per-workload API keys/accounts be provisioned, rotated, and documented? #12

haylan referenced this issue

2026-08-25 04:39:56 +00:00