Merge remote-tracking branch 'origin/research/voidllm-evaluation'

docs/research/voidllm-evaluation.md

@@ -0,0 +1,133 @@
+# Evaluation: VoidLLM as a replacement for LiteLLM proxy
+
+**Question:** Does [voidmind-io/voidllm](https://github.com/voidmind-io/voidllm)
+(the user asked us to look at it) beat the already-chosen tool (LiteLLM proxy,
+see [`docs/research/proxy-tool-choice.md`](https://git.arthurerlich.de/haylan/LLM-Server/raw/branch/research/proxy-tool-choice/docs/research/proxy-tool-choice.md)
+on branch `research/proxy-tool-choice`, and [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10))
+against this effort's requirements ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9))?
+
+**Headline: the repo is real and is a genuine, functioning AI gateway/proxy** —
+not a placeholder, not something unrelated to the name. It is young
+(created March 2026), effectively a one-person project, and it has a
+documented, explicit gap that disqualifies it for this repo's actual usage
+pattern: it cannot proxy the coding CLIs' LLM traffic at all.
+
+**Answer: stay on LiteLLM.** VoidLLM fails one hard requirement outright
+(coding-CLI routing) and has no equivalent to LiteLLM's priority-queue
+building block. It matches or is comparable on virtual keys, dashboard, and
+custom pricing, but that isn't enough to justify a switch, let alone the
+migration cost of re-doing #12–#15's completed work.
+
+## Does the repo exist and is it what it claims to be?
+
+Yes on both counts, confirmed directly against the GitHub API and repo
+content (not a blog post or secondhand summary):
+
+- Repo: `voidmind-io/voidllm`, public, not a fork, not archived. Description:
+  "Privacy-first LLM proxy and AI gateway - load balancing, multi-provider
+  routing, API key management, usage tracking, rate limiting. Self-hosted.
+  Zero knowledge of your prompts." Created 2026-03-17, last pushed
+  2026-08-25 (same week as this evaluation). Language: Go, 129 stars, 15
+  forks, 28 open issues.
+  Source: `https://api.github.com/repos/voidmind-io/voidllm` (fetched
+  directly).
+- README confirms it is exactly what the description says: a self-hosted
+  proxy sitting in front of OpenAI/Anthropic/Azure/Ollama/vLLM/custom
+  providers, with virtual API keys, RBAC (org/team/user/key), rate limits,
+  token budgets, a web dashboard (usage, keys, playground), and an MCP
+  gateway feature.
+  Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/README.md`.
+- It has real release artifacts (Linux/Windows/macOS binaries), a Helm
+  chart, CI/codecov/Go-report-card/OpenSSF-scorecard/Snyk badges, and a
+  documented `docs/` tree with real content behind every link checked
+  (providers, load balancing, API reference) — not stub pages.
+  Source: same README; `docs/models/providers.md`, `docs/api/overview.md`,
+  `docs/models/load-balancing.md`, `docs/index.md`, all fetched from
+  `raw.githubusercontent.com/voidmind-io/voidllm/main/`.
+
+**Maturity/health caveat:** this is a small, young project. Contributors
+per the GitHub API: `christianromeni` (151 commits — the sole real author),
+`dependabot[bot]` (36, automated), and two accounts with 1 commit each
+(`martinsotirov`, `SAY-5`). Effectively a solo maintainer, ~5 months old.
+The README itself discloses "This project was built with significant
+assistance from AI (Claude by Anthropic)." None of this makes it fake, but
+it is a materially less-established project than LiteLLM (widely deployed,
+100+ integrations, multi-year history) and carries the usual small-project
+risks: bus factor, slower security response, less community troubleshooting
+history.
+Source: `https://api.github.com/repos/voidmind-io/voidllm/contributors`.
+
+**License:** Business Source License 1.1, not OSI open source. Self-hosting
+for internal/production use is explicitly and unconditionally permitted
+("regardless of the number of instances, users, or volume of traffic");
+the restriction is only on reselling it as a competing hosted/managed
+service. Converts to Apache 2.0 four years after each release. Fine for
+this repo's private homelab use, but a step down from LiteLLM's plain MIT.
+Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/LICENSE`.
+
+It also has a paid tier structure (Pro €49/mo, Enterprise €149/mo, one-time
+"Founding Member" €999) gating cross-org analytics, SSO/OIDC, audit logs,
+OpenTelemetry, and Redis-backed multi-instance state behind payment. The
+features this evaluation needs (virtual keys, per-key usage, dashboard,
+custom pricing, docker-compose deploy) are all listed under the free
+Community tier, so the paywall doesn't block this repo's use case — but it's
+a different project shape than LiteLLM's free/MIT-with-optional-enterprise-
+addon model.
+Source: README "Features" table.
+
+## Requirement-by-requirement
+
+| Requirement | LiteLLM (current) | VoidLLM |
+|---|---|---|
+| OpenAI-compatible routing | Yes | Yes — `/v1/chat/completions`, embeddings, images, audio, streaming |
+| Anthropic-compatible / unified Anthropic Messages endpoint | Yes — native `/v1/messages` unified endpoint accepts Anthropic-format requests, translates to any backend | **No.** No `/v1/messages` or any Anthropic-shaped *inbound* endpoint exists. VoidLLM only accepts OpenAI-format requests and can translate *outbound* to an Anthropic-format upstream (`provider: anthropic` in config) — the reverse direction of what's needed |
+| Coding CLIs routed through the proxy | Yes — Claude Code, Kimi, OpenCode all point at LiteLLM today (issue #15, `docs/coding-cli-setup.md`) | **No — explicitly unsupported.** `docs/models/providers.md`: *"Claude Code talks directly to Anthropic's API for LLM access - you can't route its LLM requests through VoidLLM."* VoidLLM can only be added as an MCP server to Claude Code, not as its LLM backend |
+| Per-workload virtual keys with separate usage views | Yes | Yes — `vl_uk_`/`vl_tk_`/`vl_sa_`/`vl_sk_` key types, org→team→user→key RBAC hierarchy, per-key and per-team usage (`GET /api/v1/usage/me`, `GET /api/v1/orgs/:org_id/usage`) |
+| Usage/spend dashboard (not logs-only) | Yes — Admin UI `/ui`, Usage tab, free tier | Yes — Web UI with dashboard/usage/keys/playground screens, listed as Community (free) tier |
+| Custom cost-per-token pricing for local model | Yes — `model_info.input_cost_per_token`/`output_cost_per_token` in `config.yaml` (already wired in this repo's `litellm-config.yaml` against Claude Sonnet 5's published rate) | Yes, equivalent mechanism — per-model `pricing.input_per_1m`/`output_per_1m` in `voidllm.yaml` |
+| docker-compose self-hostable alongside existing stack | Yes — already running (`litellm` + `litellm-db` services in `docker-compose.yml`) | Yes — documented `docker-compose up` quick start, single Go binary, SQLite by default or Postgres |
+| Native request queuing/priority | Beta, real but flaky — scheduler with a `priority` field, known bug (leaks into provider request, closed not-planned); needs smoke test (issue #17) | **Not found.** No queuing/priority-scheduling doc page exists in VoidLLM's docs index. The only "priority" concept is a *load-balancing* strategy (which upstream **deployment** to prefer/fail over to) — not request-level queue ordering for concurrent callers hitting one backend. Rate limiting is reject-on-429, not queue-and-wait. Concretely weaker than even LiteLLM's beta scheduler for this repo's actual need (one local GPU, interactive vs. batch tiers) |
+| Room to add more LLM backends later | Yes — `model_list` of arbitrary provider entries, 100+ providers | Yes — 6 built-in provider types (OpenAI, Anthropic, Azure, Ollama, vLLM, custom-OpenAI-compatible), multi-deployment load balancing/failover per model |
+| Project health/maturity | MIT, multi-year, widely deployed, 100+ integrations | Real project, ~5 months old, effectively solo-maintained, BSL 1.1, 129 stars |
+
+Sources for the VoidLLM column: `README.md`, `docs/models/providers.md`,
+`docs/api/overview.md`, `docs/models/load-balancing.md`, `docs/index.md`
+(all `raw.githubusercontent.com/voidmind-io/voidllm/main/...`, fetched
+directly during this evaluation). LiteLLM column sourced from
+`docs/research/proxy-tool-choice.md` on branch `research/proxy-tool-choice`
+and this repo's live `litellm-config.yaml` / `docker-compose.yml`.
+
+## Why this disqualifies VoidLLM here
+
+Two failures, not one, and they hit the requirements list at its hardest
+points:
+
+1. **Coding-CLI routing is a hard requirement this repo already depends on.**
+   Issue #15 migrated Claude Code, Kimi, and OpenCode to route through the
+   proxy (`docs/coding-cli-setup.md`), and issue #9's destination explicitly
+   lists coding CLIs as one of the gateway's fronted consumers. VoidLLM's own
+   docs say plainly that Claude Code's LLM traffic cannot go through it.
+   Even setting Claude Code aside, VoidLLM has no inbound Anthropic
+   Messages-shaped endpoint at all — any Anthropic-format client (present or
+   future) is unsupported, only OpenAI-format inbound is. LiteLLM's
+   `/v1/messages` unified endpoint is a direct, working answer to this same
+   need today.
+2. **No request-priority/queuing story**, which issue #16 already settled on
+   using LiteLLM's beta scheduler for. VoidLLM has nothing documented in
+   this space beyond reject-on-limit rate limiting and load-balancer
+   deployment ordering. Switching would mean giving up even LiteLLM's shaky
+   beta feature for nothing.
+
+On top of both dealbreakers, VoidLLM is a much younger, single-maintainer
+project against an already-integrated, working LiteLLM deployment (#14/#15
+done, only the scheduler smoke test in #17 outstanding). There's no
+requirement VoidLLM meets that LiteLLM doesn't already meet as well or
+better, so there's no upside to weigh against the migration cost and the
+two outright gaps.
+
+## Recommendation
+
+**Stick with LiteLLM.** Do not switch. VoidLLM is worth a second look in the
+future only if it adds an Anthropic-format inbound endpoint (making
+coding-CLI routing possible) and a real request-queuing/priority mechanism —
+neither exists today.