From f300c5b034154b749e0b1b143e9b69a854c7f15f Mon Sep 17 00:00:00 2001 From: Haylan Date: Wed, 26 Aug 2026 08:42:23 +0200 Subject: [PATCH] research: evaluate voidllm as a LiteLLM proxy replacement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against the live GitHub repo/docs (not secondhand): real, active, functioning AI gateway, ~5 months old, effectively single-maintainer, BSL 1.1. Fails two of the requirements from docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic can't be routed through it) and no request-priority/queuing feature. Recommendation: stay on LiteLLM. Co-Authored-By: Claude Sonnet 5 --- docs/research/voidllm-evaluation.md | 133 ++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 docs/research/voidllm-evaluation.md diff --git a/docs/research/voidllm-evaluation.md b/docs/research/voidllm-evaluation.md new file mode 100644 index 0000000..3ffb032 --- /dev/null +++ b/docs/research/voidllm-evaluation.md @@ -0,0 +1,133 @@ +# Evaluation: VoidLLM as a replacement for LiteLLM proxy + +**Question:** Does [voidmind-io/voidllm](https://github.com/voidmind-io/voidllm) +(the user asked us to look at it) beat the already-chosen tool (LiteLLM proxy, +see [`docs/research/proxy-tool-choice.md`](https://git.arthurerlich.de/haylan/LLM-Server/raw/branch/research/proxy-tool-choice/docs/research/proxy-tool-choice.md) +on branch `research/proxy-tool-choice`, and [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10)) +against this effort's requirements ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9))? + +**Headline: the repo is real and is a genuine, functioning AI gateway/proxy** — +not a placeholder, not something unrelated to the name. It is young +(created March 2026), effectively a one-person project, and it has a +documented, explicit gap that disqualifies it for this repo's actual usage +pattern: it cannot proxy the coding CLIs' LLM traffic at all. + +**Answer: stay on LiteLLM.** VoidLLM fails one hard requirement outright +(coding-CLI routing) and has no equivalent to LiteLLM's priority-queue +building block. It matches or is comparable on virtual keys, dashboard, and +custom pricing, but that isn't enough to justify a switch, let alone the +migration cost of re-doing #12–#15's completed work. + +## Does the repo exist and is it what it claims to be? + +Yes on both counts, confirmed directly against the GitHub API and repo +content (not a blog post or secondhand summary): + +- Repo: `voidmind-io/voidllm`, public, not a fork, not archived. Description: + "Privacy-first LLM proxy and AI gateway - load balancing, multi-provider + routing, API key management, usage tracking, rate limiting. Self-hosted. + Zero knowledge of your prompts." Created 2026-03-17, last pushed + 2026-08-25 (same week as this evaluation). Language: Go, 129 stars, 15 + forks, 28 open issues. + Source: `https://api.github.com/repos/voidmind-io/voidllm` (fetched + directly). +- README confirms it is exactly what the description says: a self-hosted + proxy sitting in front of OpenAI/Anthropic/Azure/Ollama/vLLM/custom + providers, with virtual API keys, RBAC (org/team/user/key), rate limits, + token budgets, a web dashboard (usage, keys, playground), and an MCP + gateway feature. + Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/README.md`. +- It has real release artifacts (Linux/Windows/macOS binaries), a Helm + chart, CI/codecov/Go-report-card/OpenSSF-scorecard/Snyk badges, and a + documented `docs/` tree with real content behind every link checked + (providers, load balancing, API reference) — not stub pages. + Source: same README; `docs/models/providers.md`, `docs/api/overview.md`, + `docs/models/load-balancing.md`, `docs/index.md`, all fetched from + `raw.githubusercontent.com/voidmind-io/voidllm/main/`. + +**Maturity/health caveat:** this is a small, young project. Contributors +per the GitHub API: `christianromeni` (151 commits — the sole real author), +`dependabot[bot]` (36, automated), and two accounts with 1 commit each +(`martinsotirov`, `SAY-5`). Effectively a solo maintainer, ~5 months old. +The README itself discloses "This project was built with significant +assistance from AI (Claude by Anthropic)." None of this makes it fake, but +it is a materially less-established project than LiteLLM (widely deployed, +100+ integrations, multi-year history) and carries the usual small-project +risks: bus factor, slower security response, less community troubleshooting +history. +Source: `https://api.github.com/repos/voidmind-io/voidllm/contributors`. + +**License:** Business Source License 1.1, not OSI open source. Self-hosting +for internal/production use is explicitly and unconditionally permitted +("regardless of the number of instances, users, or volume of traffic"); +the restriction is only on reselling it as a competing hosted/managed +service. Converts to Apache 2.0 four years after each release. Fine for +this repo's private homelab use, but a step down from LiteLLM's plain MIT. +Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/LICENSE`. + +It also has a paid tier structure (Pro €49/mo, Enterprise €149/mo, one-time +"Founding Member" €999) gating cross-org analytics, SSO/OIDC, audit logs, +OpenTelemetry, and Redis-backed multi-instance state behind payment. The +features this evaluation needs (virtual keys, per-key usage, dashboard, +custom pricing, docker-compose deploy) are all listed under the free +Community tier, so the paywall doesn't block this repo's use case — but it's +a different project shape than LiteLLM's free/MIT-with-optional-enterprise- +addon model. +Source: README "Features" table. + +## Requirement-by-requirement + +| Requirement | LiteLLM (current) | VoidLLM | +|---|---|---| +| OpenAI-compatible routing | Yes | Yes — `/v1/chat/completions`, embeddings, images, audio, streaming | +| Anthropic-compatible / unified Anthropic Messages endpoint | Yes — native `/v1/messages` unified endpoint accepts Anthropic-format requests, translates to any backend | **No.** No `/v1/messages` or any Anthropic-shaped *inbound* endpoint exists. VoidLLM only accepts OpenAI-format requests and can translate *outbound* to an Anthropic-format upstream (`provider: anthropic` in config) — the reverse direction of what's needed | +| Coding CLIs routed through the proxy | Yes — Claude Code, Kimi, OpenCode all point at LiteLLM today (issue #15, `docs/coding-cli-setup.md`) | **No — explicitly unsupported.** `docs/models/providers.md`: *"Claude Code talks directly to Anthropic's API for LLM access - you can't route its LLM requests through VoidLLM."* VoidLLM can only be added as an MCP server to Claude Code, not as its LLM backend | +| Per-workload virtual keys with separate usage views | Yes | Yes — `vl_uk_`/`vl_tk_`/`vl_sa_`/`vl_sk_` key types, org→team→user→key RBAC hierarchy, per-key and per-team usage (`GET /api/v1/usage/me`, `GET /api/v1/orgs/:org_id/usage`) | +| Usage/spend dashboard (not logs-only) | Yes — Admin UI `/ui`, Usage tab, free tier | Yes — Web UI with dashboard/usage/keys/playground screens, listed as Community (free) tier | +| Custom cost-per-token pricing for local model | Yes — `model_info.input_cost_per_token`/`output_cost_per_token` in `config.yaml` (already wired in this repo's `litellm-config.yaml` against Claude Sonnet 5's published rate) | Yes, equivalent mechanism — per-model `pricing.input_per_1m`/`output_per_1m` in `voidllm.yaml` | +| docker-compose self-hostable alongside existing stack | Yes — already running (`litellm` + `litellm-db` services in `docker-compose.yml`) | Yes — documented `docker-compose up` quick start, single Go binary, SQLite by default or Postgres | +| Native request queuing/priority | Beta, real but flaky — scheduler with a `priority` field, known bug (leaks into provider request, closed not-planned); needs smoke test (issue #17) | **Not found.** No queuing/priority-scheduling doc page exists in VoidLLM's docs index. The only "priority" concept is a *load-balancing* strategy (which upstream **deployment** to prefer/fail over to) — not request-level queue ordering for concurrent callers hitting one backend. Rate limiting is reject-on-429, not queue-and-wait. Concretely weaker than even LiteLLM's beta scheduler for this repo's actual need (one local GPU, interactive vs. batch tiers) | +| Room to add more LLM backends later | Yes — `model_list` of arbitrary provider entries, 100+ providers | Yes — 6 built-in provider types (OpenAI, Anthropic, Azure, Ollama, vLLM, custom-OpenAI-compatible), multi-deployment load balancing/failover per model | +| Project health/maturity | MIT, multi-year, widely deployed, 100+ integrations | Real project, ~5 months old, effectively solo-maintained, BSL 1.1, 129 stars | + +Sources for the VoidLLM column: `README.md`, `docs/models/providers.md`, +`docs/api/overview.md`, `docs/models/load-balancing.md`, `docs/index.md` +(all `raw.githubusercontent.com/voidmind-io/voidllm/main/...`, fetched +directly during this evaluation). LiteLLM column sourced from +`docs/research/proxy-tool-choice.md` on branch `research/proxy-tool-choice` +and this repo's live `litellm-config.yaml` / `docker-compose.yml`. + +## Why this disqualifies VoidLLM here + +Two failures, not one, and they hit the requirements list at its hardest +points: + +1. **Coding-CLI routing is a hard requirement this repo already depends on.** + Issue #15 migrated Claude Code, Kimi, and OpenCode to route through the + proxy (`docs/coding-cli-setup.md`), and issue #9's destination explicitly + lists coding CLIs as one of the gateway's fronted consumers. VoidLLM's own + docs say plainly that Claude Code's LLM traffic cannot go through it. + Even setting Claude Code aside, VoidLLM has no inbound Anthropic + Messages-shaped endpoint at all — any Anthropic-format client (present or + future) is unsupported, only OpenAI-format inbound is. LiteLLM's + `/v1/messages` unified endpoint is a direct, working answer to this same + need today. +2. **No request-priority/queuing story**, which issue #16 already settled on + using LiteLLM's beta scheduler for. VoidLLM has nothing documented in + this space beyond reject-on-limit rate limiting and load-balancer + deployment ordering. Switching would mean giving up even LiteLLM's shaky + beta feature for nothing. + +On top of both dealbreakers, VoidLLM is a much younger, single-maintainer +project against an already-integrated, working LiteLLM deployment (#14/#15 +done, only the scheduler smoke test in #17 outstanding). There's no +requirement VoidLLM meets that LiteLLM doesn't already meet as well or +better, so there's no upside to weigh against the migration cost and the +two outright gaps. + +## Recommendation + +**Stick with LiteLLM.** Do not switch. VoidLLM is worth a second look in the +future only if it adds an Anthropic-format inbound endpoint (making +coding-CLI routing possible) and a real request-queuing/priority mechanism — +neither exists today.