research: evaluate voidllm as a LiteLLM proxy replacement
Verified against the live GitHub repo/docs (not secondhand): real, active, functioning AI gateway, ~5 months old, effectively single-maintainer, BSL 1.1. Fails two of the requirements from docs/research/proxy-tool-choice.md / issue #9 — no Anthropic Messages inbound endpoint (VoidLLM's own docs say Claude Code's LLM traffic can't be routed through it) and no request-priority/queuing feature. Recommendation: stay on LiteLLM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,133 @@
|
|||||||
|
# Evaluation: VoidLLM as a replacement for LiteLLM proxy
|
||||||
|
|
||||||
|
**Question:** Does [voidmind-io/voidllm](https://github.com/voidmind-io/voidllm)
|
||||||
|
(the user asked us to look at it) beat the already-chosen tool (LiteLLM proxy,
|
||||||
|
see [`docs/research/proxy-tool-choice.md`](https://git.arthurerlich.de/haylan/LLM-Server/raw/branch/research/proxy-tool-choice/docs/research/proxy-tool-choice.md)
|
||||||
|
on branch `research/proxy-tool-choice`, and [issue #10](https://git.arthurerlich.de/haylan/LLM-Server/issues/10))
|
||||||
|
against this effort's requirements ([issue #9](https://git.arthurerlich.de/haylan/LLM-Server/issues/9))?
|
||||||
|
|
||||||
|
**Headline: the repo is real and is a genuine, functioning AI gateway/proxy** —
|
||||||
|
not a placeholder, not something unrelated to the name. It is young
|
||||||
|
(created March 2026), effectively a one-person project, and it has a
|
||||||
|
documented, explicit gap that disqualifies it for this repo's actual usage
|
||||||
|
pattern: it cannot proxy the coding CLIs' LLM traffic at all.
|
||||||
|
|
||||||
|
**Answer: stay on LiteLLM.** VoidLLM fails one hard requirement outright
|
||||||
|
(coding-CLI routing) and has no equivalent to LiteLLM's priority-queue
|
||||||
|
building block. It matches or is comparable on virtual keys, dashboard, and
|
||||||
|
custom pricing, but that isn't enough to justify a switch, let alone the
|
||||||
|
migration cost of re-doing #12–#15's completed work.
|
||||||
|
|
||||||
|
## Does the repo exist and is it what it claims to be?
|
||||||
|
|
||||||
|
Yes on both counts, confirmed directly against the GitHub API and repo
|
||||||
|
content (not a blog post or secondhand summary):
|
||||||
|
|
||||||
|
- Repo: `voidmind-io/voidllm`, public, not a fork, not archived. Description:
|
||||||
|
"Privacy-first LLM proxy and AI gateway - load balancing, multi-provider
|
||||||
|
routing, API key management, usage tracking, rate limiting. Self-hosted.
|
||||||
|
Zero knowledge of your prompts." Created 2026-03-17, last pushed
|
||||||
|
2026-08-25 (same week as this evaluation). Language: Go, 129 stars, 15
|
||||||
|
forks, 28 open issues.
|
||||||
|
Source: `https://api.github.com/repos/voidmind-io/voidllm` (fetched
|
||||||
|
directly).
|
||||||
|
- README confirms it is exactly what the description says: a self-hosted
|
||||||
|
proxy sitting in front of OpenAI/Anthropic/Azure/Ollama/vLLM/custom
|
||||||
|
providers, with virtual API keys, RBAC (org/team/user/key), rate limits,
|
||||||
|
token budgets, a web dashboard (usage, keys, playground), and an MCP
|
||||||
|
gateway feature.
|
||||||
|
Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/README.md`.
|
||||||
|
- It has real release artifacts (Linux/Windows/macOS binaries), a Helm
|
||||||
|
chart, CI/codecov/Go-report-card/OpenSSF-scorecard/Snyk badges, and a
|
||||||
|
documented `docs/` tree with real content behind every link checked
|
||||||
|
(providers, load balancing, API reference) — not stub pages.
|
||||||
|
Source: same README; `docs/models/providers.md`, `docs/api/overview.md`,
|
||||||
|
`docs/models/load-balancing.md`, `docs/index.md`, all fetched from
|
||||||
|
`raw.githubusercontent.com/voidmind-io/voidllm/main/`.
|
||||||
|
|
||||||
|
**Maturity/health caveat:** this is a small, young project. Contributors
|
||||||
|
per the GitHub API: `christianromeni` (151 commits — the sole real author),
|
||||||
|
`dependabot[bot]` (36, automated), and two accounts with 1 commit each
|
||||||
|
(`martinsotirov`, `SAY-5`). Effectively a solo maintainer, ~5 months old.
|
||||||
|
The README itself discloses "This project was built with significant
|
||||||
|
assistance from AI (Claude by Anthropic)." None of this makes it fake, but
|
||||||
|
it is a materially less-established project than LiteLLM (widely deployed,
|
||||||
|
100+ integrations, multi-year history) and carries the usual small-project
|
||||||
|
risks: bus factor, slower security response, less community troubleshooting
|
||||||
|
history.
|
||||||
|
Source: `https://api.github.com/repos/voidmind-io/voidllm/contributors`.
|
||||||
|
|
||||||
|
**License:** Business Source License 1.1, not OSI open source. Self-hosting
|
||||||
|
for internal/production use is explicitly and unconditionally permitted
|
||||||
|
("regardless of the number of instances, users, or volume of traffic");
|
||||||
|
the restriction is only on reselling it as a competing hosted/managed
|
||||||
|
service. Converts to Apache 2.0 four years after each release. Fine for
|
||||||
|
this repo's private homelab use, but a step down from LiteLLM's plain MIT.
|
||||||
|
Source: `https://raw.githubusercontent.com/voidmind-io/voidllm/main/LICENSE`.
|
||||||
|
|
||||||
|
It also has a paid tier structure (Pro €49/mo, Enterprise €149/mo, one-time
|
||||||
|
"Founding Member" €999) gating cross-org analytics, SSO/OIDC, audit logs,
|
||||||
|
OpenTelemetry, and Redis-backed multi-instance state behind payment. The
|
||||||
|
features this evaluation needs (virtual keys, per-key usage, dashboard,
|
||||||
|
custom pricing, docker-compose deploy) are all listed under the free
|
||||||
|
Community tier, so the paywall doesn't block this repo's use case — but it's
|
||||||
|
a different project shape than LiteLLM's free/MIT-with-optional-enterprise-
|
||||||
|
addon model.
|
||||||
|
Source: README "Features" table.
|
||||||
|
|
||||||
|
## Requirement-by-requirement
|
||||||
|
|
||||||
|
| Requirement | LiteLLM (current) | VoidLLM |
|
||||||
|
|---|---|---|
|
||||||
|
| OpenAI-compatible routing | Yes | Yes — `/v1/chat/completions`, embeddings, images, audio, streaming |
|
||||||
|
| Anthropic-compatible / unified Anthropic Messages endpoint | Yes — native `/v1/messages` unified endpoint accepts Anthropic-format requests, translates to any backend | **No.** No `/v1/messages` or any Anthropic-shaped *inbound* endpoint exists. VoidLLM only accepts OpenAI-format requests and can translate *outbound* to an Anthropic-format upstream (`provider: anthropic` in config) — the reverse direction of what's needed |
|
||||||
|
| Coding CLIs routed through the proxy | Yes — Claude Code, Kimi, OpenCode all point at LiteLLM today (issue #15, `docs/coding-cli-setup.md`) | **No — explicitly unsupported.** `docs/models/providers.md`: *"Claude Code talks directly to Anthropic's API for LLM access - you can't route its LLM requests through VoidLLM."* VoidLLM can only be added as an MCP server to Claude Code, not as its LLM backend |
|
||||||
|
| Per-workload virtual keys with separate usage views | Yes | Yes — `vl_uk_`/`vl_tk_`/`vl_sa_`/`vl_sk_` key types, org→team→user→key RBAC hierarchy, per-key and per-team usage (`GET /api/v1/usage/me`, `GET /api/v1/orgs/:org_id/usage`) |
|
||||||
|
| Usage/spend dashboard (not logs-only) | Yes — Admin UI `/ui`, Usage tab, free tier | Yes — Web UI with dashboard/usage/keys/playground screens, listed as Community (free) tier |
|
||||||
|
| Custom cost-per-token pricing for local model | Yes — `model_info.input_cost_per_token`/`output_cost_per_token` in `config.yaml` (already wired in this repo's `litellm-config.yaml` against Claude Sonnet 5's published rate) | Yes, equivalent mechanism — per-model `pricing.input_per_1m`/`output_per_1m` in `voidllm.yaml` |
|
||||||
|
| docker-compose self-hostable alongside existing stack | Yes — already running (`litellm` + `litellm-db` services in `docker-compose.yml`) | Yes — documented `docker-compose up` quick start, single Go binary, SQLite by default or Postgres |
|
||||||
|
| Native request queuing/priority | Beta, real but flaky — scheduler with a `priority` field, known bug (leaks into provider request, closed not-planned); needs smoke test (issue #17) | **Not found.** No queuing/priority-scheduling doc page exists in VoidLLM's docs index. The only "priority" concept is a *load-balancing* strategy (which upstream **deployment** to prefer/fail over to) — not request-level queue ordering for concurrent callers hitting one backend. Rate limiting is reject-on-429, not queue-and-wait. Concretely weaker than even LiteLLM's beta scheduler for this repo's actual need (one local GPU, interactive vs. batch tiers) |
|
||||||
|
| Room to add more LLM backends later | Yes — `model_list` of arbitrary provider entries, 100+ providers | Yes — 6 built-in provider types (OpenAI, Anthropic, Azure, Ollama, vLLM, custom-OpenAI-compatible), multi-deployment load balancing/failover per model |
|
||||||
|
| Project health/maturity | MIT, multi-year, widely deployed, 100+ integrations | Real project, ~5 months old, effectively solo-maintained, BSL 1.1, 129 stars |
|
||||||
|
|
||||||
|
Sources for the VoidLLM column: `README.md`, `docs/models/providers.md`,
|
||||||
|
`docs/api/overview.md`, `docs/models/load-balancing.md`, `docs/index.md`
|
||||||
|
(all `raw.githubusercontent.com/voidmind-io/voidllm/main/...`, fetched
|
||||||
|
directly during this evaluation). LiteLLM column sourced from
|
||||||
|
`docs/research/proxy-tool-choice.md` on branch `research/proxy-tool-choice`
|
||||||
|
and this repo's live `litellm-config.yaml` / `docker-compose.yml`.
|
||||||
|
|
||||||
|
## Why this disqualifies VoidLLM here
|
||||||
|
|
||||||
|
Two failures, not one, and they hit the requirements list at its hardest
|
||||||
|
points:
|
||||||
|
|
||||||
|
1. **Coding-CLI routing is a hard requirement this repo already depends on.**
|
||||||
|
Issue #15 migrated Claude Code, Kimi, and OpenCode to route through the
|
||||||
|
proxy (`docs/coding-cli-setup.md`), and issue #9's destination explicitly
|
||||||
|
lists coding CLIs as one of the gateway's fronted consumers. VoidLLM's own
|
||||||
|
docs say plainly that Claude Code's LLM traffic cannot go through it.
|
||||||
|
Even setting Claude Code aside, VoidLLM has no inbound Anthropic
|
||||||
|
Messages-shaped endpoint at all — any Anthropic-format client (present or
|
||||||
|
future) is unsupported, only OpenAI-format inbound is. LiteLLM's
|
||||||
|
`/v1/messages` unified endpoint is a direct, working answer to this same
|
||||||
|
need today.
|
||||||
|
2. **No request-priority/queuing story**, which issue #16 already settled on
|
||||||
|
using LiteLLM's beta scheduler for. VoidLLM has nothing documented in
|
||||||
|
this space beyond reject-on-limit rate limiting and load-balancer
|
||||||
|
deployment ordering. Switching would mean giving up even LiteLLM's shaky
|
||||||
|
beta feature for nothing.
|
||||||
|
|
||||||
|
On top of both dealbreakers, VoidLLM is a much younger, single-maintainer
|
||||||
|
project against an already-integrated, working LiteLLM deployment (#14/#15
|
||||||
|
done, only the scheduler smoke test in #17 outstanding). There's no
|
||||||
|
requirement VoidLLM meets that LiteLLM doesn't already meet as well or
|
||||||
|
better, so there's no upside to weigh against the migration cost and the
|
||||||
|
two outright gaps.
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
**Stick with LiteLLM.** Do not switch. VoidLLM is worth a second look in the
|
||||||
|
future only if it adds an Anthropic-format inbound endpoint (making
|
||||||
|
coding-CLI routing possible) and a real request-queuing/priority mechanism —
|
||||||
|
neither exists today.
|
||||||
Reference in New Issue
Block a user