LLM-Server/docs/research/voidllm-evaluation.md

Evaluation: VoidLLM as a replacement for LiteLLM proxy

Question: Does voidmind-io/voidllm (the user asked us to look at it) beat the already-chosen tool (LiteLLM proxy, see docs/research/proxy-tool-choice.md on branch research/proxy-tool-choice, and issue #10) against this effort's requirements (issue #9)?

Headline: the repo is real and is a genuine, functioning AI gateway/proxy — not a placeholder, not something unrelated to the name. It is young (created March 2026), effectively a one-person project, and it has a documented, explicit gap that disqualifies it for this repo's actual usage pattern: it cannot proxy the coding CLIs' LLM traffic at all.

Answer: stay on LiteLLM. VoidLLM fails one hard requirement outright (coding-CLI routing) and has no equivalent to LiteLLM's priority-queue building block. It matches or is comparable on virtual keys, dashboard, and custom pricing, but that isn't enough to justify a switch, let alone the migration cost of re-doing #12–#15's completed work.

Does the repo exist and is it what it claims to be?

Yes on both counts, confirmed directly against the GitHub API and repo content (not a blog post or secondhand summary):

• Repo: voidmind-io/voidllm, public, not a fork, not archived. Description: "Privacy-first LLM proxy and AI gateway - load balancing, multi-provider routing, API key management, usage tracking, rate limiting. Self-hosted. Zero knowledge of your prompts." Created 2026-03-17, last pushed 2026-08-25 (same week as this evaluation). Language: Go, 129 stars, 15 forks, 28 open issues. Source: https://api.github.com/repos/voidmind-io/voidllm (fetched directly).
• README confirms it is exactly what the description says: a self-hosted proxy sitting in front of OpenAI/Anthropic/Azure/Ollama/vLLM/custom providers, with virtual API keys, RBAC (org/team/user/key), rate limits, token budgets, a web dashboard (usage, keys, playground), and an MCP gateway feature. Source: https://raw.githubusercontent.com/voidmind-io/voidllm/main/README.md.
• It has real release artifacts (Linux/Windows/macOS binaries), a Helm chart, CI/codecov/Go-report-card/OpenSSF-scorecard/Snyk badges, and a documented docs/ tree with real content behind every link checked (providers, load balancing, API reference) — not stub pages. Source: same README; docs/models/providers.md, docs/api/overview.md, docs/models/load-balancing.md, docs/index.md, all fetched from raw.githubusercontent.com/voidmind-io/voidllm/main/.

Maturity/health caveat: this is a small, young project. Contributors per the GitHub API: christianromeni (151 commits — the sole real author), dependabot[bot] (36, automated), and two accounts with 1 commit each (martinsotirov, SAY-5). Effectively a solo maintainer, ~5 months old. The README itself discloses "This project was built with significant assistance from AI (Claude by Anthropic)." None of this makes it fake, but it is a materially less-established project than LiteLLM (widely deployed, 100+ integrations, multi-year history) and carries the usual small-project risks: bus factor, slower security response, less community troubleshooting history. Source: https://api.github.com/repos/voidmind-io/voidllm/contributors.

License: Business Source License 1.1, not OSI open source. Self-hosting for internal/production use is explicitly and unconditionally permitted ("regardless of the number of instances, users, or volume of traffic"); the restriction is only on reselling it as a competing hosted/managed service. Converts to Apache 2.0 four years after each release. Fine for this repo's private homelab use, but a step down from LiteLLM's plain MIT. Source: https://raw.githubusercontent.com/voidmind-io/voidllm/main/LICENSE.

It also has a paid tier structure (Pro €49/mo, Enterprise €149/mo, one-time "Founding Member" €999) gating cross-org analytics, SSO/OIDC, audit logs, OpenTelemetry, and Redis-backed multi-instance state behind payment. The features this evaluation needs (virtual keys, per-key usage, dashboard, custom pricing, docker-compose deploy) are all listed under the free Community tier, so the paywall doesn't block this repo's use case — but it's a different project shape than LiteLLM's free/MIT-with-optional-enterprise- addon model. Source: README "Features" table.

Requirement-by-requirement

Requirement	LiteLLM (current)	VoidLLM
OpenAI-compatible routing	Yes	Yes — /v1/chat/completions, embeddings, images, audio, streaming
Anthropic-compatible / unified Anthropic Messages endpoint	Yes — native /v1/messages unified endpoint accepts Anthropic-format requests, translates to any backend	No. No /v1/messages or any Anthropic-shaped inbound endpoint exists. VoidLLM only accepts OpenAI-format requests and can translate outbound to an Anthropic-format upstream (provider: anthropic in config) — the reverse direction of what's needed
Coding CLIs routed through the proxy	Yes — Claude Code, Kimi, OpenCode all point at LiteLLM today (issue #15, docs/coding-cli-setup.md)	No — explicitly unsupported. docs/models/providers.md: "Claude Code talks directly to Anthropic's API for LLM access - you can't route its LLM requests through VoidLLM." VoidLLM can only be added as an MCP server to Claude Code, not as its LLM backend
Per-workload virtual keys with separate usage views	Yes	Yes — vl_uk_/vl_tk_/vl_sa_/vl_sk_ key types, org→team→user→key RBAC hierarchy, per-key and per-team usage (GET /api/v1/usage/me, GET /api/v1/orgs/:org_id/usage)
Usage/spend dashboard (not logs-only)	Yes — Admin UI /ui, Usage tab, free tier	Yes — Web UI with dashboard/usage/keys/playground screens, listed as Community (free) tier
Custom cost-per-token pricing for local model	Yes — model_info.input_cost_per_token/output_cost_per_token in config.yaml (already wired in this repo's litellm-config.yaml against Claude Sonnet 5's published rate)	Yes, equivalent mechanism — per-model pricing.input_per_1m/output_per_1m in voidllm.yaml
docker-compose self-hostable alongside existing stack	Yes — already running (litellm + litellm-db services in docker-compose.yml)	Yes — documented docker-compose up quick start, single Go binary, SQLite by default or Postgres
Native request queuing/priority	Beta, real but flaky — scheduler with a priority field, known bug (leaks into provider request, closed not-planned); needs smoke test (issue #17)	Not found. No queuing/priority-scheduling doc page exists in VoidLLM's docs index. The only "priority" concept is a load-balancing strategy (which upstream deployment to prefer/fail over to) — not request-level queue ordering for concurrent callers hitting one backend. Rate limiting is reject-on-429, not queue-and-wait. Concretely weaker than even LiteLLM's beta scheduler for this repo's actual need (one local GPU, interactive vs. batch tiers)
Room to add more LLM backends later	Yes — model_list of arbitrary provider entries, 100+ providers	Yes — 6 built-in provider types (OpenAI, Anthropic, Azure, Ollama, vLLM, custom-OpenAI-compatible), multi-deployment load balancing/failover per model
Project health/maturity	MIT, multi-year, widely deployed, 100+ integrations	Real project, ~5 months old, effectively solo-maintained, BSL 1.1, 129 stars

Sources for the VoidLLM column: README.md, docs/models/providers.md, docs/api/overview.md, docs/models/load-balancing.md, docs/index.md (all raw.githubusercontent.com/voidmind-io/voidllm/main/..., fetched directly during this evaluation). LiteLLM column sourced from docs/research/proxy-tool-choice.md on branch research/proxy-tool-choice and this repo's live litellm-config.yaml / docker-compose.yml.

Why this disqualifies VoidLLM here

Two failures, not one, and they hit the requirements list at its hardest points:

1. Coding-CLI routing is a hard requirement this repo already depends on. Issue #15 migrated Claude Code, Kimi, and OpenCode to route through the proxy (docs/coding-cli-setup.md), and issue #9's destination explicitly lists coding CLIs as one of the gateway's fronted consumers. VoidLLM's own docs say plainly that Claude Code's LLM traffic cannot go through it. Even setting Claude Code aside, VoidLLM has no inbound Anthropic Messages-shaped endpoint at all — any Anthropic-format client (present or future) is unsupported, only OpenAI-format inbound is. LiteLLM's /v1/messages unified endpoint is a direct, working answer to this same need today.
2. No request-priority/queuing story, which issue #16 already settled on using LiteLLM's beta scheduler for. VoidLLM has nothing documented in this space beyond reject-on-limit rate limiting and load-balancer deployment ordering. Switching would mean giving up even LiteLLM's shaky beta feature for nothing.

On top of both dealbreakers, VoidLLM is a much younger, single-maintainer project against an already-integrated, working LiteLLM deployment (#14/#15 done, only the scheduler smoke test in #17 outstanding). There's no requirement VoidLLM meets that LiteLLM doesn't already meet as well or better, so there's no upside to weigh against the migration cost and the two outright gaps.

Recommendation

Stick with LiteLLM. Do not switch. VoidLLM is worth a second look in the future only if it adds an Anthropic-format inbound endpoint (making coding-CLI routing possible) and a real request-queuing/priority mechanism — neither exists today.