LLM-Server/docs/research/colibri-ai-stack-evaluation.md

Evaluating Colibrì (JustVugg/colibri) for this stack

Date: 2026-09-08 Scope: The user flagged https://github.com/JustVugg/colibri as something that "could revolutionize" this self-hosted AI stack. What is Colibrì actually, is it compatible with this stack's AMD ROCm/HIP-only single-GPU setup, and — even if compatible — does it fill a real gap versus what llama.cpp, OmniRoute, Qdrant, Neo4j, and ComfyUI already do here?

1. What Colibrì actually is

Colibrì is a pure-C, zero-runtime-dependency inference engine whose specific trick is treating "storage, RAM, and VRAM as a single inference hierarchy" so that huge mixture-of-experts (MoE) models — far bigger than any one machine's VRAM+RAM — can still run, by keeping the small dense layers resident and streaming the (much larger) set of routed experts from disk on demand with an LRU/"hot-store" cache and router-lookahead prefetching:

"Colibrì is an open-source inference engine designed to run frontier mixture-of-experts (MoE) models on consumer hardware... The fundamental approach uses 'a JIT, but for weights' — parameters are staged across storage tiers (VRAM/RAM/NVMe) based on measured routing patterns rather than kept resident." — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md

It ships single-C-file implementations for eight specific model families — GLM-5.2/5.3, GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 (35B-A3B), and OLMoE — each requiring model weights pre-converted into Colibrì's own container format (coli convert), not arbitrary GGUF files:

"Eight model families with single C file implementations... GLM-5.2/5.3 | 744B | 372GB | 16GB+ ... Kimi K3 | 2.8T | 1.6TB | 32GB+" — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md

It's meant to be run either from prebuilt binaries/releases, built from source (./setup.sh under c/), or via Docker (docker/Dockerfile, docker/Dockerfile.slim, docker/docker-compose.yml exist in-repo — confirmed present via the GitHub contents API, https://api.github.com/repos/JustVugg/colibri/contents/docker), exposing an OpenAI- and Anthropic-compatible HTTP API (coli serve, default http://127.0.0.1:8000/v1, plus /v1/messages) — the same shape OmniRoute already expects from a provider, per third-party summaries of docs/api.md and docs/serve_protocol.md (search result summary, secondary).

It launched July 10, 2026 and went viral on Hacker News the same day (453 points) on the strength of running the 744B-parameter GLM-5.2 model on a 25GB-RAM consumer box:

"A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744 billion parameters, on a regular PC with 25GB of memory." — https://gigazine.net/gsc_news/en/20260710-colibri-glm/ (secondary coverage)

2. Hardware/runtime requirements — AMD ROCm or NVIDIA-only?

This is the load-bearing question given this stack runs llama.cpp on ROCm/HIP, not CUDA, on a single AMD Radeon AI PRO R9700. The answer is more nuanced than a flat yes/no — verified against source, not just README prose:

• The engine is CPU-first; a GPU is optional at all. docs/quickstart.md states plainly: "You do not need a GPU. A GPU only helps if you have one; the engine runs CPU-only by default." — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/quickstart.md

• AMD/ROCm support is real, shipped, and reasonably recent — not just a README claim. It was requested in issue #69 (opened 2026-07-11, "No ROCM support"), the maintainer confirmed it was mechanically straightforward since HIP closely mirrors CUDA, an initial PR (#112) added it but was closed unmerged, and a follow-up PR — tracked as #339 — landed the actual mechanism that shipped: a single shared CUDA kernel source (backend_cuda.cu) compiled either by nvcc or by hipcc against a compatibility header:

  "backend_gpu_compat.h — 'one GPU backend source, two vendors.' ... maps CUDA runtime calls to HIP equivalents when compiled by hipcc with HIP=1... handles architecture-specific guards for rocWMMA availability and matrix core support across different GPU architectures (gfx906, gfx908, gfx11xx, etc.)." — https://raw.githubusercontent.com/JustVugg/colibri/main/c/backend_gpu_compat.h (confirmed present in the current main branch — this is not a stale/unmerged branch)

  This shipped in a tagged release, not just an open PR — CHANGELOG.md lists "AMD GPU support" as part of v1.1.0 (2026-07-22): "AMD GPU support, dual-SSD streaming, fmt=5/fmt=6 quantization formats." — https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md

• Notably, the community contributor who tested it used an RX 9070 XT / gfx1201 — the same RDNA4 architecture generation as this stack's Radeon AI PRO R9700 — per the PR #339 description: "validated across CPU builds, HIP testing on gfx1201, and NVIDIA compatibility verification." That's a genuinely favorable, non-generic signal for this specific card's GPU family, better than "AMD support exists somewhere."

• But ROCm support is thinner and less documented than the CUDA/Metal paths. docs/ has cuda.md, metal.md, metal_implementation.md, and vulkan.md, but no rocm.md or hip.md (confirmed via the GitHub contents API listing of docs/, https://api.github.com/repos/JustVugg/colibri/contents/docs). Model-specific tuning docs are written CUDA-only with no AMD mention at all — e.g. docs/qwen36-cuda-tier.md references COLI_CUDA=1, backend_cuda.cu, and lists test hardware as "RTX 3070 8 GB + Quadro RTX 4000" (both NVIDIA), with zero AMD/ROCm/HIP text anywhere in that document. — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/qwen36-cuda-tier.md So: the general GPU-acceleration mechanism supports ROCm/HIP and has been community-validated on hardware close to the R9700, but the per-model tiering/tuning documentation and (presumably) most of the maintainer's own benchmarking is CUDA-first. Treat AMD support as functional-but-secondary, not a first-class, symmetrically-tested backend.

• A separate GPU-agnostic path also exists: a Vulkan backend (backend_vulkan.c, confirmed present in the c/ directory listing) that the project positions as covering "AMD via Mesa/RADV" as a vendor-neutral fallback, independent of the HIP path above — this would also be viable on the R9700 in principle, though vendor-neutral compute back ends are typically slower than a vendor SDK path (HIP/ROCm) and no R9700/gfx1201-specific Vulkan numbers were found in the docs reviewed.

Bottom line on hardware fit: not a blocker. Unlike a hard CUDA-only dependency, Colibrì's AMD/HIP path is real, shipped in a release, and specifically exercised on the same RDNA4 family as this stack's GPU — this is not a disqualifying finding the way it would be for a CUDA-only tool.

3. License

Apache License 2.0, confirmed by fetching LICENSE directly from the repo — a standard permissive license, fine for self-hosted use here (commercial or non-commercial, modification, redistribution all permitted, patent grant included, "AS IS" with no warranty). — https://raw.githubusercontent.com/JustVugg/colibri/main/LICENSE

Model weights are licensed separately from the engine — e.g. the README notes "GLM-5.2 weights released by Z.ai under MIT" (https://raw.githubusercontent.com/JustVugg/colibri/main/README.md) — so each model's own license would need checking before use, same as with any GGUF today.

4. Maturity signals

Pulled from the GitHub API (https://api.github.com/repos/JustVugg/colibri) and the changelog:

Signal	Value
Repo created	2026-07-01
First tagged release (v1.0.0)	2026-07-19
Current version (as of today)	1.10.2 (2026-09-06)
Age at time of writing	~10 weeks
Stars / Forks	27,047 / 2,963
Open issues	104
Top contributor	JustVugg — 1,077 commits
#2 contributor	ZacharyZcR — 163 commits
Total contributors	100+ (long tail, most in single digits)
License	Apache 2.0
Archived?	No

Read honestly, this is a viral, very-early-stage, single-maintainer-dominated project, not a mature or slow-burn one:

• It is ~10 weeks old today. The star count (27k) is wildly disproportionate to that age and reflects a Hacker News front-page moment (453 points the day it launched, https://gigazine.net/gsc_news/en/20260710-colibri-glm/), not organic multi-year adoption. Stars are a popularity signal, not a quality proof, and here the ratio (huge stars, ~10 weeks old, one dominant committer) is itself a maturity red flag worth naming rather than a mark in its favor.
• Commit/release activity is genuinely fast — 14 tagged releases in under 8 weeks (v1.0.0 on 2026-07-19 through v1.10.2 on 2026-09-06, https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md) — so it is actively maintained day-to-day, not abandoned. But that pace also means breaking changes and security patches are frequent: v1.6.2 (2026-08-14) was itself "a security release: six privately-reported memory-safety issues fixed... from untrusted input," and v1.10.2 (2026-09-06) again touched "security fixes for image API" — both signs of a codebase still finding its footing on hardening, not evidence of instability being the norm, but worth weighing given this stack would be exposing any such server on an internal network via OmniRoute.
• Contribution concentration is heavy: the maintainer (JustVugg) has ~6.6x the commits of the next contributor, and the rest of the 100+ contributor list trails off into single-digit-commit drive-by PRs (per the GitHub contributors API, https://api.github.com/repos/JustVugg/colibri/contributors) — a classic "one person's viral project plus a wave of small first-time PRs" shape, not an established multi-maintainer team.
• The AMD/ROCm feature specifically has a short, thin history: requested 2026-07-11, shipped 2026-07-22 (11 days later, in v1.1.0), documented only implicitly (no dedicated docs/hip.md/rocm.md, unlike every other backend) — i.e., it is the newest and least-independently-verified of the project's four GPU backends (CUDA, Metal, Vulkan, HIP).

5. What capability gap it would actually fill in this stack

Concretely comparing against what's already running (docker-compose.yml):

• llama.cpp (ROCm) already fully GPU-resides the current model — --n-gpu-layers 999 on the llama-server service means the whole Qwen3.8-27B-class GGUF sits in the R9700's VRAM and runs at normal, fast, interactive token rates. Colibrì's entire value proposition is the opposite case: models too large to fit in VRAM+RAM at all, accepted at the cost of streaming most of the model from disk on every forward pass. For a model that already fits on this GPU (which is the whole point of the current setup), Colibrì offers no benefit — llama.cpp is already doing the fast thing.

• The actual gap it could fill is running models this stack categorically cannot run today — e.g. GLM-5.2 (744B), DeepSeek V4 Flash (284B), or Kimi K3 (2.8T), none of which would ever fit on a single R9700 regardless of quantization. Colibrì's own benchmarks make the cost of that explicit and, unlike its capability claims, these are the project's self-reported numbers, not independently reproduced — flagged as such:

  "6× RTX 5090 (full residency): 5.8-6.8 tokens/second decode... 128GB CPU-only desktop: ~1.8 tokens/second (warm)... Single RTX 5070 Ti: 1.07 tokens/second... 25GB dev box: 0.05-0.1 tokens/second (baseline)." — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md (self-reported, unverified by any third party found during this research)

  At 0.05–2 tokens/second on any hardware remotely resembling a single-GPU workstation, this is not usable for the interactive coding-CLI workloads this stack is built around (Claude Code CLI, Kimi CLI, etc. routed through OmniRoute per the README) — those need low per-token latency for tool-calling round trips, not throughput measured in seconds per token. It would only be plausible as an occasional, patient, offline/batch capability (e.g. "let a huge model chew on a large doc overnight") layered in alongside, not instead of, the current llama.cpp path.

• Disk footprint is a real new cost, not a marginal one: 167GB–1.6TB per model (https://raw.githubusercontent.com/JustVugg/colibri/main/README.md), which would need to be provisioned in addition to the existing models Docker volume, GGUF downloads, Qdrant/Neo4j volumes, and ComfyUI's model files already on this box.

• No overlap or replacement value for OmniRoute, Qdrant, Neo4j, or ComfyUI — Colibrì is strictly an inference-engine alternative to llama.cpp for a narrow, specific list of very large MoE models; it does nothing related to gateway/key-management (OmniRoute's job), vector/graph storage (Qdrant/Neo4j's job), or image generation (ComfyUI's job). Its own coli serve OpenAI/Anthropic-compatible endpoint could in principle be registered as another OmniRoute provider the same way llama-server is today — that part is mechanically plausible — but it would be adding a second, much slower inference backend next to the existing fast one, not replacing or upgrading anything currently in the stack.

6. Bottom line

Not a fit for this stack right now, and the "revolutionize" framing does not hold up — but for a more specific reason than "wrong GPU vendor":

• ROCm/AMD support is real and not the blocker one might expect. It shipped in a tagged release (v1.1.0, 2026-07-22), lives in a compatibility header confirmed present on main (c/backend_gpu_compat.h), and was community-tested on an RX 9070 XT / gfx1201 — the same RDNA4 family as this stack's R9700. This is genuinely worth noting as a positive, since it's the kind of thing that usually is disqualifying for AMD-only stacks and here it isn't.
• The disqualifying issue is fit, not hardware: Colibrì solves "run a model way too big for your VRAM+RAM by streaming most of it from disk," at 0.05–2 tokens/second. This stack's actual situation is the opposite — a model sized to fully fit and run fast on a single GPU via llama.cpp. Colibrì would add no speed, capability, or reliability benefit to the model already running here, and its own numbers show it isn't fast enough to serve the interactive coding-CLI use case this stack exists for.
• It could only ever be interesting as a bolt-on, offline-only capability for occasionally running an otherwise-impossible frontier-scale model (700B–2.8T params) for patient, non-interactive tasks — at the cost of hundreds of GB to ~1.6TB of extra disk per model, on a ~10-week-old, single-maintainer, still-hardening project (two security-patch releases already) with no ROCm-specific documentation and the thinnest testing history of its four GPU backends.
• Recommendation: worth a passing watch, not worth integrating. Revisit if/when: (a) the project reaches a more established maturity point (6–12 months, broader contributor base, dedicated ROCm docs bringing it to parity with the CUDA/Metal paths), and (b) there's an actual concrete need in this stack to run a model in the 200B+ range that cannot fit on the R9700 — which isn't the case today (the current model is deliberately sized to fit fully in VRAM). Until then, integrating it would add operational surface (a new container, huge disk provisioning, a newer/less-hardened codebase) for no measurable improvement over the existing llama.cpp/ROCm path.