docs(research): evaluate Colibrì for this stack
Assess whether the Colibrì disk-streaming MoE inference engine fits the AMD ROCm single-GPU setup or fills a gap beyond the existing llama.cpp/OmniRoute/Qdrant/Neo4j/ComfyUI stack. Conclusion: worth a passing watch, not worth integrating today.
This commit is contained in:
@@ -0,0 +1,222 @@
|
||||
# Evaluating Colibrì (JustVugg/colibri) for this stack
|
||||
|
||||
**Date:** 2026-09-08
|
||||
**Scope:** The user flagged https://github.com/JustVugg/colibri as something that "could revolutionize"
|
||||
this self-hosted AI stack. What is Colibrì actually, is it compatible with this stack's AMD
|
||||
ROCm/HIP-only single-GPU setup, and — even if compatible — does it fill a real gap versus what
|
||||
llama.cpp, OmniRoute, Qdrant, Neo4j, and ComfyUI already do here?
|
||||
|
||||
## 1. What Colibrì actually is
|
||||
|
||||
Colibrì is a pure-C, zero-runtime-dependency inference engine whose specific trick is treating
|
||||
"storage, RAM, and VRAM as a single inference hierarchy" so that huge mixture-of-experts (MoE)
|
||||
models — far bigger than any one machine's VRAM+RAM — can still run, by keeping the small dense
|
||||
layers resident and streaming the (much larger) set of routed experts from disk on demand with an
|
||||
LRU/"hot-store" cache and router-lookahead prefetching:
|
||||
|
||||
> "Colibrì is an open-source inference engine designed to run frontier mixture-of-experts (MoE)
|
||||
> models on consumer hardware... The fundamental approach uses 'a JIT, but for weights' — parameters
|
||||
> are staged across storage tiers (VRAM/RAM/NVMe) based on measured routing patterns rather than kept
|
||||
> resident."
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
|
||||
|
||||
It ships single-C-file implementations for eight specific model families — GLM-5.2/5.3,
|
||||
GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 (35B-A3B), and OLMoE
|
||||
— each requiring model weights pre-converted into Colibrì's own container format (`coli convert`),
|
||||
not arbitrary GGUF files:
|
||||
|
||||
> "Eight model families with single C file implementations... GLM-5.2/5.3 | 744B | 372GB | 16GB+ ...
|
||||
> Kimi K3 | 2.8T | 1.6TB | 32GB+"
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md
|
||||
|
||||
It's meant to be run either from prebuilt binaries/releases, built from source (`./setup.sh` under
|
||||
`c/`), or via Docker (`docker/Dockerfile`, `docker/Dockerfile.slim`, `docker/docker-compose.yml` exist
|
||||
in-repo — confirmed present via the GitHub contents API, https://api.github.com/repos/JustVugg/colibri/contents/docker),
|
||||
exposing an OpenAI- and Anthropic-compatible HTTP API (`coli serve`, default `http://127.0.0.1:8000/v1`,
|
||||
plus `/v1/messages`) — the same shape OmniRoute already expects from a provider, per third-party
|
||||
summaries of `docs/api.md` and `docs/serve_protocol.md`
|
||||
([search result summary, secondary](https://github.com/JustVugg/colibri/blob/main/docs/api.md)).
|
||||
|
||||
It launched July 10, 2026 and went viral on Hacker News the same day (453 points) on the strength of
|
||||
running the 744B-parameter GLM-5.2 model on a 25GB-RAM consumer box:
|
||||
|
||||
> "A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744
|
||||
> billion parameters, on a regular PC with 25GB of memory."
|
||||
— https://gigazine.net/gsc_news/en/20260710-colibri-glm/ (secondary coverage)
|
||||
|
||||
## 2. Hardware/runtime requirements — AMD ROCm or NVIDIA-only?
|
||||
|
||||
**This is the load-bearing question given this stack runs llama.cpp on ROCm/HIP, not CUDA, on a
|
||||
single AMD Radeon AI PRO R9700.** The answer is more nuanced than a flat yes/no — verified against
|
||||
source, not just README prose:
|
||||
|
||||
- **The engine is CPU-first; a GPU is optional at all.** `docs/quickstart.md` states plainly: "You do
|
||||
**not** need a GPU. A GPU only helps if you have one; the engine runs CPU-only by default."
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/docs/quickstart.md
|
||||
- **AMD/ROCm support is real, shipped, and reasonably recent — not just a README claim.** It was
|
||||
requested in [issue #69](https://github.com/JustVugg/colibri/issues/69) (opened 2026-07-11, "No
|
||||
ROCM support"), the maintainer confirmed it was mechanically straightforward since HIP closely
|
||||
mirrors CUDA, an initial PR (#112) added it but was **closed unmerged**, and a follow-up PR — tracked
|
||||
as [#339](https://github.com/JustVugg/colibri/issues/69) — landed the actual mechanism that shipped:
|
||||
a single shared CUDA kernel source (`backend_cuda.cu`) compiled either by `nvcc` or by `hipcc`
|
||||
against a compatibility header:
|
||||
|
||||
> "backend_gpu_compat.h — 'one GPU backend source, two vendors.' ... maps CUDA runtime calls to HIP
|
||||
> equivalents when compiled by hipcc with `HIP=1`... handles architecture-specific guards for rocWMMA
|
||||
> availability and matrix core support across different GPU architectures (gfx906, gfx908, gfx11xx,
|
||||
> etc.)."
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/c/backend_gpu_compat.h (confirmed present
|
||||
in the current `main` branch — this is not a stale/unmerged branch)
|
||||
|
||||
This shipped in a **tagged release**, not just an open PR — `CHANGELOG.md` lists "AMD GPU support" as
|
||||
part of v1.1.0 (2026-07-22): "AMD GPU support, dual-SSD streaming, fmt=5/fmt=6 quantization formats."
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md
|
||||
- **Notably, the community contributor who tested it used an RX 9070 XT / gfx1201** — the same RDNA4
|
||||
architecture generation as this stack's Radeon AI PRO R9700 — per the PR #339 description: "validated
|
||||
across CPU builds, HIP testing on gfx1201, and NVIDIA compatibility verification." That's a genuinely
|
||||
favorable, non-generic signal for this specific card's GPU family, better than "AMD support exists
|
||||
somewhere."
|
||||
- **But ROCm support is thinner and less documented than the CUDA/Metal paths.** `docs/` has `cuda.md`,
|
||||
`metal.md`, `metal_implementation.md`, and `vulkan.md`, but **no `rocm.md` or `hip.md`** (confirmed via
|
||||
the GitHub contents API listing of `docs/`, https://api.github.com/repos/JustVugg/colibri/contents/docs).
|
||||
Model-specific tuning docs are written CUDA-only with no AMD mention at all — e.g.
|
||||
`docs/qwen36-cuda-tier.md` references `COLI_CUDA=1`, `backend_cuda.cu`, and lists test hardware as
|
||||
"RTX 3070 8 GB + Quadro RTX 4000" (both NVIDIA), with zero AMD/ROCm/HIP text anywhere in that
|
||||
document. — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/qwen36-cuda-tier.md
|
||||
So: the *general* GPU-acceleration mechanism supports ROCm/HIP and has been community-validated on
|
||||
hardware close to the R9700, but the *per-model* tiering/tuning documentation and (presumably) most
|
||||
of the maintainer's own benchmarking is CUDA-first. Treat AMD support as functional-but-secondary,
|
||||
not a first-class, symmetrically-tested backend.
|
||||
- **A separate GPU-agnostic path also exists**: a Vulkan backend (`backend_vulkan.c`, confirmed present
|
||||
in the `c/` directory listing) that the project positions as covering "AMD via Mesa/RADV" as a
|
||||
vendor-neutral fallback, independent of the HIP path above — this would also be viable on the R9700
|
||||
in principle, though vendor-neutral compute back ends are typically slower than a vendor SDK path
|
||||
(HIP/ROCm) and no R9700/gfx1201-specific Vulkan numbers were found in the docs reviewed.
|
||||
|
||||
**Bottom line on hardware fit: not a blocker.** Unlike a hard CUDA-only dependency, Colibrì's AMD/HIP
|
||||
path is real, shipped in a release, and specifically exercised on the same RDNA4 family as this
|
||||
stack's GPU — this is not a disqualifying finding the way it would be for a CUDA-only tool.
|
||||
|
||||
## 3. License
|
||||
|
||||
Apache License 2.0, confirmed by fetching `LICENSE` directly from the repo — a standard permissive
|
||||
license, fine for self-hosted use here (commercial or non-commercial, modification, redistribution all
|
||||
permitted, patent grant included, "AS IS" with no warranty).
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/LICENSE
|
||||
|
||||
Model weights are licensed separately from the engine — e.g. the README notes "GLM-5.2 weights
|
||||
released by Z.ai under MIT"
|
||||
(https://raw.githubusercontent.com/JustVugg/colibri/main/README.md) — so each model's own license
|
||||
would need checking before use, same as with any GGUF today.
|
||||
|
||||
## 4. Maturity signals
|
||||
|
||||
Pulled from the GitHub API (https://api.github.com/repos/JustVugg/colibri) and the changelog:
|
||||
|
||||
| Signal | Value |
|
||||
|---|---|
|
||||
| Repo created | 2026-07-01 |
|
||||
| First tagged release (v1.0.0) | 2026-07-19 |
|
||||
| Current version (as of today) | 1.10.2 (2026-09-06) |
|
||||
| Age at time of writing | ~10 weeks |
|
||||
| Stars / Forks | 27,047 / 2,963 |
|
||||
| Open issues | 104 |
|
||||
| Top contributor | JustVugg — 1,077 commits |
|
||||
| #2 contributor | ZacharyZcR — 163 commits |
|
||||
| Total contributors | 100+ (long tail, most in single digits) |
|
||||
| License | Apache 2.0 |
|
||||
| Archived? | No |
|
||||
|
||||
Read honestly, this is **a viral, very-early-stage, single-maintainer-dominated project**, not a
|
||||
mature or slow-burn one:
|
||||
|
||||
- It is ~10 weeks old today. The star count (27k) is wildly disproportionate to that age and reflects
|
||||
a Hacker News front-page moment (453 points the day it launched,
|
||||
https://gigazine.net/gsc_news/en/20260710-colibri-glm/), not organic multi-year adoption. Stars are
|
||||
a popularity signal, not a quality proof, and here the ratio (huge stars, ~10 weeks old, one dominant
|
||||
committer) is itself a maturity red flag worth naming rather than a mark in its favor.
|
||||
- Commit/release activity is genuinely fast — 14 tagged releases in under 8 weeks (v1.0.0 on 2026-07-19
|
||||
through v1.10.2 on 2026-09-06, https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md)
|
||||
— so it is actively maintained day-to-day, not abandoned. But that pace also means breaking changes
|
||||
and security patches are frequent: v1.6.2 (2026-08-14) was itself "a security release: six
|
||||
privately-reported memory-safety issues fixed... from untrusted input," and v1.10.2 (2026-09-06)
|
||||
again touched "security fixes for image API" — both signs of a codebase still finding its footing on
|
||||
hardening, not evidence of instability being the norm, but worth weighing given this stack would be
|
||||
exposing any such server on an internal network via OmniRoute.
|
||||
- Contribution concentration is heavy: the maintainer (JustVugg) has ~6.6x the commits of the next
|
||||
contributor, and the rest of the 100+ contributor list trails off into single-digit-commit
|
||||
drive-by PRs (per the GitHub contributors API,
|
||||
https://api.github.com/repos/JustVugg/colibri/contributors) — a classic "one person's viral project
|
||||
plus a wave of small first-time PRs" shape, not an established multi-maintainer team.
|
||||
- The AMD/ROCm feature specifically has a short, thin history: requested 2026-07-11, shipped 2026-07-22
|
||||
(11 days later, in v1.1.0), documented only implicitly (no dedicated `docs/hip.md`/`rocm.md`, unlike
|
||||
every other backend) — i.e., it is the newest and least-independently-verified of the project's four
|
||||
GPU backends (CUDA, Metal, Vulkan, HIP).
|
||||
|
||||
## 5. What capability gap it would actually fill in this stack
|
||||
|
||||
Concretely comparing against what's already running (`docker-compose.yml`):
|
||||
|
||||
- **llama.cpp (ROCm) already fully GPU-resides the current model** — `--n-gpu-layers 999` on the
|
||||
llama-server service means the whole Qwen3.8-27B-class GGUF sits in the R9700's VRAM and runs at
|
||||
normal, fast, interactive token rates. Colibrì's entire value proposition is the *opposite* case:
|
||||
models **too large to fit in VRAM+RAM at all**, accepted at the cost of streaming most of the model
|
||||
from disk on every forward pass. For a model that already fits on this GPU (which is the whole point
|
||||
of the current setup), Colibrì offers no benefit — llama.cpp is already doing the fast thing.
|
||||
- **The actual gap it could fill is running models this stack categorically cannot run today** — e.g.
|
||||
GLM-5.2 (744B), DeepSeek V4 Flash (284B), or Kimi K3 (2.8T), none of which would ever fit on a single
|
||||
R9700 regardless of quantization. Colibrì's own benchmarks make the cost of that explicit and, unlike
|
||||
its capability claims, these are the project's self-reported numbers, not independently reproduced —
|
||||
flagged as such:
|
||||
|
||||
> "6× RTX 5090 (full residency): 5.8-6.8 tokens/second decode... 128GB CPU-only desktop: ~1.8
|
||||
> tokens/second (warm)... Single RTX 5070 Ti: 1.07 tokens/second... 25GB dev box: 0.05-0.1
|
||||
> tokens/second (baseline)."
|
||||
— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md (self-reported, unverified by
|
||||
any third party found during this research)
|
||||
|
||||
At 0.05–2 tokens/second on any hardware remotely resembling a single-GPU workstation, this is not
|
||||
usable for the interactive coding-CLI workloads this stack is built around (Claude Code CLI, Kimi
|
||||
CLI, etc. routed through OmniRoute per the README) — those need low per-token latency for tool-calling
|
||||
round trips, not throughput measured in seconds per token. It would only be plausible as an
|
||||
occasional, patient, offline/batch capability (e.g. "let a huge model chew on a large doc overnight")
|
||||
layered in *alongside*, not instead of, the current llama.cpp path.
|
||||
- **Disk footprint is a real new cost, not a marginal one**: 167GB–1.6TB per model
|
||||
(https://raw.githubusercontent.com/JustVugg/colibri/main/README.md), which would need to be
|
||||
provisioned in addition to the existing `models` Docker volume, GGUF downloads, Qdrant/Neo4j
|
||||
volumes, and ComfyUI's model files already on this box.
|
||||
- **No overlap or replacement value for OmniRoute, Qdrant, Neo4j, or ComfyUI** — Colibrì is strictly an
|
||||
inference-engine alternative to llama.cpp for a narrow, specific list of very large MoE models; it
|
||||
does nothing related to gateway/key-management (OmniRoute's job), vector/graph storage (Qdrant/Neo4j's
|
||||
job), or image generation (ComfyUI's job). Its own `coli serve` OpenAI/Anthropic-compatible endpoint
|
||||
could in principle be registered as another OmniRoute provider the same way llama-server is today —
|
||||
that part is mechanically plausible — but it would be adding a second, much slower inference backend
|
||||
next to the existing fast one, not replacing or upgrading anything currently in the stack.
|
||||
|
||||
## 6. Bottom line
|
||||
|
||||
**Not a fit for this stack right now, and the "revolutionize" framing does not hold up** — but for a
|
||||
more specific reason than "wrong GPU vendor":
|
||||
|
||||
- **ROCm/AMD support is real and not the blocker one might expect.** It shipped in a tagged release
|
||||
(v1.1.0, 2026-07-22), lives in a compatibility header confirmed present on `main`
|
||||
(`c/backend_gpu_compat.h`), and was community-tested on an RX 9070 XT / gfx1201 — the same RDNA4
|
||||
family as this stack's R9700. This is genuinely worth noting as a positive, since it's the kind of
|
||||
thing that usually *is* disqualifying for AMD-only stacks and here it isn't.
|
||||
- **The disqualifying issue is fit, not hardware**: Colibrì solves "run a model way too big for your
|
||||
VRAM+RAM by streaming most of it from disk," at 0.05–2 tokens/second. This stack's actual situation is
|
||||
the opposite — a model sized to fully fit and run fast on a single GPU via llama.cpp. Colibrì would add
|
||||
no speed, capability, or reliability benefit to the model already running here, and its own numbers
|
||||
show it isn't fast enough to serve the interactive coding-CLI use case this stack exists for.
|
||||
- It could only ever be interesting as a *bolt-on, offline-only* capability for occasionally running an
|
||||
otherwise-impossible frontier-scale model (700B–2.8T params) for patient, non-interactive tasks — at
|
||||
the cost of hundreds of GB to ~1.6TB of extra disk per model, on a ~10-week-old, single-maintainer,
|
||||
still-hardening project (two security-patch releases already) with no ROCm-specific documentation and
|
||||
the thinnest testing history of its four GPU backends.
|
||||
- **Recommendation: worth a passing watch, not worth integrating.** Revisit if/when: (a) the project
|
||||
reaches a more established maturity point (6–12 months, broader contributor base, dedicated ROCm docs
|
||||
bringing it to parity with the CUDA/Metal paths), and (b) there's an actual concrete need in this stack
|
||||
to run a model in the 200B+ range that cannot fit on the R9700 — which isn't the case today (the
|
||||
current model is deliberately sized to fit fully in VRAM). Until then, integrating it would add
|
||||
operational surface (a new container, huge disk provisioning, a newer/less-hardened codebase) for no
|
||||
measurable improvement over the existing llama.cpp/ROCm path.
|
||||
Reference in New Issue
Block a user