diff --git a/docs/research/colibri-ai-stack-evaluation.md b/docs/research/colibri-ai-stack-evaluation.md new file mode 100644 index 0000000..7d68530 --- /dev/null +++ b/docs/research/colibri-ai-stack-evaluation.md @@ -0,0 +1,222 @@ +# Evaluating Colibrì (JustVugg/colibri) for this stack + +**Date:** 2026-09-08 +**Scope:** The user flagged https://github.com/JustVugg/colibri as something that "could revolutionize" +this self-hosted AI stack. What is Colibrì actually, is it compatible with this stack's AMD +ROCm/HIP-only single-GPU setup, and — even if compatible — does it fill a real gap versus what +llama.cpp, OmniRoute, Qdrant, Neo4j, and ComfyUI already do here? + +## 1. What Colibrì actually is + +Colibrì is a pure-C, zero-runtime-dependency inference engine whose specific trick is treating +"storage, RAM, and VRAM as a single inference hierarchy" so that huge mixture-of-experts (MoE) +models — far bigger than any one machine's VRAM+RAM — can still run, by keeping the small dense +layers resident and streaming the (much larger) set of routed experts from disk on demand with an +LRU/"hot-store" cache and router-lookahead prefetching: + +> "Colibrì is an open-source inference engine designed to run frontier mixture-of-experts (MoE) +> models on consumer hardware... The fundamental approach uses 'a JIT, but for weights' — parameters +> are staged across storage tiers (VRAM/RAM/NVMe) based on measured routing patterns rather than kept +> resident." +— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md + +It ships single-C-file implementations for eight specific model families — GLM-5.2/5.3, +GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash, Qwen3.8-Flash-Next, Qwen3.6 (35B-A3B), and OLMoE +— each requiring model weights pre-converted into Colibrì's own container format (`coli convert`), +not arbitrary GGUF files: + +> "Eight model families with single C file implementations... GLM-5.2/5.3 | 744B | 372GB | 16GB+ ... +> Kimi K3 | 2.8T | 1.6TB | 32GB+" +— https://raw.githubusercontent.com/JustVugg/colibri/main/README.md + +It's meant to be run either from prebuilt binaries/releases, built from source (`./setup.sh` under +`c/`), or via Docker (`docker/Dockerfile`, `docker/Dockerfile.slim`, `docker/docker-compose.yml` exist +in-repo — confirmed present via the GitHub contents API, https://api.github.com/repos/JustVugg/colibri/contents/docker), +exposing an OpenAI- and Anthropic-compatible HTTP API (`coli serve`, default `http://127.0.0.1:8000/v1`, +plus `/v1/messages`) — the same shape OmniRoute already expects from a provider, per third-party +summaries of `docs/api.md` and `docs/serve_protocol.md` +([search result summary, secondary](https://github.com/JustVugg/colibri/blob/main/docs/api.md)). + +It launched July 10, 2026 and went viral on Hacker News the same day (453 points) on the strength of +running the 744B-parameter GLM-5.2 model on a 25GB-RAM consumer box: + +> "A new inference engine called 'Colibrì' has emerged that can run the massive AI 'GLM-5.2,' with 744 +> billion parameters, on a regular PC with 25GB of memory." +— https://gigazine.net/gsc_news/en/20260710-colibri-glm/ (secondary coverage) + +## 2. Hardware/runtime requirements — AMD ROCm or NVIDIA-only? + +**This is the load-bearing question given this stack runs llama.cpp on ROCm/HIP, not CUDA, on a +single AMD Radeon AI PRO R9700.** The answer is more nuanced than a flat yes/no — verified against +source, not just README prose: + +- **The engine is CPU-first; a GPU is optional at all.** `docs/quickstart.md` states plainly: "You do + **not** need a GPU. A GPU only helps if you have one; the engine runs CPU-only by default." + — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/quickstart.md +- **AMD/ROCm support is real, shipped, and reasonably recent — not just a README claim.** It was + requested in [issue #69](https://github.com/JustVugg/colibri/issues/69) (opened 2026-07-11, "No + ROCM support"), the maintainer confirmed it was mechanically straightforward since HIP closely + mirrors CUDA, an initial PR (#112) added it but was **closed unmerged**, and a follow-up PR — tracked + as [#339](https://github.com/JustVugg/colibri/issues/69) — landed the actual mechanism that shipped: + a single shared CUDA kernel source (`backend_cuda.cu`) compiled either by `nvcc` or by `hipcc` + against a compatibility header: + + > "backend_gpu_compat.h — 'one GPU backend source, two vendors.' ... maps CUDA runtime calls to HIP + > equivalents when compiled by hipcc with `HIP=1`... handles architecture-specific guards for rocWMMA + > availability and matrix core support across different GPU architectures (gfx906, gfx908, gfx11xx, + > etc.)." + — https://raw.githubusercontent.com/JustVugg/colibri/main/c/backend_gpu_compat.h (confirmed present + in the current `main` branch — this is not a stale/unmerged branch) + + This shipped in a **tagged release**, not just an open PR — `CHANGELOG.md` lists "AMD GPU support" as + part of v1.1.0 (2026-07-22): "AMD GPU support, dual-SSD streaming, fmt=5/fmt=6 quantization formats." + — https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md +- **Notably, the community contributor who tested it used an RX 9070 XT / gfx1201** — the same RDNA4 + architecture generation as this stack's Radeon AI PRO R9700 — per the PR #339 description: "validated + across CPU builds, HIP testing on gfx1201, and NVIDIA compatibility verification." That's a genuinely + favorable, non-generic signal for this specific card's GPU family, better than "AMD support exists + somewhere." +- **But ROCm support is thinner and less documented than the CUDA/Metal paths.** `docs/` has `cuda.md`, + `metal.md`, `metal_implementation.md`, and `vulkan.md`, but **no `rocm.md` or `hip.md`** (confirmed via + the GitHub contents API listing of `docs/`, https://api.github.com/repos/JustVugg/colibri/contents/docs). + Model-specific tuning docs are written CUDA-only with no AMD mention at all — e.g. + `docs/qwen36-cuda-tier.md` references `COLI_CUDA=1`, `backend_cuda.cu`, and lists test hardware as + "RTX 3070 8 GB + Quadro RTX 4000" (both NVIDIA), with zero AMD/ROCm/HIP text anywhere in that + document. — https://raw.githubusercontent.com/JustVugg/colibri/main/docs/qwen36-cuda-tier.md + So: the *general* GPU-acceleration mechanism supports ROCm/HIP and has been community-validated on + hardware close to the R9700, but the *per-model* tiering/tuning documentation and (presumably) most + of the maintainer's own benchmarking is CUDA-first. Treat AMD support as functional-but-secondary, + not a first-class, symmetrically-tested backend. +- **A separate GPU-agnostic path also exists**: a Vulkan backend (`backend_vulkan.c`, confirmed present + in the `c/` directory listing) that the project positions as covering "AMD via Mesa/RADV" as a + vendor-neutral fallback, independent of the HIP path above — this would also be viable on the R9700 + in principle, though vendor-neutral compute back ends are typically slower than a vendor SDK path + (HIP/ROCm) and no R9700/gfx1201-specific Vulkan numbers were found in the docs reviewed. + +**Bottom line on hardware fit: not a blocker.** Unlike a hard CUDA-only dependency, Colibrì's AMD/HIP +path is real, shipped in a release, and specifically exercised on the same RDNA4 family as this +stack's GPU — this is not a disqualifying finding the way it would be for a CUDA-only tool. + +## 3. License + +Apache License 2.0, confirmed by fetching `LICENSE` directly from the repo — a standard permissive +license, fine for self-hosted use here (commercial or non-commercial, modification, redistribution all +permitted, patent grant included, "AS IS" with no warranty). +— https://raw.githubusercontent.com/JustVugg/colibri/main/LICENSE + +Model weights are licensed separately from the engine — e.g. the README notes "GLM-5.2 weights +released by Z.ai under MIT" +(https://raw.githubusercontent.com/JustVugg/colibri/main/README.md) — so each model's own license +would need checking before use, same as with any GGUF today. + +## 4. Maturity signals + +Pulled from the GitHub API (https://api.github.com/repos/JustVugg/colibri) and the changelog: + +| Signal | Value | +|---|---| +| Repo created | 2026-07-01 | +| First tagged release (v1.0.0) | 2026-07-19 | +| Current version (as of today) | 1.10.2 (2026-09-06) | +| Age at time of writing | ~10 weeks | +| Stars / Forks | 27,047 / 2,963 | +| Open issues | 104 | +| Top contributor | JustVugg — 1,077 commits | +| #2 contributor | ZacharyZcR — 163 commits | +| Total contributors | 100+ (long tail, most in single digits) | +| License | Apache 2.0 | +| Archived? | No | + +Read honestly, this is **a viral, very-early-stage, single-maintainer-dominated project**, not a +mature or slow-burn one: + +- It is ~10 weeks old today. The star count (27k) is wildly disproportionate to that age and reflects + a Hacker News front-page moment (453 points the day it launched, + https://gigazine.net/gsc_news/en/20260710-colibri-glm/), not organic multi-year adoption. Stars are + a popularity signal, not a quality proof, and here the ratio (huge stars, ~10 weeks old, one dominant + committer) is itself a maturity red flag worth naming rather than a mark in its favor. +- Commit/release activity is genuinely fast — 14 tagged releases in under 8 weeks (v1.0.0 on 2026-07-19 + through v1.10.2 on 2026-09-06, https://raw.githubusercontent.com/JustVugg/colibri/main/CHANGELOG.md) + — so it is actively maintained day-to-day, not abandoned. But that pace also means breaking changes + and security patches are frequent: v1.6.2 (2026-08-14) was itself "a security release: six + privately-reported memory-safety issues fixed... from untrusted input," and v1.10.2 (2026-09-06) + again touched "security fixes for image API" — both signs of a codebase still finding its footing on + hardening, not evidence of instability being the norm, but worth weighing given this stack would be + exposing any such server on an internal network via OmniRoute. +- Contribution concentration is heavy: the maintainer (JustVugg) has ~6.6x the commits of the next + contributor, and the rest of the 100+ contributor list trails off into single-digit-commit + drive-by PRs (per the GitHub contributors API, + https://api.github.com/repos/JustVugg/colibri/contributors) — a classic "one person's viral project + plus a wave of small first-time PRs" shape, not an established multi-maintainer team. +- The AMD/ROCm feature specifically has a short, thin history: requested 2026-07-11, shipped 2026-07-22 + (11 days later, in v1.1.0), documented only implicitly (no dedicated `docs/hip.md`/`rocm.md`, unlike + every other backend) — i.e., it is the newest and least-independently-verified of the project's four + GPU backends (CUDA, Metal, Vulkan, HIP). + +## 5. What capability gap it would actually fill in this stack + +Concretely comparing against what's already running (`docker-compose.yml`): + +- **llama.cpp (ROCm) already fully GPU-resides the current model** — `--n-gpu-layers 999` on the + llama-server service means the whole Qwen3.8-27B-class GGUF sits in the R9700's VRAM and runs at + normal, fast, interactive token rates. Colibrì's entire value proposition is the *opposite* case: + models **too large to fit in VRAM+RAM at all**, accepted at the cost of streaming most of the model + from disk on every forward pass. For a model that already fits on this GPU (which is the whole point + of the current setup), Colibrì offers no benefit — llama.cpp is already doing the fast thing. +- **The actual gap it could fill is running models this stack categorically cannot run today** — e.g. + GLM-5.2 (744B), DeepSeek V4 Flash (284B), or Kimi K3 (2.8T), none of which would ever fit on a single + R9700 regardless of quantization. Colibrì's own benchmarks make the cost of that explicit and, unlike + its capability claims, these are the project's self-reported numbers, not independently reproduced — + flagged as such: + + > "6× RTX 5090 (full residency): 5.8-6.8 tokens/second decode... 128GB CPU-only desktop: ~1.8 + > tokens/second (warm)... Single RTX 5070 Ti: 1.07 tokens/second... 25GB dev box: 0.05-0.1 + > tokens/second (baseline)." + — https://raw.githubusercontent.com/JustVugg/colibri/main/README.md (self-reported, unverified by + any third party found during this research) + + At 0.05–2 tokens/second on any hardware remotely resembling a single-GPU workstation, this is not + usable for the interactive coding-CLI workloads this stack is built around (Claude Code CLI, Kimi + CLI, etc. routed through OmniRoute per the README) — those need low per-token latency for tool-calling + round trips, not throughput measured in seconds per token. It would only be plausible as an + occasional, patient, offline/batch capability (e.g. "let a huge model chew on a large doc overnight") + layered in *alongside*, not instead of, the current llama.cpp path. +- **Disk footprint is a real new cost, not a marginal one**: 167GB–1.6TB per model + (https://raw.githubusercontent.com/JustVugg/colibri/main/README.md), which would need to be + provisioned in addition to the existing `models` Docker volume, GGUF downloads, Qdrant/Neo4j + volumes, and ComfyUI's model files already on this box. +- **No overlap or replacement value for OmniRoute, Qdrant, Neo4j, or ComfyUI** — Colibrì is strictly an + inference-engine alternative to llama.cpp for a narrow, specific list of very large MoE models; it + does nothing related to gateway/key-management (OmniRoute's job), vector/graph storage (Qdrant/Neo4j's + job), or image generation (ComfyUI's job). Its own `coli serve` OpenAI/Anthropic-compatible endpoint + could in principle be registered as another OmniRoute provider the same way llama-server is today — + that part is mechanically plausible — but it would be adding a second, much slower inference backend + next to the existing fast one, not replacing or upgrading anything currently in the stack. + +## 6. Bottom line + +**Not a fit for this stack right now, and the "revolutionize" framing does not hold up** — but for a +more specific reason than "wrong GPU vendor": + +- **ROCm/AMD support is real and not the blocker one might expect.** It shipped in a tagged release + (v1.1.0, 2026-07-22), lives in a compatibility header confirmed present on `main` + (`c/backend_gpu_compat.h`), and was community-tested on an RX 9070 XT / gfx1201 — the same RDNA4 + family as this stack's R9700. This is genuinely worth noting as a positive, since it's the kind of + thing that usually *is* disqualifying for AMD-only stacks and here it isn't. +- **The disqualifying issue is fit, not hardware**: Colibrì solves "run a model way too big for your + VRAM+RAM by streaming most of it from disk," at 0.05–2 tokens/second. This stack's actual situation is + the opposite — a model sized to fully fit and run fast on a single GPU via llama.cpp. Colibrì would add + no speed, capability, or reliability benefit to the model already running here, and its own numbers + show it isn't fast enough to serve the interactive coding-CLI use case this stack exists for. +- It could only ever be interesting as a *bolt-on, offline-only* capability for occasionally running an + otherwise-impossible frontier-scale model (700B–2.8T params) for patient, non-interactive tasks — at + the cost of hundreds of GB to ~1.6TB of extra disk per model, on a ~10-week-old, single-maintainer, + still-hardening project (two security-patch releases already) with no ROCm-specific documentation and + the thinnest testing history of its four GPU backends. +- **Recommendation: worth a passing watch, not worth integrating.** Revisit if/when: (a) the project + reaches a more established maturity point (6–12 months, broader contributor base, dedicated ROCm docs + bringing it to parity with the CUDA/Metal paths), and (b) there's an actual concrete need in this stack + to run a model in the 200B+ range that cannot fit on the R9700 — which isn't the case today (the + current model is deliberately sized to fit fully in VRAM). Until then, integrating it would add + operational surface (a new container, huge disk provisioning, a newer/less-hardened codebase) for no + measurable improvement over the existing llama.cpp/ROCm path.