JustVugg / colibri

Tiny engine, immense model. Run frontier MoE models — 744B to 2.8T parameters — on consumer and heterogeneous hardware, in pure C with zero engine dependencies, by treating storage, RAM, and VRAM as a single inference hierarchy (AI memory multitiering). 轻量引擎，巨型模型。通过将存储、内存（RAM）和显存（VRAM）视为单一推理层级（AI 内存多级化），在消费级及异构硬件上运行前沿 MoE 模型（744B 至 2.8T 参数）。该引擎采用纯 C 语言编写，无任何外部依赖。

Nine families run today: GLM-5.2/5.3 (744B), GLM-5.3-Flash (321B, with vision), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), DeepSeek V4.1 Flash (552B, with vision), Qwen3.8-Flash-Next (125B + 51B n-gram), Qwen3.6 (35B-A3B) and OLMoE (7B) — one C file each, the same coli chat / coli serve / coli web front end. Full roster ↓ 目前支持九大模型系列：GLM-5.2/5.3 (744B)、GLM-5.3-Flash (321B，含视觉)、Inkling (975B)、Kimi K3 (2.8T)、DeepSeek V4 Flash (284B)、DeepSeek V4.1 Flash (552B，含视觉)、Qwen3.8-Flash-Next (125B + 51B n-gram)、Qwen3.6 (35B-A3B) 以及 OLMoE (7B)。每个模型仅需一个 C 文件，并统一使用 coli chat / coli serve / coli web 前端。完整列表见下文 ↓

Colibrì is an inference engine you can run today, and an open research platform. Its primary goal is to pursue inference-side performance across the entire software/hardware boundary — model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU overlap — so large models depend less on scarce hardware and cost less to run. Colibrì 既是一个现即可用的推理引擎，也是一个开放的研究平台。其核心目标是在整个软硬件边界上追求推理性能——涵盖模型格式、内存层级、存储 I/O、部署、调度、内核、推测执行以及 CPU/GPU 重叠计算——从而降低大模型对稀缺硬件的依赖，并减少运行成本。

Colibrì treats VRAM, RAM, and storage as a single multitier hierarchy, and it is deliberately a place to test aggressive systems ideas — so there is no SLA on speed, and a hard guarantee on semantics: experiments must earn their place through reproducible end-to-end measurements, and the default policy never silently changes model precision or router semantics. Insufficient fast memory may reduce speed; it must not quietly redefine the model. Colibrì 将 VRAM、RAM 和存储视为单一的多级层级，并特意将其作为测试激进系统构想的试验场。因此，它不对速度提供 SLA（服务等级协议），但对语义有严格保证：所有实验必须通过可复现的端到端测量来证明其价值，且默认策略绝不会在用户不知情的情况下更改模型精度或路由语义。快速内存不足可能会降低速度，但绝不能悄悄地重新定义模型。

See it running

运行演示

The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 — full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown, the VRAM/RAM/disk tier bar and the live mini-brain in the corner. Web 控制面板 (./coli web)：以 4 tok/s 的速度运行 744B 模型，首字延迟 (TTFT) 为 1.6 秒，磁盘占用为 0。模型完整专家参数驻留在 6 张 RTX 5090 上，并实时显示 Token 指标、单轮时间分解、VRAM/RAM/磁盘层级条，以及角落里的实时“迷你大脑”。

The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert’s measured topic affinity. 大脑页面：将全部 19,456 个专家视为一个活跃的皮层——颜色代表存储层级，亮度代表路由热度，每一轮被路由的专家都会闪烁白色。悬停可查看该专家测得的主题亲和力。

The Atlas page: the measured expert atlas as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin. 图谱页面：将测得的专家图谱呈现为 3D 星系——包含 13,260 个特征化专家，1,041 个按主题（诗歌、法律、中文、SQL 等）聚类的复制专家。位置基于测得的路由亲和力，而非学习到的嵌入向量。支持拖拽旋转。

The research mission

研究使命

With Colibrì, private frontier model access is not limited by availability of hyperscaler-class hardware. With its multitiering features Colibrì removes proprietary hardware dependencies aggressively optimizing functional inference engine pipelines. 有了 Colibrì，私有前沿模型的访问不再受限于超大规模算力硬件的可用性。凭借其多级化特性，Colibrì 消除了对专有硬件的依赖，并积极优化功能性推理引擎流水线。

Our operational mission includes changing how weights are represented and moved, deciding what lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing launch and synchronization overhead, exploiting sparsity and reuse, and testing new decoding algorithms. Nothing is protected merely because it is conventional; nothing is adopted merely because a microbenchmark looks fast. The deciding result is end-to-end inference on real machines, with correctness and quality measured alongside throughput, latency, memory, and cost. 我们的运营使命包括：改变权重的表示和移动方式，决定哪些数据驻留在 VRAM、RAM 或存储中，重叠异构计算，减少启动和同步开销，利用稀疏性和重用性，以及测试新的解码算法。没有任何东西仅仅因为“传统”就受到保护；也没有任何东西仅仅因为微基准测试看起来快就被采用。最终的评判标准是真实机器上的端到端推理，并同时衡量正确性、质量、吞吐量、延迟、内存和成本。

The practical consequence is accessibility: run a 744B-parameter model on hardware you already own, watch every expert fire in real time, and change the code that does it. Not renting intelligence behind an API — holding it: probing it, measuring it, improving it. The engine is deliberately small enough that the next useful optimization can come from anyone willing to measure it. 其实际意义在于可访问性：在你现有的硬件上运行 744B 参数模型，实时观察每个专家的触发情况，并修改实现代码。不再是通过 API 租用智能，而是真正掌握它：探测它、测量它、改进它。该引擎被刻意设计得足够小，以便任何愿意进行测量的人都能贡献下一个有用的优化。

Core techniques and measured findings

核心技术与测量发现

One hierarchy, not limited by tier capacity. VRAM, RAM, and NVMe are placement tiers for the same weights; limited fast memory changes speed, not model semantics. 单一层级，不受层级容量限制。 VRAM、RAM 和 NVMe 是同一权重的不同放置层级；有限的快速内存只会改变速度，不会改变模型语义。
A JIT for weights. Measured routing heat drives a per-layer LRU, a learned pinned hot-store, and one-layer-ahead prefetch instead of loading every expert. It wins on repeatable workloads; history can overfit, and lookahead can lose on some hosts, so both remain measurable policies rather than promises. 权重的 JIT（即时编译）。 通过测得的路由热度驱动逐层 LRU、学习型固定热存储以及“超前一层”预取，而非加载所有专家。这在可重复的工作负载中表现优异；历史记录可能导致过拟合，超前预取在某些主机上可能失效，因此两者均作为可测量的策略而非承诺。
I/O is part of the engine. Batched expert unions, overlapped reads and compute, O_DIRECT, and weighted dual-SSD striping attack the streaming path rather than pretending storage latency is free. I/O 是引擎的一部分。 通过批量专家联合、读写与计算重叠、O_DIRECT 以及加权双 SSD 条带化来攻克流式传输路径，而不是假装存储延迟为零。
Heterogeneous execution. CPU, CUDA, Metal, NUMA memory, and partial or full expert residency share one runtime and can be combined according to the machine. 异构执行。 CPU、CUDA、Metal、NUMA 内存以及部分或全部专家驻留共享同一个运行时，并可根据机器配置进行组合。
Compressed state without a different model. Token-exact forward validation, 57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep optimization tied to correctness. 无需更换模型的压缩状态。 Token 精确的前向验证、缩小 57 倍的 MLA KV 状态、持久化热对话以及忠实的 DSA，确保优化始终与正确性挂钩。
Speculation that must earn its keep. Native MTP and grammar-forced drafts are measured end to end and can be disabled when acceptance does not repay verification. 必须证明其价值的推测执行。 原生 MTP 和语法强制草稿均经过端到端测量，当接受率无法抵消验证成本时，可将其禁用。