Colibrì proof-of-concept runs 1.5TB frontier model on 25GB RAM; hints at consumer-scale inference

An Italian engineer known as JustVugg (Vincenzo) created Colibrì, a proof-of-concept that runs the 744-billion-parameter GLM-5.2 model (a Mixture-of-Experts architecture) on modest consumer hardware: a standard CPU, 25 GB of RAM, and a 1 GB/s NVMe virtual drive. While inference speed is presently only 0.05–0.1 tokens per second (impractical for real-time use), the project demonstrates that frontier-level capability—comparable to offerings from Anthropic, OpenAI, and others—can theoretically be accessed on resource-constrained machines through clever architectural exploitation.

Colibrì works by loading model slices to RAM on demand. GLM-5.2's Mixture-of-Experts design activates only the subset of expert sub-models needed per token, not per query. By repeatedly loading and unloading the relevant experts, Colibrì trades compute and speed for memory footprint. Quantization (lossy compression) further reduces the model size. The current bottleneck is NVMe storage I/O and memory bandwidth; adding more storage speed moves the constraint to RAM, then CPU cores, and so on. The single-C-file implementation prioritizes simplicity and minimal dependencies.

Colibrì is still a proof-of-concept and does not yet run on GPUs; even GPU versions would face data-movement bottlenecks. However, the project has gained rapid adoption in its early weeks, and Vincenzo is collecting benchmark data to improve performance and address architectural issues. For now, throughput (0.05–0.1 tok/s) falls far short of the 20–30 tok/s needed for real-time conversational AI, but the research direction is clear.

For ML ops teams: Colibrì exemplifies the emerging 'local AI' design pattern—trading latency for accessibility and privacy. While impractical for production inference today, this work signals that frontier model capability may soon be runnable on edge devices or modest local machines, opening a potential new distribution model for open-weight models. Key takeaway: Mixture-of-Experts and quantization are enabling much lower-hardware inference, a trend to watch as consumer GPU and edge-compute deployments scale.