Colibri Runs GLM-5.2 on Consumer Hardware. Here Is the Catch.

Yes, Colibri can run GLM-5.2 on a machine with roughly 25 GB of RAM. No, that machine does not become a local Claude replacement. The developer's measured cold speed is 0.05 to 0.1 token per second: one generated token every 10 to 20 seconds, before counting prompt processing. A short 100-token answer takes roughly 17 to 33 minutes. A long reasoning trace can take hours.

The achievement is still real. Colibri makes a 744-billion-parameter Mixture-of-Experts model execute on hardware that cannot hold anything close to the full model in RAM or VRAM. It does that by treating NVMe storage, system RAM and optional GPU memory as tiers of one memory hierarchy. The trade is simple and brutal: capacity comes from disk; latency is paid in disk reads.

This is a technical review of the project as it exists on 14 July 2026. Colibri is moving quickly, and its own README mixes proven measurements, community measurements and forward-looking estimates. We keep those categories separate.

What is Colibri?

Colibri, styled Colibrì by the project, is a lightweight inference runtime written mainly in C for running very large sparse MoE models across storage, RAM and optional VRAM. The upstream runtime currently targets GLM-5.2's specific glm_moe_dsa architecture. It also includes an OLMoE engine used for smaller validation and quantization experiments, not a generic frontier-model loader. The project ships a command-line interface, a local web UI, a converter, benchmarking tools, an OpenAI-compatible text API and experimental CPU, CUDA and Metal execution paths.

The engine itself is released under Apache 2.0. GLM-5.2's weights carry an MIT license. That makes the combination commercially permissive, but terminology still matters: GLM-5.2 is best described as open-weight. Public weights and a permissive license do not make its training data and full training process reproducible.

The model is not being compressed into 25 GB. The full int4 checkpoint remains on disk. Colibri keeps the always-used parts in memory and loads the selected experts when the router asks for them. That distinction is the whole project.

Why use GLM-5.2 at all?

GLM-5.2 is interesting because it combines open weights with genuinely high capability. Z.ai's repository describes it as a 744B-A40B model: roughly 744 billion total parameters and 40 billion active per token. Some model indexes display 753B because parameter-counting conventions differ; this article uses the figure Colibri and Z.ai publish in their repositories. The official release reports a one-million-token context window, 81.0 on Terminal-Bench 2.1 and 62.1 on SWE-bench Pro. Those two benchmark numbers are vendor-reported, so they should not be treated as neutral proof.

There is independent support for the broader claim. Artificial Analysis ranked GLM-5.2 first among open-weight models on its Intelligence Index at launch. That does not mean it beats every closed model on every workload. It means “frontier-class open weight” is a defensible category, not just launch copy.

Do not transfer the full-precision or hosted-model benchmark score to Colibri's int4 build. Colibri's first small quality run scored 62.5% mean normalized accuracy across HellaSwag, ARC and MMLU with only 40 questions per task. The project correctly says that result is inconclusive: the evaluation protocol is poor for a reasoning model, the sample is tiny and there is no clean full-precision-versus-int4 A/B yet. Until that experiment exists, Colibri proves execution more strongly than it proves quality retention.

How does Colibri run a 744B model with 25 GB of RAM?

GLM-5.2 is sparse. For each token, its router activates a small subset of its expert feed-forward networks rather than all model parameters. Colibri exploits that sparsity at the storage layer.

1. Keep dense layers resident. Attention, embeddings, shared experts and other always-used weights occupy about 9.9 GB at int4.
2. Leave routed experts on NVMe. The project's current layout contains 21,504 routed expert blocks across the MoE layers and MTP head, each roughly 19 MB at int4.
3. Route per token. GLM-5.2 selects eight routed experts per MoE layer for the current token.
4. Load misses, reuse hits. A per-layer LRU, a learned pinned hot store and the operating system page cache keep repeatedly used experts in RAM. Optional VRAM holds another hot tier.
5. Overlap work where possible. Colibri prefetches and pipelines cold expert reads while resident experts compute, then joins the results before the layer completes.

The cold-read math explains the benchmark. Seventy-five MoE layers multiplied by eight experts and roughly 19 MB per expert is about 11.4 GB of expert data per generated token. On the developer's roughly 1 GB/s storage path, disk alone needs about 11 seconds for a cold token. That implies a ceiling near 0.09 tok/s before CPU work and overhead, which is exactly where the measured 0.05-0.1 tok/s lands.

This is why the project is more interesting than a swap-file trick. It understands the model's router, groups expert reads, preserves the dense state, maintains a learned cache and can place hot experts deliberately. Ordinary virtual memory does not know which expert will be needed next or which blocks are semantically hot.

How fast is Colibri in practice?

The honest answer is “from painfully slow to slow,” unless the machine has enough RAM and VRAM to stop using disk as the normal decode path. The following are measurements from the project's own community table, not benchmarks we reproduced.

These rates start after prompt prefill. Time to first token can be much worse, especially on a cold cache or a long prompt. The six-RTX result is a host-specific capacity experiment, not evidence that one gaming GPU will deliver 6 tok/s. Colibri's own corrected RTX 5090 result is useful here: on one AVX-512 desktop the GPU expert tier initially appeared to help, but a clean rerun showed roughly zero gain because the CPU already matched the GPU on expert matrix multiplication and storage remained the bottleneck.

For comparison, Tom's Hardware called 20-30 tok/s a real-time target. That is a reasonable conversational range, not a universal production requirement. Colibri's strongest published single-request result is still below it on hardware far beyond the 25 GB headline.

What actually controls Colibri performance?

There is no magic “best SSD” answer. The bottleneck moves as hardware improves.

1. RAM capacity and cache hit rate

At 24-32 GB, most memory is consumed by dense weights, KV state, scratch buffers and operating-system headroom. Only a tiny expert cache remains, so even a fast NVMe repeatedly serves cold experts. The project's 24 GB test reached only a 3-4% expert hit rate and 0.07 tok/s despite 2.74 GB/s measured direct-read bandwidth.

With 128 GB, tens of gigabytes can pin experts learned from the workload. That does not load the full model, but it can push hit rates above 60% and remove most disk traffic from repeated prompts. Colibri therefore rewards a consistent workload more than random one-off questions. A cache trained on code tasks may be less useful for a sudden legal-research session because expert popularity changes.

2. Real random-read throughput, not the box number

Consumer SSD marketing leads with large sequential transfers. Colibri issues parallel reads of roughly 19 MB expert blocks scattered across a 370 GB model. The relevant test is the project's iobench pattern with direct I/O on a cold shard, not the drive's maximum sequential headline.

One same-machine comparison is especially useful: moving a Ryzen 9 9950X from a 1.51 GB/s Gen3 QLC drive to an 8.81 GB/s Samsung Gen5 drive improved generation from 0.10 to 0.28 tok/s, not 5.8 times. Disk bandwidth rose 5.8x; token speed rose 2.9x because the bottleneck shifted toward matrix multiplication. Faster storage matters until something else matters more.

3. CPU kernels and memory bandwidth

Once the cache hits, experts come from RAM and the problem becomes quantized matrix multiplication plus memory bandwidth. More cores help only until they contend for the same channels. Colibri's auto-tier planner uses physical cores because simultaneous multithreading can regress memory-bound kernels. AVX-VNNI, AVX-512 and Apple Metal paths can materially change the crossover.

4. Speculation and router shortcuts

GLM-5.2 includes a multi-token-prediction head. With the correct int8 MTP files, the project reports 39-59% draft acceptance and 2.2-2.8 tokens per forward pass. The common pre-converted mirror originally shipped int4 MTP heads that silently produced almost no accepted drafts. More importantly, speculation can be slower on a cold cache because draft verification routes extra experts: the README reports expert loads rising from roughly 660 to 1,100 per token.

The --topp 0.7 router option can reduce disk traffic, and it improved one 24 GB run from 0.07 to 0.11 tok/s. It is also an explicit lossy override. If you reduce the experts the router is allowed to use, benchmark quality again. Do not present a faster altered route as identical model inference.

Will Colibri wear out an SSD?

Normal Colibri inference should not meaningfully consume an SSD's write-endurance rating. The 11 GB-per-token number is reads, not writes. NAND endurance is normally rated in TBW or DWPD because program/erase cycles wear flash. Micron's SSD endurance paper states that NAND wear from reads is negligible compared with writes. Colibri also describes expert streaming as read-only.

That does not mean the drive is immortal or that storage needs no planning.

• Initial setup writes a lot once. Downloading the pre-converted model writes about 370 GB. Converting from the 756 GB FP8 checkpoint streams shards through the drive and writes the final container, so expect well over a terabyte of host writes across download, temporary shards and output. That is visible, but small relative to the hundreds or thousands of TBW specified for many modern drives.
• Swap is the real endurance trap. If the operating system runs out of RAM and starts paging active buffers, the workload becomes write-heavy and performance collapses at the same time. Leave headroom and let Colibri's RAM budget prevent swapping.
• KV persistence writes some data. Colibri stores compressed conversation state at about 182 KB per token and updates usage history. Compared with expert reads this is small, but it is not literally zero writes.
• Sustained reads produce heat. An NVMe controller under continuous load can throttle. Samsung documents dynamic thermal throttling under high transfer workloads and recommends proper airflow. Heat reduces speed before TBW becomes relevant.
• Consumer drives can still fail. Firmware, controllers, power loss and ordinary component failure remain. Keep the original weights elsewhere or be prepared to download them again; do not treat the model drive as your only copy of important data.

The practical setup is a local NVMe with a heatsink or direct airflow, SMART temperature monitoring, swap monitoring and enough free capacity for the model plus temporary files. A dedicated 1 TB drive is more comfortable than squeezing a 370 GB model beside an operating system on a nominal 500 GB SSD. RAID 0 can raise bandwidth but doubles the number of devices whose failure loses the local copy; use it for reproducible model artifacts, not irreplaceable data.

Does a GPU make Colibri fast?

Sometimes. Colibri can place hot experts in NVIDIA VRAM, run dense work on CUDA and use a Metal backend on Apple Silicon. But a GPU does not eliminate the disk-to-host path for experts that do not fit in VRAM. Moving 19 MB blocks through storage, RAM and PCIe can cost more than multiplying them on the CPU.

A single 16-32 GB GPU holds only a small fraction of a 370 GB container. It helps when the pinned set has a high hit rate and CPU expert compute is the bottleneck. It does little when the SSD remains the bottleneck or a modern CPU already executes the quantized expert quickly. The current multi-GPU implementation also uses independent device contexts and synchronous host-staged copies; it has no P2P/NCCL path, no expert sharding and correctness-first custom kernels rather than mature Tensor Core libraries.

The no-BS buying advice: do not buy a GPU for Colibri until profiling shows expert matrix multiplication, not disk or RAM capacity, is your bottleneck. More system RAM can deliver a larger gain by preventing expert reads altogether.

Can Colibri run models other than GLM-5.2?

Not automatically. Upstream Colibri is not a general GGUF runtime like llama.cpp and it cannot point at an arbitrary Qwen, Kimi or DeepSeek checkpoint. Its engine implements GLM-5.2's tokenizer, Multi-head Latent Attention, DeepSeek Sparse Attention, router, expert layout, MTP head and custom quantized container. A model with different attention, routing or tensor names needs code and a converter.

The architecture is portable, but the implementation work is real. A third-party Colibri-Hy3 fork already applies the idea to Tencent's 295B-parameter Hy3: it adds a new C engine for GQA attention, a Hy3-specific router and a converter, then streams a roughly 142 GB int4 container. That is evidence the design can support similar sparse models. It is also evidence that “supports another model” means writing and validating another backend, not changing a model ID.

Dense models get much less benefit because nearly every parameter is needed for every token. MoE models with small always-resident sections, compact attention state and many independently stored experts are the natural fit.

Does Colibri really support a one-million-token context?

GLM-5.2 supports one million tokens according to Z.ai. That does not make one million tokens practical on a 25 GB Colibri machine. The current server defaults to a 4,096-token context, and the project's compressed KV persistence is about 182 KB per token. At one million tokens that alone is roughly 182 GB before model weights, scratch space and expert cache. Sparse attention reduces compute, but it does not make memory and prefill time disappear.

This distinction matters for commercial evaluation. “The model supports 1M context” is a model property. “This machine can serve 1M context within our latency and memory budget” is a system benchmark. Colibri currently proves the first only through the underlying model and a much smaller practical default on constrained hardware.

Is the OpenAI-compatible API production-ready?

The API is useful for integration tests. It implements model discovery, chat completions, legacy completions, server-sent-event streaming, usage counts and basic sampling controls. It can expose GLM-5.2 to existing clients without writing a custom adapter.

It is not equivalent to a production vLLM or SGLang cluster. The first version is text-only. Tools, image and audio input, custom stops, log probabilities and token penalties return errors. Requests queue because the engine executes one sequence at a time; up to 16 isolated KV slots separate conversation state, but they do not create continuous batching. Every 4K slot costs hundreds of megabytes. Authentication is a shared local API key, not a full tenancy or policy layer.

For a personal lab, that is fine. For a customer-facing system, add load tests, isolation, rate limits, observability, security review, failure recovery, model checksums, update policy, eval gates and an API fallback. Our local-model versus API break-even framework covers the operating-cost side; Colibri's single-sequence throughput makes that economic bar particularly hard to clear today.

Who should use Colibri today?

Colibri is most commercially useful today as a feasibility instrument. It lets a team answer questions that would otherwise require renting a large GPU node: does GLM-5.2 handle our private workload, how much quality does this int4 path retain, which experts become hot, and what hardware tier removes the actual bottleneck? That evidence can inform a later API, managed-endpoint or proper self-hosted deployment decision.

How should you test Colibri before buying hardware?

1. Start with the exact workload. Build 30-100 representative prompts and an acceptance rubric. Public model benchmarks do not decide whether the int4 runtime works for you.
2. Use the pre-converted checkpoint with int8 MTP heads. The repository documents exact file sizes because the older int4 MTP mirror silently disables the useful speculation path.
3. Run coli doctor and coli plan. Confirm model files, platform support, RAM projection, requested devices and the expected storage/RAM/VRAM placement before a long generation.
4. Benchmark cold storage correctly. Use the included I/O benchmark with direct I/O on an uncached shard. Do not copy a sequential SSD spec into a performance estimate.
5. Record four numbers per task. Time to first token, decode tok/s, expert hit rate and peak RSS. Add output quality and total task time; token speed alone can reward short or wrong answers.
6. Warm the real workload. Let the learned pin stabilize, then repeat the same eval. Report cold and warm results separately.
7. Change one tier at a time. More RAM, faster NVMe, GPU expert budget and CPU flags should be separate A/B tests. Otherwise you will not know what fixed the bottleneck.
8. Price the alternative. Compare the full machine, electricity and engineer time with a hosted GLM-5.2 endpoint for the same successful tasks. “Zero marginal token cost” is meaningless if one machine completes only a handful of tasks per day.

If the experiment advances beyond a lab, use an eval gate and a fallback route. The same discipline appears in our Twinsoft AI case study and our technology-selection guide: pick the architecture from measured constraints, not from the novelty of the runtime.

Sources and methodology

Runtime architecture, requirements, feature status and performance figures come from the Colibri repository and its linked issue measurements. Model architecture, license, context and launch benchmarks come from Z.ai's GLM-5 repository and official GLM-5.2 model card. We use Artificial Analysis only for an independent capability signal, not for Colibri speed. SSD endurance and thermal claims are grounded in Micron and Samsung documentation linked above.

We converted token rates into 100-token decode times with time = 100 / tokens_per_second. Those figures exclude prompt prefill, queue time and reasoning hidden before the final answer. We label repository estimates as estimates and exclude them from the measured-speed table. Facts were checked on 14 July 2026; this project is changing quickly, so verify the current README and open issues before buying parts.