Bonsai 27B Review: Can a 27B LLM Really Run on a Phone?

Yes, a 27B-class model can now run on a phone, but the useful answer needs more context than the headline. PrismML reports that its 1-bit Bonsai 27B language weights occupy about 3.9 GB and generate roughly 11 tokens per second on an iPhone 17 Pro Max. The model is derived from Qwen3.6-27B and published under Apache 2.0. It is a real on-device milestone, not proof that every 27B workload now fits comfortably inside every phone.

The same evidence also shows where the compression hurts. Across PrismML's own 15-benchmark suite, the 1-bit variant scores 89.5% of the full-precision average. Math and coding retain more of their baseline than instruction following, vision, and multi-step tool use. The quality-first ternary variant retains 94.6%, but its current GGUF deployment is about 7.2 GB, not the 5.9 GB ideal representation often repeated in launch posts.

What is Bonsai 27B?

Bonsai 27B is PrismML's low-bit version of Qwen3.6-27B, a dense multimodal model with a 27B language backbone, a vision encoder, and a native 262,144-token context window. Qwen's architecture uses linear attention in most blocks and full attention in 16 of 64 blocks. That hybrid design limits KV-cache growth compared with a conventional all-attention model.

PrismML publishes two operating points. The 1-bit build uses binary weights from {−1, +1} with one FP16 scale for each group of 128 weights. That produces 1.125 effective bits per weight and about 3.9 GB of resident language weights. The ternary build adds zero, using {−1, 0, +1}. Its representation carries 1.71 effective bits per weight and retains more quality.

This is deeper than taking a normal FP16 checkpoint and exporting a generic 1-bit file. PrismML says low-bit weights extend across embeddings, attention projections, MLP projections, and the language-model head. Custom CUDA, Metal, MLX, and WebGPU kernels consume the packed weights directly instead of expanding them back to FP16 for every operation.

How does a 27B model shrink from 54 GB to 3.9 GB?

The rough FP16 calculation is simple: 27.3 billion language parameters multiplied by two bytes is about 54 GB before runtime overhead. The binary model stores one sign bit per weight plus a 16-bit scale shared across every 128 weights. That works out to 1.125 bits per weight, roughly 14.2 times smaller than FP16.

The headline footprint is not the entire application footprint. PrismML's model card separates four memory layers:

• Resident language weights: about 3.9 GB for 1-bit, or about 7.2 GB for today's ternary GGUF pack.
• Runtime and activations: the published measurements include roughly 1.3 GB across backends.
• KV cache: grows with the active context. With 4-bit KV compression, the full 262K window still brings the 1-bit build to about 9.4 GB peak memory.
• Optional components: the compact vision projection is about 0.63 GB. The speculative-decoding drafter is another optional artifact and is not free memory.

That distinction matters commercially. A model file can fit inside a phone's app budget while its maximum advertised context cannot. Capacity planning must use weights plus context plus runtime, measured on the exact backend and device.

Does Bonsai 27B really retain 90% of Qwen3.6 intelligence?

It retains 89.5% of one vendor-reported benchmark average, not 90% of every capability. PrismML evaluated all variants in thinking mode on the same H100 setup with EvalScope and vLLM. The aggregate is useful because the conditions are matched, but it has not yet been independently reproduced and it hides large category differences.

The per-benchmark results make the risk clearer. On the 1-bit model, MATH-500 moves from 99.40 to 98.00 and BFCL v3 from 77.10 to 70.72. But the harder multi-step τ²-Bench falls from 82.90 to 61.34, IFBench from 68.03 to 52.36, and MMMU-Pro from 79.94 to 60.48. Compression loss lands precisely where a long-running agent can compound small mistakes.

PrismML's own model card says long-horizon, multi-file agentic coding is not yet a strong target for this release. That caveat should carry more weight in a procurement decision than the rounded 90% headline.

Can Bonsai 27B really run locally on a phone and laptop?

Yes, on supported high-end hardware and with the right runtime. PrismML reports about 11 tok/s for the MLX Swift build on an iPhone 17 Pro Max. For the 1-bit GGUF build, its llama-bench results report 26.0 tok/s on an M4 Pro, 44.2 on an M5 Pro, and 66.4 on an M5 Max for 128-token generation. The ternary GGUF reports 18.0, 26.2, and 44.0 tok/s on the same laptop classes.

These figures are not directly interchangeable. Phone, Metal, CUDA, and WebGPU tests use different kernels, context lengths, thermal envelopes, and measurement protocols. Prompt processing and token generation are also different phases. A fast decode rate does not guarantee a short time to first token for a long document.

What about the 90 tokens-per-second WebGPU demo?

Joshua Lochner's Hugging Face Space confirms the important part: Bonsai 27B can execute locally in a browser through custom WGSL compute shaders, and the page says Fable 5 and GPT 5.6 Sol authored and optimized the kernels before correctness tests and shape-specific benchmarks. The model and conversation stay on the device after the weights are fetched.

The widely shared 90 tok/s number is not accompanied by a hardware model, browser version, context, warm-up state, or repeatable benchmark protocol in the public demo source. Treat it as a demo result, not a capacity-planning constant. For a buying decision, record the browser, GPU, power mode, prompt length, output length, first-token latency, sustained decode rate, memory peak, and thermal throttling over repeated runs.

Where can Bonsai 27B create commercial value?

• Private document assistance: summarize, classify, or extract from sensitive local files without sending the content to a model API, provided the entire surrounding application also stays local.
• Offline field software: support technicians, inspectors, or mobile workers in locations with unreliable connectivity.
• On-device product features: make text assistance available with predictable latency and no per-token network charge after the model is deployed.
• Hybrid agent routing: keep private and bounded steps local, then route difficult reasoning or fragile tool workflows to a stronger cloud model.
• Single-machine pilots: evaluate a 27B-class model on laptops or modest GPUs before paying for a permanent inference fleet.

The valuable product is still the system around the model. Wavect's AI enablement service covers use-case selection, evals, data access, routing, deployment, observability, and team handover. Our Twinsoft AI case study shows why production value comes from the workflow and controls around the model, not its parameter count.

Which Bonsai 27B variant should a company choose?

This article intentionally owns the Bonsai-specific deployment decision. For the wider family choice, use our 2026 open-weight LLM comparison. For the financial decision between hardware and APIs, use the local-model versus API break-even calculator. Keeping those intents separate prevents one product review from pretending to answer the whole model market or your infrastructure economics.

What should a Bonsai 27B pilot measure?

1. Freeze the artifact and runtime. Record the Hugging Face revision, weight hash, backend fork, kernel commit, device, OS, and browser or MLX version.
2. Measure the whole memory envelope. Include weights, runtime, expected context, KV-cache format, vision projection, and speculative drafter.
3. Build a task eval. Score at least 50 to 100 representative tasks, including formatting constraints, tool schemas, refusal cases, and unacceptable failures.
4. Separate latency phases. Capture time to first token, prompt processing, sustained decode, and performance after repeated warm runs.
5. Test offline and privacy boundaries. Verify model fetches, analytics, crash reporting, logs, embeddings, and fallback routes. A local model inside a cloud-dependent app is not fully local.
6. Price successful outcomes. Compare device support, engineering, update distribution, monitoring, and fallback costs against a hosted API.
7. Choose a rollback path. Keep a known-good model and a server-side route for unsupported devices, overload, or quality failures.

Our MVP technology selection guide applies the same rule: decide first on the constraint that is expensive to reverse. For on-device AI, that constraint is usually the supported hardware floor and the quality of the weakest critical workflow, not the model download size.