AirLLM on 4 GB VRAM: How Layer-Wise Inference Really Works

Yes, AirLLM can execute a model that is far larger than the GPU's VRAM. It does not make the model fit on the GPU. It keeps the checkpoint on storage, moves one layer or selected experts through memory when needed, computes, releases them and repeats. The project trades memory residency for data movement.

That distinction turns the viral claim into a useful engineering question. A 4 GB peak does not tell you how much disk is required, how long the first split takes, how many seconds one token needs, whether a long context still fits or how many users the system can serve. This review was checked on 20 August 2026 and owns one narrow intent: how AirLLM's low-VRAM inference works and when it is commercially useful. For selecting an ordinary local model that fits your machine, use our local LLM hardware-fit guide.

What is AirLLM?

AirLLM is an Apache-2.0 Python inference library that decomposes supported transformer checkpoints into smaller shards and loads them on demand. The official AirLLM repository reports support for Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, Kimi K3 and other model families through an AutoModel interface. It also offers optional 4-bit or 8-bit block-wise weight compression, CPU execution, Apple Silicon support and limited prefetching.

The project's core benefit is access. A researcher can inspect output from a checkpoint that would otherwise fail at model load because the accelerator cannot hold all weights. The core cost is repeated movement. A normal GPU server keeps weights resident and reuses them for each token and request. AirLLM repeatedly fetches weights from a slower tier because capacity, not throughput, is the primary goal.

How does layer-wise inference reduce VRAM?

1. Download and split the checkpoint. AirLLM creates layer-level files. Its own FAQ warns that this step is disk-intensive and can require enough room for both the original and transformed copy.
2. Keep activations and runtime state. Input tensors, output state, attention data, the KV cache and framework buffers still need CPU or GPU memory.
3. Load the next layer. The runtime transfers the current transformer block to the compute device.
4. Run the block and release it. The activation passes through the layer, then the weight allocation can be reused for the next block.
5. Repeat for every generated token. Autoregressive decode traverses the model again for each new token. Unless weights remain cached, storage traffic returns on every step.

Peak VRAM therefore follows the largest layer plus activations and working buffers, not the sum of all parameters. Total storage still follows the checkpoint. A model can need only a few gigabytes of VRAM while occupying hundreds of gigabytes or more on disk. Long context, large batches and multiple requests also grow runtime memory even when model weights are streamed.

Why can the 2.8T model report less VRAM than a dense 70B model?

Kimi K3 is not a dense 2.8-trillion-parameter model. The official Kimi K3 model repository documents 93 layers, 896 routed experts, 16 selected experts per token and 104 billion activated parameters. AirLLM's K3 path goes below a full transformer layer and streams only routed experts. The working set can therefore be smaller than one dense 70B layer even though the full expert library is vastly larger.

The surprising ordering is plausible because peak memory depends on the largest object that must be resident at once. It does not mean the 2.8T model has fewer weights to download or less work overall. Kimi K3's published checkpoint is roughly 1.6 TB, and Moonshot's recommended production engines are vLLM, SGLang and TokenSpeed. AirLLM is an experimental access path, not the vendor's standard serving recipe.

The viral "no quantization" line needs a correction

AirLLM does not require its own quantization step for every supported model. That is the valid part. Kimi K3, however, was trained and released with MXFP4 weights and MXFP8 activations through quantization-aware training. Calling the K3 example "no quantization" confuses runtime-added compression with the format of the model that was downloaded.

The same care is needed for model size labels. The official DeepSeek-V3 repository lists 671 billion main-model parameters, 37 billion activated per token and an additional 14 billion parameters in its multi-token-prediction module. "671B on 12 GB" is useful shorthand for peak VRAM, but it is not a statement that all 671 billion weights are resident in 12 GB.

What does 4 GB VRAM hide?

Disk capacity

The original checkpoint and AirLLM's transformed shards may coexist during setup. For a 70B full-precision model, that can already mean well over 100 GB. For Kimi K3, the source checkpoint alone is around 1.6 TB. Capacity planning must include downloads, transformed weights, cache, temporary space and rollback.

Storage bandwidth and latency

Every cold layer or expert must cross storage, host memory and possibly PCIe before compute. Sequential NVMe marketing numbers are not enough. The workload includes repeated file access, allocation, deserialization and synchronization. Prefetching can overlap part of that path, but it cannot make slow storage behave like local GPU memory.

Time to first token and decode speed

The AirLLM README publishes exact peak-VRAM numbers but no reproducible tokens-per-second table for its 70B, 405B, DeepSeek-V3 or Kimi K3 headlines. A public repository evidence audit also notes that the linked 405B notebook has no executed timing or memory output and uses a pre-quantized 4-bit model. Treat every speed estimate as unknown until the exact checkpoint, revision, prompt, context, hardware and output length are measured.

Context and concurrency

Layer streaming reduces weight residency. It does not eliminate KV cache, activation or batching costs. A single short prompt that produces five tokens is not evidence for a 32K document, tool-calling agent or five concurrent users. In fact, repeated loading makes concurrency especially important: batching can amortize one layer load across several sequences, while isolated requests pay the movement cost separately.

AirLLM versus a model that fits

Research on low-resource inference reaches the same broader conclusion: storage offload creates capacity, then systems work is needed to recover speed. The PRIMA.CPP paper uses memory mapping, pipelining, prefetching and device-aware layer placement to reduce token latency on 30B to 70B models. That is not an AirLLM benchmark. It is evidence that the difference between "runs" and "runs usefully" is an inference architecture problem, not a VRAM headline.

When is an AirLLM pilot commercially sensible?

• Rare offline access: your team needs occasional output from a model that cannot otherwise load, and minutes of latency may be acceptable.
• Model research: you are checking tokenizer, architecture, quality or compatibility before funding proper serving hardware.
• Large-batch experiments: one layer load can process many sequences, so throughput may matter more than single-user latency.
• Hardware-constrained education: the goal is to understand a model and memory hierarchy, not operate an SLA.

Skip the pilot when a customer expects interactive responses, several users need simultaneous access, the model handles regulated data without a complete security review, or nobody owns storage health, dependency pinning and failure recovery. Also skip it when a 7B to 70B quantized model already passes the same task eval. The winning architecture is the least complex system that meets the product requirement.

A decision-grade benchmark plan

1. Pin the artifact. Record model ID, revision, weight format, AirLLM commit, Python, Torch, Transformers, CUDA and driver versions.
2. Measure all memory tiers. Capture peak VRAM, peak resident RAM, swap, original checkpoint size, transformed shard size and temporary setup space.
3. Separate phases. Report download, split, cold start, prompt prefill, time to first token and decode tokens per second.
4. Test realistic inputs. Use the expected context, output length, tools, languages and structured-output schema.
5. Test batches and concurrency. Compare one request, a useful offline batch and the maximum simultaneous load you promise.
6. Evaluate quality. Compare the exact streamed artifact with a hosted baseline on accepted tasks, not a generic leaderboard.
7. Price the whole system. Include SSD capacity, bandwidth, electricity, engineer time, failures, monitoring and fallback API spend.

Use the local-model versus API break-even framework after the benchmark, not before it. If the project is specifically about Kimi K3 procurement, API terms or data location, the Kimi K3 EU production review owns that decision. For another NVMe expert-streaming architecture with published throughput data, compare our Colibri hardware analysis.

Source boundaries

AirLLM features, compatibility, setup warnings and peak-memory claims come from the project repository. Kimi K3 architecture, native MXFP4 format and recommended serving engines come from Moonshot AI. DeepSeek-V3 parameter counts come from DeepSeek. The evidence gap for the 405B notebook comes from the linked public repository audit. PRIMA.CPP provides independent context on low-resource disk offload, not validation of AirLLM's headline measurements. Wavect did not download terabyte-scale checkpoints or reproduce AirLLM's GPU-memory results for this article.