--- title: "AirLLM on 4 GB VRAM: Layer-Wise Inference" canonical: https://wavect.io/blog/airllm-layer-wise-inference-low-vram/ language: en description: "Can AirLLM run 70B to 2.8T models on 4 GB VRAM? See how layer-wise inference works, what storage and speed cost, and when a pilot makes sense." image: "https://wavect.io/img/blog/headers/header_airllm-layer-wise-inference-low-vram.png" --- [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 10 min read · 20 Aug 2026 Last reviewed August 20, 2026 [**Next**](/blog/moneyprinterturbo-review-2026/) # AirLLM on 4 GB VRAM: How Layer-Wise Inference Really Works TL;DR AirLLM can reduce peak GPU memory by loading only the layer, or for supported sparse models the routed experts, needed at that moment. Its repository reports 70B on about 4 GB VRAM, Llama 3.1 405B on about 8 GB, DeepSeek-V3 on about 12 GB and Kimi K3 on 3.72 GB. Those are execution and peak-memory claims, not production throughput guarantees. The full checkpoint still needs storage, every generated token can trigger repeated weight transfers, context and runtime buffers still consume memory, and the project publishes no reproducible tokens-per-second result for those headline models. Kimi K3 also ships with native MXFP4 weights, so "no quantization" does not accurately describe that example. Use AirLLM for research, offline evaluation and model-access experiments. For a customer-facing service, choose only after measuring time to first token, decode speed, disk footprint, concurrency, quality and total cost against a smaller quantized model or API. **Yes, AirLLM can execute a model that is far larger than the GPU's VRAM.** It does not make the model fit on the GPU. It keeps the checkpoint on storage, moves one layer or selected experts through memory when needed, computes, releases them and repeats. The project trades memory residency for data movement. That distinction turns the viral claim into a useful engineering question. A 4 GB peak does not tell you how much disk is required, how long the first split takes, how many seconds one token needs, whether a long context still fits or how many users the system can serve. This review was checked on **20 August 2026** and owns one narrow intent: how AirLLM's low-VRAM inference works and when it is commercially useful. For selecting an ordinary local model that fits your machine, use our [local LLM hardware-fit guide](/blog/llmfit-local-llm-hardware-guide/). | Claim | What the available evidence supports | What it does not prove | | --- | --- | --- | | 70B on 4 GB VRAM | The runtime keeps one layer resident at a time | Interactive or production speed | | Llama 3.1 405B on 8 GB | The project publishes a code path and notebook | A reproducible full-precision benchmark | | DeepSeek-V3 671B on about 12 GB | Current AirLLM support and a peak-memory claim | Latency, concurrency or service reliability | | Kimi K3 2.8T on 3.72 GB | Maintainer-reported end-to-end peak on one RTX 6000 Ada | That the 1.6 TB checkpoint fits in VRAM or storage disappears | | No quantization required | AirLLM can stream some full-precision checkpoints without adding compression | Kimi K3 is unquantized; its released weights are natively MXFP4 | ## What is AirLLM? AirLLM is an Apache-2.0 Python inference library that decomposes supported transformer checkpoints into smaller shards and loads them on demand. The [official AirLLM repository](https://github.com/lyogavin/airllm) reports support for Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, Kimi K3 and other model families through an `AutoModel` interface. It also offers optional 4-bit or 8-bit block-wise weight compression, CPU execution, Apple Silicon support and limited prefetching. The project's core benefit is access. A researcher can inspect output from a checkpoint that would otherwise fail at model load because the accelerator cannot hold all weights. The core cost is repeated movement. A normal GPU server keeps weights resident and reuses them for each token and request. AirLLM repeatedly fetches weights from a slower tier because capacity, not throughput, is the primary goal. ## How does layer-wise inference reduce VRAM? 1. **Download and split the checkpoint.** AirLLM creates layer-level files. Its own FAQ warns that this step is disk-intensive and can require enough room for both the original and transformed copy. 2. **Keep activations and runtime state.** Input tensors, output state, attention data, the KV cache and framework buffers still need CPU or GPU memory. 3. **Load the next layer.** The runtime transfers the current transformer block to the compute device. 4. **Run the block and release it.** The activation passes through the layer, then the weight allocation can be reused for the next block. 5. **Repeat for every generated token.** Autoregressive decode traverses the model again for each new token. Unless weights remain cached, storage traffic returns on every step. Peak VRAM therefore follows the largest layer plus activations and working buffers, not the sum of all parameters. Total storage still follows the checkpoint. A model can need only a few gigabytes of VRAM while occupying hundreds of gigabytes or more on disk. Long context, large batches and multiple requests also grow runtime memory even when model weights are streamed. ## Why can the 2.8T model report less VRAM than a dense 70B model? Kimi K3 is not a dense 2.8-trillion-parameter model. The [official Kimi K3 model repository](https://github.com/MoonshotAI/Kimi-K3) documents 93 layers, 896 routed experts, 16 selected experts per token and 104 billion activated parameters. AirLLM's K3 path goes below a full transformer layer and streams only routed experts. The working set can therefore be smaller than one dense 70B layer even though the full expert library is vastly larger. The surprising ordering is plausible because peak memory depends on the largest object that must be resident at once. It does not mean the 2.8T model has fewer weights to download or less work overall. Kimi K3's published checkpoint is roughly 1.6 TB, and Moonshot's recommended production engines are vLLM, SGLang and TokenSpeed. AirLLM is an experimental access path, not the vendor's standard serving recipe. ## The viral "no quantization" line needs a correction AirLLM does not require its own quantization step for every supported model. That is the valid part. Kimi K3, however, was trained and released with MXFP4 weights and MXFP8 activations through quantization-aware training. Calling the K3 example "no quantization" confuses runtime-added compression with the format of the model that was downloaded. The same care is needed for model size labels. The [official DeepSeek-V3 repository](https://github.com/deepseek-ai/DeepSeek-V3) lists 671 billion main-model parameters, 37 billion activated per token and an additional 14 billion parameters in its multi-token-prediction module. "671B on 12 GB" is useful shorthand for peak VRAM, but it is not a statement that all 671 billion weights are resident in 12 GB. ## What does 4 GB VRAM hide? ### Disk capacity The original checkpoint and AirLLM's transformed shards may coexist during setup. For a 70B full-precision model, that can already mean well over 100 GB. For Kimi K3, the source checkpoint alone is around 1.6 TB. Capacity planning must include downloads, transformed weights, cache, temporary space and rollback. ### Storage bandwidth and latency Every cold layer or expert must cross storage, host memory and possibly PCIe before compute. Sequential NVMe marketing numbers are not enough. The workload includes repeated file access, allocation, deserialization and synchronization. Prefetching can overlap part of that path, but it cannot make slow storage behave like local GPU memory. ### Time to first token and decode speed The AirLLM README publishes exact peak-VRAM numbers but no reproducible tokens-per-second table for its 70B, 405B, DeepSeek-V3 or Kimi K3 headlines. A public [repository evidence audit](https://github.com/lyogavin/airllm/issues/295) also notes that the linked 405B notebook has no executed timing or memory output and uses a pre-quantized 4-bit model. Treat every speed estimate as unknown until the exact checkpoint, revision, prompt, context, hardware and output length are measured. ### Context and concurrency Layer streaming reduces weight residency. It does not eliminate KV cache, activation or batching costs. A single short prompt that produces five tokens is not evidence for a 32K document, tool-calling agent or five concurrent users. In fact, repeated loading makes concurrency especially important: batching can amortize one layer load across several sequences, while isolated requests pay the movement cost separately. ## AirLLM versus a model that fits | Need | Better default | Why | | --- | --- | --- | | Inspect one giant checkpoint on existing hardware | AirLLM pilot | Access matters more than response time | | Private personal chat | Smaller quantized model in llama.cpp or MLX | Resident weights usually give far better latency | | Multi-user API | vLLM, SGLang or a managed API | Batching, scheduling and observability are first-class concerns | | Offline batch classification | Benchmark both | Layer loading may be amortized across a large batch | | Production decision under uncertain quality | Smallest model that passes the eval | Capability per accepted task matters more than parameter count | Research on low-resource inference reaches the same broader conclusion: storage offload creates capacity, then systems work is needed to recover speed. The [PRIMA.CPP paper](https://arxiv.org/abs/2504.08791) uses memory mapping, pipelining, prefetching and device-aware layer placement to reduce token latency on 30B to 70B models. That is not an AirLLM benchmark. It is evidence that the difference between "runs" and "runs usefully" is an inference architecture problem, not a VRAM headline. ## When is an AirLLM pilot commercially sensible? - **Rare offline access:** your team needs occasional output from a model that cannot otherwise load, and minutes of latency may be acceptable. - **Model research:** you are checking tokenizer, architecture, quality or compatibility before funding proper serving hardware. - **Large-batch experiments:** one layer load can process many sequences, so throughput may matter more than single-user latency. - **Hardware-constrained education:** the goal is to understand a model and memory hierarchy, not operate an SLA. Skip the pilot when a customer expects interactive responses, several users need simultaneous access, the model handles regulated data without a complete security review, or nobody owns storage health, dependency pinning and failure recovery. Also skip it when a 7B to 70B quantized model already passes the same task eval. The winning architecture is the least complex system that meets the product requirement. ## A decision-grade benchmark plan 1. **Pin the artifact.** Record model ID, revision, weight format, AirLLM commit, Python, Torch, Transformers, CUDA and driver versions. 2. **Measure all memory tiers.** Capture peak VRAM, peak resident RAM, swap, original checkpoint size, transformed shard size and temporary setup space. 3. **Separate phases.** Report download, split, cold start, prompt prefill, time to first token and decode tokens per second. 4. **Test realistic inputs.** Use the expected context, output length, tools, languages and structured-output schema. 5. **Test batches and concurrency.** Compare one request, a useful offline batch and the maximum simultaneous load you promise. 6. **Evaluate quality.** Compare the exact streamed artifact with a hosted baseline on accepted tasks, not a generic leaderboard. 7. **Price the whole system.** Include SSD capacity, bandwidth, electricity, engineer time, failures, monitoring and fallback API spend. Use the [local-model versus API break-even framework](/blog/local-models-vs-apis-break-even-eu-2026/) after the benchmark, not before it. If the project is specifically about Kimi K3 procurement, API terms or data location, the [Kimi K3 EU production review](/blog/kimi-k3-eu-api-production-review/) owns that decision. For another NVMe expert-streaming architecture with published throughput data, compare our [Colibri hardware analysis](/blog/colibri-glm-5-2-consumer-hardware/). ## Source boundaries AirLLM features, compatibility, setup warnings and peak-memory claims come from the project repository. Kimi K3 architecture, native MXFP4 format and recommended serving engines come from Moonshot AI. DeepSeek-V3 parameter counts come from DeepSeek. The evidence gap for the 405B notebook comes from the linked public repository audit. PRIMA.CPP provides independent context on low-resource disk offload, not validation of AirLLM's headline measurements. Wavect did not download terabyte-scale checkpoints or reproduce AirLLM's GPU-memory results for this article. ## Frequently Asked Questions ### Can AirLLM really run a 70B model on a 4 GB GPU? AirLLM can execute supported 70B checkpoints with a low VRAM peak by loading one layer at a time. The full model remains on storage, and low peak VRAM does not guarantee interactive generation speed. ### How does AirLLM work? AirLLM splits a checkpoint into layer shards. During inference it loads the current layer to the compute device, processes activations, releases the weights and repeats. Supported sparse models can stream selected experts at a finer granularity. ### Does AirLLM use quantization? Quantization is optional in AirLLM, which offers 4-bit and 8-bit block-wise compression. Some downloaded models are already quantized. Kimi K3 ships with native MXFP4 weights, so its 3.72 GB example is not an unquantized checkpoint. ### Why does Kimi K3 need less VRAM than Llama 70B? AirLLM reports per-expert streaming for Kimi K3. Only the experts selected for a token need to be resident, while a dense model may require a complete layer. The full Kimi K3 checkpoint is still roughly 1.6 TB. ### How fast is AirLLM? There is no reproducible upstream tokens-per-second table for the headline 70B, 405B, DeepSeek-V3 and Kimi K3 claims. Measure cold start, time to first token and decode speed on the exact model, context and hardware before making a decision. ### Is AirLLM production-ready? AirLLM is best treated as a research and access tool unless your own tests prove the required latency, throughput, quality, reliability and operations. Multi-user production usually favors a resident smaller model, a serving engine or a managed API. ### When should a company test AirLLM? Test it for rare offline access to an otherwise unloadable model, architecture research or batch experiments where one layer load can serve several sequences. Compare against the smallest quantized model that passes the same eval. ## Final thoughts AirLLM makes an important capability possible: executing a checkpoint that cannot reside in GPU memory. Its 4 GB, 8 GB, 12 GB and 3.72 GB numbers describe peak memory under reported setups. They do not make the checkpoint small, storage fast or production serving automatic. Treat AirLLM as an access architecture. Verify the exact weight format, reserve disk for both original and split shards, measure cold start and every stage of generation, and compare cost per accepted task with a smaller resident model or API. If latency, concurrency and reliability matter, the largest model that starts is rarely the best system to ship. ## You may also like.. [**Which local LLM fits your hardware?** Shortlist models that fit before testing an extreme layer-streaming path.](/blog/llmfit-local-llm-hardware-guide/) [**Local models versus APIs** Calculate when self-hosting wins after latency, utilization and engineering cost are measured.](/blog/local-models-vs-apis-break-even-eu-2026/) Models and infrastructure ## Continue through this cluster Model selection, inference economics, local deployment, compression and serving architecture. [Start with the cornerstone**Self-Hosting LLMs in the EU: When Open Weights Actually Pay Off**](/blog/self-hosting-llms-eu-cost/) - [Qwen3.8-27B: Self-Hosted Computer-Use Agents Without Exporting Screenshots](/blog/qwen3-8-27b-self-hosted-computer-use-agents/) - [Netflix's vLLM and Triton Stack: 7 Production Lessons](/blog/netflix-vllm-triton-inference-stack/) - [Transformers.js Browser AI: When Local Inference Belongs in Your Product](/blog/transformers-js-browser-ai-guide/) - [How to Self-Host LiteLLM in Production: 2026 Guide](/blog/self-host-litellm-production-2026/) - [AI-Ready Company Wiki: Architecture and Build Guide](/blog/ai-ready-company-wiki/) Inbox, without the noise ## Follow the work that matters to you Get a short email when we publish something new. Follow the whole blog or only the problems you care about. [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 10 min read · 20 Aug 2026 Last reviewed August 20, 2026 [**Next**](/blog/moneyprinterturbo-review-2026/) New posts by email × × Get new posts by email A short email when we publish. Free, no tracking. ## Structured Data ```json { "@context": "https://schema.org", "@graph": [ { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ], "employee": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "founder": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "legalRepresentative": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "name": "Wavect GmbH", "subjectOf": { "@id": "https://wavect.io/verified-claims.json#dataset", "@type": "Dataset", "creator": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "description": "A machine-readable registry of quantitative and qualitative claims published by Wavect, with review dates, localized page appearances and public third-party citations where available.", "inLanguage": "en", "isAccessibleForFree": true, "license": "https://creativecommons.org/licenses/by/4.0/", "name": "Wavect verified publication claims", "url": "https://wavect.io/verified-claims.json" }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/#website", "@type": "WebSite", "inLanguage": [ "en", "de", "es", "zh" ], "name": "Wavect", "potentialAction": { "@type": "SearchAction", "query-input": "required name=search_term_string", "target": { "@type": "EntryPoint", "urlTemplate": "https://wavect.io/search/?q={search_term_string}" } }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/blog/airllm-layer-wise-inference-low-vram/#webpage", "@type": "WebPage", "dateModified": "2026-08-20", "inLanguage": "en", "isPartOf": { "@id": "https://wavect.io/#website", "@type": "WebSite" }, "lastReviewed": "2026-08-20", "url": "https://wavect.io/blog/airllm-layer-wise-inference-low-vram/" } ] } ``` ```json { "@context": "https://schema.org", "@type": "BlogPosting", "abstract": "AirLLM can reduce peak GPU memory by loading only the layer, or for supported sparse models the routed experts, needed at that moment. Its repository reports 70B on about 4 GB VRAM, Llama 3.1 405B on about 8 GB, DeepSeek-V3 on about 12 GB and Kimi K3 on 3.72 GB. Those are execution and peak-memory claims, not production throughput guarantees. The full checkpoint still needs storage, every generated token can trigger repeated weight transfers, context and runtime buffers still consume memory, and the project publishes no reproducible tokens-per-second result for those headline models. Kimi K3 also ships with native MXFP4 weights, so \"no quantization\" does not accurately describe that example. Use AirLLM for research, offline evaluation and model-access experiments. For a customer-facing service, choose only after measuring time to first token, decode speed, disk footprint, concurrency, quality and total cost against a smaller quantized model or API.", "articleBody": " Blog overview/AI and agents/Models and infrastructure AirLLM on 4 GB VRAM: How Layer-Wise Inference Really Works TL;DR AirLLM can reduce peak GPU memory by loading only the layer, or for supported sparse models the routed experts, needed at that moment. Its repository reports 70B on about 4 GB VRAM, Llama 3.1 405B on about 8 GB, DeepSeek-V3 on about 12 GB and Kimi K3 on 3.72 GB. Those are execution and peak-memory claims, not production throughput guarantees. The full checkpoint still needs storage, every generated token can trigger repeated weight transfers, context and runtime buffers still consume memory, and the project publishes no reproducible tokens-per-second result for those headline models. Kimi K3 also ships with native MXFP4 weights, so \"no quantization\" does not accurately describe that example. Use AirLLM for research, offline evaluation and model-access experiments. For a customer-facing service, choose only after measuring time to first token, decode speed, disk footprint, concurrency, quality and total cost against a smaller quantized model or API. Yes, AirLLM can execute a model that is far larger than the GPU's VRAM. It does not make the model fit on the GPU. It keeps the checkpoint on storage, moves one layer or selected experts through memory when needed, computes, releases them and repeats. The project trades memory residency for data movement. That distinction turns the viral claim into a useful engineering question. A 4 GB peak does not tell you how much disk is required, how long the first split takes, how many seconds one token needs, whether a long context still fits or how many users the system can serve. This review was checked on 20 August 2026 and owns one narrow intent: how AirLLM's low-VRAM inference works and when it is commercially useful. For selecting an ordinary local model that fits your machine, use our local LLM hardware-fit guide. AirLLM's headline claims and the decision-grade interpretation ClaimWhat the available evidence supportsWhat it does not prove 70B on 4 GB VRAMThe runtime keeps one layer resident at a timeInteractive or production speed Llama 3.1 405B on 8 GBThe project publishes a code path and notebookA reproducible full-precision benchmark DeepSeek-V3 671B on about 12 GBCurrent AirLLM support and a peak-memory claimLatency, concurrency or service reliability Kimi K3 2.8T on 3.72 GBMaintainer-reported end-to-end peak on one RTX 6000 AdaThat the 1.6 TB checkpoint fits in VRAM or storage disappears No quantization requiredAirLLM can stream some full-precision checkpoints without adding compressionKimi K3 is unquantized; its released weights are natively MXFP4 What is AirLLM? AirLLM is an Apache-2.0 Python inference library that decomposes supported transformer checkpoints into smaller shards and loads them on demand. The official AirLLM repository reports support for Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, Kimi K3 and other model families through an AutoModel interface. It also offers optional 4-bit or 8-bit block-wise weight compression, CPU execution, Apple Silicon support and limited prefetching. The project's core benefit is access. A researcher can inspect output from a checkpoint that would otherwise fail at model load because the accelerator cannot hold all weights. The core cost is repeated movement. A normal GPU server keeps weights resident and reuses them for each token and request. AirLLM repeatedly fetches weights from a slower tier because capacity, not throughput, is the primary goal. How does layer-wise inference reduce VRAM? Download and split the checkpoint. AirLLM creates layer-level files. Its own FAQ warns that this step is disk-intensive and can require enough room for both the original and transformed copy. Keep activations and runtime state. Input tensors, output state, attention data, the KV cache and framework buffers still need CPU or GPU memory. Load the next layer. The runtime transfers the current transformer block to the compute device. Run the block and release it. The activation passes through the layer, then the weight allocation can be reused for the next block. Repeat for every generated token. Autoregressive decode traverses the model again for each new token. Unless weights remain cached, storage traffic returns on every step. Peak VRAM therefore follows the largest layer plus activations and working buffers, not the sum of all parameters. Total storage still follows the checkpoint. A model can need only a few gigabytes of VRAM while occupying hundreds of gigabytes or more on disk. Long context, large batches and multiple requests also grow runtime memory even when model weights are streamed. Why can the 2.8T model report less VRAM than a dense 70B model? Kimi K3 is not a dense 2.8-trillion-parameter model. The official Kimi K3 model repository documents 93 layers, 896 routed experts, 16 selected experts per token and 104 billion activated parameters. AirLLM's K3 path goes below a full transformer layer and", "articleSection": "Engineering", "author": { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/" }, "citation": [ { "@type": "WebPage", "name": "official AirLLM repository", "url": "https://github.com/lyogavin/airllm" }, { "@type": "WebPage", "name": "official Kimi K3 model repository", "url": "https://github.com/MoonshotAI/Kimi-K3" }, { "@type": "WebPage", "name": "official DeepSeek-V3 repository", "url": "https://github.com/deepseek-ai/DeepSeek-V3" }, { "@type": "WebPage", "name": "repository evidence audit", "url": "https://github.com/lyogavin/airllm/issues/295" }, { "@type": "WebPage", "name": "PRIMA.CPP paper", "url": "https://arxiv.org/abs/2504.08791" } ], "dateModified": "2026-08-20", "datePublished": "2026-08-20", "description": "AirLLM can reduce peak GPU memory by loading only the layer, or for supported sparse models the routed experts, needed at that moment. Its repository reports 70B on about 4 GB VRAM, Llama 3.1 405B on about 8 GB, DeepSeek-V3 on about 12 GB and Kimi K3 on 3.72 GB. Those are execution and peak-memory claims, not production throughput guarantees. The full checkpoint still needs storage, every generated token can trigger repeated weight transfers, context and runtime buffers still consume memory, and the project publishes no reproducible tokens-per-second result for those headline models. Kimi K3 also ships with native MXFP4 weights, so \"no quantization\" does not accurately describe that example. Use AirLLM for research, offline evaluation and model-access experiments. For a customer-facing service, choose only after measuring time to first token, decode speed, disk footprint, concurrency, quality and total cost against a smaller quantized model or API.", "headline": "AirLLM on 4 GB VRAM: How Layer-Wise Inference Really Works", "image": "https://wavect.io/img/blog/headers/header_airllm-layer-wise-inference-low-vram.svg", "inLanguage": "en", "keywords": "AirLLM, Local LLM, GPU Inference", "mainEntityOfPage": { "@id": "https://wavect.io/blog/airllm-layer-wise-inference-low-vram/", "@type": "WebPage" }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/blog/airllm-layer-wise-inference-low-vram/", "wordCount": 2426 } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "item": "https://wavect.io/", "name": "Home", "position": 1 }, { "@type": "ListItem", "item": "https://wavect.io/blog/overview/", "name": "Blog overview", "position": 2 }, { "@type": "ListItem", "item": "https://wavect.io/blog/topics/ai-agents/", "name": "AI and agents", "position": 3 }, { "@type": "ListItem", "item": "https://wavect.io/blog/clusters/models-infrastructure/", "name": "Models and infrastructure", "position": 4 }, { "@type": "ListItem", "item": "https://wavect.io/blog/airllm-layer-wise-inference-low-vram/", "name": "AirLLM on 4 GB VRAM: Layer-Wise Inference | ", "position": 5 } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "AirLLM can execute supported 70B checkpoints with a low VRAM peak by loading one layer at a time. The full model remains on storage, and low peak VRAM does not guarantee interactive generation speed." }, "name": "Can AirLLM really run a 70B model on a 4 GB GPU?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "AirLLM splits a checkpoint into layer shards. During inference it loads the current layer to the compute device, processes activations, releases the weights and repeats. Supported sparse models can stream selected experts at a finer granularity." }, "name": "How does AirLLM work?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Quantization is optional in AirLLM, which offers 4-bit and 8-bit block-wise compression. Some downloaded models are already quantized. Kimi K3 ships with native MXFP4 weights, so its 3.72 GB example is not an unquantized checkpoint." }, "name": "Does AirLLM use quantization?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "AirLLM reports per-expert streaming for Kimi K3. Only the experts selected for a token need to be resident, while a dense model may require a complete layer. The full Kimi K3 checkpoint is still roughly 1.6 TB." }, "name": "Why does Kimi K3 need less VRAM than Llama 70B?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "There is no reproducible upstream tokens-per-second table for the headline 70B, 405B, DeepSeek-V3 and Kimi K3 claims. Measure cold start, time to first token and decode speed on the exact model, context and hardware before making a decision." }, "name": "How fast is AirLLM?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "AirLLM is best treated as a research and access tool unless your own tests prove the required latency, throughput, quality, reliability and operations. Multi-user production usually favors a resident smaller model, a serving engine or a managed API." }, "name": "Is AirLLM production-ready?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Test it for rare offline access to an otherwise unloadable model, architecture research or batch experiments where one layer load can serve several sequences. Compare against the smallest quantized model that passes the same eval." }, "name": "When should a company test AirLLM?" } ] } ```