--- title: "FreeToken AI Review: Hardware, Benchmarks & Setup" canonical: https://wavect.io/blog/freetoken-ai-inference-engine-review/ language: en description: "FreeToken AI runs large MoE models across GPU, CPU and RAM. Check hardware requirements, author benchmarks, setup, limits and production fit." image: "https://wavect.io/img/blog/headers/header_freetoken-ai-inference-engine-review.png" --- [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 13 min read · 25 Aug 2026 Last reviewed August 25, 2026 [**Next**](/blog/airllm-layer-wise-inference-low-vram/) # FreeToken AI Review: Run Frontier MoE Models on Consumer GPUs? TL;DR FreeToken is an Apache-2.0 inference engine from FlashML for running large Mixture-of-Experts models across an NVIDIA GPU, CPU, system RAM and PCIe link. Its authors report 39.3 tokens per second for Qwen3.6-35B-A3B on an 8 GB RTX 4060 laptop and 22 to 25 tokens per second for DeepSeek-V4-Flash on an RTX 5090, but these are project-paper results, not Wavect benchmarks. The current CLI requires Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13 and Python 3.10 or newer. FreeToken can remove a per-token API invoice, but not hardware, RAM, storage, electricity, operations or model-validation costs. Pilot it when a supported MoE model passes your task eval and local control or sustained usage justifies the infrastructure. **FreeToken is a new open-source inference engine for running large Mixture-of-Experts models across the hardware in one personal computer.** It uses GPU VRAM as a fast expert cache, keeps the full expert pool in system memory, and splits remaining work between PCIe transfers and CPU execution. The result can make models that do not fit in VRAM usable at interactive speeds on selected NVIDIA systems. This article covers the FlashML project at `FlashML-org/FreeToken`, not similarly named token services or SDKs. It was reviewed on **25 August 2026**. The primary intent is a decision-grade FreeToken AI review: hardware requirements, installation, author-reported benchmarks, limits and commercial fit. For the broader question of which model fits a machine, use our [local LLM hardware-fit guide](/blog/llmfit-local-llm-hardware-guide/). | Question | Short answer | | --- | --- | | Is FreeToken free? | The engine is Apache-2.0 open source. Hardware, model storage, RAM, electricity and operations are not free. | | Does it provide free API tokens? | No. It serves open-weight models on your own machine, so there is no provider token invoice for local requests. | | What hardware does the CLI need? | Currently Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13, Python 3.10 or newer, and enough RAM and storage for the checkpoint. | | Who should test it? | Teams evaluating a supported MoE for coding agents, private automation, research or sustained local inference. | | Who should wait? | Apple Silicon or AMD users, teams needing mature multi-user operations, and buyers without task-specific model and cost evaluations. | ## What is FreeToken AI? FreeToken is an edge-native serving runtime from FlashML. The [official FreeToken GitHub repository](https://github.com/FlashML-org/FreeToken) describes support for consumer laptops, gaming desktops and workstation GPUs, plus OpenAI-compatible and Anthropic-compatible endpoints for coding and tool-using agents. The code is licensed under Apache 2.0. The project is designed around sparse MoE models. An MoE checkpoint contains many expert networks, but its router activates only a small subset for each token. The arithmetic required for one token can therefore be much smaller than the full parameter count. The hard part is memory: all experts still have to live somewhere, even when most are inactive. FreeToken treats the entire computer as one inference system. Non-expert weights and frequently used experts stay on the GPU. The complete expert pool stays in host RAM. Cache misses are either moved over PCIe and executed on the GPU or computed where they already reside on the CPU. ## How does FreeToken run a model larger than GPU VRAM? The [FreeToken research paper](https://arxiv.org/abs/2608.16157) describes five mechanisms that work together: 1. **A shared GPU expert cache:** FreeToken tracks recently used layer-expert pairs with an LRU policy instead of fixing every expert placement at startup. 2. **Bandwidth-adaptive execution:** a measured policy divides cache misses between PCIe transfer plus GPU compute and direct CPU compute. The split is calibrated for the actual machine. 3. **Pipelined prefill:** two layer buffers overlap expert movement with GPU computation so long prompts do not expose every transfer as idle time. 4. **Semantic state anchors:** checkpoints at tool calls, thinking blocks and conversation boundaries let an agent reuse more context after its history is edited. 5. **Elastic memory pools:** the runtime can rebalance VRAM between the expert cache and KV cache as the context grows, without reloading the host-resident expert pool. This architecture is different from claiming that a 284B or 753B checkpoint fits inside a small GPU. It does not. The full model still requires host memory or storage. FreeToken tries to keep the active path fast by placing the right work on the GPU, CPU and interconnect at each phase. ## What are the FreeToken hardware requirements? The current [FreeToken installation requirements](https://github.com/FlashML-org/FreeToken/blob/main/docs/install.md) specify Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13 and Python 3.10 or newer. CUDA kernels compile on first use, so the CUDA 13 toolkit and `nvcc` must be available. Those are software minimums, not a promise that a chosen model will fit. Capacity planning also needs: - **System RAM:** enough for the complete host-resident expert pool, runtime allocations and the operating system. - **VRAM:** enough for non-expert weights, KV cache, execution buffers and a useful expert cache. - **Fast storage:** enough for the model checkpoint, optional converted FTW weights, caches and rollback space. - **Host bandwidth:** CPU memory bandwidth and PCIe bandwidth directly influence how misses should be divided. - **Power and cooling:** a sustained local agent workload can keep CPU, GPU and memory active for long periods. | System | GPU | Host RAM | Demonstrated model | Author-reported result | | --- | --- | --- | --- | --- | | Laptop | RTX 4060 Laptop, 8 GB | 32 GiB | Qwen3.6-35B-A3B NVFP4 | 39.3 decode tokens/s | | Gaming desktop | RTX 5090, 32 GB | 192 GiB | DeepSeek-V4-Flash, 284B total | Interactive serving demonstrated | | Workstation | RTX PRO 6000, 96 GB | 512 GiB | GLM-5.2, 753B total | 14.9 tokens/s versus 7.3 for llama.cpp | Read the table as evidence of a tested configuration, not a buying chart. The laptop used an official NVFP4 model while several larger-system comparisons used different formats. Model architecture, quantization, context, agent trajectory, host RAM, CPU bandwidth and PCIe generation all affect the result. ## Which models and backends does FreeToken support? The [official supported-model list](https://github.com/FlashML-org/FreeToken/blob/main/docs/models.md) currently includes DeepSeek-V4-Flash, GLM-5.2, GLM-4.7, Qwen3.6 and Qwen3.5 MoE variants, Qwen3-MoE, gpt-oss, Gemma-4, MiniMax-M2.5 and Muse-Glimmer checkpoints. Support is architecture and format specific. Multimodal checkpoints are currently served as text only. The MoE backend can be `fused`, `offload`, `cpu`, `hybrid` or `auto`. Fused requires the experts to fit in VRAM. Offload streams misses from host memory. CPU computes misses in place. Hybrid overlaps both paths. Auto starts with offload and can select hybrid after a bandwidth benchmark recommends it. That model list will change quickly. Pin the FreeToken version, model repository, exact revision, quantization and license in every evaluation. The engine's Apache license does not replace the model's own license or acceptable-use terms. ## How strong are the FreeToken benchmark claims? The paper evaluates six systems, two main MoE models, four agentic workloads and four baseline engines. Its authors report 77 to 83 tokens/s for Qwen3.6-35B-A3B and 22 to 25 tokens/s for DeepSeek-V4-Flash on an RTX 5090. Across five consumer systems, FreeToken reportedly led the strongest supported baseline by 1.3 to 2.1 times. Its worst time to first token stayed below 44 seconds in the tested cells, while every baseline crossed 150 seconds somewhere. The evaluation is more useful than a single short-prompt tokens-per-second screenshot because it includes coding agents, tool calls, long contexts and tail latency. It also aligns model formats across engines where supported. Still, it is a version-one arXiv preprint and the measurements come from the project authors. Wavect did not reproduce the GPU results for this article. Treat them as a strong reason to benchmark, not as a service-level guarantee. ## FreeToken versus Ollama, AirLLM and a smaller resident model | Decision | Best starting point | Why | | --- | --- | --- | | Serve a supported frontier MoE on mixed NVIDIA, CPU and RAM resources | FreeToken pilot | Its cache and hybrid execution are designed for that exact bottleneck. | | Run a common local model with a simple workflow | Ollama or llama.cpp | Start with the least complex runtime that meets the requirement. | | Inspect a huge checkpoint with extremely little VRAM | [AirLLM layer-wise inference](/blog/airllm-layer-wise-inference-low-vram/) | Access can matter more than interactive latency for research. | | Choose a model before choosing a runtime | [llmfit hardware guide](/blog/llmfit-local-llm-hardware-guide/) | First find the smallest model and quantization that can pass the task. | | Decide whether to self-host at all | [Local model versus API calculator](/blog/local-models-vs-apis-break-even-eu-2026/) | Utilization, operations and accepted-task cost determine the commercial answer. | FreeToken is especially relevant when the chosen model is sparse and agentic contexts cause repeated prefill. It is less compelling when a smaller quantized dense model already meets quality and latency targets in VRAM. It also does not remove the need to compare the model itself. For a model-specific example, our [DeepSeek V4 Flash local deployment guide](/blog/deepseek-v4-flash-0731-local-ai-pc/) owns the memory, quantization and hardware-fit decision. ## How do you install and test FreeToken? The [official FreeToken quick start](https://github.com/FlashML-org/FreeToken/blob/main/docs/quickstart.md) exposes OpenAI-compatible and Anthropic-compatible endpoints. A minimal evaluation path is: ``` uv venv source .venv/bin/activate uv pip install "freetoken[accel]" ft bench bw ft serve --model Qwen/Qwen3.6-35B-A3B ``` Then verify `/v1/models`, send a short completion, and only after that connect an agent. The CLI can launch Codex, Claude Code, OpenCode, OpenClaw and other supported harnesses against the local server. Use `ft launch... --dry-run` first so configuration changes are visible before they are applied. Do not start with the biggest possible checkpoint. Begin with the smallest supported artifact that represents the target workload. Record the download size, cold-start time, time to first token, prompt throughput, decode speed, peak VRAM, peak RAM, context capacity, power draw and failure behavior. ## Is local inference really free? Local requests do not create a hosted-provider token charge. That is different from zero cost. The full cost per accepted task includes hardware depreciation, system RAM, NVMe storage, electricity, cooling, engineering time, monitoring, model updates, security controls, downtime, idle capacity and a fallback path. For one developer with an existing workstation, the marginal cost may feel close to zero. For a customer-facing product, redundancy and utilization usually dominate. Compare the exact FreeToken system with the API it would replace, using the same prompts, tool calls, success criteria and concurrency. A faster free token is not useful if the model fails the task or the server misses its latency target. ## When is a FreeToken pilot commercially sensible? - **Sustained agent workloads:** repeated coding or tool calls create enough token volume to justify owned infrastructure. - **Local control:** data location, offline operation or predictable access matters more than immediate model switching. - **Existing hardware:** the company already owns suitable NVIDIA hardware, host RAM and fast storage. - **Supported MoE quality:** one of the supported models passes a task-specific evaluation against the current API. - **Engineering ownership:** someone owns patching, model revisions, observability, authentication and recovery. Use a managed API or smaller resident model when demand is low or bursty, the product requires the latest frontier capability, Apple or AMD hardware is fixed, multimodal serving is mandatory, or the team cannot operate a local inference service. A hybrid architecture is often the safest commercial default: local inference for predictable private tasks, with a controlled cloud fallback for difficult or overloaded requests. ## Production-readiness checklist 1. **Pin every artifact:** FreeToken commit or release, model ID, revision, quantization, CUDA, driver and dependencies. 2. **Run task evaluations:** compare accepted-task rate, not parameter count or generic leaderboard position. 3. **Measure agent traces:** include long context, tool calls, concurrent requests, retries and history compaction. 4. **Secure the endpoint:** keep the default loopback binding unless a protected network service is intentional. Add authentication, authorization, rate limits and audit logs before remote access. 5. **Observe all bottlenecks:** capture queue time, prefill, time to first token, decode, cache hits, RAM, VRAM, storage and power. 6. **Test memory pressure:** verify behavior when another application takes VRAM or the KV cache grows. 7. **Plan failure and fallback:** define what happens on model load failure, timeout, low quality, overload or host outage. 8. **Recalculate total cost:** use measured utilization and successful tasks before purchasing more hardware. For the context-specific performance problem, see our analysis of [shared KV-cache latency in multi-turn LLM inference](/blog/shared-kv-cache-llm-inference-latency/). FreeToken adds semantic anchors and elastic cache allocation, but the product still needs end-to-end measurements around the agent harness. ## Source boundaries Features, licensing and broad hardware support come from the official FreeToken repository. Architecture and benchmark figures come from the authors' arXiv paper. Software prerequisites come from the installation guide. Model and backend compatibility come from the supported-model documentation. Commands and API compatibility come from the quick start. Wavect did not reproduce the paper's hardware benchmarks and does not claim independent validation. Project support, models and requirements can change after the review date. ## Frequently Asked Questions ### What is FreeToken AI? FreeToken is an open-source inference engine from FlashML for serving large Mixture-of-Experts models across an NVIDIA GPU, CPU, system RAM and PCIe connection. It exposes OpenAI-compatible and Anthropic-compatible APIs. ### Does FreeToken give you free AI tokens? No. It runs open-weight models on hardware you control, so local requests do not create a provider token invoice. You still pay for hardware, RAM, storage, electricity and operations. ### Can FreeToken run on 8 GB VRAM? The authors report Qwen3.6-35B-A3B NVFP4 at 39.3 decode tokens per second on an RTX 4060 laptop with 8 GB VRAM and 32 GiB of system memory. Treat that as a tested project configuration and reproduce it on your exact machine. ### What operating systems and GPUs does FreeToken support? The current CLI installation guide requires Linux x86_64 and an NVIDIA GPU with driver r580 or newer plus CUDA 13. The project also advertises a Windows desktop app, but CLI and model compatibility should be checked against the current release. ### Is FreeToken faster than Ollama? In the authors' tested MoE workloads, FreeToken reports higher decode throughput and lower tail time to first token than supported baselines, including Ollama in applicable cells. That does not make it faster for every model or workload. Benchmark the exact model, format, prompt and hardware. ### Which models does FreeToken support? The current list includes selected DeepSeek-V4, GLM, Qwen MoE, gpt-oss, Gemma-4, MiniMax and Muse-Glimmer checkpoints. Compatibility is architecture and format specific, and multimodal checkpoints are currently text-only. ### Is FreeToken ready for production? It is promising enough for a controlled pilot. Production still requires task evaluations, pinned artifacts, authentication, authorization, observability, concurrency tests, recovery, fallback and a measured total-cost comparison. ## Final thoughts FreeToken changes the local-AI question from whether a model fits in GPU VRAM to whether one computer can orchestrate its GPU, CPU, RAM and PCIe link well enough for the workload. The authors' results make that a credible engineering direction, especially for sparse models and long-running agents. Do not buy hardware from a parameter headline. Confirm software support, reserve host RAM and storage, reproduce the benchmark with the exact checkpoint, and compare cost per accepted task with a smaller resident model and a managed API. If FreeToken wins that test, local frontier MoE inference can become a practical product architecture rather than a lab demonstration. ## You may also like.. [**Which local LLM fits your hardware?** Shortlist models and quantizations before choosing an inference engine.](/blog/llmfit-local-llm-hardware-guide/) [**Local models versus APIs** Calculate when owned inference beats API cost after utilization and operations.](/blog/local-models-vs-apis-break-even-eu-2026/) Models and infrastructure ## Continue through this cluster Model selection, inference economics, local deployment, compression and serving architecture. [Start with the cornerstone**Self-Hosting LLMs in the EU: When Open Weights Actually Pay Off**](/blog/self-hosting-llms-eu-cost/) - [Darkbloom AI Review: Private Inference on Idle Macs](/blog/darkbloom-ai-private-inference-mac/) - [Ox Alpha Free AI Model: Setup, Privacy and Buyer Guide](/blog/ox-alpha-free-ai-model-guide-2026/) - [Pika Audio Models API Pricing: Is SFX Really 20x Cheaper?](/blog/pika-audio-models-api-pricing-2026/) - [Thunder Compute's $13M GPU Virtualization Bet: Enterprise Buyer's Guide](/blog/thunder-compute-gpu-virtualization-series-a/) - [AirLLM on 4 GB VRAM: How Layer-Wise Inference Really Works](/blog/airllm-layer-wise-inference-low-vram/) Inbox, without the noise ## Follow the work that matters to you Get a short email when we publish something new. Follow the whole blog or only the problems you care about. [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 13 min read · 25 Aug 2026 Last reviewed August 25, 2026 [**Next**](/blog/airllm-layer-wise-inference-low-vram/) New posts by email × × Get new posts by email A short email when we publish. Free, no tracking. ## Structured Data ```json { "@context": "https://schema.org", "@graph": [ { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ], "employee": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "founder": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "legalRepresentative": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "name": "Wavect GmbH", "subjectOf": { "@id": "https://wavect.io/verified-claims.json#dataset", "@type": "Dataset", "creator": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "description": "A machine-readable registry of quantitative and qualitative claims published by Wavect, with review dates, localized page appearances and public third-party citations where available.", "inLanguage": "en", "isAccessibleForFree": true, "license": "https://creativecommons.org/licenses/by/4.0/", "name": "Wavect verified publication claims", "url": "https://wavect.io/verified-claims.json" }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/#website", "@type": "WebSite", "inLanguage": [ "en", "de", "es", "zh" ], "name": "Wavect", "potentialAction": { "@type": "SearchAction", "query-input": "required name=search_term_string", "target": { "@type": "EntryPoint", "urlTemplate": "https://wavect.io/search/?q={search_term_string}" } }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/blog/freetoken-ai-inference-engine-review/#webpage", "@type": "WebPage", "dateModified": "2026-08-25", "inLanguage": "en", "isPartOf": { "@id": "https://wavect.io/#website", "@type": "WebSite" }, "lastReviewed": "2026-08-25", "url": "https://wavect.io/blog/freetoken-ai-inference-engine-review/" } ] } ``` ```json { "@context": "https://schema.org", "@type": "BlogPosting", "abstract": "FreeToken is an Apache-2.0 inference engine from FlashML for running large Mixture-of-Experts models across an NVIDIA GPU, CPU, system RAM and PCIe link. Its authors report 39.3 tokens per second for Qwen3.6-35B-A3B on an 8 GB RTX 4060 laptop and 22 to 25 tokens per second for DeepSeek-V4-Flash on an RTX 5090, but these are project-paper results, not Wavect benchmarks. The current CLI requires Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13 and Python 3.10 or newer. FreeToken can remove a per-token API invoice, but not hardware, RAM, storage, electricity, operations or model-validation costs. Pilot it when a supported MoE model passes your task eval and local control or sustained usage justifies the infrastructure.", "articleBody": " Blog overview/AI and agents/Models and infrastructure FreeToken AI Review: Run Frontier MoE Models on Consumer GPUs? TL;DR FreeToken is an Apache-2.0 inference engine from FlashML for running large Mixture-of-Experts models across an NVIDIA GPU, CPU, system RAM and PCIe link. Its authors report 39.3 tokens per second for Qwen3.6-35B-A3B on an 8 GB RTX 4060 laptop and 22 to 25 tokens per second for DeepSeek-V4-Flash on an RTX 5090, but these are project-paper results, not Wavect benchmarks. The current CLI requires Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13 and Python 3.10 or newer. FreeToken can remove a per-token API invoice, but not hardware, RAM, storage, electricity, operations or model-validation costs. Pilot it when a supported MoE model passes your task eval and local control or sustained usage justifies the infrastructure. FreeToken is a new open-source inference engine for running large Mixture-of-Experts models across the hardware in one personal computer. It uses GPU VRAM as a fast expert cache, keeps the full expert pool in system memory, and splits remaining work between PCIe transfers and CPU execution. The result can make models that do not fit in VRAM usable at interactive speeds on selected NVIDIA systems. This article covers the FlashML project at FlashML-org/FreeToken, not similarly named token services or SDKs. It was reviewed on 25 August 2026. The primary intent is a decision-grade FreeToken AI review: hardware requirements, installation, author-reported benchmarks, limits and commercial fit. For the broader question of which model fits a machine, use our local LLM hardware-fit guide. FreeToken decision summary QuestionShort answer Is FreeToken free?The engine is Apache-2.0 open source. Hardware, model storage, RAM, electricity and operations are not free. Does it provide free API tokens?No. It serves open-weight models on your own machine, so there is no provider token invoice for local requests. What hardware does the CLI need?Currently Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13, Python 3.10 or newer, and enough RAM and storage for the checkpoint. Who should test it?Teams evaluating a supported MoE for coding agents, private automation, research or sustained local inference. Who should wait?Apple Silicon or AMD users, teams needing mature multi-user operations, and buyers without task-specific model and cost evaluations. What is FreeToken AI? FreeToken is an edge-native serving runtime from FlashML. The official FreeToken GitHub repository describes support for consumer laptops, gaming desktops and workstation GPUs, plus OpenAI-compatible and Anthropic-compatible endpoints for coding and tool-using agents. The code is licensed under Apache 2.0. The project is designed around sparse MoE models. An MoE checkpoint contains many expert networks, but its router activates only a small subset for each token. The arithmetic required for one token can therefore be much smaller than the full parameter count. The hard part is memory: all experts still have to live somewhere, even when most are inactive. FreeToken treats the entire computer as one inference system. Non-expert weights and frequently used experts stay on the GPU. The complete expert pool stays in host RAM. Cache misses are either moved over PCIe and executed on the GPU or computed where they already reside on the CPU. How does FreeToken run a model larger than GPU VRAM? The FreeToken research paper describes five mechanisms that work together: A shared GPU expert cache: FreeToken tracks recently used layer-expert pairs with an LRU policy instead of fixing every expert placement at startup. Bandwidth-adaptive execution: a measured policy divides cache misses between PCIe transfer plus GPU compute and direct CPU compute. The split is calibrated for the actual machine. Pipelined prefill: two layer buffers overlap expert movement with GPU computation so long prompts do not expose every transfer as idle time. Semantic state anchors: checkpoints at tool calls, thinking blocks and conversation boundaries let an agent reuse more context after its history is edited. Elastic memory pools: the runtime can rebalance VRAM between the expert cache and KV cache as the context grows, without reloading the host-resident expert pool. This architecture is different from claiming that a 284B or 753B checkpoint fits inside a small GPU. It does not. The full model still requires host memory or storage. FreeToken tries to keep the active path fast by placing the right work on the GPU, CPU and interconnect at each phase. What are the FreeToken hardware requirements? The current FreeToken installation requirements specify Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13 and Python 3.10 or newer. CUDA kernels compile on first use, so the CUDA 13 toolkit and nvcc must be available. Those are software minimums, not a promise that a chosen model will fit. Capacity planning also needs: System RAM: enough for the", "articleSection": "AI Infrastructure", "author": { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/" }, "citation": [ { "@type": "WebPage", "name": "official FreeToken GitHub repository", "url": "https://github.com/FlashML-org/FreeToken" }, { "@type": "WebPage", "name": "FreeToken research paper", "url": "https://arxiv.org/abs/2608.16157" }, { "@type": "WebPage", "name": "FreeToken installation requirements", "url": "https://github.com/FlashML-org/FreeToken/blob/main/docs/install.md" }, { "@type": "WebPage", "name": "official supported-model list", "url": "https://github.com/FlashML-org/FreeToken/blob/main/docs/models.md" }, { "@type": "WebPage", "name": "official FreeToken quick start", "url": "https://github.com/FlashML-org/FreeToken/blob/main/docs/quickstart.md" } ], "dateModified": "2026-08-25", "datePublished": "2026-08-25", "description": "FreeToken is an Apache-2.0 inference engine from FlashML for running large Mixture-of-Experts models across an NVIDIA GPU, CPU, system RAM and PCIe link. Its authors report 39.3 tokens per second for Qwen3.6-35B-A3B on an 8 GB RTX 4060 laptop and 22 to 25 tokens per second for DeepSeek-V4-Flash on an RTX 5090, but these are project-paper results, not Wavect benchmarks. The current CLI requires Linux x86_64, an NVIDIA GPU, driver r580 or newer, CUDA 13 and Python 3.10 or newer. FreeToken can remove a per-token API invoice, but not hardware, RAM, storage, electricity, operations or model-validation costs. Pilot it when a supported MoE model passes your task eval and local control or sustained usage justifies the infrastructure.", "headline": "FreeToken AI Review: Run Frontier MoE Models on Consumer GPUs?", "image": "https://wavect.io/img/blog/headers/header_freetoken-ai-inference-engine-review.svg", "inLanguage": "en", "keywords": "FreeToken, Local AI, AI Infrastructure", "mainEntityOfPage": { "@id": "https://wavect.io/blog/freetoken-ai-inference-engine-review/", "@type": "WebPage" }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/blog/freetoken-ai-inference-engine-review/", "wordCount": 2681 } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "item": "https://wavect.io/", "name": "Home", "position": 1 }, { "@type": "ListItem", "item": "https://wavect.io/blog/overview/", "name": "Blog overview", "position": 2 }, { "@type": "ListItem", "item": "https://wavect.io/blog/topics/ai-agents/", "name": "AI and agents", "position": 3 }, { "@type": "ListItem", "item": "https://wavect.io/blog/clusters/models-infrastructure/", "name": "Models and infrastructure", "position": 4 }, { "@type": "ListItem", "item": "https://wavect.io/blog/freetoken-ai-inference-engine-review/", "name": "FreeToken AI Review: Hardware, Benchmarks & Setup | ", "position": 5 } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "FreeToken is an open-source inference engine from FlashML for serving large Mixture-of-Experts models across an NVIDIA GPU, CPU, system RAM and PCIe connection. It exposes OpenAI-compatible and Anthropic-compatible APIs." }, "name": "What is FreeToken AI?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "No. It runs open-weight models on hardware you control, so local requests do not create a provider token invoice. You still pay for hardware, RAM, storage, electricity and operations." }, "name": "Does FreeToken give you free AI tokens?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The authors report Qwen3.6-35B-A3B NVFP4 at 39.3 decode tokens per second on an RTX 4060 laptop with 8 GB VRAM and 32 GiB of system memory. Treat that as a tested project configuration and reproduce it on your exact machine." }, "name": "Can FreeToken run on 8 GB VRAM?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The current CLI installation guide requires Linux x86_64 and an NVIDIA GPU with driver r580 or newer plus CUDA 13. The project also advertises a Windows desktop app, but CLI and model compatibility should be checked against the current release." }, "name": "What operating systems and GPUs does FreeToken support?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "In the authors' tested MoE workloads, FreeToken reports higher decode throughput and lower tail time to first token than supported baselines, including Ollama in applicable cells. That does not make it faster for every model or workload. Benchmark the exact model, format, prompt and hardware." }, "name": "Is FreeToken faster than Ollama?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The current list includes selected DeepSeek-V4, GLM, Qwen MoE, gpt-oss, Gemma-4, MiniMax and Muse-Glimmer checkpoints. Compatibility is architecture and format specific, and multimodal checkpoints are currently text-only." }, "name": "Which models does FreeToken support?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "It is promising enough for a controlled pilot. Production still requires task evaluations, pinned artifacts, authentication, authorization, observability, concurrency tests, recovery, fallback and a measured total-cost comparison." }, "name": "Is FreeToken ready for production?" } ] } ```