--- title: "Shared KV Cache Cut LLM Inference Latency 14x" canonical: https://wavect.io/blog/shared-kv-cache-llm-inference-latency/ language: en description: "How LMCache's shared KV cache cut mean TTFT from 3.98s to 0.29s on Qwen3-235B, with the same GPUs and memory, and how to tell if it helps your inference." image: "https://wavect.io/img/blog/headers/header_shared-kv-cache-llm-inference-latency.png" --- [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 10 min read · 23 Jul 2026 [**Next**](/blog/self-hosting-llms-eu-cost/) # How a Shared KV Cache Cut LLM Inference Latency 14x, No New GPUs TL;DR To serve a big model fast you run one process per GPU, and by default each of those data-parallel ranks keeps a private KV cache. So when a later turn of a conversation lands on a different rank, the model re-prefills the whole history even though another rank already computed it: the state exists on the same server, in the wrong process. LMCache's Multi-Process mode moves the cache out of the individual processes into one shared host-side layer every process can read. On an 8x H100 server running Qwen3-235B, that cut mean time-to-first-token from 3.98s to 0.29s (about 14x) and tripled decoding speed, with the same model, hardware, and 400GB total cache. The lesson is bigger than one library: a lot of inference latency is your system recomputing answers it already holds, discarded because the cache is split by process instead of shared by node. It pays off in proportion to how much your requests overlap: large for chat, agents, and long shared prompts, small for unique one-shot prompts. Measure reuse and price the GPU hours saved before buying faster hardware. **The short version:** an open-source KV cache layer called LMCache cut mean time-to-first-token by roughly 14x on a large mixture-of-experts model, and it did it without a faster GPU, a smaller model, or more memory. The only thing that changed was where the cache lived. Eight serving processes that each hoarded a private cache became eight processes sharing one. That is the whole lesson, and it is bigger than one library. Most inference optimization is sold as buying faster hardware. A lot of the latency you pay for is your own system recomputing an answer it already has, in a cache that belongs to the wrong process. This article explains what changed, what the numbers do and do not prove, and how to tell whether the same architecture would help your own [LLM](/glossary/llm/) serving. ## The expensive default: every rank hoards its own cache When a model serves a conversation, it does not reread the whole history on every turn. It stores the intermediate attention state for the tokens it has already seen. That store is the KV cache (key-value cache), and reusing it is why the second turn of a chat is far faster than the first. Recomputing it is called prefill, and prefill is the expensive part of answering a long prompt. Here is the trap. In the benchmark topology, vLLM used eight data-parallel ranks with automatic expert parallelism: attention was replicated across GPUs while the mixture-of-experts layers were sharded. Each rank ran in its own process and kept a private KV cache. The caches never talked to each other. LMCache's own benchmark makes the cost concrete. Qwen3-235B-A22B ran across eight H100 GPUs. Each rank got 50 GB of CPU memory for KV cache offloading, so the server held 400 GB in aggregate. On paper that is a large cache. In practice it was eight isolated 50 GB caches, not one shared 400 GB cache. Now picture a real multi-turn conversation. Turn one is routed to rank 3, which computes and caches the context. Turn two of the same conversation gets routed to rank 6. Rank 6 cannot see what rank 3 cached. The exact state it needs is sitting in host memory on the same physical server, but it belongs to another process, so the model does the only thing it can: it prefills the whole conversation again from scratch. You paid for that computation once already. You pay for it again on every turn that lands on a different rank. ## What LMCache changed: one cache instead of eight [LMCache](https://github.com/LMCache/LMCache) is an open-source, Apache-2.0 KV cache layer for inference engines like [vLLM](https://github.com/vllm-project/vllm) and SGLang. Its job is to move KV cache out of scarce GPU memory into a tiered hierarchy of CPU memory, local disk, and remote stores, and to let cache be reused across requests, sessions, and engine instances. The change that produced the 14x number is architectural, not algorithmic. In the conventional in-process setup, a separate cache library is embedded inside every serving process, which is why the caches are isolated. LMCache's Multi-Process mode pulls the cache out of the individual processes and runs it as one standalone service. Every serving process on the node registers with it and reads from the same host-side pool. So the same 400 GB stops being eight walled-off buckets and becomes one shared layer. When turn two lands on rank 6, rank 6 asks the shared cache, finds the state rank 3 already computed, and skips the prefill. The model, the hardware, and the total cache capacity are all identical. The computation is simply no longer thrown away. ## The benchmark numbers These figures come from LMCache's published multi-turn conversation benchmark on Qwen3-235B-A22B-Instruct-2507-FP8, running on eight NVIDIA H100 80GB GPUs with vLLM 0.18.1 and LMCache 0.4.3-dev. They compare the conventional in-process offload against Multi-Process mode. They are author-reported results on one hardware and workload; treat them as a strong directional signal, not a guarantee for your traffic. | Metric | In-process (isolated caches) | Multi-Process (shared cache) | Change | | --- | --- | --- | --- | | Mean time-to-first-token | 3.98 s | 0.29 s | About 14x faster | | P99 time-to-first-token | 13.55 s | 1.30 s | About 10x faster | | Mean decoding speed | 9.81 tokens/s | 37.47 tokens/s | About 3.8x faster | | P99 decoding speed | 34.27 tokens/s | 45.14 tokens/s | About 1.3x faster | Read the shape, not just the headline. The gain is largest where the isolated caches hurt most: cold turns that would otherwise re-prefill a long history. Mean TTFT dropping from about four seconds to under a third of a second is the difference between a chat that feels sluggish and one that feels instant. The P99 improvement matters even more for a product, because the tail is what your slowest users actually experience. ## Why this matters even if you never touch LMCache The point is not that you should install one specific library. The point is where the waste was hiding. The server had already done the work. It had the memory to keep the result. It threw the result away because the cache was partitioned by process instead of shared by node. That pattern shows up all over self-hosted inference: computation that was already paid for, discarded because of how the system was assembled rather than how much hardware it has. Before anyone approves a bigger GPU order, the cheaper question is whether the current boxes are recomputing answers they already hold. Buying capacity is the expensive way to solve a problem that reuse often solves for free. We make the same argument about spend in our guide to [reducing LLM token costs](/blog/reduce-llm-token-costs-2026/), and about hardware in the [local models versus APIs break-even analysis](/blog/local-models-vs-apis-break-even-eu-2026/). If reuse is already exhausted and one model serves a stable, latency-critical workload, specialization becomes the next question. Our [Taalas HC1 hardwired LLM ASIC review](/blog/taalas-hc1-llm-asic-review/) explains what 17,000 tokens/s proves and what a buyer must validate before trading model flexibility for speed. ## Where a shared KV cache helps, and where it does not Cache reuse pays off in direct proportion to how much your requests overlap. It is a lever, not a law, so be honest about your own traffic before you expect a 14x. | Workload | How much shared cache helps | Why | | --- | --- | --- | | Multi-turn chat and agents | Large | Every later turn re-sends the same growing history; reuse skips the repeated prefill, exactly the benchmark case. | | Long shared system prompts or RAG context | Large | A big fixed prefix repeated across many requests is prefilled once and reused, instead of once per request per rank. | | Many short, unique, one-shot prompts | Small | Little overlap between requests means little cache to reuse; the win shrinks toward the cost of running the cache layer. | | Single-process, single-GPU serving | None from this change | There are no separate ranks to unify; in-process prefix caching already covers you. | Sharing a cache is not free either. A standalone cache service adds an inter-process coordination path, a moving part to operate and monitor, and a lookup that costs a little latency on a miss. When requests barely overlap, that overhead can outweigh the saved prefill. The architecture is a strong default for conversational and long-context serving, not a universal upgrade. ## Does this actually cut your inference bill? Faster is not the same as cheaper. A latency win becomes a cost win only when it removes something on the invoice or lets you serve more from the same box. Price the change against what you actually run: `monthly benefit = GPU hours no longer spent re-prefilling + higher throughput per existing GPU + deferred hardware purchase - cache service overhead and ops time` Reused prefill frees GPU time, and freed GPU time is either a smaller cluster or more traffic served on the current one. The clean win is crossing a threshold: a latency target you can now meet without adding a node, a concurrency level the same GPUs can now sustain, or a planned hardware upgrade you can postpone. If your requests rarely overlap, the honest answer is that the saving is small and your effort is better spent elsewhere. For the full own-versus-rent picture that this feeds into, work through [what self-hosting LLMs in the EU actually costs](/blog/self-hosting-llms-eu-cost/). ## A short evaluation before you rewire serving 1. **Measure your reuse.** Before touching architecture, log how often requests share a prefix or continue a conversation. Overlap is the entire size of the prize; if it is low, stop here. 2. **Baseline the tail, not just the mean.** Record P50 and P99 TTFT and decoding speed on your real traffic. The benchmark's biggest gains were on the tail, and the tail is what users feel. 3. **Confirm the topology matches.** The 14x came from unifying multiple data-parallel ranks on one node. If you run one process per GPU today, you have the fragmentation this fixes; if you run a single process, you do not. 4. **Pilot on a shadow or slice.** Route a copy or a fraction of traffic through the shared-cache setup and compare against the baseline on the same hardware and workload. 5. **Price it, do not just time it.** Convert the measured latency and throughput change into GPU hours, node counts, or deferred purchases, minus the cost of operating the cache service. ## Questions to ask before you adopt it - What fraction of our requests actually share a prefix or continue an existing conversation? - Are we running multiple data-parallel ranks per node, so isolated caches are even a problem for us? - What are our current P50 and P99 TTFT and decoding numbers on production traffic, not a benchmark? - What does the standalone cache service cost us in latency on a miss, in operational surface, and in failure modes? - Does the latency gain convert into fewer GPUs, more throughput, or a deferred purchase, and by how much? - What is the maturity, license, and maintenance state of the cache layer, and can we operate or fork it if needed? ## Sources and claim boundaries The architecture, the 400 GB fragmentation example, and all benchmark numbers come from LMCache's write-up, ["LMCache's New Architecture Boosts MoE Inference Performance by 10x"](https://blog.lmcache.ai/en/2026/04/03/lmcaches-new-architecture-boosts-moe-inference-performance-by-10x/), and the project's [GitHub repository](https://github.com/LMCache/LMCache) and [documentation](https://docs.lmcache.ai/). The reported latency and throughput figures belong to their authors and test system, an 8x H100 server running vLLM 0.18.1 and LMCache 0.4.3-dev on Qwen3-235B-A22B; Wavect did not reproduce them. Facts were checked on 23 July 2026. ## Frequently Asked Questions ### What is a KV cache in LLM inference? The KV cache (key-value cache) stores the intermediate attention state a model computes for tokens it has already processed. Reusing it means the model does not recompute the whole prompt on every step, which is why later turns of a conversation are much faster than the first. Recomputing that state is called prefill. ### Why did sharing the KV cache make inference 14x faster? In the conventional setup each data-parallel rank keeps a private cache, so when a later turn is routed to a different rank the model re-prefills the whole conversation even though another rank already computed it. A shared cache lets any rank reuse that work, cutting mean time-to-first-token from 3.98 s to 0.29 s in LMCache's benchmark, with the same model, hardware, and total cache capacity. ### What is LMCache? LMCache is an open-source, Apache-2.0 KV cache layer for inference engines such as vLLM and SGLang. It moves KV cache out of GPU memory into a tiered hierarchy of CPU memory, disk, and remote stores, and lets cache be reused across requests, sessions, and engine instances. Its Multi-Process mode is the shared-cache architecture behind the results in this article. ### Will a shared KV cache speed up my workload? It helps in proportion to how much your requests overlap. Multi-turn chat, agents, and long shared system prompts or RAG context benefit most. Many short unique one-shot prompts benefit little, and single-process single-GPU serving gets nothing from this specific change because there are no separate ranks to unify. Measure your prefix reuse before expecting a large gain. ### Does faster inference automatically mean cheaper inference? No. A latency win becomes a cost win only when it removes GPU hours you were paying for, lets the same GPUs serve more traffic, or defers a hardware purchase, after subtracting the overhead of running the cache service. Price the change against your real bill rather than assuming speed equals savings. ## Final thoughts The most quoted line from this result is that inference got 14x faster with no new GPUs. The more useful line is why: the system was already doing the work and already had the memory to keep it, then discarded the result because the cache was split by process instead of shared by node. That is an architecture problem wearing a hardware problem's clothing. Before you approve faster GPUs, measure how often your serving stack recomputes answers it already holds. A shared KV cache is a strong default for conversational and long-context workloads, a weak one for unique short prompts, and worth pricing either way. The headline is the 14x. The lesson is to reuse the computation you have already paid for. ## You may also like.. [**Distributed LLM inference across multiple computers** How to spread one model over several machines when a single box cannot hold it, and what it costs in latency.](/blog/mesh-llm-distributed-inference-multiple-computers/) [**Cut RAG vector memory 16x** Another reuse-the-memory-you-have win: training-free quantization shrinks a retrieval corpus 16x without a bigger machine.](/blog/rag-vector-memory-quantization/) Models and infrastructure ## Continue through this cluster [Start with the cornerstone**Self-Hosting LLMs in the EU: When Open Weights Actually Pay Off**](/blog/self-hosting-llms-eu-cost/) - [Does Claude Watermark Text? The 2026 API Answer](/blog/claude-text-watermark-api-2026/) - [OpenKB Review: Knowledge Compiler vs RAG](/blog/openkb-review-vs-rag/) - [Unsloth Desktop Review: A Private Local AI Workstation?](/blog/unsloth-desktop-local-ai-workstation-review/) - [NeMo Switchyard 0.2: Agent Model Routing Without Training?](/blog/nemo-switchyard-model-router/) - [Firecrawl AnyDoc Review: 14 Formats to Markdown](/blog/firecrawl-anydoc-review/) Inbox, without the noise ## Follow the work that matters to you Get a short email when we publish something new. Follow the whole blog or only the problems you care about. [**Back**](/blog/overview/) [![Kevin Riedl](/img/team/kevin.webp)](/team/kevin-riedl/) [Kevin Riedl](/team/kevin-riedl/) https://linkedin.com/in/wsdt 10 min read · 23 Jul 2026 [**Next**](/blog/self-hosting-llms-eu-cost/) New posts by email × × Get new posts by email A short email when we publish. Free, no tracking. ## Structured Data ```json { "@context": "https://schema.org", "@graph": [ { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ], "employee": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "founder": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "legalRepresentative": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "name": "Wavect GmbH", "subjectOf": { "@id": "https://wavect.io/verified-claims.json#dataset", "@type": "Dataset", "creator": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "description": "A machine-readable registry of quantitative and qualitative claims published by Wavect, with review dates, localized page appearances and public third-party citations where available.", "inLanguage": "en", "isAccessibleForFree": true, "license": "https://creativecommons.org/licenses/by/4.0/", "name": "Wavect verified publication claims", "url": "https://wavect.io/verified-claims.json" }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/#website", "@type": "WebSite", "inLanguage": [ "en", "de", "es", "zh" ], "name": "Wavect", "potentialAction": { "@type": "SearchAction", "query-input": "required name=search_term_string", "target": { "@type": "EntryPoint", "urlTemplate": "https://wavect.io/search/?q={search_term_string}" } }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/blog/shared-kv-cache-llm-inference-latency/#webpage", "@type": "WebPage", "dateModified": "2026-07-23", "inLanguage": "en", "isPartOf": { "@id": "https://wavect.io/#website", "@type": "WebSite" }, "lastReviewed": "2026-07-23", "url": "https://wavect.io/blog/shared-kv-cache-llm-inference-latency/" } ] } ``` ```json { "@context": "https://schema.org", "@type": "BlogPosting", "abstract": "To serve a big model fast you run one process per GPU, and by default each of those data-parallel ranks keeps a private KV cache. So when a later turn of a conversation lands on a different rank, the model re-prefills the whole history even though another rank already computed it: the state exists on the same server, in the wrong process. LMCache's Multi-Process mode moves the cache out of the individual processes into one shared host-side layer every process can read. On an 8x H100 server running Qwen3-235B, that cut mean time-to-first-token from 3.98s to 0.29s (about 14x) and tripled decoding speed, with the same model, hardware, and 400GB total cache. The lesson is bigger than one library: a lot of inference latency is your system recomputing answers it already holds, discarded because the cache is split by process instead of shared by node. It pays off in proportion to how much your requests overlap: large for chat, agents, and long shared prompts, small for unique one-shot prompts. Measure reuse and price the GPU hours saved before buying faster hardware.", "articleBody": " Blog overview/AI and agents/Models and infrastructure How a Shared KV Cache Cut LLM Inference Latency 14x, No New GPUs TL;DR To serve a big model fast you run one process per GPU, and by default each of those data-parallel ranks keeps a private KV cache. So when a later turn of a conversation lands on a different rank, the model re-prefills the whole history even though another rank already computed it: the state exists on the same server, in the wrong process. LMCache's Multi-Process mode moves the cache out of the individual processes into one shared host-side layer every process can read. On an 8x H100 server running Qwen3-235B, that cut mean time-to-first-token from 3.98s to 0.29s (about 14x) and tripled decoding speed, with the same model, hardware, and 400GB total cache. The lesson is bigger than one library: a lot of inference latency is your system recomputing answers it already holds, discarded because the cache is split by process instead of shared by node. It pays off in proportion to how much your requests overlap: large for chat, agents, and long shared prompts, small for unique one-shot prompts. Measure reuse and price the GPU hours saved before buying faster hardware. The short version: an open-source KV cache layer called LMCache cut mean time-to-first-token by roughly 14x on a large mixture-of-experts model, and it did it without a faster GPU, a smaller model, or more memory. The only thing that changed was where the cache lived. Eight serving processes that each hoarded a private cache became eight processes sharing one. That is the whole lesson, and it is bigger than one library. Most inference optimization is sold as buying faster hardware. A lot of the latency you pay for is your own system recomputing an answer it already has, in a cache that belongs to the wrong process. This article explains what changed, what the numbers do and do not prove, and how to tell whether the same architecture would help your own LLM serving. The expensive default: every rank hoards its own cache When a model serves a conversation, it does not reread the whole history on every turn. It stores the intermediate attention state for the tokens it has already seen. That store is the KV cache (key-value cache), and reusing it is why the second turn of a chat is far faster than the first. Recomputing it is called prefill, and prefill is the expensive part of answering a long prompt. Here is the trap. In the benchmark topology, vLLM used eight data-parallel ranks with automatic expert parallelism: attention was replicated across GPUs while the mixture-of-experts layers were sharded. Each rank ran in its own process and kept a private KV cache. The caches never talked to each other. LMCache's own benchmark makes the cost concrete. Qwen3-235B-A22B ran across eight H100 GPUs. Each rank got 50 GB of CPU memory for KV cache offloading, so the server held 400 GB in aggregate. On paper that is a large cache. In practice it was eight isolated 50 GB caches, not one shared 400 GB cache. Now picture a real multi-turn conversation. Turn one is routed to rank 3, which computes and caches the context. Turn two of the same conversation gets routed to rank 6. Rank 6 cannot see what rank 3 cached. The exact state it needs is sitting in host memory on the same physical server, but it belongs to another process, so the model does the only thing it can: it prefills the whole conversation again from scratch. You paid for that computation once already. You pay for it again on every turn that lands on a different rank. What LMCache changed: one cache instead of eight LMCache is an open-source, Apache-2.0 KV cache layer for inference engines like vLLM and SGLang. Its job is to move KV cache out of scarce GPU memory into a tiered hierarchy of CPU memory, local disk, and remote stores, and to let cache be reused across requests, sessions, and engine instances. The change that produced the 14x number is architectural, not algorithmic. In the conventional in-process setup, a separate cache library is embedded inside every serving process, which is why the caches are isolated. LMCache's Multi-Process mode pulls the cache out of the individual processes and runs it as one standalone service. Every serving process on the node registers with it and reads from the same host-side pool. So the same 400 GB stops being eight walled-off buckets and becomes one shared layer. When turn two lands on rank 6, rank 6 asks the shared cache, finds the state rank 3 already computed, and skips the prefill. The model, the hardware, and the total cache capacity are all identical. The computation is simply no longer thrown away. The benchmark numbers These figures come from LMCache's published multi-turn conversation benchmark on Qwen3-235B-A22B-Instruct-2507-FP8, running on eight NVIDIA H100 80GB GPUs with vLLM 0.18.1 and LMCache 0.4.3-dev. They compare the conventional in-process offload against Multi-Process mode. They are author-reported results on one", "articleSection": "Engineering", "author": { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/" }, "dateModified": "2026-07-23", "datePublished": "2026-07-23", "description": "To serve a big model fast you run one process per GPU, and by default each of those data-parallel ranks keeps a private KV cache. So when a later turn of a conversation lands on a different rank, the model re-prefills the whole history even though another rank already computed it: the state exists on the same server, in the wrong process. LMCache's Multi-Process mode moves the cache out of the individual processes into one shared host-side layer every process can read. On an 8x H100 server running Qwen3-235B, that cut mean time-to-first-token from 3.98s to 0.29s (about 14x) and tripled decoding speed, with the same model, hardware, and 400GB total cache. The lesson is bigger than one library: a lot of inference latency is your system recomputing answers it already holds, discarded because the cache is split by process instead of shared by node. It pays off in proportion to how much your requests overlap: large for chat, agents, and long shared prompts, small for unique one-shot prompts. Measure reuse and price the GPU hours saved before buying faster hardware.", "headline": "Shared KV Cache Cut LLM Inference Latency 14x, With No New GPUs", "image": "https://wavect.io/img/blog/headers/header_shared-kv-cache-llm-inference-latency.svg", "inLanguage": "en", "mainEntityOfPage": { "@id": "https://wavect.io/blog/shared-kv-cache-llm-inference-latency/", "@type": "WebPage" }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/blog/shared-kv-cache-llm-inference-latency/", "wordCount": 2613 } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "item": "https://wavect.io/", "name": "Home", "position": 1 }, { "@type": "ListItem", "item": "https://wavect.io/blog/overview/", "name": "Blog overview", "position": 2 }, { "@type": "ListItem", "item": "https://wavect.io/blog/topics/ai-agents/", "name": "AI and agents", "position": 3 }, { "@type": "ListItem", "item": "https://wavect.io/blog/clusters/models-infrastructure/", "name": "Models and infrastructure", "position": 4 }, { "@type": "ListItem", "item": "https://wavect.io/blog/shared-kv-cache-llm-inference-latency/", "name": "Shared KV Cache Cut LLM Inference Latency 14x | ", "position": 5 } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "The KV cache (key-value cache) stores the intermediate attention state a model computes for tokens it has already processed. Reusing it means the model does not recompute the whole prompt on every step, which is why later turns of a conversation are much faster than the first. Recomputing that state is called prefill." }, "name": "What is a KV cache in LLM inference?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "In the conventional setup each data-parallel rank keeps a private cache, so when a later turn is routed to a different rank the model re-prefills the whole conversation even though another rank already computed it. A shared cache lets any rank reuse that work, cutting mean time-to-first-token from 3.98 s to 0.29 s in LMCache's benchmark, with the same model, hardware, and total cache capacity." }, "name": "Why did sharing the KV cache make inference 14x faster?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "LMCache is an open-source, Apache-2.0 KV cache layer for inference engines such as vLLM and SGLang. It moves KV cache out of GPU memory into a tiered hierarchy of CPU memory, disk, and remote stores, and lets cache be reused across requests, sessions, and engine instances. Its Multi-Process mode is the shared-cache architecture behind the results in this article." }, "name": "What is LMCache?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "It helps in proportion to how much your requests overlap. Multi-turn chat, agents, and long shared system prompts or RAG context benefit most. Many short unique one-shot prompts benefit little, and single-process single-GPU serving gets nothing from this specific change because there are no separate ranks to unify. Measure your prefix reuse before expecting a large gain." }, "name": "Will a shared KV cache speed up my workload?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "No. A latency win becomes a cost win only when it removes GPU hours you were paying for, lets the same GPUs serve more traffic, or defers a hardware purchase, after subtracting the overhead of running the cache service. Price the change against your real bill rather than assuming speed equals savings." }, "name": "Does faster inference automatically mean cheaper inference?" } ] } ```