How a Shared KV Cache Cut LLM Inference Latency 14x, No New GPUs

The short version: an open-source KV cache layer called LMCache cut mean time-to-first-token by roughly 14x on a large mixture-of-experts model, and it did it without a faster GPU, a smaller model, or more memory. The only thing that changed was where the cache lived. Eight serving processes that each hoarded a private cache became eight processes sharing one.

That is the whole lesson, and it is bigger than one library. Most inference optimization is sold as buying faster hardware. A lot of the latency you pay for is your own system recomputing an answer it already has, in a cache that belongs to the wrong process. This article explains what changed, what the numbers do and do not prove, and how to tell whether the same architecture would help your own LLM serving.

The expensive default: every rank hoards its own cache

When a model serves a conversation, it does not reread the whole history on every turn. It stores the intermediate attention state for the tokens it has already seen. That store is the KV cache (key-value cache), and reusing it is why the second turn of a chat is far faster than the first. Recomputing it is called prefill, and prefill is the expensive part of answering a long prompt.

Here is the trap. To serve a big model fast, you run several copies of it in parallel on one server, one per GPU, each in its own process. In the conventional setup, every one of those data-parallel ranks keeps its own private KV cache. The caches never talk to each other.

LMCache's own benchmark makes the cost concrete. Qwen3-235B-A22B ran across eight H100 GPUs. Each rank got 50 GB of CPU memory for KV cache offloading, so the server held 400 GB in aggregate. On paper that is a large cache. In practice it was eight isolated 50 GB caches, not one shared 400 GB cache.

Now picture a real multi-turn conversation. Turn one is routed to rank 3, which computes and caches the context. Turn two of the same conversation gets routed to rank 6. Rank 6 cannot see what rank 3 cached. The exact state it needs is sitting in host memory on the same physical server, but it belongs to another process, so the model does the only thing it can: it prefills the whole conversation again from scratch. You paid for that computation once already. You pay for it again on every turn that lands on a different rank.

What LMCache changed: one cache instead of eight

LMCache is an open-source, Apache-2.0 KV cache layer for inference engines like vLLM and SGLang. Its job is to move KV cache out of scarce GPU memory into a tiered hierarchy of CPU memory, local disk, and remote stores, and to let cache be reused across requests, sessions, and engine instances.

The change that produced the 14x number is architectural, not algorithmic. In the conventional in-process setup, a separate cache library is embedded inside every serving process, which is why the caches are isolated. LMCache's Multi-Process mode pulls the cache out of the individual processes and runs it as one standalone service. Every serving process on the node registers with it and reads from the same host-side pool.

So the same 400 GB stops being eight walled-off buckets and becomes one shared layer. When turn two lands on rank 6, rank 6 asks the shared cache, finds the state rank 3 already computed, and skips the prefill. The model, the hardware, and the total cache capacity are all identical. The computation is simply no longer thrown away.

The benchmark numbers

These figures come from LMCache's published multi-turn conversation benchmark on Qwen3-235B-A22B-Instruct-2507-FP8, running on eight NVIDIA H100 80GB GPUs with vLLM 0.18.1 and LMCache 0.4.3-dev. They compare the conventional in-process offload against Multi-Process mode. They are author-reported results on one hardware and workload; treat them as a strong directional signal, not a guarantee for your traffic.

Read the shape, not just the headline. The gain is largest where the isolated caches hurt most: cold turns that would otherwise re-prefill a long history. Mean TTFT dropping from about four seconds to under a third of a second is the difference between a chat that feels sluggish and one that feels instant. The P99 improvement matters even more for a product, because the tail is what your slowest users actually experience.

Why this matters even if you never touch LMCache

The point is not that you should install one specific library. The point is where the waste was hiding. The server had already done the work. It had the memory to keep the result. It threw the result away because the cache was partitioned by process instead of shared by node.

That pattern shows up all over self-hosted inference: computation that was already paid for, discarded because of how the system was assembled rather than how much hardware it has. Before anyone approves a bigger GPU order, the cheaper question is whether the current boxes are recomputing answers they already hold. Buying capacity is the expensive way to solve a problem that reuse often solves for free. We make the same argument about spend in our guide to reducing LLM token costs, and about hardware in the local models versus APIs break-even analysis.

Where a shared KV cache helps, and where it does not

Cache reuse pays off in direct proportion to how much your requests overlap. It is a lever, not a law, so be honest about your own traffic before you expect a 14x.

Sharing a cache is not free either. A standalone cache service adds a network hop, a moving part to operate and monitor, and a lookup that costs a little latency on a miss. When requests barely overlap, that overhead can outweigh the saved prefill. The architecture is a strong default for conversational and long-context serving, not a universal upgrade.

Does this actually cut your inference bill?

Faster is not the same as cheaper. A latency win becomes a cost win only when it removes something on the invoice or lets you serve more from the same box. Price the change against what you actually run:

monthly benefit = GPU hours no longer spent re-prefilling + higher throughput per existing GPU + deferred hardware purchase - cache service overhead and ops time

Reused prefill frees GPU time, and freed GPU time is either a smaller cluster or more traffic served on the current one. The clean win is crossing a threshold: a latency target you can now meet without adding a node, a concurrency level the same GPUs can now sustain, or a planned hardware upgrade you can postpone. If your requests rarely overlap, the honest answer is that the saving is small and your effort is better spent elsewhere. For the full own-versus-rent picture that this feeds into, work through what self-hosting LLMs in the EU actually costs.

A short evaluation before you rewire serving

1. Measure your reuse. Before touching architecture, log how often requests share a prefix or continue a conversation. Overlap is the entire size of the prize; if it is low, stop here.
2. Baseline the tail, not just the mean. Record P50 and P99 TTFT and decoding speed on your real traffic. The benchmark's biggest gains were on the tail, and the tail is what users feel.
3. Confirm the topology matches. The 14x came from unifying multiple data-parallel ranks on one node. If you run one process per GPU today, you have the fragmentation this fixes; if you run a single process, you do not.
4. Pilot on a shadow or slice. Route a copy or a fraction of traffic through the shared-cache setup and compare against the baseline on the same hardware and workload.
5. Price it, do not just time it. Convert the measured latency and throughput change into GPU hours, node counts, or deferred purchases, minus the cost of operating the cache service.

Questions to ask before you adopt it

• What fraction of our requests actually share a prefix or continue an existing conversation?
• Are we running multiple data-parallel ranks per node, so isolated caches are even a problem for us?
• What are our current P50 and P99 TTFT and decoding numbers on production traffic, not a benchmark?
• What does the standalone cache service cost us in latency on a miss, in operational surface, and in failure modes?
• Does the latency gain convert into fewer GPUs, more throughput, or a deferred purchase, and by how much?
• What is the maturity, license, and maintenance state of the cache layer, and can we operate or fork it if needed?

Sources and claim boundaries

The architecture, the 400 GB fragmentation example, and all benchmark numbers come from LMCache's write-up, "LMCache's New Architecture Boosts MoE Inference Performance by 10x", and the project's GitHub repository and documentation. The reported latency and throughput figures belong to their authors and test system, an 8x H100 server running vLLM 0.18.1 and LMCache 0.4.3-dev on Qwen3-235B-A22B; Wavect did not reproduce them. Facts were checked on 23 July 2026.