LMCache Reported 13.7x Lower Mean TTFT with a Shared KV Cache

The short version: LMCache reported mean time-to-first-token (TTFT) falling from 3.98 seconds to 0.29 seconds, a 13.7x ratio, in one multi-turn benchmark on a large mixture-of-experts model. The project kept the model, eight H100 GPUs, and aggregate 400 GB host-cache capacity fixed while changing isolated per-rank pools into one shared host-side pool.

This is a useful architecture result, not a universal 13.7x promise. The workload, software versions, request routing, cache hits, and system configuration came from the LMCache team, and Wavect did not reproduce the test. This article explains the reported mechanism, claim boundaries, and evidence required to decide whether the same pattern helps your own LLM serving.

The benchmark problem: each rank had an isolated cache

During autoregressive inference, a KV cache stores intermediate attention keys and values for tokens already processed. When a later request has a reusable exact prefix and the serving stack can access compatible cached blocks, it may avoid repeating part of prompt prefill. The benefit depends on prompt length, model, hardware, batching, cache hit rate, transfer path, and concurrency.

Here is the trap. In the benchmark topology, vLLM used eight data-parallel ranks with automatic expert parallelism: attention was replicated across GPUs while the mixture-of-experts layers were sharded. Each rank ran in its own process and kept a private KV cache. The caches never talked to each other.

LMCache's own benchmark makes the cost concrete. Qwen3-235B-A22B ran across eight H100 GPUs. Each rank got 50 GB of CPU memory for KV cache offloading, so the server held 400 GB in aggregate. On paper that is a large cache. In practice it was eight isolated 50 GB caches, not one shared 400 GB cache.

Consider a later turn whose exact reusable prefix was cached by rank 3 but is routed to rank 6. In the benchmark's isolated configuration, rank 6 could not access rank 3's host-side cache. It therefore could not reuse that cached prefix through this path and had to perform the applicable prefill work again.

What LMCache changed: one cache instead of eight

LMCache is an open-source, Apache-2.0 KV cache layer for inference engines like vLLM and SGLang. Its job is to move KV cache out of scarce GPU memory into a tiered hierarchy of CPU memory, local disk, and remote stores, and to let cache be reused across requests, sessions, and engine instances.

The change behind the reported 13.7x mean-TTFT ratio is architectural, not a change to the model algorithm. In the benchmark's conventional in-process setup, a separate cache library was embedded inside each serving process, leaving the host-side pools isolated. LMCache's Multi-Process mode moved cache management into a standalone service that registered the serving processes against one shared host-side pool.

So the same aggregate 400 GB changes from eight 50 GB process-local pools into one shared layer. On a compatible cache hit, another registered process can retrieve reusable KV blocks instead of repeating the corresponding prefill work. The benchmark kept the model and GPU hardware fixed; it changed the cache architecture and related serving configuration.

The benchmark numbers

These figures come from LMCache's published multi-turn conversation benchmark on Qwen3-235B-A22B-Instruct-2507-FP8, running on eight NVIDIA H100 80GB GPUs with vLLM 0.18.1 and LMCache 0.4.3-dev. They compare vLLM in-process offload with LMCache Multi-Process mode at a stated two requests per second for 120 seconds. They are project-author results for one system and workload, not an independent benchmark or a guarantee for your traffic.

Read the shape and the limits, not just the headline. The published workload was designed to create multi-turn prefix reuse, so it directly exercised the fragmentation problem. Mean and P99 TTFT both improved in that test. The reported decoding-speed statistic also changed, but the post does not supply enough independent runs, uncertainty analysis, raw traces, or alternate workloads to generalize the ratios.

Why this matters even if you never touch LMCache

The point is not that you should install one specific library. The point is where the waste was hiding. The server had already done the work. It had the memory to keep the result. It threw the result away because the cache was partitioned by process instead of shared by node.

Similar waste can occur in self-hosted inference when reusable computation is inaccessible across process or node boundaries. Before approving more GPU capacity, measure repeated prefixes, cache hits, prefill time, GPU utilization, and routing. Reuse is not free: it adds memory, transfer, lookup, coordination, isolation, correctness, and operational costs. We apply the same complete-cost discipline in our guide to reducing LLM token costs and the local models versus APIs break-even analysis.

If reuse is already exhausted and one model serves a stable, latency-critical workload, specialization becomes the next question. Our Taalas HC1 hardwired LLM ASIC review explains what 17,000 tokens/s proves and what a buyer must validate before trading model flexibility for speed.

Where a shared KV cache helps, and where it does not

Request overlap creates an opportunity for cache reuse, but overlap alone does not determine the result. Compatible prefixes, routing, eviction, transfer cost, cache capacity, concurrency, and the baseline implementation all affect the measured benefit.

A standalone cache service adds an inter-process coordination path, another component to secure and monitor, capacity and eviction decisions, and lookup or transfer work on hits and misses. When compatible reuse is low or transfer is expensive, overhead can outweigh saved prefill. Treat shared caching as a workload-dependent option, not a default for every conversational or long-context service.

Does this actually cut your inference bill?

Faster is not the same as cheaper. A latency win becomes a cost win only when it removes something on the invoice or lets you serve more from the same box. Price the change against what you actually run:

estimated monthly benefit = avoided compute and capacity cost - cache infrastructure, transfer, engineering, security, and operations cost

Reused prefill frees GPU time, and freed GPU time is either a smaller cluster or more traffic served on the current one. The clean win is crossing a threshold: a latency target you can now meet without adding a node, a concurrency level the same GPUs can now sustain, or a planned hardware upgrade you can postpone. If your requests rarely overlap, the honest answer is that the saving is small and your effort is better spent elsewhere. For the full own-versus-rent picture that this feeds into, work through what self-hosting LLMs in the EU actually costs.

A short evaluation before you rewire serving

1. Measure your reuse. Before touching architecture, log how often requests share a prefix or continue a conversation. Overlap is the entire size of the prize; if it is low, stop here.
2. Baseline the tail, not just the mean. Record P50 and P99 TTFT and decoding speed on your real traffic. The benchmark's biggest gains were on the tail, and the tail is what users feel.
3. Confirm the topology matches. The 13.7x mean-TTFT ratio came from unifying eight data-parallel rank caches on one node. Check whether your routing and cache architecture create the same fragmentation before assuming the mechanism applies.
4. Pilot on a shadow or slice. Route a copy or a fraction of traffic through the shared-cache setup and compare against the baseline on the same hardware and workload.
5. Price it, do not just time it. Convert the measured latency and throughput change into GPU hours, node counts, or deferred purchases, minus the cost of operating the cache service.

Questions to ask before you adopt it

• What fraction of our requests actually share a prefix or continue an existing conversation?
• Are we running multiple data-parallel ranks per node, so isolated caches are even a problem for us?
• What are our current P50 and P99 TTFT and decoding numbers on production traffic, not a benchmark?
• What does the standalone cache service cost us in latency on a miss, in operational surface, and in failure modes?
• Does the latency gain convert into fewer GPUs, more throughput, or a deferred purchase, and by how much?
• What is the maturity, license, and maintenance state of the cache layer, and can we operate or fork it if needed?

Sources and claim boundaries

The architecture, 400 GB fragmentation example, and benchmark numbers come from LMCache's Multi-Process benchmark write-up, the project's GitHub repository, and its documentation. The reported latency and throughput figures belong to the project authors and their test system, an 8x H100 server running vLLM 0.18.1 and LMCache 0.4.3-dev on Qwen3-235B-A22B. Wavect did not reproduce them. Facts were rechecked on 2 September 2026.