Ilmu Komputer & AI editorial
Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries
The core problem
Replicated large-model inference services typically route each request to one of several identical replicas. When a request arrives at a replica that has not seen its prefix, the replica must re-run prefill over the entire prompt before it can emit the first content token. For long-context workloads this prefill dominates latency: at 128k and 256k input lengths, the cost is measured in tens of seconds. Shared host-memory caching is the natural remedy. If the key/value (KV) state produced by one replica can be stored in a pool that other replicas can read, a request that moves between replicas can skip prefill entirely.
The premise of this study is that shared caching is useful only if two independent conditions hold simultaneously. First, the state transfer must be *correct*: the KV pages written by one replica must be readable by another without violating the ordering guarantees of the underlying accelerator runtime. Second, the transfer must be *worth it*: the request must actually move between replicas often enough that lost prefix locality does not erase the benefit. The paper examines both conditions on a concrete deployment: two single-GPU 27B vLLM replicas sharing a 256 GiB
Innovation
The correctness results are unambiguous. Controlled byte tests fail when an imposed delay is applied to the raw-pointer fallback path, and pass once the dependency on the current CUDA stream is restored. The existing mixed allocator supplies a working deployment path, and full-pool allocation checks plus service regression complete the validation. In other words, the failure is not a capacity or accounting problem in the pool; it is an ordering problem in the transfer path.
The performance results are equally clear at the single-request level. In the four-block OFF-ON-ON-OFF comparison, median cross-replica time to first content token falls from **31.715 s to 0.605 s at 128k input**, and from **92.047 s to 0.790 s at 256k input**. These are reductions of roughly 98% and 99% respectively. The absolute saving grows with context length because prefill cost grows with context length, while the cached path replaces prefill with a page transfer whose cost is comparatively small.
The session-level results are more nuanced. Six-turn synthetic sessions that alternate replicas improve by approximately **35%** and **45%** at initial contexts of **32k** and **128k**. The improvement is small
Why it matters
The study's central claim is that shared KV caching is governed by two independent gates: correctness of state transfer and preservation of prefix locality. The correctness gate is binary and unforgiving. A raw-pointer fallback that omits the CUDA stream dependency can pass casual testing and fail under imposed delay, which is exactly the kind of latent ordering bug that produces intermittent, hard-to-reproduce corruption in production. The remedy is not a new allocator but an explicit ordering edge; the existing mixed allocator already provides a working deployment path.
The performance gate is conditional. The headline numbers — 31.715 s to 0.605 s at 128k and 92.047 s to 0.790 s at 256k — describe the best case: a single long-context request that moves to a replica which has the prefix cached. The session results, 35% and 45% at 32k and 128k, describe a more realistic case in which only some turns cross replicas. The fixed-placement control showing little benefit confirms that the mechanism, not a generic serving improvement, drives the gains.
The practical implication is a routing policy question. Shared caching pays off when the router sends a request to a replica that holds its prefix, or when sessions alternate replicas often enough that prefill would otherwise be repeated. If the router pins sessions to a single replica, the pool is idle and the benefit vanishes. The engineering recommendation is therefore twofold: validate the transfer path with controlled delay tests that force stream-ordering bugs to surface, and instrument cross-replica routing rates before expecting session-level gains.
A useful mental model of the deployment is shown below. The router decides whether a request's prefix is present in the shared pool; if it is, the replica loads KV pages instead of re-running prefill.
The remaining open question is how far the boundary extends. The study covers two replicas, one pool size, and two context regimes. Whether the same locality conditions hold at higher replica counts, with heterogeneous prefix distributions, or under admission control that evicts pages from the pool is left to future work. What the case study establishes is that the correctness gate must be closed first, and that the performance gate is a property of routing, not of the cache alone.
Who should read this
Opening member content…