Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management

Large language model (LLM) inference increasingly spans models that differ in size, capability, price, and provider. This shift creates two costs for developers. One is the integration cost of choosing among and switching between many models. The other is the inference cost of rebuilding a KV cache when it is unavailable or incompatible with the selected model.

The paper defines and analyzes the **Unified AI Gateway** as a system setting for an edge-deployed AI traffic hub. It coordinates model routing, KV cache management, and compute placement across end devices, edge resources, and cloud model services. At request time, the gateway jointly selects a target model, an execution site, and a KV cache action under task-quality, latency, cost, and resource constraints. In parallel, background cache-management actions optimize KV cache placement, replication, retrieval, and lifecycle decisions for subsequent requests.

The work synthesizes existing evidence on KV cache reuse, compression, cross-model mapping, distributed storage, and transfer, and discusses the remaining challenges of integrating these capabilities into one system.

Across eight typical workload profiles, the workload-level analytical simulation reports TTFT speedups of $1.25\times$–$13.28\times$ and input-cost benefits of $1.20\times$–$6.16\times$. These ranges indicate that joint optimization of model routing, execution placement, and KV cache actions can yield substantial latency and cost improvements relative to baseline strategies that treat these decisions independently.

The results are derived from an analytical simulation rather than a deployed system, so they represent workload-level projections under the modeled constraints. The paper does not report per-profile breakdowns in the abstract; the ranges summarize the observed benefits across the eight profiles.