Ilmu Komputer & AI editorial
LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
The core problem
As large language model (LLM) inference shifts from single-tenant deployments to multi-tenant GPU clusters, co-batching has become the default mechanism for improving throughput. By packing requests from many tenants into a single forward pass, operators amortize kernel launch overhead and raise hardware utilization. However, this very efficiency obscures per-tenant resource usage: once requests are fused into a batch, the scheduler loses visibility into how much latency, memory bandwidth, and compute each individual request actually consumed.
This opacity limits control. Without an accurate per-request attribution primitive, providers cannot enforce fair-share policies, cannot bill tenants on true consumption, and cannot make fine-grained scheduling decisions such as preemption or admission control. The core requirement is therefore a **real-time, per-request attribution primitive** that is both accurate and light enough to execute inside the scheduling loop itself.
The authors present **LLMVisor**, a roofline-guided latency attribution model. LLMVisor captures the two dominant regimes of LLM inference — the memory-bound phase and the compute-bound phase — via a concise piecewis
Innovation
LLMVisor is evaluated against a token-count baseline across two model families and two GPU generations. The headline results are strong: LLMVisor attains **near-perfect R-squared** for latency attribution, indicating that the piecewise-linear roofline form explains almost all variance in observed batch latency.
Relative error reductions versus the token-count baseline are substantial and consistent across percentiles:
- **Prefill:** up to **2.5x** reduction at p90 and **3.3x** at p99.
- **Decode:** up to **3.5x** reduction at p90 and **4.4x** at p99.
These gains hold *despite* batching variability and sequence divergence — the two conditions that most stress attribution models, since heterogeneous batches mix requests with very different compute and memory profiles. The fact that the improvement is larger at p99 than at p90 indicates that LLMVisor is particularly effective at taming tail latency attribution errors, which are precisely the errors that matter most for SLA enforcement and fair-share scheduling.
The decode improvements exceeding prefill improvements are consistent with the roofline intuition: decode is memory-bound and more sensitive to weight-loading traffic, so a
Why it matters
The significance of LLMVisor lies less in any single accuracy number and more in the **primitive** it provides. Multi-tenant LLM serving has been stuck in a regime where co-batching improves throughput but destroys observability. LLMVisor restores observability at the granularity that matters — the individual request — and does so cheaply enough to sit inside the scheduler.
Several design choices deserve attention. First, the **piecewise-linear** form is a deliberate compromise: it is expressive enough to capture the compute-bound/memory-bound transition, yet simple enough to fit and evaluate in microseconds. A fully nonlinear model might fit marginally better but would be too slow for the scheduling loop. Second, the **additive decomposition** is what makes fractional sharing possible; without it, attribution would yield a batch-level number that cannot be assigned to tenants. Third, the use of **FLOPs and memory I/O** as features, rather than token counts, is the key modeling insight — token count is a proxy that breaks down precisely when batches are heterogeneous.
The evaluation's breadth — Llama 3.1-8B and Qwen 2.5-14B/32B, A100 and H100, varying tensor parallelism and workload mixes — supports the claim that the approach is not overfit to a single configuration. The larger gains at p99 than p90 suggest LLMVisor is robust to the tail cases that dominate real-world SLA violations.
Limitations are implicit in the scope: the model is validated on dense transformer inference at the evaluated scales, and its coefficients must be fitted per regime and per hardware configuration. Whether the piecewise-linear form generalizes to mixture-of-experts routing, speculative decoding, or disaggregated prefill/decode architectures remains open. Nonetheless, as a real-time attribution primitive for fractional sharing, LLMVisor establishes a practical baseline that token-count heuristics cannot match.
Who should read this
Opening member content…