Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms

A decomposed energy model reveals why token-normalized metrics mislead energy-aware LLM serving on NVIDIA H100 and H200 GPUs

Prabhu Vellaisamy; Vanessa Lam; Shawn Blanton; John Paul Shen· 2026· DOI 10.48550/arXiv.2608.28044

The core problem

Large language model (LLM) inference serving is commercially priced by tokens, yet the underlying GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete: average output-token energy can decrease even when total request energy increases. The authors argue that energy-aware serving must therefore jointly optimize both request energy and token energy, rather than only reducing per-token energy cost. The study is motivated by the observation that a single request incurs a one-time prefill cost plus a fixed generation setup cost, while each subsequent output-token generation step adds only marginal step energy. Formally, the total request energy can be expressed as: E_{request} = E_{prefill} + E_{setup} + i = 1 \sum N E_{step, i} where N is the output length. The token-normalized energy is then E_{token} = E_{request} / N, which is highly sensitive to N and to batching and context length. The paper characterizes this behavior across model type (M), phase (P), batch size (B), context length (C), and output length (N) on NVIDIA H100 and H200 GPUs, covering both dens

The results demonstrate that token energy and request energy can move in opposite directions. For Llama-3.2-1B on H200 at batch-16 and context-4K, increasing output length from 10 to 512 tokens reduces token energy from 7.46 to 0.72 J/token, while total batched inference-window energy increases from 1.19 to 5.93 kJ. This is a direct illustration of the accounting mismatch: per-token energy improves by roughly an order of magnitude, yet the absolute energy consumed per request grows by about 5x.Batching also reduces token energy, but the gain is context-bounded. At 10 output tokens, the batch-16 to batch-1 gain falls from 6.31x at context-512 to 1.17x at context-4K. In other words, the benefit of batching diminishes as context length grows, because the fixed prefill and setup costs become a larger fraction of the total window energy.MoE models amplify this effect. Sparse routing and fragmented expert execution increase fixed energy at low concurrency. However, batching spreads that fixed energy across more generated tokens and substantially narrows the dense-vs.-MoE token-energy gap. The reported metrics are functions of model type (), phase (), batch size (), context l

Large language model (LLM) inference serving is commercially priced by tokens, yet the underlying GPU energy is consumed over inference windows. This accounting mismatch makes token-normalized metrics incomplete: average output-token energy can decrease even when total request energy increases. The authors argue that energy-aware serving must therefore jointly optimize both request energy and token energy, rather than only reducing per-token energy cost.

The study is motivated by the observation that a single request incurs a one-time prefill cost plus a fixed generation setup cost, while each subsequent output-token generation step adds only marginal step energy. Formally, the total request energy can be expressed as:

The central implication is that energy-aware serving should jointly optimize request energy and token energy. Optimizing only per-token energy can hide rising total request energy, especially when output length increases or when context length is large. The decomposed model provides a principled way to reason about trade-offs: fixed prefill and setup costs are amortized over more tokens as N grows, but the marginal step energy still accumulates linearly. For operators, the practical guidance is to treat batching and context length as coupled knobs. Batching reduces token energy, but the gain is context-bounded; at long contexts, the fixed prefill cost dominates and batching yields only modest token-energy improvements. For MoE models, low-concurrency deployments are particularly inefficient because sparse routing fragments expert execution and raises fixed energy, whereas batching recovers much of the dense-vs.-MoE gap. The authors conclude that token-normalized metrics are incomplete for energy-aware serving. A dual-objective view—request energy and token energy—is needed to avoid misleading conclusions. The taxonomy candidates for this work include Architecture, Cybersecurity, Network, and Cryptography, though the primary contribution is in energy characterization for LLM inference on GPU platforms.

CS practitioners and researchers

Opening member content…