Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

Greedy decoding from large language models (LLMs) is commonly treated as a deterministic operation: given the same model weights, prompt, and decoding algorithm, one expects identical outputs. This paper challenges that assumption by demonstrating that greedy decoding is **not precision-invariant**. The same model, prompt, and decoding algorithm produce different outputs when run in BF16 versus FP16 on identical hardware. Across six models (1.1B–7B parameters, four families; divergence additionally characterized at 12B) and three benchmarks, the authors find that 49–100% of prompts diverge between the two precisions. A single token flip often cascades into trajectory-level divergence, where subsequent tokens differ entirely. This phenomenon undermines reproducibility and reliability in LLM inference, especially in applications requiring exact output matching. The authors develop an empirical error-propagation analysis to explain why these divergences occur and propose a targeted intervention to mitigate them. The work is motivated by the need for practical solutions that preserve computational efficiency while improving cross-precision agreement.

The authors conduct a systematic empirical study of cross-precision divergence in greedy decoding. They evaluate six models ranging from 1.1B to 7B parameters across four model families, with additional characterization at 12B parameters. Three benchmarks are used to assess divergence rates. The core methodology involves running identical prompts through the same model in BF16 and FP16 on identical hardware and comparing the generated token sequences. To understand the underlying causes, they perform an error-propagation analysis across the model's layers. They find that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps. Instead, the outcome depends primarily on the **top-two logit margin** at the LM head relative to the directional perturbation between the top-two candidates. This insight leads to five testable predictions about intervention outcomes, including the counterintuitive prediction that applying more FP32 compute (broader scope) makes agreement worse. The experiments match all five predictions. Based on this analysis, they design a low-overhead intervention: **selective FP32 LM head recomputation**, triggered only when the margin falls below a threshold. They evaluate this method across six models and four batch sizes, measuring exact agreement and latency overhead. The analysis can be formalized as follows: let $z_1$ and $z_2$ be the top-two logits, and let $\Delta =z_1-z_2$ be the margin. The directional perturbation $\delta$ between the top-two candidates in reduced precision determines whether a flip occurs. A flip happens when $\Delta <\delta$. The selective recomputation triggers when $\Delta <\tau$, where $\tau$ is a threshold. The intervention recomputes the LM head in FP32 for those steps, reducing the effective perturbation. The authors also map the applicability boundary across models and batch sizes, hypothesizing that training-time precision stability is a determining factor.