Ilmu Komputer & AI editorial
Open AccessOA2026
Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark
A leave-one-agent-out calibration audit reveals that calibration-transfer failures are not broad but concentrated in specific target agent/head combinations, with no cross-benchmark replication established.
YanZe Cao· 2026· DOI 10.48550/arXiv.2609.25647
The core problem
Early outcome prediction from agent trajectories can reduce evaluation costs by terminating runs once the outcome is sufficiently predictable, provided the predictor's confidence is well-calibrated. Calibration is jeopardized when a predictor is applied to an agent it was never trained on, yet it remains unclear whether such transfer failures are widespread across agent systems or concentrated in specific target agent/head combinations. This work conducts a testing-driven reliability audit using public SWE-bench Verified trajectories and a frozen dual-head early-outcome prediction pipeline. The central hypothesis is that calibration-transfer errors may be target-specific rather than broad, and that such errors may not replicate across benchmarks. The study employs a leave-one-agent-out calibration audit, a shared-predictor leave-two-agents-out control, oracle prior correction, and a robustness battery over training cohorts, task resampling, task halves, jackknife, and thresholds. A fixed-scaffold TerminalBench analysis serves as a pre-registered boundary test to assess cross-benchmark replication.
Innovation
Broad same-predictor pairwise heterogeneity was not supported. The median pairwise corrected-gap differences were 0.0180 for the SUCCESS head (45 pairs) and 0.0385 for the FAILURE head (35 pairs). The pre-registered heterogeneity criterion was not met on either head, indicating that calibration-transfer errors are not broadly distributed across agent pairs. However, two specific combinations exhibited persistent calibration-transfer errors: gpt-5-mini/SUCCESS and claude-opus-4.6/FAILURE, with median corrected gaps of 0.1377 and 0.1107, respectively. These errors did not show a sign reversal under any frozen control, including the leave-two-agents-out control and oracle prior correction. The robustness battery confirmed the persistence of these target-specific errors across training cohorts, task resampling, task halves, jackknife, and thresholds. In the TerminalBench boundary test, the success target produced zero decisions (INDETERMINATE), and the failure target did not satisfy the pre-registered persistence criterion, thus failing to establish cross-benchmark replication. The following Mermaid diagram summarizes the audit workflow:
Early outcome prediction from agent trajectories can reduce evaluation costs by terminating runs once the outcome is sufficiently predictable, provided the predictor's confidence is well-calibrated. Calibration is jeopardized when a predictor is applied to an agent it was never trained on, yet it remains unclear whether such transfer failures are widespread across agent systems or concentrated in specific target agent/head combinations. This work conducts a testing-driven reliability audit using public SWE-bench Verified trajectories and a frozen dual-head early-outcome prediction pipeline. The central hypothesis is that calibration-transfer errors may be target-specific rather than broad, and that such errors may not replicate across benchmarks. The study employs a leave-one-agent-out calibration audit, a shared-predictor leave-two-agents-out control, oracle prior correction, and a robustness battery over training cohorts, task resampling, task halves, jackknife, and thresholds. A fixed-scaffold TerminalBench analysis serves as a pre-registered boundary test to assess cross-benchmark replication.
The audit leverages public SWE-bench Verified trajectories and a frozen dual-head early-outcome prediction pipeline, which outputs separate predictions for SUCCESS and FAILURE outcomes. The primary analysis is a leave-one-agent-out calibration audit: for each target agent, the predictor is trained on all other agents and evaluated on the held-out agent. To control for shared predictor effects, a leave-two-agents-out control is also performed. Oracle prior correction adjusts for base rate differences. A robustness battery tests the stability of findings across variations in training cohorts, task resampling, task halves, jackknife resampling, and decision thresholds. The pre-registered heterogeneity criterion assesses whether pairwise corrected-gap differences are broadly distributed. Additionally, a fixed-scaffold TerminalBench analysis serves as a boundary test for cross-benchmark replication. The key metric is the corrected gap, defined as the difference between predicted confidence and observed accuracy, with calibration-transfer error indicated by persistent nonzero gaps. Formally, for a target agent and head
, the corrected gap is
, where
is the predicted probability and is the true outcome. The audit evaluates whether significantly differs from zero after correction.
Why it matters
The findings demonstrate that a strong target-specific calibration-transfer error can exist within one frozen environment, but the evidence does not establish that the error is intrinsic to the model or general across benchmarks. The absence of broad heterogeneity suggests that calibration-transfer failures are not a universal property of the predictor but are concentrated in particular agent/head combinations. The persistent errors for gpt-5-mini/SUCCESS and claude-opus-4.6/FAILURE indicate that certain targets may require tailored calibration or additional training data. The failure to replicate on TerminalBench highlights the importance of benchmark-specific validation and cautions against assuming cross-benchmark generalization. The pre-registered boundary test produced an INDETERMINATE result for the success target, meaning no decisions were made, and the failure target did not meet the persistence criterion. This underscores the need for further research to determine whether such errors are intrinsic to the model or arise from environment-specific factors. Practically, the results imply that early-outcome prediction pipelines should be audited per target agent and head, and that calibration transfer should not be assumed without explicit testing. The study's limitations include the use of a single benchmark for the primary audit and the frozen nature of the predictor, which may not reflect retrained models. Future work should explore cross-benchmark replication with larger and more diverse agent sets.
Who should read this
CS practitioners and researchers
Opening member content…