When Are Aggregate Agent Traces Diagnosable? Traffic-Governed Interpretation and Calibrated Abstention

A diagnosability-first framework that gates aggregate fault interpretation on clean-policy exposure and abstains when traces cannot support a claim

Peiying Zhu; Sidi Chang· 2026· DOI 10.48550/arXiv.2609.25806

The core problem

Runtime traces are often treated as transparent windows into agent behavior, yet a closed-loop policy determines which states are visited and which failures become visible. The authors study a simulated hotel-pricing agent that maps time, inventory, and market state to discrete price actions under varying demand regimes. A fault may leave no aggregate trace when the policy rarely visits affected cells, making naive scoring or localization unreliable. The central claim is that entry into aggregate-only fault interpretation should be treated as a **diagnosability decision** that precedes scoring or localization. The paper formalizes this as a gated process: first establish exposure, then score change, and abstain when the trace cannot support the claim. This reframes the problem from "what is the fault?" to "is the trace even capable of revealing it?"

In a frozen one-shot heldout evaluation, 55/72 (76.4%) regime-component units were reference-admitted, representing 20 physical components. Of these, 54/55 passed matched runtime admission, while the rejected unit abstained. Stable false admission was 0/20, with a one-sided exact 95% upper bound of 0.1391, meeting the frozen 0.20 criterion. Across 540 repeated unit-arm rows nested in those 20 clusters, affected clean traffic reduced negative log likelihood by 29.3% relative to cell coverage, a gain of 0.1264 nats per row (cluster-bootstrap 95% interval [0.0593, 0.1918]). Adding mask family and its interaction improved log loss by 0.0015 nats per row (one-sided upper bound 0.0066), below the frozen 0.01 practical-sufficiency margin. These results show that the gated approach achieves calibrated abstention while preserving detection power.

Runtime traces are often treated as transparent windows into agent behavior, yet a closed-loop policy determines which states are visited and which failures become visible. The authors study a simulated hotel-pricing agent that maps time, inventory, and market state to discrete price actions under varying demand regimes. A fault may leave no aggregate trace when the policy rarely visits affected cells, making naive scoring or localization unreliable. The central claim is that entry into aggregate-only fault interpretation should be treated as a **diagnosability decision** that precedes scoring or localization. The paper formalizes this as a gated process: first establish exposure, then score change, and abstain when the trace cannot support the claim. This reframes the problem from "what is the fault?" to "is the trace even capable of revealing it?"

The framework uses two sequential gates. A **reference-map gate** requires repeated clean-policy support for a regime-component unit. A **matched runtime gate** then requires joint support in both clean and current streams. Signal analysis occurs only after both gates pass. False admission is calibrated on a disjoint clean stream at the physical-component level, and detection is modeled by affected clean traffic rather than nominal cell coverage.

The findings support a bounded rule for interpreting aggregate agent behavior: first establish exposure, then score change, and abstain when the trace cannot support the claim. The reference-map and matched runtime gates provide a principled way to avoid false confidence in aggregate-only fault interpretation. The use of affected clean traffic as the detection model is a key innovation, as it accounts for the policy's visitation pattern rather than assuming uniform coverage. The development audit's finding that exact minimum hitting set and greedy selection chose identical supports in 12/12 scenarios suggests that singleton evidence often resolves conflicts, simplifying implementation. However, the approach is limited to the simulated hotel-pricing domain and may require adaptation for other closed-loop systems. Future work could extend the framework to continuous state spaces and multi-agent settings.

CS practitioners and researchers

Opening member content…