Ilmu Komputer & AI editorial
Open AccessOA2026
Update from Hell: Can Coding Agents Survive Hidden Breakage in Dependency Upgrades?
DEPBENCH exposes a stark capability gap: the best coding agent configuration resolves only 51.2% of real-world dependency-upgrade tasks involving hidden code-level changes.
Zijian Luo; Runzhi He; Pengfei Gao; Yu Kang; Zeqi Lin; Minghua Ma; Qingwei Lin; Saravan Rajmohan; Yongqiang Tian· 2026· DOI 10.48550/arXiv.2608.30300
The core problem
Modern software systems rely heavily on third-party dependencies, yet upgrading those dependencies remains a costly and error-prone maintenance activity. Dependency upgrades do not always preserve the function signatures, type systems, APIs, or runtime semantics assumed by existing code. As a result, developers must often perform source code adaptations to accommodate dependency-induced changes. Critically, such code-level changes are frequently not explicitly communicated to project maintainers, posing a significant challenge to software reliability. Meanwhile, coding agents have emerged as a new form of software development tool, increasingly adopted for their automation capabilities. This paper introduces DEPBENCH, a benchmark of 203 real-world dependency-upgrade tasks across five package ecosystems and five language communities, each involving hidden code-level changes that require source code adaptation. The central research question is: can coding agents survive hidden breakage in dependency upgrades? The authors evaluate mainstream coding agents on DEPBENCH to quantify the gap between current agent capabilities and real-world software maintenance needs.
Innovation
The evaluation of mainstream coding agents on DEPBENCH reveals a stark capability gap. The best completed configuration solved only 104 out of 203 tasks, corresponding to a success rate of 51.2%. This means that nearly half of the real-world dependency-upgrade tasks involving hidden code-level changes were not resolved by the most effective agent configuration tested. Performance varied substantially across agent harnesses, models, and ecosystems. The authors report "substantial variation across agent harnesses, models, and ecosystems," indicating that no single configuration dominates across all contexts. The results highlight that current coding agents struggle with the nuanced adaptations required when dependency upgrades introduce hidden breakage. The 51.2% success rate underscores that while agents can automate some aspects of dependency upgrades, they are far from reliable for real-world software maintenance. The benchmark's diversity—five package ecosystems and five language communities—ensures that the findings are not artifacts of a single environment, strengthening the conclusion that the gap is systemic.
Modern software systems rely heavily on third-party dependencies, yet upgrading those dependencies remains a costly and error-prone maintenance activity. Dependency upgrades do not always preserve the function signatures, type systems, APIs, or runtime semantics assumed by existing code. As a result, developers must often perform source code adaptations to accommodate dependency-induced changes. Critically, such code-level changes are frequently not explicitly communicated to project maintainers, posing a significant challenge to software reliability. Meanwhile, coding agents have emerged as a new form of software development tool, increasingly adopted for their automation capabilities. This paper introduces DEPBENCH, a benchmark of 203 real-world dependency-upgrade tasks across five package ecosystems and five language communities, each involving hidden code-level changes that require source code adaptation. The central research question is: can coding agents survive hidden breakage in dependency upgrades? The authors evaluate mainstream coding agents on DEPBENCH to quantify the gap between current agent capabilities and real-world software maintenance needs.
The authors constructed DEPBENCH, a benchmark comprising 203 real-world dependency-upgrade tasks. These tasks span five package ecosystems, covering five language communities, ensuring diversity in dependency management practices and language semantics. Each task involves hidden code-level changes—alterations in function signatures, type systems, APIs, or runtime semantics—that are not explicitly communicated to maintainers but require source code adaptation for the upgrade to succeed. The benchmark is designed to reflect the complexity of real-world maintenance scenarios. The authors then evaluated mainstream coding agents on DEPBENCH, systematically varying agent harnesses and underlying models to measure performance. The evaluation metric is task completion: whether the agent successfully adapts the source code to accommodate the dependency-induced changes. The best completed configuration solved only 104 out of 203 tasks, yielding a success rate of 51.2%. The authors also analyzed variation across agent harnesses, models, and ecosystems to identify factors influencing performance. This methodology enables a rigorous assessment of coding agents' ability to handle hidden breakage in dependency upgrades.
Why it matters
The findings from DEPBENCH carry significant implications for software reliability and the adoption of coding agents. The 51.2% success rate indicates that current agents are not yet capable of reliably handling hidden breakage in dependency upgrades, which are common in modern software maintenance. The substantial variation across agent harnesses, models, and ecosystems suggests that performance is highly context-dependent, and no universal solution exists. This variability may stem from differences in how agents interpret error messages, access documentation, or reason about type systems and APIs. The hidden nature of the code-level changes—not explicitly communicated to maintainers—poses a particular challenge for agents, which must infer required adaptations from incomplete information. The authors emphasize "an important gap between current agent capabilities and real-world software maintenance needs." This gap implies that developers cannot yet fully delegate dependency upgrades to agents without oversight. Future work could focus on improving agents' ability to detect and reason about hidden breakage, perhaps by integrating richer static analysis or runtime feedback. The benchmark itself provides a foundation for tracking progress in this area. Overall, DEPBENCH serves as a critical wake-up call for the software engineering community, highlighting the need for more robust automation in dependency management.
Who should read this
CS practitioners and researchers
Opening member content…