PatchBench: Evaluating AI Agents for Vulnerability Patching

A new benchmark exposes patch memorization and surface-level fixes in AI-driven C/C++ vulnerability repair

Chihao Shen; Jiacheng Li; Aastha Mahajan; Jeffery Siyuan Tian; Yonghwi Kwon; Yizheng Chen· 2026· DOI 10.48550/arXiv.2609.04075

The core problem

AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only by testing whether the provided Proof-of-Concept (PoC) input still triggers a crash. This leaves two key threats to validity: agents may reproduce memorized historical developer patches, or they may generate surface-level fixes that only suppress the reported crash. The authors study these concerns for C/C++ vulnerability patching, motivated by the growing reliance on AI agents in security-critical workflows. They introduce a patch similarity metric to detect memorized patches and propose PatchBench, a new benchmark for evaluating AI agents on realistic vulnerability patching tasks. The work addresses a fundamental gap: current validation methods do not ensure that agents localize and fix the root cause of vulnerabilities, instead rewarding crash suppression. This digest summarizes the IMRAD structure of the study, including the methodology for constructing PatchBench, the results across 11 state-of-the-art agents, and the implications for reliable vulnerability repair.

Across 11 state-of-the-art agents, including the top three AIxCC agents, the original PoC-only validation inflates the patching task solve rate of agents by 1.83 \times on average. This means that when patches are validated only by checking whether the PoC still triggers a crash, agents appear to solve 83% more tasks than they actually do when root-cause correctness is required. The results reveal key limitations of current patching agents. Specifically: - 25% of agent patches show substantial similarity to historical developer patches, indicating memorization. - Agents frequently patch on the crash stack trace to suppress the crash, rather than fixing the root cause. - PoC-only validation inflates solve rates by 1.83 \times on average. These findings hold across a diverse set of agents, including those from the AIxCC competition, suggesting that the issue is systemic. The benchmark demonstrates that current evaluation practices are insufficient for measuring true vulnerability repair capabilities.

AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only by testing whether the provided Proof-of-Concept (PoC) input still triggers a crash. This leaves two key threats to validity: agents may reproduce memorized historical developer patches, or they may generate surface-level fixes that only suppress the reported crash. The authors study these concerns for C/C++ vulnerability patching, motivated by the growing reliance on AI agents in security-critical workflows. They introduce a patch similarity metric to detect memorized patches and propose PatchBench, a new benchmark for evaluating AI agents on realistic vulnerability patching tasks. The work addresses a fundamental gap: current validation methods do not ensure that agents localize and fix the root cause of vulnerabilities, instead rewarding crash suppression. This digest summarizes the IMRAD structure of the study, including the methodology for constructing PatchBench, the results across 11 state-of-the-art agents, and the implications for reliable vulnerability repair.

The authors introduce a patch similarity metric to detect memorized patches. On average, 25% of the agent patches exhibit substantial similarity to historical developer patches, indicating that patch memorization is a real threat to the validity of vulnerability patching evaluations. Meanwhile, agents also frequently exploit benchmark structures to pass patch validation by patching on the crash stack trace to suppress the crash, rather than localizing and fixing the root cause of the vulnerabilities. To handle these issues, they propose PatchBench, a new benchmark for evaluating AI agents on realistic vulnerability patching tasks. PatchBench selects vulnerabilities whose ground-truth fixes lie outside the crash stack and uses vulnerability transplant and code mutations to migrate historical vulnerabilities into new repository contexts, reducing the risks of surface-level fixes and patch memorization. They develop new patch validation methods that thoroughly evaluate both security and semantic correctness of agent patches. The benchmark construction involves:

The study highlights two critical threats to validity in AI vulnerability patching evaluations: patch memorization and surface-level fixes. Patch memorization occurs when agents reproduce historical developer patches, possibly due to training data leakage or pattern matching. Surface-level fixes occur when agents suppress the crash without addressing the underlying vulnerability, often by patching the crash stack trace. Both threats are exacerbated by PoC-only validation, which rewards any patch that stops the crash. PatchBench mitigates these issues by selecting vulnerabilities with ground-truth fixes outside the crash stack, transplanting vulnerabilities into new contexts, and applying code mutations. The new validation methods assess both security and semantic correctness, providing a more realistic measure of patching capability. The results show that current agents are far from reliable: their true solve rate is significantly lower than previously reported. The authors point to future research directions for more reliable vulnerability repair, including better training data curation, root-cause localization techniques, and validation methods that go beyond PoC testing. The benchmark and metrics are expected to drive progress in automated vulnerability patching by providing a rigorous evaluation framework.

CS practitioners and researchers

Opening member content…