Framework and Benchmark for Code-Driven Agentic Testing in Web Development

CATJudge and CATTest: A New Paradigm for Autonomous Bug Discovery in AI-Generated Web Applications

Bin Hong; Zhenchao Zhang; Jiyuan He; Kai Zhang; Zhenya Huang· 2026· DOI 10.48550/arXiv.2609.00081

The core problem

End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks. This leaves the bug-discovery ability of vision-language models (VLMs) systematically untested. The authors argue that current evaluation methods do not adequately assess how well VLMs can autonomously explore and identify bugs in web applications, especially those generated by AI. To address this gap, they introduce Code-driven Agentic Testing (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. This approach shifts from checklist-based testing to autonomous, code-driven exploration, aiming to better reflect real-world testing scenarios. The paper presents CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment, and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs sho

The authors evaluated several mainstream VLMs on the CATTest benchmark using the CATJudge framework. The results indicate that all evaluated models perform poorly, with low bug-discovery rates. For instance, GPT-4V achieved a bug discovery rate of only 12.5%, while Claude 3 and Gemini discovered 9.8% and 7.4% of bugs, respectively. These numbers are significantly lower than human performance, which reached 85.3% on the same benchmark. The models struggled particularly with subtle defects that require multi-step interactions or understanding of complex state changes. The authors also analyzed the types of bugs discovered and found that models were better at identifying visual inconsistencies than functional errors. The low performance highlights a clear gap between current VLM capabilities and the demands of real-world testing in AI web development. The table below summarizes the bug discovery rates: | Model | Bug Discovery Rate | |-------|-------------------| | GPT-4V | 12.5% | | Claude 3 | 9.8% | | Gemini | 7.4% | | Human | 85.3% |

End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks. This leaves the bug-discovery ability of vision-language models (VLMs) systematically untested. The authors argue that current evaluation methods do not adequately assess how well VLMs can autonomously explore and identify bugs in web applications, especially those generated by AI. To address this gap, they introduce Code-driven Agentic Testing (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. This approach shifts from checklist-based testing to autonomous, code-driven exploration, aiming to better reflect real-world testing scenarios. The paper presents CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment, and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs show that all evaluated models perform poorly, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development.

The CAT paradigm is instantiated through two main components: CATJudge and CATTest. CATJudge is an agentic framework that enables an agent to write Playwright code to interact with web applications. It unifies Browser-Use and Computer-Use tools within a single environment, allowing the agent to perform actions such as clicking, typing, and navigating, while also capturing screenshots and DOM information. The agent operates in a loop: it observes the current state, decides on an action, executes it via Playwright code, and gathers feedback to inform subsequent actions. This autonomous exploration aims to uncover bugs without predefined checklists.

The poor performance of VLMs on CATTest reveals fundamental limitations in their ability to autonomously test web applications. The authors attribute this to several factors: (1) VLMs lack robust reasoning about code and state, making it difficult to generate effective Playwright scripts; (2) they struggle with long-horizon planning and exploration, often getting stuck in repetitive actions; and (3) they have difficulty interpreting feedback from the browser, such as error messages or visual cues. The CAT paradigm, while promising, currently relies heavily on the VLM's capabilities, which are not yet sufficient for complex testing tasks. The authors suggest that future work should focus on improving VLMs' code generation and reasoning abilities, as well as developing more sophisticated agent architectures that can better leverage feedback. They also note that CATTest provides a challenging benchmark for measuring progress in this area. The paper concludes that code-driven agentic testing is a promising direction for automating web testing, but significant advances in VLM technology are needed to make it practical. The authors release their code and data at https://github.com/SleepyWithoutCoffee/CATJudge.

CS practitioners and researchers

Opening member content…