Ilmu Komputer & AI editorial
Evaluating Coding Agents on Kernel Exploit Generation
The core problem
Coding agents have demonstrated the ability to find real vulnerabilities in production software. However, bug discovery results do not measure whether agents can construct exploit primitives—the fundamental building blocks required to turn a vulnerability into a working exploit. This paper introduces **KEX-bench**, a benchmark designed to evaluate coding agents on exploit primitive generation against real operating-system kernels.
KEX-bench contains **45 task instances** across **40 Linux and Windows CVEs**, covering five primitive categories: kernel address leak, instruction-pointer control, heap read, heap write, and arbitrary address write. Each task runs in an isolated virtual machine, exposes controlled tools, and uses a deterministic verifier to check primitive-specific success. The benchmark aims to provide reproducible research on AI-assisted exploitation.
The central research question is: *Can state-of-the-art coding agents, paired with frontier and open-weight models, generate exploit primitives under fixed tool-call budgets?* The authors evaluate multiple configurations and report success rates with and without a reference proof of concept (PoC).
Innovation
The evaluation reveals a stark performance gap between Windows and Linux tasks, and between settings with and without a reference PoC.
**Without a reference PoC:**
- The strongest configuration solves **1 of 20 Windows tasks (5.0%)**.
- The strongest configuration solves **14 of 25 Linux tasks (56.0%)**.
**With a reference PoC:**
- The strongest configuration solves **31 of 45 tasks (68.9%)**.
These results highlight that agents can often reach kernel crashes but fail to shape kernel state into exploit primitives. The presence of a reference PoC dramatically improves success, suggesting that agents struggle with the creative and state-manipulation aspects of primitive generation.
A summary of the results is shown below:
| Setting | Windows | Linux | Overall |
|---------|---------|-------|---------|
| Without PoC | 1/20 (5.0%) | 14/25 (56.0%) | 15/45 (33.3%) |
| With PoC | — | — | 31/45 (68.9%) |
*Note: The paper does not provide a breakdown of Windows vs. Linux for the with-PoC setting.*
Why it matters
The results expose a critical limitation in current coding agents: while they can discover vulnerabilities and trigger kernel crashes, they lack the capability to reliably convert those crashes into controlled exploit primitives. This gap is especially pronounced on Windows, where only 5.0% of tasks are solved without a reference PoC, compared to 56.0% on Linux. The difference may stem from the complexity of Windows kernel internals, differences in available tooling, or the nature of the CVEs selected.
The substantial improvement when a reference PoC is provided (68.9% overall) indicates that agents benefit greatly from guidance. This suggests that current agents are better at adapting and refining existing exploit strategies than at inventing new ones from scratch. The fixed tool-call budget further constrains exploration, emphasizing efficiency.
KEX-bench provides a reproducible platform for measuring progress in AI-assisted exploitation. Future work could focus on improving agents' ability to manipulate kernel state, generalize across operating systems, and operate under tighter constraints. The benchmark's release at https://kex-bench.github.io aims to catalyze research in this direction.
A high-level architecture of the benchmark is illustrated below:
This architecture ensures that each task is evaluated consistently and securely.
Who should read this
Opening member content…