CornerCase: Automated Extremal Testing of Protocol Implementations using LLMs

A two-stage LLM-driven approach for generating boundary test cases from RFCs and uncovering critical bugs in HTTP, DNS, BGP, SMTP, and QUIC implementations.

Rathin Singha; Kuan Qian; Srinath Saikrishnan; Tracy Zhao; Soheil Abbasloo; Ryan Beckett; Siva Kesava Reddy Kakarla; Todd Millstein; George Varghese· 2026· DOI 10.48550/arXiv.2606.29124

The core problem

Network protocol implementations frequently harbor bugs that manifest only at the boundaries of their specification—inputs that are just within or just outside allowed ranges, or messages that are valid in isolation but invalid in a given state. Historical vulnerabilities such as the SSL Heartbleed exploit and TCP Christmas Tree packets demonstrate the criticality of boundary testing. Despite this, existing techniques like fuzzing and model-based testing often fail to systematically target these extremal behaviors. CornerCase addresses this gap by leveraging large language models (LLMs) to automatically extract explicit validity constraints from protocol specifications (e.g., RFCs) and generate test cases at or near the boundaries of those constraints. The approach is evaluated on widely used implementations of HTTP, DNS, BGP, SMTP, and QUIC, revealing numerous previously unknown bugs. For instance, the HTTP server h2o enters a redirect loop when processing URLs containing encoded null bytes. Overall, CornerCase identified and filed 42 anomalies, with 26 acknowledged as bugs and 18 fixed to date.

Innovation

CornerCase was evaluated on widely used implementations of HTTP, DNS, BGP, SMTP, and QUIC. The tool uncovered many previously unknown bugs. For example, the HTTP server h2o enters a redirect loop when processing URLs containing encoded null bytes. Overall, CornerCase identified and filed 42 anomalies. To date, 26 have been acknowledged as bugs by the respective maintainers, and 18 have been fixed, with others under active investigation. These results demonstrate the effectiveness of the approach in finding real-world bugs that impact protocol implementations. The table below summarizes the anomalies per protocol (illustrative): | Protocol | Anomalies Filed | Acknowledged Bugs | Fixed | |----------|----------------|-------------------|-------| | HTTP | 12 | 8 | 6 | | DNS | 8 | 5 | 3 | | BGP | 7 | 4 | 3 | | SMTP | 6 | 4 | 3 | | QUIC | 9 | 5 | 3 | | **Total**| **42** | **26** | **18**| Note: The per-protocol breakdown is illustrative based on the total numbers reported; exact distr

Network protocol implementations frequently harbor bugs that manifest only at the boundaries of their specification—inputs that are just within or just outside allowed ranges, or messages that are valid in isolation but invalid in a given state. Historical vulnerabilities such as the SSL Heartbleed exploit and TCP Christmas Tree packets demonstrate the criticality of boundary testing. Despite this, existing techniques like fuzzing and model-based testing often fail to systematically target these extremal behaviors. CornerCase addresses this gap by leveraging large language models (LLMs) to automatically extract explicit validity constraints from protocol specifications (e.g., RFCs) and generate test cases at or near the boundaries of those constraints. The approach is evaluated on widely used implementations of HTTP, DNS, BGP, SMTP, and QUIC, revealing numerous previously unknown bugs. For instance, the HTTP server h2o enters a redirect loop when processing URLs containing encoded null bytes. Overall, CornerCase identified and filed 42 anomalies, with 26 acknowledged as bugs and 18 fixed to date.

CornerCase decomposes test generation into two stages. First, an LLM extracts explicit validity constraints from protocol specifications in a structured, section-by-section manner. Each constraint is represented as a predicate over message fields or protocol states, often involving numeric ranges, string patterns, or state transitions. For example, a constraint might specify that a length field must satisfy 0 \leq L \leq 65535 or that a particular flag is only valid when the connection is in the ESTABLISHED state. Second, extremal test cases are generated at or near the boundary of each constraint. This includes values exactly at the boundary (e.g., L = 65535), just inside (e.g., L = 65534), and just outside (e.g., L = 65536). The generated tests are then executed across multiple implementations of the same protocol, and differential testing identifies inconsistencies in behavior. The overall workflow is illustrated below:

Why it matters

The success of CornerCase highlights the value of combining LLMs with structured constraint extraction for automated testing. By focusing on extremal values, the approach efficiently targets the most error-prone parts of protocol implementations. The use of differential testing across multiple implementations further amplifies its bug-finding capability, as inconsistencies often reveal genuine bugs. However, the approach relies on the quality of the LLM's constraint extraction; incomplete or incorrect constraints could lead to missed bugs or false positives. Future work could explore iterative refinement of constraints using feedback from test execution, as well as extending the technique to other domains such as cryptographic protocols or file formats. The taxonomy candidates for this work include Architecture, Cybersecurity, Network, and Cryptography, reflecting its broad applicability. Overall, CornerCase represents a significant step towards automated, specification-guided testing of protocol implementations.

Who should read this

CS practitioners and researchers

Opening member content…