How It's Made: Uncovering Detection Engineering Processes for Network Intrusion Detection Rules

A large-scale study of Suricata rule engineering reveals three phases, limited impact of prior experience, and the critical role of labeled data.

Koen T. W. Teuwen; Emmanuele Zambon; Luca Allodi· 2026· DOI 10.48550/arXiv.2609.25901

The core problem

Many Security Operations Centers (SOCs) rely on signature-based Network Intrusion Detection Systems (NIDS) such as Suricata to detect malicious network activity. Despite their widespread use, the process of engineering detection rules remains understudied. This paper addresses that gap by investigating how engineers create rules for NIDS. The authors introduce SuriCap, a platform designed for rule engineering exercises, and conduct CTF-style workshops with 60 participants, including trained MSc students and experienced SOC professionals. Participants created rules for four scenarios, producing 3,146 valid rules. The study aims to uncover the methods, performance, and iteration patterns of rule engineering, and to identify common phases and patterns that can inform SOC managers and improve expectations regarding engineer expertise.

The analysis of 3,146 valid rules revealed several key findings. Surprisingly, prior experience had limited impact on rule quality, suggesting that less experienced engineers can produce rules comparable to experts. This challenges common assumptions about the necessity of extensive experience in detection engineering. The study also observed significant challenges in generalizing rules beyond the available tests, underscoring the need for sufficient labeled data to develop robust rules. Participants exhibited varying iteration patterns, and the authors identified three phases in the rule engineering process: an initial exploration phase, a refinement phase, and a validation phase. A common pattern emerged across participants, indicating a structured approach to rule creation. The performance of rules varied, but the overall quality was sufficient to provide insights into the engineering process.

Many Security Operations Centers (SOCs) rely on signature-based Network Intrusion Detection Systems (NIDS) such as Suricata to detect malicious network activity. Despite their widespread use, the process of engineering detection rules remains understudied. This paper addresses that gap by investigating how engineers create rules for NIDS. The authors introduce SuriCap, a platform designed for rule engineering exercises, and conduct CTF-style workshops with 60 participants, including trained MSc students and experienced SOC professionals. Participants created rules for four scenarios, producing 3,146 valid rules. The study aims to uncover the methods, performance, and iteration patterns of rule engineering, and to identify common phases and patterns that can inform SOC managers and improve expectations regarding engineer expertise.

The authors developed SuriCap, a platform that facilitates rule engineering exercises by providing a controlled environment for participants to write Suricata rules. They hosted CTF-style workshops where 60 participants, comprising trained MSc students and experienced SOC professionals, were tasked with creating detection rules for four distinct scenarios. The scenarios likely represented different types of network intrusions, requiring participants to analyze network traffic and craft appropriate signatures. The platform captured all rule submissions, resulting in 3,146 valid rules. The study then analyzed these rules to assess their quality, performance, and the iteration patterns employed by participants. The methodology also included comparing the outcomes based on prior experience to determine its impact on rule quality. Additionally, the authors examined the challenges participants faced in generalizing rules beyond the provided tests, emphasizing the need for sufficient labeled data.

The findings suggest that rule engineering is a structured process that can be learned and executed effectively regardless of prior experience. The three identified phases—exploration, refinement, and validation—offer a framework for SOC managers to structure training and workflows. The limited impact of experience implies that organizations can broaden their hiring and training to include less experienced engineers, potentially alleviating skill shortages. However, the difficulty in generalizing rules highlights the importance of investing in labeled data and comprehensive testing environments. The authors propose that SOC managers use these insights to set realistic expectations and improve their detection engineering processes. Future work could explore automated assistance and better data labeling techniques to enhance rule generalization. Overall, the study provides a foundational understanding of NIDS rule engineering and offers practical implications for cybersecurity operations.

CS practitioners and researchers

Opening member content…