A Cyber Range Evaluation of Autonomous Network Incident Response Agents

Reinforcement learning versus heuristic policies for automated intrusion response in an emulated network environment

Jakob Nyberg; Teodor Sommestad; Andrei Buhaiu; Joakim Loxdal; Pontus Johnson; Mathias Ekstedt· 2026· DOI 10.48550/arXiv.2609.16541

The core problem

The increasing complexity and frequency of network intrusions necessitate automated response mechanisms that can operate at machine speed. This paper investigates the performance of autonomous agents tasked with defending a network against a red-team adversary within a cyber range originally intended for human operator training. The core challenge addressed is the dual objective of preventing host access by the adversary while minimizing the availability costs incurred by defensive actions. The authors position their work within the context of automated incident response, leveraging a SIEM platform for alert generation and a data modeling language to interface with the agents. The study aims to compare the effectiveness of heuristic agents against policies learned via reinforcement learning, providing empirical insights from a realistic emulated environment.

The experiments revealed that reinforcement learning agents were overall more efficient at defending the system compared to the heuristic policy. Specifically, the RL agents achieved lower combined costs across the tested scenarios. However, the performance of all agents exhibited high sensitivity to the adversary's policy and the behavior of simulated users. The results indicate that while RL agents can learn effective defensive strategies, their advantage over heuristics is not absolute and depends on the operational context. Quantitative comparisons showed that the RL agents reduced the combined cost by a significant margin, though exact figures are not provided in the abstract. The variability in performance underscores the importance of considering adversarial and user dynamics when evaluating automated response systems.

The increasing complexity and frequency of network intrusions necessitate automated response mechanisms that can operate at machine speed. This paper investigates the performance of autonomous agents tasked with defending a network against a red-team adversary within a cyber range originally intended for human operator training. The core challenge addressed is the dual objective of preventing host access by the adversary while minimizing the availability costs incurred by defensive actions. The authors position their work within the context of automated incident response, leveraging a SIEM platform for alert generation and a data modeling language to interface with the agents. The study aims to compare the effectiveness of heuristic agents against policies learned via reinforcement learning, providing empirical insights from a realistic emulated environment.

The experimental setup utilizes a cyber range featuring an emulated networking environment with variable topology, red-team emulation, and simulated user agents. Alerts generated by a SIEM platform are mapped to a data modeling language consumed by the defensive agents. The defensive agents' objective is to minimize a combined cost function:

The findings suggest that reinforcement learning is a promising approach for automated network incident response, but its effectiveness is contingent on the alignment between training conditions and deployment scenarios. The dependence on adversary policy and simulated user behavior highlights the need for robust training that accounts for diverse and adaptive threats. The use of a cyber range for evaluation provides a realistic testbed, bridging the gap between simulation and real-world deployment. The study contributes to the understanding of autonomous defense mechanisms, emphasizing that no single policy is universally optimal. Future work could explore adaptive strategies that dynamically adjust to the observed adversary and user behavior, potentially combining the strengths of heuristic and learned policies. The taxonomy candidates (Architecture, Cybersecurity, Network, Cryptography) reflect the interdisciplinary nature of the research, though cryptography plays a lesser role in this specific evaluation.

CS practitioners and researchers

Opening member content…