Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents

AI agents are being deployed rapidly, accompanied by a growing number of AI-specific attacks and corresponding incidents. As incident reporting becomes increasingly important for legal compliance, governance, accountability, and security, current frameworks must be adapted to the unique characteristics of AI agents. This paper addresses the gap by comparing AI systems and AI agents and, drawing on input from 23 experts in academia and industry, identifies the information required for reporting incidents where the security of AI agents is harmed. The authors emphasize that potential reporting elements include agent memory and memory accesses, actual and potential levels of autonomy, and tool usage. The work is motivated by the need for incident reporting frameworks that can handle the distinct properties of AI agents, which differ from traditional AI systems in their autonomy, interactivity, and tool use. The paper also identifies several open research questions, including how to efficiently record incidents and how to determine whether vulnerabilities and incidents generalize. Expert feedback highlighted potential reporting weaknesses, such as risks of data leakage and attacks targeting the reporting infrastructure itself, creating additional research needs. Finally, the authors summarize privacy requirements and outline research directions for the secure and trustworthy deployment of AI agents.

The methodology centers on a comparative analysis of AI systems and AI agents, followed by expert elicitation. Two editorial authors conducted the comparison, and then gathered input from 23 experts in academia and industry. The expert input was used to identify the information required for reporting incidents where the security of AI agents is harmed. The process likely involved structured discussions or surveys to elicit expert opinions on reporting elements, open research questions, and potential weaknesses. The paper does not specify the exact elicitation technique (e.g., Delphi method, workshops), but it emphasizes that the findings are based on expert consensus. The authors also identify several open research questions, such as how to efficiently record incidents and how to determine whether vulnerabilities and incidents generalize. Expert feedback highlighted potential reporting weaknesses, including risks of data leakage and attacks targeting the reporting infrastructure itself. The methodology is qualitative and exploratory, aiming to set a research agenda rather than provide empirical validation. The inclusion of 23 experts suggests a broad range of perspectives, which strengthens the validity of the identified reporting elements and research directions.