Computer Science editorial
Open AccessOA2026
DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security
A benchmark and evaluation protocol for measuring agent security under dual-control interaction, where both agent and user influence the shared environment state.
Ivan Aleksandrov; German Kochnev; Sabrina Sadiekh; Yaroslav Rogozaยท 2026ยท DOI 10.48550/arXiv.2609.24662
The core problem
LLM-based agents increasingly operate in environments where they interact with users, tools, and external systems. Yet most security evaluations assume passive users and static control, ignoring the interactive dynamics that shape real agent behavior. The authors introduce **DUMA-Bench**, a benchmark and evaluation protocol for measuring agent security under *dual-control* interaction, where both the agent and the user can influence the shared environment state. The core research question is whether agent security is solely a property of the model or emerges from the interaction between model, user, and environment. DUMA-Bench addresses this by extending ฯยฒ-bench with adversarial environments covering eight vulnerability classes, including RAG poisoning, cross-agent manipulation, and unsafe output handling. The benchmark evaluates 14 models from five model families (OpenAI, Anthropic, DeepSeek, Qwen, and Z.ai) across eight domains and multiple user-behavior regimes. The key finding is that introducing dual-control interaction increases the attack success rate from 26.9% to 41.1%, demonstrating that security is not solely a model property but emerges from interaction dynamics.
Innovation
Across experiments, introducing dual-control interaction increases the attack success rate from **26.9%** to **41.1%**. This 14.2 percentage point increase demonstrates that interactive dynamics significantly amplify security vulnerabilities. The evaluation covers 14 models from five model families (OpenAI, Anthropic, DeepSeek, Qwen, and Z.ai) across eight domains and multiple user-behavior regimes. The results show that agent security is not solely a property of the model but emerges from the interaction between the model, the user, and the environment. The benchmark reveals that models which appear secure under static evaluation may become vulnerable when users actively influence the environment. The eight vulnerability classes, including RAG poisoning, cross-agent manipulation, and unsafe output handling, were tested across all models. The attack success rate varied by model family and domain, but the overall trend of increased vulnerability under dual-control was consistent. These findings highlight the need for evaluation protocols that account for interactive dynamics in realistic agent deployments.
LLM-based agents increasingly operate in environments where they interact with users, tools, and external systems. Yet most security evaluations assume passive users and static control, ignoring the interactive dynamics that shape real agent behavior. The authors introduce **DUMA-Bench**, a benchmark and evaluation protocol for measuring agent security under *dual-control* interaction, where both the agent and the user can influence the shared environment state. The core research question is whether agent security is solely a property of the model or emerges from the interaction between model, user, and environment. DUMA-Bench addresses this by extending ฯยฒ-bench with adversarial environments covering eight vulnerability classes, including RAG poisoning, cross-agent manipulation, and unsafe output handling. The benchmark evaluates 14 models from five model families (OpenAI, Anthropic, DeepSeek, Qwen, and Z.ai) across eight domains and multiple user-behavior regimes. The key finding is that introducing dual-control interaction increases the attack success rate from 26.9% to 41.1%, demonstrating that security is not solely a model property but emerges from interaction dynamics.
DUMA-Bench extends the ฯยฒ-bench framework by introducing dual-control interaction, where both the agent and the user can influence the shared environment state. The benchmark includes adversarial environments covering eight vulnerability classes: RAG poisoning, cross-agent manipulation, unsafe output handling, and five others. The evaluation protocol measures attack success rate (ASR) under different user-behavior regimes. Formally, let be the agent, the user, and the environment. In dual-control, the state transition is , where is the agent action and is the user action. The attack success rate is defined as:
Why it matters
The results indicate that agent security is not solely a property of the model but emerges from the interaction between the model, the user, and the environment. This has significant implications for the design of secure LLM agent systems. Traditional security evaluations that assume passive users and static control may underestimate vulnerabilities. DUMA-Bench provides a missing evaluation layer for studying security in realistic agent deployments. The benchmark's dual-control interaction model captures the dynamic nature of real-world agent deployments, where users can influence the environment state. The eight vulnerability classes cover a range of attack vectors, from RAG poisoning to cross-agent manipulation. The increase in attack success rate from 26.9% to 41.1% underscores the importance of considering user behavior in security evaluations. Future work could extend the benchmark to additional vulnerability classes and model families. The authors suggest that security evaluations should incorporate dual-control interaction to better reflect real-world scenarios. DUMA-Bench is positioned as a foundational tool for evaluating and improving the security of LLM-based agents in interactive environments.
Who should read this
CS practitioners and researchers
Opening member contentโฆ