Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction

A Think-Act-Speak framework that lifts task success to 82.0% and cuts background-speech false responses from 73.0% to 13.0%

Lujia Bao; Qian Chen; Luyao Cheng; Chong Deng; Yuxiang Kong; Xiangang Li; Xu Li; Jiaqing Liu; Chao-Hong Tan; Haoyu Wang; Wen Wang; Xilou Wang; Haoxiang Xu; Junhao Xu; Liang Yi; Binbin Zhang; Qinglin Zhang; Qiquan Zhang· 2026· DOI 10.48550/arXiv.2609.25176

The core problem

Real-time voice assistants are no longer simple speech recognizers or text-to-speech front ends. They must reason over evolving requests, execute actions through tools, and obey conversational rules such as when to speak, when to stay silent, and when to interrupt. The authors identify three coupled requirements — **Think**, **Act**, and **Speak and Coordinate** — and argue that prior systems optimize them in isolation, producing assistants that either reason poorly, fail to use tools reliably, or respond at inappropriate moments. Qwen-Audio-3.1-Realtime is presented as an integrated answer to these requirements. The paper evaluates the model across six axes: audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. The central empirical claims are concrete: compared with Qwen-Audio-3.0-Realtime, version 3.1 raises overall task success from **78.4% to 82.0%** on a half-duplex speech-to-text adaptation of τ -Voice, and on speech-to-speech Full-Duplex-Bench v1.5 it reduces the response rate to background speech from **73.0% to 13.0%**. The work also introduces a separate Voice Harness prototype that uses Qwen-Audio-3.0-R

Innovation

The paper reports evaluations across audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Two headline comparisons against Qwen-Audio-3.0-Realtime are given. On the half-duplex speech-to-text adaptation of τ -Voice, overall task success improves from **78.4%** to **82.0%**, a gain of 3.6 percentage points. This benchmark exercises the Think and Act capabilities: the assistant must interpret a spoken request, reason about it, and complete a task. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech drops from **73.0%** to **13.0%**. This is a reduction of 60 percentage points, and it directly measures the Speak and Coordinate capability: an assistant that responds to background speech is failing to distinguish the user from ambient audio. The magnitude of the change suggests that the coordination stage, not just the acoustic front end, is doing substantial work. The Voice Harness prototype is presented separately rather than as a scored result. It uses Qwen-Audio-3.0-Realtime as its foreground and extends spoken interaction to persistent tasks through foreground-background coordinati

Real-time voice assistants are no longer simple speech recognizers or text-to-speech front ends. They must reason over evolving requests, execute actions through tools, and obey conversational rules such as when to speak, when to stay silent, and when to interrupt. The authors identify three coupled requirements — **Think**, **Act**, and **Speak and Coordinate** — and argue that prior systems optimize them in isolation, producing assistants that either reason poorly, fail to use tools reliably, or respond at inappropriate moments.

Qwen-Audio-3.1-Realtime is presented as an integrated answer to these requirements. The paper evaluates the model across six axes: audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. The central empirical claims are concrete: compared with Qwen-Audio-3.0-Realtime, version 3.1 raises overall task success from **78.4% to 82.0%** on a half-duplex speech-to-text adaptation of τ -Voice, and on speech-to-speech Full-Duplex-Bench v1.5 it reduces the response rate to background speech from **73.0% to 13.0%**. The work also introduces a separate Voice Harness prototype that uses Qwen-Audio-3.0-Realtime as its foreground and extends spoken interaction to persistent tasks through foreground-background coordination and memory.

Why it matters

The results support the paper's central claim that reasoning, action, and conversational coordination should be trained together rather than as separate modules. The τ -Voice improvement is modest in absolute terms but occurs on a task-success metric where the baseline is already high, so the remaining headroom is limited. The Full-Duplex-Bench v1.5 result is the more striking one: cutting background-speech responses from 73.0% to 13.0% addresses a failure mode that users notice immediately, because an assistant that answers the television is perceived as broken regardless of its reasoning quality. Several limitations follow from the reported evidence. The evaluation is described as covering six axes, but only two quantitative comparisons are given in the abstract, and both are against the immediately preceding version rather than against external systems. The half-duplex τ -Voice adaptation is a speech-to-text setup, so it does not by itself demonstrate full-duplex task success. The Voice Harness is a prototype and is not reported with benchmark numbers. Finally, the safety axis is listed among the evaluations but no safety metric is disclosed in the available text. For practitioners, the design suggests a practical recipe: use supervised fine-tuning to anchor behavior, use multi-teacher on-policy distillation to transfer language ability into the audio model, use GRPO with self-evolving executable environments to teach tool use, and reserve a dedicated alignment stage for turn-taking. The taxonomy candidates for this work — Architecture, Cybersecurity, Network, Cryptography — are only partially apt; the paper is primarily an architecture and agentic-interaction contribution, with safety evaluation as the closest link to the security-oriented categories.

Who should read this

CS practitioners and researchers

Opening member content…