Benchmarking Neural Defend ARCAS 1B: A Foundational Multimodal Deepfake Detection Model

AI-generated imagery evolves faster than benchmark-specific detector evaluations, making a single score an incomplete account of generalization. The authors (Sivashankar Selvarajan, Piyush Verma, Sumit Kumar, and Sharayu N. Deshmukh) address this gap by evaluating **Neural Defend ARCAS 1B**, a foundational multimodal deepfake detection model, across benchmark families **without benchmark-specific parameter updates**. The central research question is: how well does a single frozen detector generalize across heterogeneous evaluation protocols, and what do aggregate scores hide about test-population, class-balance, and missing-record coverage?

The study's methodological stance is deliberately conservative. Rather than optimizing for any leaderboard, it retains each benchmark's **native aggregation** and supplements it with **record-level measures**, **coverage accounting**, and **subgroup diagnostics**. This design makes visible the differences that pooled summaries typically obscure. The authors explicitly restrict cross-paper comparisons to aligned evidence, treating differences in release, population, preprocessing, training, or benchmark exposure as **context rather than rank**.

The Results are presented benchmark by benchmark, each subsection identifying the release and evaluation population, reporting the official metric, and describing observed error patterns. Because the model is evaluated **without benchmark-specific parameter updates**, the reported numbers reflect zero-shot transfer rather than tuned performance. The authors emphasize that differences across benchmarks in release, population, preprocessing, training, or benchmark exposure are **context rather than rank**—a benchmark with a higher score is not necessarily a harder or easier test in any absolute sense.

Key observed patterns include: (1) native aggregation scores vary across benchmark families, but this variation is partly attributable to differences in class balance and evaluation population rather than to model capability alone; (2) record-level measures reveal error concentrations that aggregate scores obscure, particularly in subgroups with skewed class distributions; and (3) coverage accounting shows that missing-record gaps differ across benchmarks, meaning that pooled summaries computed over unequal evaluated populations are not directly comparable. The combined analysis synthe