MarkSec: Capability-Aware Evaluation of Adversarial Attacks Against LLM Watermarks

A unified framework reveals that attack rankings shift when text quality is enforced, challenging isolated evaluations of stealing, scrubbing, and spoofing.

Kairong Li; Zhikun Zhang; Xiao Ren; Yunjun Gao· 2026· DOI 10.48550/arXiv.2609.16681

The core problem

Large language model (LLM) watermarking embeds detectable signals into generated text to trace provenance. However, adversaries can mount three classes of attacks: **stealing** (recovering watermark information), **scrubbing** (removing watermark signals), and **spoofing** (forging text accepted as watermarked). Prior work studies these attacks in isolation, obscuring their connections and making comparisons unreliable due to inconsistent detector calibration, metric definitions, and reporting protocols. Moreover, evaluating attack success and text quality separately fails to identify attacks that are both effective and quality-preserving. This paper introduces **MarkSec**, a general framework that unifies the analysis of stealing, scrubbing, and spoofing under a common reporting protocol. MarkSec also proposes a **quality-constrained attack success metric** that jointly assesses effectiveness and text quality. The authors evaluate representative watermark families, attacks, LLMs, and datasets, yielding three key findings: (1) attacks that appear strongest by watermark removal alone can fall behind general rewriting when text quality is required; (2) general rewriting remains a str

Innovation

Experiments across representative watermark families, attacks, LLMs, and datasets yield three main findings. **First**, attacks that appear strongest by watermark removal alone can fall behind general rewriting when success also requires acceptable text quality. This indicates that conventional attack success metrics overestimate the practical threat of some attacks. **Second**, general rewriting remains a strong baseline across watermark families, while its advantage over other scrubbers varies by family. For example, on some watermark families, general rewriting achieves higher quality-constrained success than specialized scrubbers, whereas on others the gap narrows. **Third**, in a case study of one watermark family, stealing-based scrubbers often underperform the best general-scrubbing baselines when text quality is required. This suggests that stealing watermark information does not necessarily translate to better scrubbing under quality constraints. The results are consistent across multiple LLMs and datasets, underscoring the importance of joint evaluation. Quantitatively, the quality-constrained success rates reveal that some attacks with high raw success rates suffer signi

Large language model (LLM) watermarking embeds detectable signals into generated text to trace provenance. However, adversaries can mount three classes of attacks: **stealing** (recovering watermark information), **scrubbing** (removing watermark signals), and **spoofing** (forging text accepted as watermarked). Prior work studies these attacks in isolation, obscuring their connections and making comparisons unreliable due to inconsistent detector calibration, metric definitions, and reporting protocols. Moreover, evaluating attack success and text quality separately fails to identify attacks that are both effective and quality-preserving. This paper introduces **MarkSec**, a general framework that unifies the analysis of stealing, scrubbing, and spoofing under a common reporting protocol. MarkSec also proposes a **quality-constrained attack success metric** that jointly assesses effectiveness and text quality. The authors evaluate representative watermark families, attacks, LLMs, and datasets, yielding three key findings: (1) attacks that appear strongest by watermark removal alone can fall behind general rewriting when text quality is required; (2) general rewriting remains a strong baseline across watermark families, though its advantage over other scrubbers varies; and (3) in a case study, stealing-based scrubbers often underperform the best general-scrubbing baselines when text quality is enforced. These results highlight that apparent attack winners depend on text-quality constraints, attack generality, and capability assumptions.

MarkSec provides a unified framework for evaluating stealing, scrubbing, and spoofing attacks against LLM watermarks. The framework standardizes detector calibration, metric definitions, and reporting protocols to enable fair comparisons. Central to MarkSec is a **quality-constrained attack success metric** that jointly measures attack effectiveness and text quality. Formally, let A denote an attack, W a watermarking scheme, and D a detector. The attack success rate is defined as:

Why it matters

The MarkSec framework exposes critical limitations in prior evaluations that treat attack success and text quality separately. By introducing a quality-constrained metric, the authors show that apparent attack winners depend on text-quality constraints, attack generality, and capability assumptions. **Text-quality constraints**: When quality is enforced, attacks that aggressively remove watermarks often degrade text, reducing their practical utility. **Attack generality**: General rewriting methods, which are not tailored to specific watermark families, remain competitive or superior across families, suggesting that specialized attacks may overfit to particular watermarks. **Capability assumptions**: Stealing-based scrubbers assume the adversary can recover watermark information; however, even with this capability, they do not consistently outperform simpler general-scrubbing baselines when quality matters. These insights imply that future watermarking and attack research should adopt unified, quality-aware evaluation protocols. The MarkSec framework provides a foundation for such standardization, enabling fair comparisons and revealing trade-offs between effectiveness and quality. Limitations include the scope of watermark families and attacks tested, and the choice of quality metrics, which may affect rankings. Nonetheless, the findings highlight the need to consider text quality as a first-class constraint in adversarial evaluations.

Who should read this

CS practitioners and researchers

Opening member content…