Ilmu Komputer & AI editorial
Open AccessOA2026
Conformal Privacy Auditing: Calibrated Re-identification Attacks with Statistical Guarantees
A distribution-free framework for certifying document-level re-identification risk against LLM-empowered adversaries
Shuo Huang; Gholamreza Haffari; Xingliang Yuan; Ting Yu; Lizhen Qu· 2026· DOI 10.48550/arXiv.2609.21340
The core problem
Empirical identity leakage from released text is increasingly driven by attackers that combine large language models (LLMs) with auxiliary knowledge to link documents to individuals. Existing audits typically report success rates for specific attack pipelines but lack finite-sample statistical guarantees, while training-time protections such as differential privacy are difficult to translate into release-time decisions for individual natural-language documents. This gap motivates a distribution-free calibration framework that provides a statistical certificate of re-identification risk for each released document against LLM-empowered adversaries. The authors introduce Conformal Privacy Auditing (CPA), which outputs a conformal ambiguity set of candidate identities guaranteed to contain the true identity with user-chosen confidence under exchangeability, together with an interpretable leakage proxy derived from set size. CPA supports both logit-access and sampling-only attackers, enabling audits of open-source models and proprietary API models in a unified framework.
Innovation
Across multiple release benchmarks and attacker configurations, CPA achieves calibrated coverage, meaning that the empirical coverage of the ambiguity sets closely matches the nominal confidence level . The authors report that CPA reveals sharp shifts in certified identifiability as auxiliary knowledge, LLM augmentation, and release mechanisms vary. For instance, increasing the amount of auxiliary knowledge available to the attacker significantly reduces the average size of the ambiguity sets, indicating higher re-identification risk. Similarly, LLM augmentation—where the attacker uses an LLM to enhance linking—leads to smaller sets compared to baseline attacks. The framework also exposes differences between release mechanisms: documents released with stronger privacy protections yield larger ambiguity sets, reflecting lower certified identifiability. These results provide a statistically grounded basis for reporting and comparing release-time linkage risk across attacker configurations, datasets, and release mechanisms alike.
Empirical identity leakage from released text is increasingly driven by attackers that combine large language models (LLMs) with auxiliary knowledge to link documents to individuals. Existing audits typically report success rates for specific attack pipelines but lack finite-sample statistical guarantees, while training-time protections such as differential privacy are difficult to translate into release-time decisions for individual natural-language documents. This gap motivates a distribution-free calibration framework that provides a statistical certificate of re-identification risk for each released document against LLM-empowered adversaries. The authors introduce Conformal Privacy Auditing (CPA), which outputs a conformal ambiguity set of candidate identities guaranteed to contain the true identity with user-chosen confidence under exchangeability, together with an interpretable leakage proxy derived from set size. CPA supports both logit-access and sampling-only attackers, enabling audits of open-source models and proprietary API models in a unified framework.
CPA operates by calibrating a conformal predictor on a set of documents with known identities, using a nonconformity score that measures how well a candidate identity explains the document under an LLM-empowered attacker. Given a user-chosen confidence level , the conformal ambiguity set
for a new document is constructed as:
Why it matters
The CPA framework addresses a critical gap in privacy auditing by offering finite-sample statistical guarantees for re-identification risk in natural-language documents. Unlike differential privacy, which provides training-time guarantees that are often too conservative or difficult to interpret for individual releases, CPA delivers release-time certificates that are directly interpretable via the ambiguity set size. The distribution-free nature of conformal prediction ensures validity under minimal assumptions (exchangeability), making CPA applicable to a wide range of models and datasets. However, the coverage guarantee is marginal over the randomness in calibration and test points, and may not hold conditionally on specific documents. Future work could explore conditional coverage and extend CPA to structured data or multimodal releases. The unified support for logit-access and sampling-only attackers makes CPA practical for auditing both open-source and proprietary models, facilitating transparent risk communication. Overall, CPA provides a rigorous and flexible tool for privacy auditing in the era of LLM-empowered adversaries.
Who should read this
CS practitioners and researchers
Opening member content…