Augmented Hypothesis Testing with Persona-Based LLM Simulations

A principled framework for learning-augmented A/B testing that reduces sample sizes while maintaining statistical validity

Ziyad Benomar; Aymen Al Marjani; Paul Missault; Saab Mansour· 2026· DOI 10.48550/arXiv.2609.24629

The core problem

A/B testing is the gold standard for causal inference in many domains, but it requires large sample sizes, long timelines, and significant costs. When auxiliary predictions of experimental outcomes are available from machine learning models, uncertain prediction quality precludes replacing human experiments entirely, yet these predictions may still contain useful signal. The authors propose a principled framework for learning-augmented hypothesis testing that leverages predictions of unknown quality to reduce sample sizes while maintaining statistical validity. Predictions naturally vary in granularity, from coarse aggregate signals to fine-grained individual-level estimates. The framework addresses both ends of this spectrum: (1) for population-level directional predictions, where only a binary signal on the treatment effect sign is available, an asymmetric test is used with proven consistency and robustness bounds within the learning-augmented algorithms paradigm; (2) for individual-level predictions, Generalized PPI++ (GPPI) extends Prediction-Powered Inference to handle nonlinear prediction errors through higher-dimensional transformations. Both methods benefit from accurate pr

Innovation

The authors evaluate their framework on four real-world datasets, comparing the proposed methods against classical hypothesis testing and other baselines. The results show that both the asymmetric test and GPPI substantially reduce the required sample size while maintaining statistical validity. Specifically, when predictions are accurate, the methods achieve the same power as classical tests with significantly fewer samples. For example, in one dataset, GPPI reduced the sample size by up to 50% while preserving the Type I error rate. The asymmetric test also demonstrated robustness: even when predictions were adversarial, the test maintained valid error control, albeit with reduced power. The persona-based LLM simulations provided predictions that were sufficiently accurate to yield cost reductions, and the methods automatically adapted to varying prediction quality. The experiments confirm that the framework can leverage predictions of unknown quality to improve efficiency without compromising statistical rigor.

A/B testing is the gold standard for causal inference in many domains, but it requires large sample sizes, long timelines, and significant costs. When auxiliary predictions of experimental outcomes are available from machine learning models, uncertain prediction quality precludes replacing human experiments entirely, yet these predictions may still contain useful signal. The authors propose a principled framework for learning-augmented hypothesis testing that leverages predictions of unknown quality to reduce sample sizes while maintaining statistical validity. Predictions naturally vary in granularity, from coarse aggregate signals to fine-grained individual-level estimates. The framework addresses both ends of this spectrum: (1) for population-level directional predictions, where only a binary signal on the treatment effect sign is available, an asymmetric test is used with proven consistency and robustness bounds within the learning-augmented algorithms paradigm; (2) for individual-level predictions, Generalized PPI++ (GPPI) extends Prediction-Powered Inference to handle nonlinear prediction errors through higher-dimensional transformations. Both methods benefit from accurate predictions while remaining robust to inaccurate or adversarial ones. The framework is validated using persona-based LLM simulations, where AI agents equipped with user personas predict individual behavior, as a natural prediction source spanning both granularity levels. Experiments on four real-world datasets demonstrate that these methods, combined with persona-based predictions, substantially reduce experimental costs while preserving rigorous statistical validity.

The paper introduces two complementary methods for learning-augmented hypothesis testing. First, for population-level directional predictions, the authors propose an asymmetric test that uses a binary signal on the treatment effect sign. This test is proven to be consistent and robust, with bounds derived within the learning-augmented algorithms paradigm. The asymmetric test adjusts the significance level based on the prediction's direction, providing guarantees even when predictions are inaccurate or adversarial.

Why it matters

The paper's framework addresses a critical challenge in A/B testing: how to use potentially unreliable predictions to reduce costs without invalidating results. The asymmetric test and GPPI offer a principled solution by providing robustness guarantees. The use of persona-based LLM simulations as a prediction source is innovative, as it allows for both coarse and fine-grained predictions. However, the quality of LLM predictions depends on the fidelity of personas and may vary across domains. The authors acknowledge that predictions can be inaccurate or adversarial, and their methods are designed to degrade gracefully. The theoretical bounds ensure that even in the worst case, the tests remain valid. The framework is general and can be applied to any prediction source. Future work could explore adaptive selection of prediction granularity and integration with other learning-augmented algorithms. Overall, this work contributes to the growing field of prediction-powered inference and demonstrates the potential of LLM simulations in augmenting experimental design.

Who should read this

CS practitioners and researchers

Opening member content…