DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation

The first benchmark targeting execution efficiency in LLM-generated data science code reveals that correctness alone fails to characterize efficiency, with 79.1% of efficiency deficits stemming from domain-specific root causes.

Zhihao Gong; Junzhe Yu; Dong Huang; Zeyu Sun; Jie M. Zhang; Dan Hao· 2026· DOI 10.48550/arXiv.2608.30248

The core problem

Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. This gap is critical: two functionally correct implementations of the same data science task can differ in runtime by factors of 10x or more, yet existing benchmarks assign them identical scores. The authors introduce **DSEffi-Bench**, the first benchmark specifically targeting execution efficiency in LLM-generated DS code. It comprises 1,000 instances across 10+ DS libraries, with stress-testing harnesses and human-validated references. The central research question is whether correctness alone can characterize the efficiency of LLM-generated data science code. The answer, based on evaluation of 16 models across 3 tiers, is a resounding no. This digest follows the IMRAD structure to present the methodology, results, and analysis of this benchmark.

Innovation

Evaluating 16 models across 3 tiers yields several striking findings: - **Correctness does not predict efficiency**: GPT-5.4 leads in correctness (Pass = 66.9%) but its efficiency score (B ∣ P = 71.7%) nearly matches GPT-5.4-mini (71.6%), which solves 47 fewer tasks. This indicates that solving more tasks correctly does not necessarily translate to more efficient code. - **Efficiency leaders differ from correctness leaders**: Kimi-K2.5 ranks lowest in correctness among frontier models (Pass = 40.2%) yet achieves the highest efficiency score (B ∣ P = 73.6%) across all 16 models. This counterintuitive result underscores that efficiency and correctness are largely orthogonal capabilities. - **Taxonomy distribution**: A human-annotated five-category taxonomy reveals that 79.1% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes. The remaining 20.9% are attributable to algorithmic complexity alone. Distinct failure profiles emerge across model tiers and libraries, suggesting that efficiency issues are not uniform but depend on the model's training and the library's API design. - **Improvement experiments**: Two exploratory experiments provide init

Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. This gap is critical: two functionally correct implementations of the same data science task can differ in runtime by factors of 10x or more, yet existing benchmarks assign them identical scores. The authors introduce **DSEffi-Bench**, the first benchmark specifically targeting execution efficiency in LLM-generated DS code. It comprises 1,000 instances across 10+ DS libraries, with stress-testing harnesses and human-validated references. The central research question is whether correctness alone can characterize the efficiency of LLM-generated data science code. The answer, based on evaluation of 16 models across 3 tiers, is a resounding no. This digest follows the IMRAD structure to present the methodology, results, and analysis of this benchmark.

DSEffi-Bench is constructed with three core components:

Why it matters

The findings from DSEffi-Bench challenge the prevailing assumption that correctness is a sufficient proxy for code quality in data science. The decoupling of correctness and efficiency has profound implications:1. **Benchmark design**: Future benchmarks must incorporate efficiency metrics to avoid rewarding inefficient solutions. The BP metric provides a principled way to measure efficiency conditioned on correctness.2. **Model evaluation**: Ranking models solely by correctness can be misleading. Kimi-K2.5's high efficiency despite low correctness suggests that some models may prioritize different optimization objectives. Practitioners should consider both dimensions when selecting models for data science tasks.3. **Root cause analysis**: The five-category taxonomy reveals that most efficiency deficits are domain-specific, not just algorithmic. This means that generic optimizations (e.g., reducing time complexity) may not address the majority of issues. Instead, library-specific knowledge and API best practices are crucial.4. **Actionable improvements**: The two exploratory experiments show that diagnostics can lead to tangible gains. Taxonomy-guided optimization directly addresses identified deficits, while library-conditioned routing leverages model strengths per library. The latter achieves near-state-of-the-art efficiency at a fraction of the cost, highlighting the potential for cost-effective deployment.However, the study has limitations. The taxonomy, while human-annotated, may not cover all possible efficiency issues. The improvement experiments are exploratory and need further validation. Future work could expand the benchmark to more libraries and tasks, and investigate automated methods for taxonomy annotation.In conclusion, DSEffi-Bench provides a critical step toward understanding and improving the efficiency of LLM-generated data science code. It calls for a shift from correctness-only evaluation to a dual focus on correctness and efficiency, with domain-specific diagnostics guiding optimization.

Who should read this

CS practitioners and researchers

Opening member content…