What Survives the Next Model? Benchmarking LLM-Based Techniques Against Single-Prompts

A meta-analysis of 35 ICSE 2026 papers reveals that 37-63% of engineered LLM techniques are outperformed by a single prompt on a newer model, exposing the fragility of workarounds to temporary model deficits.

Nahian Salsabil; Joy Saha; Simantika Bhattacharjee Dristi; Nicholas Phair; Nusrat Jahan Mozumder; Matthew B. Dwyer; Sebastian Elbaum· 2026· DOI 10.48550/arXiv.2609.00468

The core problem

The software engineering (SE) research community has rapidly integrated Large Language Models (LLMs) into complex techniques to tackle diverse tasks. However, the strategic value of this investment is uncertain: as frontier model generations advance, their native capabilities may render existing techniques obsolete. This paper investigates whether the elaborate tooling proposed in recent LLM-based SE research can be outperformed by the simplest possible alternative—a single, automatically generated prompt executed on a newer generation model, without iterative refinement. The authors analyze 35 LLM-based technique papers from ICSE 2026 to assess the durability of these research contributions. The central question is: what survives the next model? The findings aim to inform the community about the cost-benefit proposition of techniques designed as workarounds to temporary model deficits, and to highlight the need for focusing on enduring challenges that scale synergistically with future model generations.

Innovation

The analysis reveals that for between 37% and 63% of the papers, a newer model with a single prompt natively outperforms the heavily engineered tooling proposed just a year prior. This wide range reflects variation across task types and evaluation metrics. Constructive techniques—such as code generation or repair—are found to be more amenable to substitution by a single prompt. In contrast, a surviving set of papers relies on strategies that provide additional insights to the model, where newer LLMs amplify the proposed technique rather than replace it. These surviving techniques often incorporate domain knowledge, static analysis, or iterative feedback that complements the model's native abilities. The results suggest that the effectiveness of many techniques is tightly coupled to the specific model generation they were designed for, and that model improvements can rapidly erode their advantage.

The software engineering (SE) research community has rapidly integrated Large Language Models (LLMs) into complex techniques to tackle diverse tasks. However, the strategic value of this investment is uncertain: as frontier model generations advance, their native capabilities may render existing techniques obsolete. This paper investigates whether the elaborate tooling proposed in recent LLM-based SE research can be outperformed by the simplest possible alternative—a single, automatically generated prompt executed on a newer generation model, without iterative refinement. The authors analyze 35 LLM-based technique papers from ICSE 2026 to assess the durability of these research contributions. The central question is: what survives the next model? The findings aim to inform the community about the cost-benefit proposition of techniques designed as workarounds to temporary model deficits, and to highlight the need for focusing on enduring challenges that scale synergistically with future model generations.

The study conducts a systematic benchmark of 35 LLM-based technique papers published at ICSE 2026. For each paper, the authors extract the proposed technique and its evaluation setup. They then generate a single, automatically crafted prompt that captures the essence of the task, and execute it on a newer generation LLM without any iterative refinement or complex orchestration. The performance of this single-prompt baseline is compared against the original technique's reported results. The comparison is designed to be fair: the newer model is used zero-shot, with no fine-tuning or task-specific engineering. The authors also categorize the techniques by task type (e.g., code generation, repair, testing) to identify patterns in substitutability. The source code and results are publicly available for reproducibility.

Why it matters

The findings raise critical questions about the cost-benefit proposition of LLM-based SE research. Techniques designed as workarounds to temporary model deficits—such as prompt engineering tricks or multi-step pipelines that compensate for limited context or reasoning—may not survive the next model generation. The authors argue that the community should focus on enduring challenges that scale synergistically with future model generations. This includes techniques that provide additional insights to the model, such as integrating formal methods, program analysis, or human expertise, which are likely to remain valuable as models improve. The study also highlights the need for benchmarking against simple baselines to avoid over-engineering. The taxonomy of techniques that survive versus those that are substituted can guide future research directions. Ultimately, the paper calls for a strategic re-evaluation of how LLM-based techniques are designed and evaluated, emphasizing robustness to model evolution. Substitution Rate = \frac{Number of Techniques Outperformed by Single Prompt}{Total Techniques Evaluated} \times 100% With 35 papers, a substitution rate of 37% corresponds to approximately 13 papers, while 63% corresponds to approximately 22 papers.

Who should read this

CS practitioners and researchers

Opening member content…