Ilmu Komputer & AI editorial
Open AccessOA2026
Revisiting Feedback-Driven LLM Code Repair: A Replication and Exploratory Java Extension
A partial replication of FeedbackEval on Python and an exploratory Java extension reveal that feedback effectiveness rankings are sensitive to benchmark construction, feedback representation, and tooling ecosystem.
Louis Lalonde; Wassim Keddache; Thomas Perron Touchette; Leuson Da Silva; Foutse Khomh· 2026· DOI 10.48550/arXiv.2609.00362
The core problem
Large Language Models (LLMs) have become increasingly popular for automating software engineering tasks, including code repair. While initial results are promising, concerns about reproducibility and generalizability remain underexplored. This work addresses these concerns by partially reproducing the FeedbackEval benchmark, which evaluates how LLMs leverage different feedback types for Python code repair, and by conducting an exploratory extension to Java. The central research questions are: (1) Can the main qualitative trends of FeedbackEval be reproduced? (2) Do these trends generalize to Java, a different programming language with distinct tooling ecosystems? The study aims to assess the robustness of feedback-driven LLM code repair across languages and to identify factors that influence feedback effectiveness.
Innovation
In the Python replication, the main qualitative trends from the original study are reproduced: test feedback remains the strongest feedback type, significantly improving repair success over no feedback. Specifically, the repair success rate with test feedback is substantially higher than with no feedback, and additional information further enhances performance. However, in the Java extension, the ranking differs: simple test feedback and JUnit-based test feedback do not differ significantly in repair success rate. This suggests that the effectiveness of feedback types may not generalize across languages. Furthermore, the study finds that lighter prompts (i.e., shorter prompts with less context) reduce cost without significant differences in repair effectiveness. This indicates that prompt engineering can be optimized for cost without sacrificing performance. The results also show that the choice of LLM (GPT-4o vs. Claude 3.5 Sonnet) affects absolute performance but not the relative ranking of feedback types in Python.
Large Language Models (LLMs) have become increasingly popular for automating software engineering tasks, including code repair. While initial results are promising, concerns about reproducibility and generalizability remain underexplored. This work addresses these concerns by partially reproducing the FeedbackEval benchmark, which evaluates how LLMs leverage different feedback types for Python code repair, and by conducting an exploratory extension to Java. The central research questions are: (1) Can the main qualitative trends of FeedbackEval be reproduced? (2) Do these trends generalize to Java, a different programming language with distinct tooling ecosystems? The study aims to assess the robustness of feedback-driven LLM code repair across languages and to identify factors that influence feedback effectiveness.
The study comprises two main parts. First, a partial replication of the original FeedbackEval study on 394 repair tasks using two LLMs: GPT-4o and Claude 3.5 Sonnet. The replication follows the original experimental setup, including the types of feedback provided: no feedback, simple test feedback, and test feedback with additional information (e.g., error messages, stack traces). Second, an exploratory Java extension is conducted by constructing 100 erroneous repair instances from 50 Java tasks. These instances are evaluated with the same feedback types, adapted to Java's tooling ecosystem, including simple test feedback and JUnit-based test feedback. The effectiveness of each feedback type is measured by the repair success rate. The study also investigates the impact of prompt length on repair effectiveness and cost. The experimental design allows for a controlled comparison between Python and Java, although the Java extension is exploratory due to the smaller sample size and potential differences in task complexity.
Why it matters
The findings highlight that previous conclusions from Python may be sensitive to benchmark construction, feedback representation, and tooling ecosystem. The discrepancy between Python and Java results may be explained by differences in feedback informativeness and tooling ecosystems. For instance, the verbosity of test frameworks (e.g., JUnit in Java vs. pytest in Python) could affect how much useful information is conveyed to the LLM. In Java, JUnit test failures often produce lengthy stack traces that may overwhelm the model, whereas Python's test output might be more concise. Additionally, the Java extension used a smaller set of tasks, which may limit statistical power. The study hypothesizes that the effectiveness of feedback types depends on the specific language and tooling, motivating the need for more controlled multilingual benchmarks. The finding that lighter prompts reduce cost without significant performance loss suggests that practitioners can adopt cost-saving measures. Overall, the study calls for more rigorous multilingual evaluation and careful feedback design in LLM-based repair systems. Future work should include larger Java benchmarks and other languages to validate these observations.
Who should read this
CS practitioners and researchers
Opening member content…