LyndHub
Jadwal Sholat

Memuat jadwal sholat…

← All journalsSearch index

Ilmu Komputer & AI editorial

Open AccessOA2026

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

AutoSciRub: An Evaluation-First Framework for Autonomous Scientific Research
Xuehai Wang; Haowei Qin; Tongxin Liu; Junkai Li; Buqiang Xu; Jintian Zhang; Yijun Chen; Zirui Xue; Shumin Deng· 2026· DOI 10.48550/arXiv.2608.31076

The core problem

Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address this problem, the authors present **AutoSciRub**, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification, as well as iterative revision. The core insight is that explicit evaluation criteria should be established *before* the research process, rather than relying on post-hoc assessment. This shifts the paradigm from "improve then evaluate" to "evaluate before improving."

Innovation

The authors evaluate AutoSciRub on two benchmarks: **ResearchClawBench** and **AstaBench E2E Discovery**.

- On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of **2.08 points** across three backbone LLMs under the fixed Codex harness, and **2.95 points** across three agent harnesses using a fixed DeepSeek-V4-Flash backbone.
- On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub achieves an average improvement of **16.8 points** across three agent harnesses, while maintaining or increasing the number of successfully completed tasks.

These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research. The improvements are consistent across different backbone models and agent harnesses, indicating the robustness of the approach.

Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address this problem, the authors present **AutoSciRub**, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification, as well as iterative revision. The core insight is that explicit evaluation criteria should be established *before* the research process, rather than relying on post-hoc assessment. This shifts the paradigm from "improve then evaluate" to "evaluate before improving."
AutoSciRub operates in three main stages:

Why it matters

The key contribution of AutoSciRub is the shift from post-hoc evaluation to pre-execution rubric induction. By making implicit requirements explicit, the framework reduces the risk of missing analyses, inappropriate methods, and unsupported conclusions. The rubric serves as a contract between the user's intent and the agent's execution, enabling targeted revision.

The results show that the benefits are more pronounced on the more challenging AstaBench E2E Discovery subset, with an average improvement of 16.8 points, compared to 2.08–2.95 points on ResearchClawBench. This suggests that evaluation-first guidance is particularly valuable when tasks are complex and underspecified.

However, the approach relies on the quality of the induced rubric. If the rubric is incomplete or incorrect, the agent may still fail to meet the true requirements. Future work could explore interactive rubric refinement and learning from human feedback. Additionally, the computational overhead of rubric induction and verification should be considered in resource-constrained settings.

Overall, AutoSciRub represents a promising direction for controlling autonomous research agents, and the code is available at https://github.com/zjunlp/AutoSciRub.

Who should read this

CS practitioners and researchers

Opening member content…