Beneath the Diff: Diagnosing and Mitigating Algorithmic Mode Collapse in Code-Level Autonomous Research Loops

Code-level autonomous research loops (ARLs) have emerged as a concrete object of study in automated machine learning research. In such loops, an LLM agent proposes modifications to an experimental training pipeline, executes the modified pipeline, and retains edits that improve a verifiable in-loop metric. Although executable metrics may appear to provide a reliable signal of progress, it remains unclear whether repeated metric-driven code editing leads to genuine improvements that generalize beyond the loop. This paper provides a systematic diagnosis of this question. Across various experiment settings, the authors identify a robust failure mode called **algorithmic mode collapse**. In this regime, surface-level edit diversity remains stable, but semantic and mechanism-level diversity collapse: the agent continues to edit different lines of code while repeatedly proposing the same kinds of algorithmic changes. This collapse is accompanied by a widening gap between in-loop metric gains and gains measured on independent held-out evaluations. The authors then propose Diversity-Aware Proposal Sampling (DAPS), a lightweight mitigation that combines category-coverage reweighting, persistent edit memory, and a validation gate. Under a three-tier protocol separating the in-loop metric, the audit metric read by the gate, and a blind metric no loop component ever accesses, DAPS reduces semantic-cluster decay of edits by 69.1% and improves relative faithfulness by 83.7% blind and 81.6% audited, while preserving in-loop optimization speed.

The authors conduct a systematic diagnosis of algorithmic mode collapse in code-level ARLs. They design experiments across various settings where an LLM agent iteratively proposes code edits to a training pipeline, executes the modified pipeline, and retains edits that improve a verifiable in-loop metric. To measure diversity, they distinguish between surface-level edit diversity (e.g., lines of code changed) and semantic/mechanism-level diversity (e.g., the types of algorithmic changes proposed). They track the evolution of these diversity measures over loop iterations and compare in-loop metric gains with gains on independent held-out evaluations.