Ilmu Komputer & AI editorial
HARTS: Efficient Agentic Reinforcement Learning for Hybrid-Attention Models over Arbitrary Rollout Trees
The core problem
Agentic reinforcement learning (RL) frequently generates irregular rollout trees in which many root-to-leaf trajectories share long common histories. Training each trajectory independently causes the shared prefixes to be recomputed repeatedly, wasting computation. Existing systems are designed primarily for full-attention models and do not provide dense, differentiable hybrid-attention execution that is compatible with activation recomputation.
The paper introduces **HARTS** (*Hybrid-Attention RL over Tree Structures*), a system that targets this gap. HARTS is presented as the first system to demonstrate arbitrary-rollout-tree prefix-sharing speedups on a real hybrid-attention model. The work is motivated by the observation that hybrid-attention architectures combine full attention with chunkwise linear attention, and that this combination requires careful state management when shared prefixes are compressed rather than replayed.
The central problem can be stated as follows: given a rollout tree with shared prefixes, how can one execute forward and backward passes over all branches efficiently while preserving the numerical behavior of trajectory-wise training? HARTS answers thi
Innovation
The evaluation uses an Agentic RL workload generated from SWE-bench tasks. On this workload, HARTS achieves **4.81โ4.87ร** forward/backward/gradient speedup with activation recomputation across multiple parallel configurations. This speedup is attributed to prefix sharing over arbitrary rollout trees and to the packed execution of all branches in one call per round.
Numerical differences between HARTS and the baseline are reported to be comparable to baseline self-rerun variation. This indicates that the bounded state replay and differentiable state handoffs preserve numerical behavior to a degree similar to ordinary run-to-run variation.
In addition, the reward trend of HARTS is similar to the baseline over the first 120 steps of -Bench training. This suggests that the efficiency gains do not come at the cost of degraded learning behavior in the early phase of training. The results are reported across multiple parallel configurations, supporting the generality of the speedup claim.
Why it matters
The main technical contribution of HARTS is the combination of prefix compression with dense, differentiable hybrid-attention execution. By avoiding replay of shared prefixes, the system reduces redundant computation that would otherwise scale with the number of branches sharing a prefix. The linear-time chunkwise algorithm ensures that the number of sequential linear-attention calls is minimized under the packed execution model, which is important for latency and throughput.
A key design choice is the preservation of chunkwise state partitioning. HARTS does not repeat projections, MLP/MoE computation, or final outputs; it performs only bounded state replay for numerical alignment. This keeps the computation close to trajectory-wise training while enabling tree-level batching. The support for activation recomputation is also significant, because it allows memory savings without breaking the differentiable state handoffs.
The treatment of MoE routing is another notable aspect. For deterministic, no-token-drop top- MoE routing, semantic multiplicities restore MoE-objective token weights and load statistics. This ensures that the MoE objective remains consistent even when tokens are packed across branches.
Limitations are not explicitly enumerated in the abstract, but the scope is hybrid-attention models and arbitrary rollout trees. The claim of being the first system to demonstrate arbitrary-rollout-tree prefix-sharing speedups on a real hybrid-attention model suggests that prior work either targeted full-attention models or did not support dense differentiable hybrid-attention execution with activation recomputation. Future work could extend the approach to other attention variants or to broader agentic RL workloads.
Who should read this
Opening member contentโฆ