RODS: Reward-Driven Online Data Synthesis cuts data 20x

RODS, by Ruishan Fang, Siyuan Lu, Chenyi Zhuang and Tao Lin (arXiv:2606.19047, submitted 17 Jun 2026), closes the loop between reinforcement learning training and data generation for multi-turn tool-use agents. The method uses reward variance from rollouts as a zero-cost detector of the agent's capability boundary, synthesizes new multi-turn variants that match structural complexity, and manages a dynamic replay buffer that co-evolves with the policy.

What problem does RODS solve?

RODS addresses the rapid depletion of informative samples in static datasets during multi-turn tool-use reinforcement learning. The authors show the GRPO gradient signal concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound, meaning samples near the agent’s capability boundary produce disproportionately large policy gradients. As the policy improves this boundary shifts, and a static dataset loses those high-value samples.

The paper quantifies the data-efficiency outcome: starting from 400 human seeds and maintaining an active training pool of approximately 800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories, and it improves over fixed-data RL and environment augmentation in the authors’ controlled setting.

How does RODS work?

RODS repurposes reward variance computed from routine rollouts as a boundary detector, requires no extra inference beyond those rollouts, synthesizes multi-turn variants aligned to skill and structural complexity (for example API topology and dependency depth), and uses a dynamic replay buffer that evolves with the policy. The pipeline continuously identifies boundary samples, resamples and synthesizes new variants, and feeds them back into training.

Concretely, the method detects high-variance tasks from GRPO rollouts, selects those samples near the capability boundary, and applies a skill-aligned resampling pipeline to generate multi-turn data that preserves structural features such as API call topology and dependency depth. Those synthesized variants populate an active training pool that the policy trains on, and the replay buffer is managed dynamically so the dataset co-evolves with agent performance.

Why it matters

RODS converts a training-side signal already available in rollouts into a practical mechanism for targeted data generation, removing the need for massive static offline corpora. The paper demonstrates that a small, actively managed pool (starting from 400 seeds and ~800 active samples) can match a much larger offline dataset of 17,000 samples while cutting required trajectories by about 20x. That implies lower data collection cost for multi-turn tool-use RL in controlled settings, and a way to focus synthesis where it will most affect learning: the agent’s capability boundary.

What to watch

Look for code and replication materials linked to the arXiv entry and for follow-up work evaluating RODS beyond the authors’ controlled setting. The next concrete milestone is external reproduction of the claimed parity with a 17K-sample offline pipeline and the roughly 20x trajectory reduction on other tool-use environments.

Additional details and the full technical exposition appear in the arXiv submission (arXiv:2606.19047), which includes the motivation tied to GRPO reward-variance concentration and the description of the skill-aligned resampling and dynamic replay buffer designs.