Reward as an Agent (2026): DynDiff-GRPO in Embodied World Models

Pu Li, Zhigang Lin, Qiang Wu, Yongxuan Lv, Fei Wang and Shan You posted Reward as An Agent for Embodied World Models to arXiv (arXiv:2606.19990) on 18 Jun 2026. The paper argues that broader exploration in model-based reinforcement learning can scale successfully when it is grounded in robust verification and demonstrates its approach across multiple open-source world models.

What did the paper introduce?

The paper introduces two linked contributions: Reward as an Agent, an agentic reward framework that actively evaluates generated behaviors to provide robust reward signals, and DynDiff-GRPO, a Dynamic-Aware Rollout Diversification method that expands action-space exploration to diversify trajectories. The authors instantiate both ideas in embodied world models and report that the unified approach mitigates reward hacking under distribution shifts and yields significant accuracy gains across multiple open-source world models.

The submission appears on arXiv as 2606.19990, submitted 18 Jun 2026, and the upload is listed at 11,233 KB. The six authors frame the problem as less about exploration per se and more about the absence of reliable verification strategies that let expanded exploration avoid exploitation of imperfect rewards.

How do Reward as an Agent and DynDiff-GRPO work together?

Reward as an Agent supplies verification while DynDiff-GRPO supplies diversified sampling, the paper says: the verifier actively evaluates generated behaviors to provide robust reward signals, and DynDiff-GRPO explicitly expands action-space exploration to broaden state-action coverage and encourage richer embodied behaviors beyond conservative rollouts. Put simply, one component seeks to prevent reward hacking by checking behavior, the other pushes the policy into new parts of the dynamics.

The authors instantiate this pairing in embodied world-model testbeds, where physical plausibility and task completion offer stringent checks on whether increased exploration produces genuine improvements rather than exploited reward artifacts. The unified system is presented as a means to allow reinforcement learning to operate on a more reliable reward foundation while sampling substantially diversified trajectories.

Why it matters

The paper reframes a common constraint in model-based RL: conservative rollouts have been used to limit distribution shift, but that conservatism restricts behavioral diversity and discovery. By shifting attention from raw exploration to verification, the authors provide a concrete path to scale exploration without triggering reward-hacking failure modes. If robust, this approach could let practitioners explore farther from training distributions in embodied settings while retaining meaningful signals for policy improvement.

That matters because embodied world models are sensitive to physical plausibility and dynamics; the paper leverages those constraints as a rigorous testbed to validate that an agentic reward mechanism plus explicit rollout diversification can produce "significant accuracy gains" across open-source models, per the authors.

What to watch

Watch for code or reproducibility artifacts tied to the arXiv entry (arXiv:2606.19990) and for independent evaluations on additional open-source world models. Confirmation will come from community replication of the claimed accuracy gains and from demonstrations that the verifier prevents reward hacking under broader distribution shifts.

Notes

• Paper title: Reward as An Agent for Embodied World Models.
• Authors: Pu Li, Zhigang Lin, Qiang Wu, Yongxuan Lv, Fei Wang, Shan You.
• arXiv identifier: arXiv:2606.19990 [cs.AI], submitted 18 Jun 2026, file 11,233 KB.