WebStep benchmark for web agents with semantic state

Jiwan Chung, JiHyuk Byun, Vibhav Vineet and Seon Joo Kim submitted WebStep on 8 Apr 2026: a benchmark of 1,800 task instances that attaches an automatic semantic state tracker to websites so researchers can evaluate web agents at the process level rather than only by terminal success.

What is WebStep and how does it work?

WebStep is a benchmark of 1,800 task instances where each website exposes a deterministic semantic MDP alongside the GUI; the agent interacts with the interface while the environment records high-level states and transitions in the background, enabling fine-grained, automatic analysis without manual annotation. The semantic MDP records the environment-level state and transitions during an interaction trajectory, producing a semantic trajectory researchers can use to compute process metrics such as exploration reach, execution accuracy and per-skill success.

The paper frames WebStep as a controlled dataset with difficulty tiers, allowing decomposition of agent behavior by skill and by the sequence of actions that lead to success or failure. Because the semantic MDP is deterministic and recorded by the environment, the benchmark avoids costly manual labeling while still providing a process-level signal tied to the agent's GUI behavior.

What did the process-level analysis reveal?

Process metrics reveal differences invisible to terminal success: three agents whose overall success rates cluster within 31-33% diverge sharply on where they fail and why. The authors first show that although several agents achieve similar end-to-end success rates, they differ in whether they explore more of the interface or execute actions accurately once they reach the right area.

Decomposing performance by skill produces concrete contrasts. On the Housing website, OpenAI CUA outperforms Qwen3.5 by 23.7% on commit actions yet underperforms the same model by 15.6% on filtering. Bifurcation analysis further isolates the single decisive error in a trajectory that flips success to failure, and the paper finds that this decisive error is agent-specific rather than shared across models. The authors also note that these distinctions widen with task difficulty: success rates remain similar on easy tasks but separate sharply as exploration demands increase.

Those process-level signals let researchers point to specific skills or action types to improve. An agent with high exploration reach but low execution accuracy needs different fixes than an agent that reaches the target area rarely but acts correctly once there.

Why it matters

Benchmarks that report only terminal success mask complementary failure modes. WebStep supplies an automated way to measure where a web agent goes wrong at the action and skill level, turning otherwise opaque failures into actionable diagnostics. That matters for both model developers who need targeted interventions and for benchmark designers who want metrics that isolate exploration from execution and expose error localization across long interaction sequences.

By making the semantic state explicit and deterministic, WebStep reduces the manual effort required to get process-level data, which should accelerate iterative improvement cycles when teams tune agent policies or training curricula based on specific skill gaps.

What to watch

Look for other research groups to reuse WebStep's semantic-state approach to build benchmarks that separate exploration and execution, or to publish follow-up evaluations that apply the paper's bifurcation and per-skill analyses to larger agent families. The clearest check of WebStep's influence will be whether subsequent papers use its semantic MDP traces to propose targeted fixes for the exact skill deficits it surfaces.