HAS-Bench: Evaluating LLM Human-Agent Systems across Six Domains

HAS-Bench and HAS-Framework, introduced in a paper (arXiv:2607.04329) submitted on 5 Jul 2026 by Yaozu Wu and 13 other authors, formalize and evaluate human participation in LLM-based human-agent systems across six domains. The work presents a graph-based framework and a benchmark that measure both task outcomes and process-level collaboration behavior.

What are HAS-Framework and HAS-Bench?

HAS-Framework is a graph-based framework that represents humans and LLM-powered agents as first-class participants with explicit roles, permissions, communication paths, and action authority. HAS-Bench builds on that framework: it is a benchmark for Human-Agent Systems that evaluates configurable human participation across agency levels, interaction channels, and persona policies.

HAS-Framework makes structure explicit so systems can vary who may act, who may speak, and how control transfers between people and agents. HAS-Bench converts those design choices into measurable scenarios and records both whether tasks succeed and how the collaboration unfolded.

How did the authors evaluate human participation?

HAS-Bench measures task outcomes and process-level collaboration behavior, including clarification quality, feedback utilization, control calibration, safety, initiative, and interaction cost. The benchmark lets researchers configure human participation along three axes: agency levels, interaction channels, and persona policies, then observe both outcome metrics and interaction processes.

The paper lists these six process-level behaviors as core measurements. The authors emphasize that HAS-Bench tracks not only whether a task completes but also how clarification and feedback were used, how control was calibrated between humans and agents, whether interactions remained safe, who took initiative, and how much interaction cost was incurred.

What did the experiments find?

Experiments across six domains show that human participation can substantially improve task completion and failure recovery, but the gains depend on when, how, and by whom human input is exercised. The authors report that human input helps recovery from failures and improves completion rates in many settings, though the benchmark reveals substantial variance tied to the timing and source of that input.

The paper frames those findings by comparing configurable participation modes. By varying agency and channel, HAS-Bench surfaces trade-offs: some configurations raise interaction cost or change initiative patterns even as they boost task success. The benchmark therefore captures both positive and negative process-level consequences of adding human collaborators to LLM-driven workflows.

Why it matters

HAS-Bench reframes evaluation for systems where humans and LLMs act as peers. Measuring clarification quality, feedback utilization, control calibration, safety, initiative, and interaction cost pushes assessments beyond single-shot task accuracy toward collaboration dynamics. That matters for researchers building systems intended to operate with human oversight, and for practitioners who need to decide when and how to insert human judgment into automated flows.

By formalizing roles, permissions, communication paths, and action authority, HAS-Framework gives designers a common vocabulary and experimental surface. HAS-Bench then makes those choices comparable across studies, which could change how teams benchmark real-world deployments.

What to watch

Look for others to adopt HAS-Bench in new domains and for public release of code and datasets linked to the arXiv entry (the paper's arXiv page lists sections for Code, Data and Media). A concrete signal that the approach is taking hold will be independent replications that apply HAS-Bench’s configurable participation axes and report consistent effects on task completion and failure recovery.

Paper and provenance: arXiv:2607.04329, DOI https://doi.org/10.48550/arXiv.2607.04329, submitted on 5 Jul 2026, authored by Yaozu Wu and 13 other authors.