MapSatisfyBench: Benchmarking satisfaction-aware map agents

MapSatisfyBench, published as arXiv:2606.17453 on 16 Jun 2026, is a new benchmark built from large-scale, real-world anonymized user data that evaluates map agents on user satisfaction beyond task completion. The paper, authored by Lubin Bai, Mengyu Cao, Sixue Wang, Zhongwei Wan, Yue Pan, Jiale Hou, Xiang Li and Xiuyuan Zhang, frames evaluation around implicit decision factors that often go unspoken in everyday map queries.

What is MapSatisfyBench and what does it measure?

MapSatisfyBench is a benchmark designed to move map-agent evaluation from strict task completion to satisfaction-aware spatial decision making, measuring whether agents recover and act on implicit decision factors present in everyday queries. The benchmark converts satisfaction-relevant factors into objective, quantifiable evaluation targets and supplies ground truth annotated from five dimensions, enabling full-chain evaluation of satisfaction-aware map agents.

The authors define implicit decision factors as needs that affect user acceptance but are frequently not specified by users. The benchmark assesses whether an agent can proactively recover those factors from information available before it responds, rather than relying solely on clarification questions.

How was the benchmark constructed and annotated?

The dataset and evaluation pipeline were built via a restore-identify-filter framework that reconstructs complete user needs from behavior-chain evidence, identifies implicit decision factors, and retains only those supported by pre-query evidence. This framework underpins MapSatisfyBench and is the methodological core the authors present for producing evaluable implicit factors.

MapSatisfyBench is constructed from large-scale, real-world anonymized user data and includes ground-truth annotations across five dimensions. The paper was submitted to arXiv on 16 Jun 2026 as arXiv:2606.17453 and the submission package is 3,914 KB in size. The authors position the restore-identify-filter pipeline as necessary because a factor is only evaluable if it affects user acceptance and can be recovered from information available to the agent before responding.

How do current map agents perform on satisfaction-aware tasks?

According to experiments reported in the paper, current agents generally perform well on explicit task completion but remain limited in satisfying implicit decision factors and in proactively acquiring the evidence needed for satisfaction-aware decisions. The authors emphasize that clarification is effective but increases user burden in daily interactions, and that capable agents should first attempt to recover implicit factors from available sources.

The experiments thus reveal a gap: explicit task success does not imply satisfaction-aware behavior. The benchmark is intended to surface that gap by converting implicit needs into measurable targets and evaluating agent performance across the full decision chain the authors reconstruct.

Why it matters

MapSatisfyBench shifts evaluation toward the kinds of everyday scenarios where map services are most used, where users issue underspecified, informal queries and implicitly expect agents to infer unspoken preferences. By grounding implicit decision factors in behavior-chain evidence and insisting they be recoverable from pre-query information, the benchmark sets a clearer standard for agents that must operate without burdening users with extra clarification. This matters for providers who deploy agents in consumer-facing map services and for researchers trying to close the gap between task completion and actual user satisfaction.

What to watch

Watch for subsequent papers and model evaluations that report performance on the five annotated dimensions defined by MapSatisfyBench and for implementations of the restore-identify-filter framework in production map agents. Progress on agents that proactively acquire pre-query evidence, rather than relying on clarification, will be the clearest signal that satisfaction-aware evaluation is influencing development.

References: MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factors, Bai et al., arXiv:2606.17453, submitted 16 Jun 2026.