TickingCollabBench: Time-Sensitive Multi-Agent Minecraft Benchmark

Juheon Yi, Jinglu Wang, Xiaoyi Zhang and Yan Lu submitted a paper on 14 Jun 2026 introducing TickingCollabBench, a Minecraft-based multi-agent benchmark for time-sensitive complementary collaboration tasks (arXiv:2606.15684, DOI https://doi.org/10.48550/arXiv.2606.15684). The authors build a TickingCollab framework and an automated, feasibility-aware pipeline to generate dynamic tasks and evaluate multi-agent coordination under strict real-time constraints.

What is TickingCollabBench?

TickingCollabBench is a benchmark and framework that models a novel class of time-sensitive complementary collaboration tasks in Minecraft, capturing four core characteristics: agent heterogeneity, mandatory collaboration, dynamic environments, and strict real-time constraints with failure risks. The benchmark is intended to reflect these real-world collaboration aspects so researchers can evaluate how agents coordinate when roles differ, environments change, and timing matters.

The paper positions the benchmark as more than scenarios. It provides primitives and task composition tools so researchers can express declarative task layouts and stress coordination under partial observability and time pressure.

How does the framework and benchmark work?

The TickingCollab framework supports generation of diverse dynamic environments and abstracts Minecraft's primitive APIs to enable declarative YAML task specifications for composing events. On top of that, the authors design a feasibility-aware automated benchmark generation pipeline where an LLM drafts structurally diverse task configurations and a feasibility verifier filters out invalid ones using approximate constraints.

Concretely, the workflow described in the paper has an LLM produce task configurations, the feasibility verifier rejects configurations that violate approximate constraints, and the framework uses YAML specifications to map those tasks to Minecraft events by calling abstracted primitive APIs. The arXiv entry also indicates the manuscript includes links and toggles for code, data and media associated with the article, suggesting the authors intended reproducible assets alongside the paper (submission bundle size 9,344 KB).

What did evaluations find?

Evaluations show language-model-driven agents struggle when tasks are time-sensitive and environments are dynamic: the authors report that language latency and the inherent difficulty of coordinating under partial observability and agent heterogeneity cause LLMs to frequently fail under dynamic environments and fall significantly short of a global-knowledge oracle. The paper frames that gap as a core outcome of applying current LLM-based approaches to complementary, time-constrained multi-agent problems in Minecraft.

The evaluation finding ties the benchmark design to the observed failure modes: strict real-time constraints and the need for mandatory, role-complementary actions expose limits of LLM coordination when agents lack shared global state and when message or planning latency matters.

Why it matters

TickingCollabBench makes a case that multi-agent benchmarks must encode timing, role complementarity and environmental dynamics to reveal coordination weaknesses that static or synchronous scenarios hide. The paper shows that even structurally diverse tasks generated by an LLM and pruned by a feasibility verifier still expose substantial performance gaps between decentralized LLM agents and an oracle with global knowledge. For researchers building multi-agent systems, that highlights where improvements—lower-latency coordination, better partial-observability strategies, or tighter role allocation—need to focus.

What to watch

Look for the authors to release the framework artifacts linked on the arXiv entry and for follow-up work that benchmarks methods to close the gap with the global-knowledge oracle. Future evaluations that vary latency, observability and agent heterogeneity will be the clearest signals of progress.

Reference note: the paper was submitted to arXiv on 14 Jun 2026 as arXiv:2606.15684 and is available via DOI https://doi.org/10.48550/arXiv.2606.15684.