Multimodal LLM evaluation: four missing capabilities (2026)

Po-han Li, Shenghui Chen, Sandeep Chinchali and Ufuk Topcu submitted a paper titled "What We are Missing in Multimodal LLM Evaluation?" to arXiv on 24 Jun 2026 (arXiv:2606.26348, file size 5,001 KB). The paper argues that while multimodal large language models, which accept text, images, audio and video and produce text, have advanced rapidly, existing evaluation benchmarks remain narrow and task-isolated.

What gaps did the authors identify?

The paper names four specific shortfalls that current benchmarks do not capture well: temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention. The opening of the abstract lists those four areas, calling out "temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention." The authors contrast these integration-focused requirements with the prevailing approach of isolated task benchmarks, which reveal little about a model's ability to combine information across modalities.

How do current evaluations fall short?

Current benchmarks largely measure isolated skills, the paper states, and therefore fail to test whether a model integrates modalities over time or space. The abstract emphasizes that most existing evaluation benchmarks are limited to isolated tasks and reveal little about a model’s cross-modal integration. The result is blind spots: a model may score well on single-modality tasks yet fail when required to track objects across frames, reason about physical constraints, maintain consistency between modalities, or focus attention on the relevant subset of inputs.

The paper frames these failures as concrete categories. Temporal-spatial coherence covers synchronization across frames and spatial relations in video or image sequences. Physical world understanding refers to commonsense physical reasoning when modalities interact. Multimodal consistency addresses whether outputs stay aligned across text, image and audio inputs. Selective attention concerns whether models can ignore irrelevant inputs and concentrate on the signals that matter.

Why it matters

Measuring only isolated tasks risks overstating progress. If benchmarks do not test integration, models that excel on single tasks can still fail in real-world multimodal settings where signals arrive together and must be fused reliably. The authors say addressing these gaps is essential for measuring real progress in multimodal intelligence and exposing capability boundaries. That matters for researchers building evaluation suites and for engineers choosing models for systems that require cross-modal reasoning, physical-world constraints, or time-sensitive perception.

What to watch

Watch for follow-up benchmark proposals and datasets that explicitly target the four gaps the paper enumerates. The arXiv entry notes the submission as version v1 on 24 Jun 2026 and registers a DOI via DataCite pending registration, so revisions or companion data and code links may appear. Concrete next signals will be new tasks or challenge sets measuring temporal-spatial coherence, physical reasoning across modalities, consistency checks across inputs, or selective-attention scenarios.

The paper itself serves as a compact call to expand evaluation beyond isolated tasks. Its authorship and the explicit list of missing areas give a clear starting point for both benchmark designers and practitioners who need to know whether a model truly integrates multimodal information or only performs well on siloed tasks.