DigitalCoach dataset: 72 coaching sessions, 22,752 dialogue turns

DigitalCoach, a multimodal dataset submitted to arXiv on 30 Jun 2026, captures 72 human expert-novice computer use coaching sessions totaling 22,752 dialogue turns and 28.1 hours of synchronized screen and input event recordings across five software applications. The dataset was designed to test whether current agentic models can teach humans how to use software, and to measure differences in communication and grounding between models and people.

What is DigitalCoach?

DigitalCoach is a paired dialogue and interaction corpus: 72 expert-novice coaching sessions, 22,752 dialogue turns, and 28.1 hours of screen and input event recordings spanning five software applications. The dataset aligns utterances with visual context and input traces so researchers can evaluate both conversational behavior and visual grounding in computer use coaching.

The collection emphasizes real coaching interactions rather than scripted exchanges, producing a multimodal resource that links spoken or typed guidance to the learner's on-screen state and inputs. The arXiv submission lists seven authors: Meng Chen, Anya Ji, Tsung-Han Wu, Tobias Maringgele, David M. Chan, Alane Suhr, and Amy Pavel.

How do state-of-the-art models compare to human coaches?

Automated evaluation shows a clear behavioral gap: "models provide more direct instructions, but fewer explanations, error diagnoses, and knowledge-check questions." When coaching method is fixed, models produce utterances similar to human references in style, yet their outputs are poorly grounded in the visual context captured by the dataset.

Interactive evaluation amplifies those findings. Model coaches tend to cause learners to follow instructions passively rather than engage in deeper problem solving, and the models underperform on visually grounded guidance. The paper frames these differences along two axes: communicative content (direct instruction versus explanation and diagnosis) and grounding quality (utterance relevance to on-screen state and input events).

Why it matters

DigitalCoach exposes where current agentic systems fall short at teaching practical skills. Models that favor direct instructions over explanations and diagnostic checks can produce superficial task completion without building user understanding. Poor visual grounding means an agent might give plausible-sounding guidance that does not map to the learner's actual screen state, increasing user confusion.

Those failings affect anyone building tutoring or in-application assistants: designers need metrics and data that capture explanation frequency, error diagnosis, knowledge checks, and grounding fidelity. DigitalCoach supplies those raw signals by combining dialogue with 28.1 hours of recorded interactions across five applications, enabling targeted improvements to model training and evaluation.

What to watch

Look for model evaluations that report improvements on the specific axes DigitalCoach highlights: increases in explanations, error diagnoses, and knowledge-check questions, and measurable gains in visual grounding during interactive tests. Another concrete signal will be follow-up work that uses the dataset to reduce passive user behavior when interacting with model-based coaches.

DigitalCoach lays a foundation for evaluating collaborative and proactive computer use coaching agents, offering a concrete benchmark of 72 sessions and 22,752 turns to measure both what agents say and whether what they say maps to what users actually see and do.