Frame Rate Effects on LSTM/GRU for SSBD Autism Hand Motions

Raunak Mondal and Peter Washington submitted a paper on 8 Jul 2026 showing that sampling video every 15 frames yields peak sequence-based classification accuracies of 97.5% for LSTM and 98.75% for GRU on the Self-Stimulatory Behavior Diagnosis (SSBD) dataset. The study evaluates sequence models, temporal sampling rates, and augmentation strategies for detecting autism-related self-stimulatory hand behaviors from video.

How did frame rate change model accuracy?

Sampling every 15 frames produced the best results: the LSTM model reached 97.5% accuracy and the GRU reached 98.75% accuracy on pose-derived features from SSBD. The authors trained both LSTM and GRU models at sampling intervals of 1, 5, 15, 30, 45, and 90 frames and compared them to prior convolutional neural network baselines reported at 62 to 76% accuracy.

The paper frames the comparison around sequence-based recurrent architectures for temporal dynamics. Both LSTM and GRU exceeded the earlier CNN baselines, with the strongest performance concentrated at the 15-frame sampling interval, suggesting a mid-range temporal sampling favours capturing the idiosyncratic hand motions in SSBD-derived clips.

Which data augmentation strategies helped and how were they tested?

Ten data augmentation strategies were applied within an I3D transfer learning pipeline and evaluated by ablation to quantify marginal contributions. Horizontal flip was the single-best standalone augmentation, achieving 48.78% accuracy on its own, while removal of upsampling from the augmentation pipeline produced the largest degradation in performance, indicating upsampling was necessary for effective augmentation on these behavioral videos.

The authors report the augmentation study separately from the LSTM/GRU experiments: the I3D transfer learning pipeline served as the testbed for the ablation study, isolating the impact of each augmentation technique on a data-scarce clinical task.

What about personalization and per-subject models?

A personalized machine learning approach, training per-subject models and testing on temporally split segments of each video, produced consistent predictions with a reported mean loss of 1.84 and a standard deviation of 0.79. That result emphasizes stable within-subject model behaviour when segments are split temporally rather than mixed across subjects.

The paper bundles these results with architecture and sampling recommendations to guide practitioners working with small behavioral datasets, where subject variability and limited clips make augmentation and sampling choices critical.

Why it matters

Automated detection of autism-related self-stimulatory behaviors is constrained by small labeled datasets and subtle temporal dynamics. Demonstrating near-100% accuracy for sequence models at a concrete sampling interval, and showing which augmentations matter, gives clinicians and engineers actionable settings: prefer sequence models over the reported CNN baselines and sample at a cadence similar to every 15 frames when using pose-derived features on SSBD-like data. The finding that exclusion of upsampling substantially harms augmentation pipelines highlights a technical lever for improving performance when raw video diversity is limited.

What to watch

Look for follow-up work that publishes per-interval accuracy curves for LSTM and GRU across the full set of tested sampling intervals and for code or checkpoints from the I3D augmentation ablation. The submission lists a DOI (https://doi.org/10.48550/arXiv.2607.07957) and notes a preliminary version was presented as a poster at the AMIA 2024 Informatics Summit; the full 15-page paper includes five figures and three tables for deeper inspection.