Computer vision system from MIT Sea Grant for fish monitoring

MIT Sea Grant and collaborators have demonstrated a deep learning computer vision system that analyzes citizen-science video to detect and classify fish species in coastal waters. The project, developed with the Woodwell Climate Research Center and additional partners, converts volunteer-submitted footage into structured observations to support monitoring at scales that manual review cannot match.

The team’s pipeline ingests community video and extracts frames for annotation, trains convolutional neural networks to locate and identify fish, and aggregates detections into time-stamped presence and count records. Researchers emphasized integration with citizen science workflows so volunteers both supply source footage and participate in dataset curation.

How the system works

Video from recreational divers, shore-based observers, and baited remote underwater cameras is first preprocessed for quality and frame rate. Annotators label frames to create a training set used to optimize object-detection and classification models. At inference, the pipeline performs per-frame detection, links detections across frames to reduce double counts, and outputs species-level occurrences and simple abundance indicators.

The stack described by the team includes standard computer vision building blocks: frame extraction and filtering, annotation tools for volunteers and experts, model training and validation, and a lightweight inference service for batch processing of new clips. Outputs are produced in a formats compatible with ecological databases so observations can feed into existing monitoring repositories and analyses.

Researchers reported addressing common field challenges: highly variable lighting and turbidity, off-axis and partial views of fish, and uneven taxonomic representation in training data. To reduce annotation cost, the group combined volunteer labeling with expert review and used iterative rounds of model-assisted annotation, where the model proposes labels that humans verify.

Field tests and collaboration

The system was exercised on coastal datasets provided by project partners and citizen scientists. In demonstrations, automated detections produced species lists and temporal occurrence records that aligned with expert-derived summaries for the same clips. The team highlights that automated processing enables rapid review of larger video volumes, shifting human effort toward validation and hard cases rather than exhaustive frame-by-frame scoring.

Woodwell Climate Research Center contributed ecological expertise and baseline monitoring datasets that helped validate the models in temperate coastal conditions. Other collaborators provided deployment guidance and helped design volunteer-facing labeling tools so that community contributors can inspect, correct, and enrich model outputs.

The researchers are presenting the pipeline as a modular approach intended to be adapted for different regions and gear types. They note that performance varies by species and environment, and that ongoing data collection and targeted annotation remain necessary to expand taxonomic coverage and reduce bias toward commonly observed species.

Why it matters

Automating video analysis lowers the manual burden of converting growing volumes of citizen footage into usable ecological data, enabling more frequent and broader geographic monitoring. For managers and researchers, that means faster access to occurrence and relative-abundance signals, while volunteers gain clearer pathways to contribute usable scientific observations.