Detecting floating litter in freshwater bodies with semi-supervised deep learning.

Researchers and practitioners have extensively utilized supervised Deep Learning methods to quantify floating litter in rivers and canals. These methods require the availability of large amount of labeled data for training. The labeling work is expensive and laborious, resulting in small open datasets available in the field compared to the comprehensive datasets for computer vision, e.g., ImageNet. Fine-tuning models pre-trained on these larger datasets helps improve litter detection performances and reduces data requirements. Yet, the effectiveness of using features learned from generic datasets is limited in large-scale monitoring, where automated detection must adapt across different locations, environmental conditions, and sensor settings. To address this issue, we propose a two-stage semi-supervised learning method to detect floating litter based on the Swapping Assignments between multiple Views of the same image (SwAV). SwAV is a self-supervised learning approach that learns the underlying feature representation from unlabeled data. In the first stage, we used SwAV to pre-train a ResNet50 backbone architecture on about 100k unlabeled images. In the second stage, we added new layers to the pre-trained ResNet50 to create a Faster R-CNN architecture, and fine-tuned it with a limited number of labeled images (≈1.8k images with 2.6k annotated litter items). We developed and validated our semi-supervised floating litter detection methodology for images collected in canals and waterways of Delft (the Netherlands) and Jakarta (Indonesia). We tested for out-of-domain generalization performances in a zero-shot fashion using additional data from Ho Chi Minh City (Vietnam), Amsterdam and Groningen (the Netherlands). We benchmarked our results against the same Faster R-CNN architecture trained via supervised learning alone by fine-tuning ImageNet pre-trained weights. The findings indicate that the semi-supervised learning method matches or surpasses the supervised learning benchmark when tested on new images from the same training locations. We measured better performances when little data (≈200 images with about 300 annotated litter items) is available for fine-tuning and with respect to reducing false positive predictions. More importantly, the proposed approach demonstrates clear superiority for generalization on the unseen locations, with improvements in average precision of up to 12.7%. We attribute this superior performance to the more effective high-level feature extraction from SwAV pre-training from relevant unlabeled images. Our findings highlight a promising direction to leverage semi-supervised learning for developing foundational models, which have revolutionized artificial intelligence applications in most fields. By scaling our proposed approach with more data and compute, we can make significant strides in monitoring to address the global challenge of litter pollution in water bodies.

Deep learning for detecting macroplastic litter in water bodies: A review.

Plastic pollution in water bodies is an unresolved environmental issue that damages all aquatic environments, and causes economic and health problems. Accurate detection of macroplastic litter (plastic items >5 mm) in water is essential to estimate the quantities, compositions and sources, identify emerging trends, and design preventive measures or mitigation strategies. In recent years, researchers have demonstrated the potential of computer vision (CV) techniques based on deep learning (DL) for automated detection of macroplastic litter in water bodies. However, a systematic review to describe the state-of-the-art of the field is lacking. Here we provide such a review, and we highlight current knowledge gaps and suggest promising future research directions. The review compares 34 papers with respect to their application and modeling related criteria. The results show that the researchers have employed a variety of DL architectures implementing different CV techniques to detect macroplastic litter in various aquatic environments. However, key knowledge gaps must be addressed to overcome the lack of: (i) DL-based macroplastic litter detection models with sufficient generalization capability, (ii) DL-based quantification of macroplastic (mass) fluxes and hotspots and (iii) scalable macroplastic litter monitoring strategies based on robust DL-based quantification. We advocate for the exploration of data-centric artificial intelligence approaches and semi-supervised learning to develop models with improved generalization capabilities. These models can boost the development of new methods for the quantification of macroplastic (mass) fluxes and hotspots, and allow for structural monitoring strategies that leverage robust DL-based quantification. While the identified gaps concern all bodies of water, we recommend increased efforts with respect to riverine ecosystems, considering their major role in transport and storage of litter.