ConvNets for counting: Object detection of transient phenomena in steelpan drums.

We train an object detector built from convolutional neural networks to count interference fringes in elliptical antinode regions in frames of high-speed video recordings of transient oscillations in Caribbean steelpan drums, illuminated by electronic speckle pattern interferometry (ESPI). The annotations provided by our model aim to contribute to the understanding of time-dependent behavior in such drums by tracking the development of sympathetic vibration modes. The system is trained on a dataset of crowdsourced human-annotated images obtained from the Zooniverse Steelpan Vibrations Project. Due to the small number of human-annotated images and the ambiguity of the annotation task, we also evaluate the model on a large corpus of synthetic images whereby the properties have been matched to the real images by style transfer using a Generative Adversarial Network. Applying the model to thousands of unlabeled video frames, we measure oscillations consistent with audio recordings of these drum strikes. One unanticipated result is that sympathetic oscillations of higher-octave notes significantly precede the rise in sound intensity of the corresponding second harmonic tones; the mechanism responsible for this remains unidentified. This paper primarily concerns the development of the predictive model; further exploration of the steelpan images and deeper physical insights await its further application.

Individual identification in acoustic recordings.

Recent advances in bioacoustics combined with acoustic individual identification (AIID) could open frontiers for ecological and evolutionary research because traditional methods of identifying individuals are invasive, expensive, labor-intensive, and potentially biased. Despite overwhelming evidence that most taxa have individual acoustic signatures, the application of AIID remains challenging and uncommon. Furthermore, the methods most commonly used for AIID are not compatible with many potential AIID applications. Deep learning in adjacent disciplines suggests opportunities to advance AIID, but such progress is limited by training data. We suggest that broadscale implementation of AIID is achievable, but researchers should prioritize methods that maximize the potential applications of AIID, and develop case studies with easy taxa at smaller spatiotemporal scales before progressing to more difficult scenarios.

Tracking transients in steelpan strikes using surveillance technology.

This paper presents advancements in tracking features in high-speed videos of Caribbean steelpans illuminated by electronic speckle pattern interferometry, made possible by incorporating robust computer vision libraries for object detection and image segmentation, and cleaning of the training dataset. Besides increasing the accuracy of fringe counts by 10% or more compared to previous work, this paper introduces a segmentation-regression map for the entire drum surface yielding interference fringe counts comparable to those obtained via object detection. Once trained, this model can count fringes for musical instruments not part of the training set, including those with non-elliptical antinode shapes.