Individual canopy tree species maps for the National Ecological Observatory Network.

The ecology of forest ecosystems depends on the composition of trees. Capturing fine-grained information on individual trees at broad scales provides a unique perspective on forest ecosystems, forest restoration, and responses to disturbance. Individual tree data at wide extents promises to increase the scale of forest analysis, biogeographic research, and ecosystem monitoring without losing details on individual species composition and abundance. Computer vision using deep neural networks can convert raw sensor data into predictions of individual canopy tree species through labeled data collected by field researchers. Using over 40,000 individual tree stems as training data, we create landscape-level species predictions for over 100 million individual trees across 24 sites in the National Ecological Observatory Network (NEON). Using hierarchical multi-temporal models fine-tuned for each geographic area, we produce open-source data available as 1 km2 shapefiles with individual tree species prediction, as well as crown location, crown area, and height of 81 canopy tree species. Site-specific models had an average performance of 79% accuracy covering an average of 6 species per site, ranging from 3 to 15 species per site. All predictions are openly archived and have been uploaded to Google Earth Engine to benefit the ecology community and overlay with other remote sensing assets. We outline the potential utility and limitations of these data in ecology and computer vision research, as well as strategies for improving predictions using targeted data sampling.

Hyperspectral signals in the soil: Plant-soil hydraulic connection and disequilibrium as mechanisms of drought tolerance and rapid recovery.

Predicting soil water status remotely is appealing due to its low cost and large-scale application. During drought, plants can disconnect from the soil, causing disequilibrium between soil and plant water potentials at pre-dawn. The impact of this disequilibrium on plant drought response and recovery is not well understood, potentially complicating soil water status predictions from plant spectral reflectance. This study aimed to quantify drought-induced disequilibrium, evaluate plant responses and recovery, and determine the potential for predicting soil water status from plant spectral reflectance. Two species were tested: sweet corn (Zea mays), which disconnected from the soil during intense drought, and peanut (Arachis hypogaea), which did not. Sweet corn's hydraulic disconnection led to an extended 'hydrated' phase, but its recovery was slower than peanut's, which remained connected to the soil even at lower water potentials (-5 MPa). Leaf hyperspectral reflectance successfully predicted the soil water status of peanut consistently, but only until disequilibrium occurred in sweet corn. Our results reveal different hydraulic strategies for plants coping with extreme drought and provide the first example of using spectral reflectance to quantify rhizosphere water status, emphasizing the need for species-specific considerations in soil water status predictions from canopy reflectance.

Spectral ecophysiology: hyperspectral pressure-volume curves to estimate leaf turgor loss.

Turgor loss point (TLP) is an important proxy for plant drought tolerance, species habitat suitability, and drought-induced plant mortality risk. Thus, TLP serves as a critical tool for evaluating climate change impacts on plants, making it imperative to develop high-throughput and in situ methods to measure TLP. We developed hyperspectral pressure-volume curves (PV curves) to estimate TLP using leaf spectral reflectance. We used partial least square regression models to estimate water potential (Ψ) and relative water content (RWC) for two species, Frangula caroliniana and Magnolia grandiflora. RWC and Ψ's model for each species had R2 ≥ 0.7 and %RMSE = 7-10. We constructed PV curves with model estimates and compared the accuracy of directly measured and spectra-predicted TLP. Our findings indicate that leaf spectral measurements are an alternative method for estimating TLP. F. caroliniana TLP's values were -1.62 ± 0.15 (means ± SD) and -1.62 ± 0.34 MPa for observed and reflectance predicted, respectively (P > 0.05), while M. grandiflora were -1.78 ± 0.34 and -1.66 ± 0.41 MPa (P > 0.05). The estimation of TLP through leaf reflectance-based PV curves opens a broad range of possibilities for future research aimed at understanding and monitoring plant water relations on a large scale with spectral ecophysiology.

Data science competition for cross-site individual tree species identification from airborne remote sensing data.

Data on individual tree crowns from remote sensing have the potential to advance forest ecology by providing information about forest composition and structure with a continuous spatial coverage over large spatial extents. Classifying individual trees to their taxonomic species over large regions from remote sensing data is challenging. Methods to classify individual species are often accurate for common species, but perform poorly for less common species and when applied to new sites. We ran a data science competition to help identify effective methods for the task of classification of individual crowns to species identity. The competition included data from three sites to assess each methods' ability to generalize patterns across two sites simultaneously and apply methods to an untrained site. Three different metrics were used to assess and compare model performance. Six teams participated, representing four countries and nine individuals. The highest performing method from a previous competition in 2017 was applied and used as a baseline to understand advancements and changes in successful methods. The best species classification method was based on a two-stage fully connected neural network that significantly outperformed the baseline random forest and gradient boosting ensemble methods. All methods generalized well by showing relatively strong performance on the trained sites (accuracy = 0.46-0.55, macro F1 = 0.09-0.32, cross entropy loss = 2.4-9.2), but generally failed to transfer effectively to the untrained site (accuracy = 0.07-0.32, macro F1 = 0.02-0.18, cross entropy loss = 2.8-16.3). Classification performance was influenced by the number of samples with species labels available for training, with most methods predicting common species at the training sites well (maximum F1 score of 0.86) relative to the uncommon species where none were predicted. Classification errors were most common between species in the same genus and different species that occur in the same habitat. Most methods performed better than the baseline in detecting if a species was not in the training data by predicting an untrained mixed-species class, especially in the untrained site. This work has highlighted that data science competitions can encourage advancement of methods, particularly by bringing in new people from outside the focal discipline, and by providing an open dataset and evaluation criteria from which participants can learn.

Plant Parasitic Nematode Identification in Complex Samples with Deep Learning.

Plant parasitic nematodes are significant contributors to yield loss worldwide, causing devastating losses to every crop species, in every climate. Mitigating these losses requires swift and informed management strategies, centered on identification and quantification of field populations. Current plant parasitic nematode identification methods rely heavily on manual analyses of microscope images by a highly trained nematologist. This mode is not only expensive and time consuming, but often excludes the possibility of widely sharing and disseminating results to inform regional trends and potential emergent issues. This work presents a new public dataset containing annotated images of plant parasitic nematodes from heterologous soil extractions. This dataset serves to propagate new automated methodologies or speedier plant parasitic nematode identification using multiple deep learning object detection models and offers a path towards widely shared tools, results, and meta-analyses.

Perennial grass root system specializes for multiple resource acquisitions with differential elongation and branching patterns.

Roots optimize the acquisition of limited soil resources, but relationships between root forms and functions have often been assumed rather than demonstrated. Furthermore, how root systems co-specialize for multiple resource acquisitions is unclear. Theory suggests that trade-offs exist for the acquisition of different resource types, such as water and certain nutrients. Measurements used to describe the acquisition of different resources should then account for differential root responses within a single system. To demonstrate this, we grew Panicum virgatum in split-root systems that vertically partitioned high water availability from nutrient availability so that root systems must absorb the resources separately to fully meet plant demands. We evaluated root elongation, surface area, and branching, and we characterized traits using an order-based classification scheme. Plants allocated approximately 3/4th of primary root length towards water acquisition, whereas lateral branches were progressively allocated towards nutrients. However, root elongation rates, specific root length, and mass fraction were similar. Our results support the existence of differential root functioning within perennial grasses. Similar responses have been recorded in many plant functional types suggesting a fundamental relationship. Root responses to resource availability can be incorporated into root growth models via maximum root length and branching interval parameters.

Spatial and Texture Analysis of Root System distribution with Earth mover's Distance (STARSEED).

PURPOSE: Root system architectures are complex and challenging to characterize effectively for agronomic and ecological discovery. METHODS: We propose a new method, Spatial and Texture Analysis of Root SystEm distribution with Earth mover's Distance (STARSEED), for comparing root system distributions that incorporates spatial information through a novel application of the Earth Mover's Distance (EMD). RESULTS: We illustrate that the approach captures the response of sesame root systems for different genotypes and soil moisture levels. STARSEED provides quantitative and visual insights into changes that occur in root architectures across experimental treatments. CONCLUSION: STARSEED can be generalized to other plants and provides insight into root system architecture development and response to varying growth conditions not captured by existing root architecture metrics and models. The code and data for our experiments are publicly available: https://github.com/GatorSense/STARSEED .

Computer vision for assessing species color pattern variation from web-based community science images.

Openly available community science digital vouchers provide a wealth of data to study phenotypic change across space and time. However, extracting phenotypic data from these resources requires significant human effort. Here, we demonstrate a workflow and computer vision model for automatically categorizing species color pattern from community science images. Our work is focused on documenting the striped/unstriped color polymorphism in the Eastern Red-backed Salamander (Plethodon cinereus). We used an ensemble convolutional neural network model to analyze this polymorphism in 20,318 iNaturalist images. Our model was highly accurate (∼98%) despite image heterogeneity. We used the resulting annotations to document extensive niche overlap between morphs, but wider niche breadth for striped morphs at the range-wide scale. Our work showcases key design principles for using machine learning with heterogeneous community science image data to address questions at an unprecedented scale.

Evaluation of Postharvest Senescence of Broccoli via Hyperspectral Imaging.

Fresh fruit and vegetables are invaluable for human health; however, their quality often deteriorates before reaching consumers due to ongoing biochemical processes and compositional changes. We currently lack any objective indices which indicate the freshness of fruit or vegetables resulting in limited capacity to improve product quality eventually leading to food loss and waste. In this conducted study, we hypothesized that certain proteins and compounds, such as glucosinolates, could be used as one potential indicator to monitor the freshness of broccoli following harvest. To support our study, glucosinolate contents in broccoli based on HPLC measurement and transcript expression of glucosinolate biosynthetic genes in response to postharvest stresses were evaluated. We found that the glucosinolate biosynthetic pathway coincided with the progression of senescence in postharvest broccoli during storage. Additionally, we applied machine learning-based hyperspectral image (HSI) analysis, unmixing, and subpixel target detection approaches to evaluate glucosinolate level to detect postharvest senescence in broccoli. This study provides an accessible approach to precisely estimate freshness in broccoli through machine learning-based hyperspectral image analysis. Such a tool would further allow significant advancement in postharvest logistics and bolster the availability of high-quality, nutritious fresh produce.

Jointly Optimized Spatial Histogram UNET Architecture (JOSHUA) for Adipose Tissue Segmentation.

Objective. We aim to develop a machine learning algorithm to quantify adipose tissue deposition at surgical sites as a function of biomaterial implantation. Impact Statement. To our knowledge, this study is the first investigation to apply convolutional neural network (CNN) models to identify and segment adipose tissue in histological images from silk fibroin biomaterial implants. Introduction. When designing biomaterials for the treatment of various soft tissue injuries and diseases, one must consider the extent of adipose tissue deposition. In this work, we analyzed adipose tissue accumulation in histological images of sectioned silk fibroin-based biomaterials excised from rodents following subcutaneous implantation for 1, 2, 4, or 8 weeks. Current strategies for quantifying adipose tissue after biomaterial implantation are often tedious and prone to human bias during analysis. Methods. We used CNN models with novel spatial histogram layer(s) that can more accurately identify and segment regions of adipose tissue in hematoxylin and eosin (H&E) and Masson's trichrome stained images, allowing for determination of the optimal biomaterial formulation. We compared the method, Jointly Optimized Spatial Histogram UNET Architecture (JOSHUA), to the baseline UNET model and an extension of the baseline model, attention UNET, as well as to versions of the models with a supplemental attention-inspired mechanism (JOSHUA+ and UNET+). Results. The inclusion of histogram layer(s) in our models shows improved performance through qualitative and quantitative evaluation. Conclusion. Our results demonstrate that the proposed methods, JOSHUA and JOSHUA+, are highly beneficial for adipose tissue identification and localization. The new histological dataset and code used in our experiments are publicly available.

A benchmark dataset for canopy crown detection and delineation in co-registered airborne RGB, LiDAR and hyperspectral imagery from the National Ecological Observation Network.

Broad scale remote sensing promises to build forest inventories at unprecedented scales. A crucial step in this process is to associate sensor data into individual crowns. While dozens of crown detection algorithms have been proposed, their performance is typically not compared based on standard data or evaluation metrics. There is a need for a benchmark dataset to minimize differences in reported results as well as support evaluation of algorithms across a broad range of forest types. Combining RGB, LiDAR and hyperspectral sensor data from the USA National Ecological Observatory Network's Airborne Observation Platform with multiple types of evaluation data, we created a benchmark dataset to assess crown detection and delineation methods for canopy trees covering dominant forest types in the United States. This benchmark dataset includes an R package to standardize evaluation metrics and simplify comparisons between methods. The benchmark dataset contains over 6,000 image-annotated crowns, 400 field-annotated crowns, and 3,000 canopy stem points from a wide range of forest types. In addition, we include over 10,000 training crowns for optional use. We discuss the different evaluation data sources and assess the accuracy of the image-annotated crowns by comparing annotations among multiple annotators as well as overlapping field-annotated crowns. We provide an example submission and score for an open-source algorithm that can serve as a baseline for future methods.

Predictive models to identify Holstein cows at risk of metritis and clinical cure and reproductive/productive failure following antimicrobial treatment.

Precision dairy farming, specifically the design of management strategies according to the animal's needs, may soon become the norm since automated technologies that generate large amounts of data for each individual are becoming more affordable. Our objectives were to determine whether the use of behavioral changes could improve the accuracy of prediction of the risk of metritis and the risk of clinical cure of cows diagnosed with metritis. Addition of behavioral data to the algorithms to predict the outcomes of interest increased their accuracy by 7 to 32%. The incidence of metritis in postpartum dairy cows ranges from 20 to 40%. Unfortunately, approximately 30% of cows treated with antimicrobials following the diagnosis of metritis fail to cure and have impaired reproductive performance. Automated behavior monitoring devices have become more affordable and accessible. In the current study, we investigated whether behavioral changes recorded by automated devices improve models for the prediction, within 42 h of calving, of metritis and acute metritis. Furthermore, we determined whether behavioral changes aid on the prediction, 24 h before the diagnosis of metritis, of cure in response to antimicrobial treatments and the reproductive (failure to become pregnant)/productive (bottom quartile of milk yield) success within 200 d in milk (DIM). At enrollment, Holstein cows (n = 555) from two farms were fitted with an automated device (HR-LDn tag, SCR Engineers Ltd., Netanya, Israel) 21 d before the expected calving date. Cows were examined for metritis (fetid, watery, red/brown uterine discharge) and were randomly assigned to receive ampicillin trihydrate or ceftiofur crystalline free acid treatments. Contemporary cows with no clinical diseases (NoCD = 362) were paired with cows with metritis. Cure from metritis was defined as the absence of fetid, watery, pink/brown uterine discharge and rectal temperature < 39.5 °C, 11 d after diagnosis. In addition, cows in the lowest quartile of milk production, within lactation and farm, and that were not pregnant by 200 DIM were classified as failure. We built models containing: routinely-available data [lactation number (1, 2, ≥3), calf sex, still birth, twining, dystocia, vaginal laceration score, days on the close-up diets], body condition score (BCS) and BCS change from enrollment to calving (ΔBCS), behavior (feeding, rumination, idle, and active time), and their interactions. The area under the curve (AUC) of the models containing routinely-available data, ΔBCS, and behavior data at 2 DIM to predict metritis [AUC = 0.82, 95% confidence interval (CI) = 0.78, 0.85] and acute metritis (AUC = 0.87, 95% CI = 0.83, 0.89) were (P < 0.01) excellent; whereas the models predicting cure (AUC = 0.92, 95% CI = 0.85, 0.95) and failure (AUC = 0.90, 95% CI = 0.84, 0.94) were outstanding. Behavioral changes peripartum contribute for the identification of cows at risk for metritis, allowing the development of preventive strategies. In addition, predicting whether cows will respond to antimicrobial treatment and succeed during lactation may allow for earlier decision-making regarding treatment and culling.

Non-Invasive Heart Rate Estimation From Ballistocardiograms Using Bidirectional LSTM Regression.

Non-invasive heart rate estimation is of great importance in daily monitoring of cardiovascular diseases. In this paper, a bidirectional long short term memory (bi-LSTM) regression network is developed for non-invasive heart rate estimation from the ballistocardiograms (BCG) signals. The proposed deep regression model provides an effective solution to the existing challenges in BCG heart rate estimation, such as the mismatch between the BCG signals and ground-truth reference, multi-sensor fusion and effective time series feature learning. Allowing label uncertainty in the estimation can reduce the manual cost of data annotation while further improving the heart rate estimation performance. Compared with the state-of-the-art BCG heart rate estimation methods, the strong fitting and generalization ability of the proposed deep regression model maintains better robustness to noise (e.g., sensor noise) and perturbations (e.g., body movements) in the BCG signals and provides a more reliable solution for long term heart rate monitoring.

A remote sensing derived data set of 100 million individual tree crowns for the National Ecological Observatory Network.

Forests provide biodiversity, ecosystem, and economic services. Information on individual trees is important for understanding forest ecosystems but obtaining individual-level data at broad scales is challenging due to the costs and logistics of data collection. While advances in remote sensing techniques allow surveys of individual trees at unprecedented extents, there remain technical challenges in turning sensor data into tangible information. Using deep learning methods, we produced an open-source data set of individual-level crown estimates for 100 million trees at 37 sites across the United States surveyed by the National Ecological Observatory Network's Airborne Observation Platform. Each canopy tree crown is represented by a rectangular bounding box and includes information on the height, crown area, and spatial location of the tree. These data have the potential to drive significant expansion of individual-level research on trees by facilitating both regional analyses and cross-region comparisons encompassing forest types from most of the United States.

RhizoVision Crown: An Integrated Hardware and Software Platform for Root Crown Phenotyping.

Root crown phenotyping measures the top portion of crop root systems and can be used for marker-assisted breeding, genetic mapping, and understanding how roots influence soil resource acquisition. Several imaging protocols and image analysis programs exist, but they are not optimized for high-throughput, repeatable, and robust root crown phenotyping. The RhizoVision Crown platform integrates an imaging unit, image capture software, and image analysis software that are optimized for reliable extraction of measurements from large numbers of root crowns. The hardware platform utilizes a backlight and a monochrome machine vision camera to capture root crown silhouettes. The RhizoVision Imager and RhizoVision Analyzer are free, open-source software that streamline image capture and image analysis with intuitive graphical user interfaces. The RhizoVision Analyzer was physically validated using copper wire, and features were extensively validated using 10,464 ground-truth simulated images of dicot and monocot root systems. This platform was then used to phenotype soybean and wheat root crowns. A total of 2,799 soybean (Glycine max) root crowns of 187 lines and 1,753 wheat (Triticum aestivum) root crowns of 186 lines were phenotyped. Principal component analysis indicated similar correlations among features in both species. The maximum heritability was 0.74 in soybean and 0.22 in wheat, indicating that differences in species and populations need to be considered. The integrated RhizoVision Crown platform facilitates high-throughput phenotyping of crop root crowns and sets a standard by which open plant phenotyping platforms can be benchmarked.

Super resolution for root imaging.

PREMISE: High-resolution cameras are very helpful for plant phenotyping as their images enable tasks such as target vs. background discrimination and the measurement and analysis of fine above-ground plant attributes. However, the acquisition of high-resolution images of plant roots is more challenging than above-ground data collection. An effective super-resolution (SR) algorithm is therefore needed for overcoming the resolution limitations of sensors, reducing storage space requirements, and boosting the performance of subsequent analyses. METHODS: We propose an SR framework for enhancing images of plant roots using convolutional neural networks. We compare three alternatives for training the SR model: (i) training with non-plant-root images, (ii) training with plant-root images, and (iii) pretraining the model with non-plant-root images and fine-tuning with plant-root images. The architectures of the SR models were based on two state-of-the-art deep learning approaches: a fast SR convolutional neural network and an SR generative adversarial network. RESULTS: In our experiments, we observed that the SR models improved the quality of low-resolution images of plant roots in an unseen data set in terms of the signal-to-noise ratio. We used a collection of publicly available data sets to demonstrate that the SR models outperform the basic bicubic interpolation, even when trained with non-root data sets. DISCUSSION: The incorporation of a deep learning-based SR model in the imaging process enhances the quality of low-resolution images of plant roots. We demonstrate that SR preprocessing boosts the performance of a machine learning system trained to separate plant roots from their background. Our segmentation experiments also show that high performance on this task can be achieved independently of the signal-to-noise ratio. We therefore conclude that the quality of the image enhancement depends on the desired application.

Hyperspectral tree crown classification using the multiple instance adaptive cosine estimator.

Tree species classification using hyperspectral imagery is a challenging task due to the high spectral similarity between species and large intra-species variability. This paper proposes a solution using the Multiple Instance Adaptive Cosine Estimator (MI-ACE) algorithm. MI-ACE estimates a discriminative target signature to differentiate between a pair of tree species while accounting for label uncertainty. Multi-class species classification is achieved by training a set of one-vs-one MI-ACE classifiers corresponding to the classification between each pair of tree species and a majority voting on the classification results from all classifiers. Additionally, the performance of MI-ACE does not rely on parameter settings that require tuning resulting in a method that is easy to use in application. Results presented are using training and testing data provided by a data analysis competition aimed at encouraging the development of methods for extracting ecological information through remote sensing obtained through participation in the competition. The experimental results using one-vs-one MI-ACE technique composed of a hierarchical classification, where a tree crown is first classified to one of the genus classes and one of the species classes. The species-level rank-1 classification accuracy is 86.4% and cross entropy is 0.9395 on the testing data, provided by the competition organizer, without the release of ground truth for testing data. Similarly, the same evaluation metrics are computed on the training data, where the rank-1 classification accuracy is 95.62% and the cross entropy is 0.2649. The results show that the presented approach can not only classify the majority species classes, but also classify the rare species classes.

Multiple Instance Dictionary Learning for Beat-to-Beat Heart Rate Monitoring From Ballistocardiograms.

A multiple instance dictionary learning approach, dictionary learning using functions of multiple instances (DL-FUMI), is used to perform beat-to-beat heart rate estimation and to characterize heartbeat signatures from ballistocardiogram (BCG) signals collected with a hydraulic bed sensor. DL-FUMI estimates a "heartbeat concept" that represents an individual's personal ballistocardiogram heartbeat pattern. DL-FUMI formulates heartbeat detection and heartbeat characterization as a multiple instance learning problem to address the uncertainty inherent in aligning BCG signals with ground truth during training. Experimental results show that the estimated heartbeat concept obtained by DL-FUMI is an effective heartbeat prototype and achieves superior performance over comparison algorithms.

Discriminative Multiple Instance Hyperspectral Target Characterization.

In this paper, two methods for discriminative multiple instance target characterization, MI-SMF and MI-ACE, are presented. MI-SMF and MI-ACE estimate a discriminative target signature from imprecisely-labeled and mixed training data. In many applications, such as sub-pixel target detection in remotely-sensed hyperspectral imagery, accurate pixel-level labels on training data is often unavailable and infeasible to obtain. Furthermore, since sub-pixel targets are smaller in size than the resolution of a single pixel, training data is comprised only of mixed data points (in which target training points are mixtures of responses from both target and non-target classes). Results show improved, consistent performance over existing multiple instance concept learning methods on several hyperspectral sub-pixel target detection problems.

Partial Membership Latent Dirichlet Allocation for Soft Image Segmentation.

Topic models [e.g., probabilistic latent semantic analysis, latent Dirichlet allocation (LDA), and supervised LDA] have been widely used for segmenting imagery. However, these models are confined to crisp segmentation, forcing a visual word (i.e., an image patch) to belong to one and only one topic. Yet, there are many images in which some regions cannot be assigned a crisp categorical label (e.g., transition regions between a foggy sky and the ground or between sand and water at a beach). In these cases, a visual word is best represented with partial memberships across multiple topics. To address this, we present a partial membership LDA (PM-LDA) model and an associated parameter estimation algorithm. This model can be useful for imagery, where a visual word may be a mixture of multiple topics. Experimental results on visual and sonar imagery show that PM-LDA can produce both crisp and soft semantic image segmentations; a capability previous topic modeling methods do not have.