Integrating multi-modal remote sensing, deep learning, and attention mechanisms for yield prediction in plant breeding experiments.

In both plant breeding and crop management, interpretability plays a crucial role in instilling trust in AI-driven approaches and enabling the provision of actionable insights. The primary objective of this research is to explore and evaluate the potential contributions of deep learning network architectures that employ stacked LSTM for end-of-season maize grain yield prediction. A secondary aim is to expand the capabilities of these networks by adapting them to better accommodate and leverage the multi-modality properties of remote sensing data. In this study, a multi-modal deep learning architecture that assimilates inputs from heterogeneous data streams, including high-resolution hyperspectral imagery, LiDAR point clouds, and environmental data, is proposed to forecast maize crop yields. The architecture includes attention mechanisms that assign varying levels of importance to different modalities and temporal features that, reflect the dynamics of plant growth and environmental interactions. The interpretability of the attention weights is investigated in multi-modal networks that seek to both improve predictions and attribute crop yield outcomes to genetic and environmental variables. This approach also contributes to increased interpretability of the model's predictions. The temporal attention weight distributions highlighted relevant factors and critical growth stages that contribute to the predictions. The results of this study affirm that the attention weights are consistent with recognized biological growth stages, thereby substantiating the network's capability to learn biologically interpretable features. Accuracies of the model's predictions of yield ranged from 0.82-0.93 R2 ref in this genetics-focused study, further highlighting the potential of attention-based models. Further, this research facilitates understanding of how multi-modality remote sensing aligns with the physiological stages of maize. The proposed architecture shows promise in improving predictions and offering interpretable insights into the factors affecting maize crop yields, while demonstrating the impact of data collection by different modalities through the growing season. By identifying relevant factors and critical growth stages, the model's attention weights provide valuable information that can be used in both plant breeding and crop management. The consistency of attention weights with biological growth stages reinforces the potential of deep learning networks in agricultural applications, particularly in leveraging remote sensing data for yield prediction. To the best of our knowledge, this is the first study that investigates the use of hyperspectral and LiDAR UAV time series data for explaining/interpreting plant growth stages within deep learning networks and forecasting plot-level maize grain yield using late fusion modalities with attention mechanisms.

Row selection in remote sensing from four-row plots of maize and sorghum based on repeatability and predictive modeling.

Remote sensing enables the rapid assessment of many traits that provide valuable information to plant breeders throughout the growing season to improve genetic gain. These traits are often extracted from remote sensing data on a row segment (rows within a plot) basis enabling the quantitative assessment of any row-wise subset of plants in a plot, rather than a few individual representative plants, as is commonly done in field-based phenotyping. Nevertheless, which rows to include in analysis is still a matter of debate. The objective of this experiment was to evaluate row selection and plot trimming in field trials conducted using four-row plots with remote sensing traits extracted from RGB (red-green-blue), LiDAR (light detection and ranging), and VNIR (visible near infrared) hyperspectral data. Uncrewed aerial vehicle flights were conducted throughout the growing seasons of 2018 to 2021 with data collected on three years of a sorghum experiment and two years of a maize experiment. Traits were extracted from each plot based on all four row segments (RS) (RS1234), inner rows (RS23), outer rows (RS14), and individual rows (RS1, RS2, RS3, and RS4). Plot end trimming of 40 cm was an additional factor tested. Repeatability and predictive modeling of end-season yield were used to evaluate performance of these methodologies. Plot trimming was never shown to result in significantly different outcomes from non-trimmed plots. Significant differences were often observed based on differences in row selection. Plots with more row segments were often favorable for increasing repeatability, and excluding outer rows improved predictive modeling. These results support long-standing principles of experimental design in agronomy and should be considered in breeding programs that incorporate remote sensing.

A novel transfer learning framework for sorghum biomass prediction using UAV-based remote sensing data and genetic markers.

Yield for biofuel crops is measured in terms of biomass, so measurements throughout the growing season are crucial in breeding programs, yet traditionally time- and labor-consuming since they involve destructive sampling. Modern remote sensing platforms, such as unmanned aerial vehicles (UAVs), can carry multiple sensors and collect numerous phenotypic traits with efficient, non-invasive field surveys. However, modeling the complex relationships between the observed phenotypic traits and biomass remains a challenging task, as the ground reference data are very limited for each genotype in the breeding experiment. In this study, a Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) model is proposed for sorghum biomass prediction. The architecture is designed to exploit the time series remote sensing and weather data, as well as static genotypic information. As a large number of features have been derived from the remote sensing data, feature importance analysis is conducted to identify and remove redundant features. A strategy to extract representative information from high-dimensional genetic markers is proposed. To enhance generalization and minimize the need for ground reference data, transfer learning strategies are proposed for selecting the most informative training samples from the target domain. Consequently, a pre-trained model can be refined with limited training samples. Field experiments were conducted over a sorghum breeding trial planted in multiple years with more than 600 testcross hybrids. The results show that the proposed LSTM-based RNN model can achieve high accuracies for single year prediction. Further, with the proposed transfer learning strategies, a pre-trained model can be refined with limited training samples from the target domain and predict biomass with an accuracy comparable to that from a trained-from-scratch model for both multiple experiments within a given year and across multiple years.

Estimating Leaf Area Index in Row Crops Using Wheel-Based and Airborne Discrete Return Light Detection and Ranging Data.

Leaf area index (LAI) is an important variable for characterizing plant canopy in crop models. It is traditionally defined as the total one-sided leaf area per unit ground area and is estimated by both direct and indirect methods. This paper explores the effectiveness of using light detection and ranging (LiDAR) data to estimate LAI for sorghum and maize with different treatments at multiple times during the growing season from both a wheeled vehicle and Unmanned Aerial Vehicles. Linear and nonlinear regression models are investigated for prediction utilizing statistical and plant structure-based features extracted from the LiDAR point cloud data with ground reference obtained from an in-field plant canopy analyzer (indirect method). Results based on the value of the coefficient of determination (R 2) and root mean squared error for predictive models ranged from ∼0.4 in the early season to ∼0.6 for sorghum and ∼0.5 to 0.80 for maize from 40 Days after Sowing to harvest.

An Out-of-Sample Extension to Manifold Learning via Meta-Modelling.

Unsupervised manifold learning has become accepted as an important tool for reducing dimensionality of a dataset by finding its meaningful low-dimensional representation lying on an unknown nonlinear subspace. Most manifold learning methods only embed an existing dataset, but do not provide an explicit mapping function for novel out-of-sample data, thereby potentially resulting in an ineffective tool for classification purposes, particularly for iterative methods such as active learning. To address this issue, out-of-sample extension methods have been introduced to generalize an existing embedding of new samples. In this work, a novel out-of-sample method is introduced by utilizing High Dimensional Model Representation (HDMR) as a nonlinear multivariate regression with the Tikhonov regularizer for unsupervised manifold learning algorithms. The proposed method was extensively analyzed using illustrative datasets sampled from known manifolds. Several experiments with 3D synthetic datasets and face recognition datasets were also conducted, and the performance of the proposed method was compared to several well-known out-of-sample methods. The results obtained with Locally Linear Embedding (LLE), Laplacian Eigenmaps (LE), and t-Distributed Stochastic Neighbor Embedding (t-SNE) showed that the proposed method achieves competitive even better performance than the other out-of-sample methods.

Unsupervised multistage image classification using hierarchical clustering with a Bayesian similarity measure.

A new multistage method using hierarchical clustering for unsupervised image classification is presented. In the first phase, the multistage method performs segmentation using a hierarchical clustering procedure which confines merging to spatially adjacent clusters and generates an image partition such that no union of any neighboring segments has homogeneous intensity values. In the second phase, the segments resulting from the first stage are classified into a small number of distinct states by a sequential merging operation. The region-merging procedure in the first phase makes use of spatial contextual information by characterizing the geophysical connectedness of a digital image structure with a Markov random field, while the second phase employs a context-free similarity measure in the clustering process. The segmentation procedure of region merging is implemented as a hierarchical clustering algorithm whereby a multiwindow approach using a pyramid-like structure is employed to increase computational efficiency while maintaining spatial connectivity in merging. From experiments with both simulated and remotely sensed data, the proposed method was determined to be quite effective for unsupervised analysis. In particular, the region-merging approach based on spatial contextual information was shown to provide more accurate classification of images with smooth spatial patterns.