Out-of-Distribution Detection Algorithms for Robust Insect Classification.

Plants encounter a variety of beneficial and harmful insects during their growth cycle. Accurate identification (i.e., detecting insects' presence) and classification (i.e., determining the type or class) of these insect species is critical for implementing prompt and suitable mitigation strategies. Such timely actions carry substantial economic and environmental implications. Deep learning-based approaches have produced models with good insect classification accuracy. Researchers aim to implement identification and classification models in agriculture, facing challenges when input images markedly deviate from the training distribution (e.g., images like vehicles, humans, or a blurred image or insect class that is not yet trained on). Out-of-distribution (OOD) detection algorithms provide an exciting avenue to overcome these challenges as they ensure that a model abstains from making incorrect classification predictions on images that belong to non-insect and/or untrained insect classes. As far as we know, no prior in-depth exploration has been conducted on the role of the OOD detection algorithms in addressing agricultural issues. Here, we generate and evaluate the performance of state-of-the-art OOD algorithms on insect detection classifiers. These algorithms represent a diversity of methods for addressing an OOD problem. Specifically, we focus on extrusive algorithms, i.e., algorithms that wrap around a well-trained classifier without the need for additional co-training. We compared three OOD detection algorithms: (a) maximum softmax probability, which uses the softmax value as a confidence score; (b) Mahalanobis distance (MAH)-based algorithm, which uses a generative classification approach; and (c) energy-based algorithm, which maps the input data to a scalar value, called energy. We performed an extensive series of evaluations of these OOD algorithms across three performance axes: (a) Base model accuracy: How does the accuracy of the classifier impact OOD performance? (b) How does the level of dissimilarity to the domain impact OOD performance? (c) Data imbalance: How sensitive is OOD performance to the imbalance in per-class sample size? Evaluating OOD algorithms across these performance axes provides practical guidelines to ensure the robust performance of well-trained models in the wild, which is a key consideration for agricultural applications. Based on this analysis, we proposed the most effective OOD algorithm as wrapper for the insect classifier with highest accuracy. We presented the results of its OOD detection performance in the paper. Our results indicate that OOD detection algorithms can significantly enhance user trust in insect pest classification by abstaining classification under uncertain conditions.

Predicting the onset of preeclampsia by longitudinal monitoring of metabolic changes throughout pregnancy with Raman spectroscopy.

Preeclampsia is a life-threatening pregnancy disorder. Current clinical assays cannot predict the onset of preeclampsia until the late 2nd trimester, which often leads to poor maternal and neonatal outcomes. Here we show that Raman spectroscopy combined with machine learning in pregnant patient plasma enables rapid, highly sensitive maternal metabolome screening that predicts preeclampsia as early as the 1st trimester with >82% accuracy. We identified 12, 15 and 17 statistically significant metabolites in the 1st, 2nd and 3rd trimesters, respectively. Metabolic pathway analysis shows multiple pathways corresponding to amino acids, fatty acids, retinol, and sugars are enriched in the preeclamptic cohort relative to a healthy pregnancy. Leveraging Pearson's correlation analysis, we show for the first time with Raman Spectroscopy that metabolites are associated with several clinical factors, including patients' body mass index, gestational age at delivery, history of preeclampsia, and severity of preeclampsia. We also show that protein quantification alone of proinflammatory cytokines and clinically relevant angiogenic markers are inadequate in identifying at-risk patients. Our findings demonstrate that Raman spectroscopy is a powerful tool that may complement current clinical assays in early diagnosis and in the prognosis of the severity of preeclampsia to ultimately enable comprehensive prenatal care for all patients.

Direct Growth Control of Antibiotic-Resistant Bacteria Using Visible-Light-Responsive Novel Photoswitchable Antibiotics.

In addition to the discovery of new (modified) potent antibiotics to combat antibiotic resistance, there is a critical need to develop novel strategies that would restrict their off-target effects and unnecessary exposure to bacteria in our body and environment. We report a set of new photoswitchable arylazopyrazole-modified norfloxacin antibiotics that present a high degree of bidirectional photoisomerization, impressive fatigue resistance and reasonably high cis half-lives. The irradiated isomers of most compounds were found to exhibit nearly equal or higher antibacterial activity than norfloxacin against Gram-positive bacteria. Notably, against norfloxacin-resistant S. aureus bacteria, the visible-light-responsive p-SMe-substituted derivative showed remarkably high antimicrobial potency (MIC of 0.25 µg/mL) in the irradiated state, while the potency was reduced by 24-fold in case of its non-irradiated state. The activity was estimated to be retained for more than 7 hours. This is the first report to demonstrate direct photochemical control of the growth of antibiotic-resistant bacteria and to show the highest activity difference between irradiated and non-irradiated states of a photoswitchable antibiotic. Additionally, both isomers were found to be non-harmful to human cells. Molecular modellings were performed to identify the underlying reason behind the high-affinity binding of the irradiated isomer to topoisomerase IV enzyme.

Cyber-agricultural systems for crop breeding and sustainable production.

The cyber-agricultural system (CAS) represents an overarching framework of agriculture that leverages recent advances in ubiquitous sensing, artificial intelligence, smart actuators, and scalable cyberinfrastructure (CI) in both breeding and production agriculture. We discuss the recent progress and perspective of the three fundamental components of CAS - sensing, modeling, and actuation - and the emerging concept of agricultural digital twins (DTs). We also discuss how scalable CI is becoming a key enabler of smart agriculture. In this review we shed light on the significance of CAS in revolutionizing crop breeding and production by enhancing efficiency, productivity, sustainability, and resilience to changing climate. Finally, we identify underexplored and promising future directions for CAS research and development.

Dominating Set Model Aggregation for communication-efficient decentralized deep learning.

Decentralized deep learning algorithms leverage peer-to-peer communication of model parameters and/or gradients over communication graphs among the learning agents with access to their private data sets. The majority of the studies in this area focus on achieving high accuracy, with many at the expense of increased communication overhead among the agents. However, large peer-to-peer communication overhead often becomes a practical challenge, especially in harsh environments such as for an underwater sensor network. In this paper, we aim to reduce communication overhead while achieving similar performance as the state-of-the-art algorithms. To achieve this, we use the concept of Minimum Connected Dominating Set from graph theory that is applied in ad hoc wireless networks to address communication overhead issues. Specifically, we propose a new decentralized deep learning algorithm called minimum connected Dominating Set Model Aggregation (DSMA). We investigate the efficacy of our method for different communication graph topologies with a small to large number of agents using varied neural network model architectures. Empirical results on benchmark data sets show a significant (up to 100X) reduction in communication time while preserving the accuracy or in some cases, increasing it compared to the state-of-the-art methods. We also present an analysis to show the convergence of our proposed algorithm.

First Trimester Prediction of Preterm Birth in Patient Plasma with Machine-Learning-Guided Raman Spectroscopy and Metabolomics.

Preterm birth (PTB) is the leading cause of infant deaths globally. Current clinical measures often fail to identify women who may deliver preterm. Therefore, accurate screening tools are imperative for early prediction of PTB. Here, we show that Raman spectroscopy is a promising tool for studying biological interfaces, and we examine differences in the maternal metabolome of the first trimester plasma of PTB patients and those that delivered at term (healthy). We identified fifteen statistically significant metabolites that are predictive of the onset of PTB. Mass spectrometry metabolomics validates the Raman findings identifying key metabolic pathways that are enriched in PTB. We also show that patient clinical information alone and protein quantification of standard inflammatory cytokines both fail to identify PTB patients. We show for the first time that synergistic integration of Raman and clinical data guided with machine learning results in an unprecedented 85.1% accuracy of risk stratification of PTB in the first trimester that is currently not possible clinically. Correlations between metabolites and clinical features highlight the body mass index and maternal age as contributors of metabolic rewiring. Our findings show that Raman spectral screening may complement current prenatal care for early prediction of PTB, and our approach can be translated to other patient-specific biological interfaces.

Systematic Performance Evaluation of Reinforcement Learning Algorithms Applied to Wastewater Treatment Control Optimization.

Treatment of wastewater using activated sludge relies on several complex, nonlinear processes. While activated sludge systems can provide high levels of treatment, including nutrient removal, operating these systems is often challenging and energy intensive. Significant research investment has been made in recent years into improving control optimization of such systems, through both domain knowledge and, more recently, machine learning. This study leverages a novel interface between a common process modeling software and a Python reinforcement learning environment to evaluate four common reinforcement learning algorithms for their ability to minimize treatment energy use while maintaining effluent compliance within the Benchmark Simulation Model No. 1 (BSM1) simulation. Three of the algorithms tested, deep Q-learning, proximal policy optimization, and synchronous advantage actor critic, generally performed poorly over the scenarios tested in this study. In contrast, the twin delayed deep deterministic policy gradient (TD3) algorithm consistently produced a high level of control optimization while maintaining the treatment requirements. Under the best selection of state observation features, TD3 control optimization reduced aeration and pumping energy requirements by 14.3% compared to the BSM1 benchmark control, outperforming the advanced domain-based control strategy of ammonia-based aeration control, although future work is necessary to improve robustness of RL implementation.

"Canopy fingerprints" for characterizing three-dimensional point cloud data of soybean canopies.

Advances in imaging hardware allow high throughput capture of the detailed three-dimensional (3D) structure of plant canopies. The point cloud data is typically post-processed to extract coarse-scale geometric features (like volume, surface area, height, etc.) for downstream analysis. We extend feature extraction from 3D point cloud data to various additional features, which we denote as 'canopy fingerprints'. This is motivated by the successful application of the fingerprint concept for molecular fingerprints in chemistry applications and acoustic fingerprints in sound engineering applications. We developed an end-to-end pipeline to generate canopy fingerprints of a three-dimensional point cloud of soybean [Glycine max (L.) Merr.] canopies grown in hill plots captured by a terrestrial laser scanner (TLS). The pipeline includes noise removal, registration, and plot extraction, followed by the canopy fingerprint generation. The canopy fingerprints are generated by splitting the data into multiple sub-canopy scale components and extracting sub-canopy scale geometric features. The generated canopy fingerprints are interpretable and can assist in identifying patterns in a database of canopies, querying similar canopies, or identifying canopies with a certain shape. The framework can be extended to other modalities (for instance, hyperspectral point clouds) and tuned to find the most informative fingerprint representation for downstream tasks. These canopy fingerprints can aid in the utilization of canopy traits at previously unutilized scales, and therefore have applications in plant breeding and resilient crop production.

Few-shot deep learning for AFM force curve characterization of single-molecule interactions.

Deep learning (DL)-based analytics has the scope to transform the field of atomic force microscopy (AFM) with regard to fast and bias-free measurement characterization. For example, AFM force-distance curves can help estimate important parameters of binding kinetics, such as the most probable rupture force, binding probability, association, and dissociation constants, as well as receptor density on live cells. Other than the ideal single-rupture event in the force-distance curves, there can be no-rupture, double-rupture, or multiple-rupture events. The current practice is to go through such datasets manually, which can be extremely tedious work for the experimentalists. We address this issue by adopting a few-shot learning approach to build sample-efficient DL models that demonstrate better performance than shallow ML models while matching the performance of moderately trained humans. We also release our AFM force curve dataset and annotations publicly as a benchmark for the research community.

Deep learning-based phenotyping for genome wide association studies of sudden death syndrome in soybean.

Using a reliable and accurate method to phenotype disease incidence and severity is essential to unravel the complex genetic architecture of disease resistance in plants, and to develop disease resistant cultivars. Genome-wide association studies (GWAS) involve phenotyping large numbers of accessions, and have been used for a myriad of traits. In field studies, genetic accessions are phenotyped across multiple environments and replications, which takes a significant amount of labor and resources. Deep Learning (DL) techniques can be effective for analyzing image-based tasks; thus DL methods are becoming more routine for phenotyping traits to save time and effort. This research aims to conduct GWAS on sudden death syndrome (SDS) of soybean [Glycine max L. (Merr.)] using disease severity from both visual field ratings and DL-based (using images) severity ratings collected from 473 accessions. Images were processed through a DL framework that identified soybean leaflets with SDS symptoms, and then quantified the disease severity on those leaflets into a few classes with mean Average Precision of 0.34 on unseen test data. Both visual field ratings and image-based ratings identified significant single nucleotide polymorphism (SNP) markers associated with disease resistance. These significant SNP markers are either in the proximity of previously reported candidate genes for SDS or near potentially novel candidate genes. Four previously reported SDS QTL were identified that contained a significant SNPs, from this study, from both a visual field rating and an image-based rating. The results of this study provide an exciting avenue of using DL to capture complex phenotypic traits from images to get comparable or more insightful results compared to subjective visual field phenotyping of traits for disease symptoms.

Deep Learning for Live Cell Shape Detection and Automated AFM Navigation.

Atomic force microscopy (AFM) provides a platform for high-resolution topographical imaging and the mechanical characterization of a wide range of samples, including live cells, proteins, and other biomolecules. AFM is also instrumental for measuring interaction forces and binding kinetics for protein-protein or receptor-ligand interactions on live cells at a single-molecule level. However, performing force measurements and high-resolution imaging with AFM and data analytics are time-consuming and require special skill sets and continuous human supervision. Recently, researchers have explored the applications of artificial intelligence (AI) and deep learning (DL) in the bioimaging field. However, the applications of AI to AFM operations for live-cell characterization are little-known. In this work, we implemented a DL framework to perform automatic sample selection based on the cell shape for AFM probe navigation during AFM biomechanical mapping. We also established a closed-loop scanner trajectory control for measuring multiple cell samples at high speed for automated navigation. With this, we achieved a 60× speed-up in AFM navigation and reduced the time involved in searching for the particular cell shape in a large sample. Our innovation directly applies to many bio-AFM applications with AI-guided intelligent automation through image data analysis together with smart navigation.

A Modular Vision Language Navigation and Manipulation Framework for Long Horizon Compositional Tasks in Indoor Environment.

In this paper we propose a new framework-MoViLan (Modular Vision and Language) for execution of visually grounded natural language instructions for day to day indoor household tasks. While several data-driven, end-to-end learning frameworks have been proposed for targeted navigation tasks based on the vision and language modalities, performance on recent benchmark data sets revealed the gap in developing comprehensive techniques for long horizon, compositional tasks (involving manipulation and navigation) with diverse object categories, realistic instructions and visual scenarios with non reversible state changes. We propose a modular approach to deal with the combined navigation and object interaction problem without the need for strictly aligned vision and language training data (e.g., in the form of expert demonstrated trajectories). Such an approach is a significant departure from the traditional end-to-end techniques in this space and allows for a more tractable training process with separate vision and language data sets. Specifically, we propose a novel geometry-aware mapping technique for cluttered indoor environments, and a language understanding model generalized for household instruction following. We demonstrate a significant increase in success rates for long horizon, compositional tasks over recent works on the recently released benchmark data set -ALFRED.

Sugar and stops in drivers with insulin-dependent type 1 diabetes.

BACKGROUND: Diabetes is a major public health challenge, affecting millions of people worldwide. Abnormal physiology in diabetes, particularly hypoglycemia, can cause driver impairments that affect safe driving. While diabetes driver safety has been previously researched, few studies link real-time physiologic changes in drivers with diabetes to objective real-world driver safety, particularly at high-risk areas like intersections. To address this, we investigated the role of acute physiologic changes in drivers with type 1 diabetes mellitus (T1DM) on safe stopping at stop intersections. METHODS: 18 T1DM drivers (21-52 years, µ = 31.2 years) and 14 controls (21-55 years, µ = 33.4 years) participated in a 4-week naturalistic driving study. At induction, each participant's personal vehicle was instrumented with a camera and sensor system to collect driving data (e.g., GPS, video, speed). Video was processed with computer vision algorithms detecting traffic elements (e.g., traffic signals, stop signs). Stop intersections were geolocated with clustering methods, state intersection databases, and manual review. Videos showing driver stop intersection approaches were extracted and manually reviewed to classify stopping behavior (full, rolling, and no stop) and intersection traffic characteristics. RESULTS: Mixed-effects logistic regression models determined how diabetes driver stopping safety (safe vs. unsafe stop) was affected by 1) disease and 2) at-risk, acute physiology (hypo- and hyperglycemia). Diabetes drivers who were acutely hyperglycemic (≥ 300 mg/dL) had 2.37 increased odds of unsafe stopping (95% CI: 1.26-4.47, p = 0.008) compared to those with normal physiology. Acute hypoglycemia did not associate with unsafe stopping (p = 0.537), however the lower frequency of hypoglycemia (vs. hyperglycemia) warrants a larger sample of drivers to investigate this effect. Critically, presence of diabetes alone did not associate with unsafe stopping, underscoring the need to evaluate driver physiology in licensing guidelines. CONCLUSION: This study links acute, abnormal physiologic fluctuations in drivers with diabetes to driver safety based on unsafe stopping at stop-controlled intersections, providing recommendations for clinicians aimed at improving patient safety, fair licensing guidelines, and targets for developing advanced driver assistance systems.

On Consensus-Optimality Trade-offs in Collaborative Deep Learning.

In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality. In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the incremental consensus-based distributed stochastic gradient descent (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the generalized consensus-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning.

A high-fidelity residential building occupancy detection dataset.

This paper describes development of a data acquisition system used to capture a range of occupancy related modalities from single-family residences, along with the dataset that was generated. The publicly available dataset includes: grayscale images at 32-by-32 pixels, captured every second; audio files, which have undergone processing to remove personally identifiable information; indoor environmental readings, captured every ten seconds; and ground truth binary occupancy status. The data acquisition system, coined the mobile human presence detection (HPDmobile) system, was deployed in six homes for a minimum duration of one month each, and captured all modalities from at least four different locations concurrently inside each home. The environmental modalities are available as captured, but to preserve the privacy and identity of the occupants, images were downsized and audio files went through a series of processing steps, as described in this paper. This dataset adds to a very small body of existing data, with applications to energy efficiency and indoor environmental quality.

Using Machine Learning to Develop a Fully Automated Soybean Nodule Acquisition Pipeline (SNAP).

Nodules form on plant roots through the symbiotic relationship between soybean (Glycine max L. Merr.) roots and bacteria (Bradyrhizobium japonicum) and are an important structure where atmospheric nitrogen (N2) is fixed into bioavailable ammonia (NH3) for plant growth and development. Nodule quantification on soybean roots is a laborious and tedious task; therefore, assessment is frequently done on a numerical scale that allows for rapid phenotyping, but is less informative and suffers from subjectivity. We report the Soybean Nodule Acquisition Pipeline (SNAP) for nodule quantification that combines RetinaNet and UNet deep learning architectures for object (i.e., nodule) detection and segmentation. SNAP was built using data from 691 unique roots from diverse soybean genotypes, vegetative growth stages, and field locations and has a good model fit (R 2 = 0.99). SNAP reduces the human labor and inconsistencies of counting nodules, while acquiring quantifiable traits related to nodule growth, location, and distribution on roots. The ability of SNAP to phenotype nodules on soybean roots at a higher throughput enables researchers to assess the genetic and environmental factors, and their interactions on nodulation from an early development stage. The application of SNAP in research and breeding pipelines may lead to more nitrogen use efficiency for soybean and other legume species cultivars, as well as enhanced insight into the plant-Bradyrhizobium relationship.

Deep Multiview Image Fusion for Soybean Yield Estimation in Breeding Applications.

Reliable seed yield estimation is an indispensable step in plant breeding programs geared towards cultivar development in major row crops. The objective of this study is to develop a machine learning (ML) approach adept at soybean (Glycine max L. (Merr.)) pod counting to enable genotype seed yield rank prediction from in-field video data collected by a ground robot. To meet this goal, we developed a multiview image-based yield estimation framework utilizing deep learning architectures. Plant images captured from different angles were fused to estimate the yield and subsequently to rank soybean genotypes for application in breeding decisions. We used data from controlled imaging environment in field, as well as from plant breeding test plots in field to demonstrate the efficacy of our framework via comparing performance with manual pod counting and yield estimation. Our results demonstrate the promise of ML models in making breeding decisions with significant reduction of time and human effort and opening new breeding method avenues to develop cultivars.

UAS-Based Plant Phenotyping for Research and Breeding Applications.

Unmanned aircraft system (UAS) is a particularly powerful tool for plant phenotyping, due to reasonable cost of procurement and deployment, ease and flexibility for control and operation, ability to reconfigure sensor payloads to diversify sensing, and the ability to seamlessly fit into a larger connected phenotyping network. These advantages have expanded the use of UAS-based plant phenotyping approach in research and breeding applications. This paper reviews the state of the art in the deployment, collection, curation, storage, and analysis of data from UAS-based phenotyping platforms. We discuss pressing technical challenges, identify future trends in UAS-based phenotyping that the plant research community should be aware of, and pinpoint key plant science and agronomic questions that can be resolved with the next generation of UAS-based imaging modalities and associated data analysis pipelines. This review provides a broad account of the state of the art in UAS-based phenotyping to reduce the barrier to entry to plant science practitioners interested in deploying this imaging modality for phenotyping in plant breeding and research areas.

Crop yield prediction integrating genotype and weather variables using deep learning.

Accurate prediction of crop yield supported by scientific and domain-relevant insights, is useful to improve agricultural breeding, provide monitoring across diverse climatic conditions and thereby protect against climatic challenges to crop production. We used performance records from Uniform Soybean Tests (UST) in North America to build a Long Short Term Memory (LSTM)-Recurrent Neural Network based model that leveraged pedigree relatedness measures along with weekly weather parameters to dissect and predict genotype response in multiple-environments. Our proposed models outperformed other competing machine learning models such as Support Vector Regression with Radial Basis Function kernel (SVR-RBF), least absolute shrinkage and selection operator (LASSO) regression and the data-driven USDA model for yield prediction. Additionally, for providing interpretability of the important time-windows in the growing season, we developed a temporal attention mechanism for LSTM models. The outputs of such interpretable models could provide valuable insights to plant breeders.

A fast saddle-point dynamical system approach to robust deep learning.

Recent focus on robustness to adversarial attacks for deep neural networks produced a large variety of algorithms for training robust models. Most of the effective algorithms involve solving the min-max optimization problem for training robust models (min step) under worst-case attacks (max step). However, they often suffer from high computational cost from running several inner maximization iterations (to find an optimal attack) inside every outer minimization iteration. Therefore, it becomes difficult to readily apply such algorithms for moderate to large size real world data sets. To alleviate this, we explore the effectiveness of iterative descent-ascent algorithms where the maximization and minimization steps are executed in an alternate fashion to simultaneously obtain the worst-case attack and the corresponding robust model. Specifically, we propose a novel discrete-time dynamical system-based algorithm that aims to find the saddle point of a min-max optimization problem in the presence of uncertainties. Under the assumptions that the cost function is convex and uncertainties enter concavely in the robust learning problem, we analytically show that our algorithm converges asymptotically to the robust optimal solution under a general adversarial budget constraints as induced by ℓp norm, for 1≤p≤∞. Based on our proposed analysis, we devise a fast robust training algorithm for deep neural networks. Although such training involves highly non-convex robust optimization problems, empirical results show that the algorithm can achieve significant robustness compared to other state-of-the-art robust models on benchmark data sets.