A review of ultrasonic sensing and machine learning methods to monitor industrial processes.

Supervised machine learning techniques are increasingly being combined with ultrasonic sensor measurements owing to their strong performance. These techniques also offer advantages over calibration procedures of more complex fitting, improved generalisation, reduced development time, ability for continuous retraining, and the correlation of sensor data to important process information. However, their implementation requires expertise to extract and select appropriate features from the sensor measurements as model inputs, select the type of machine learning algorithm to use, and find a suitable set of model hyperparameters. The aim of this article is to facilitate implementation of machine learning techniques in combination with ultrasonic measurements for in-line and on-line monitoring of industrial processes and other similar applications. The article first reviews the use of ultrasonic sensors for monitoring processes, before reviewing the combination of ultrasonic measurements and machine learning. We include literature from other sectors such as structural health monitoring. This review covers feature extraction, feature selection, algorithm choice, hyperparameter selection, data augmentation, domain adaptation, semi-supervised learning and machine learning interpretability. Finally, recommendations for applying machine learning to the reviewed processes are made.

Volumetric Segmentation of Cell Cycle Markers in Confocal Images Using Machine Learning and Deep Learning.

Understanding plant growth processes is important for many aspects of biology and food security. Automating the observations of plant development-a process referred to as plant phenotyping-is increasingly important in the plant sciences, and is often a bottleneck. Automated tools are required to analyze the data in microscopy images depicting plant growth, either locating or counting regions of cellular features in images. In this paper, we present to the plant community an introduction to and exploration of two machine learning approaches to address the problem of marker localization in confocal microscopy. First, a comparative study is conducted on the classification accuracy of common conventional machine learning algorithms, as a means to highlight challenges with these methods. Second, a 3D (volumetric) deep learning approach is developed and presented, including consideration of appropriate loss functions and training data. A qualitative and quantitative analysis of all the results produced is performed. Evaluation of all approaches is performed on an unseen time-series sequence comprising several individual 3D volumes, capturing plant growth. The comparative analysis shows that the deep learning approach produces more accurate and robust results than traditional machine learning. To accompany the paper, we are releasing the 4D point annotation tool used to generate the annotations, in the form of a plugin for the popular ImageJ (FIJI) software. Network models and example datasets will also be available online.

Three Dimensional Root CT Segmentation using Multi-Resolution Encoder-Decoder Networks.

We address the complex problem of reliably segmenting root structure from soil in X-ray Computed Tomography (CT) images. We utilise a deep learning approach, and propose a state-of-the-art multi-resolution architecture based on encoderdecoders. While previous work in encoder-decoders implies the use of multiple resolutions simply by downsampling and upsampling images, we make this process explicit, with branches of the network tasked separately with obtaining local high-resolution segmentation, and wider low-resolution contextual information. The complete network is a memory efficient implementation that is still able to resolve small root detail in large volumetric images. We compare against a number of different encoder-decoder based architectures from the literature, as well as a popular existing image analysis tool designed for root CT segmentation. We show qualitatively and quantitatively that a multi-resolution approach offers substantial accuracy improvements over a both a small receptive field size in a deep network, or a larger receptive field in a shallower network. We then further improve performance using an incremental learning approach, in which failures in the original network are used to generate harder negative training examples. Our proposed method requires no user interaction, is fully automatic, and identifies large and fine root material throughout the whole volume.

Deep convolutional neural networks for image-based Convolvulus sepium detection in sugar beet fields.

BACKGROUND: Convolvulus sepium (hedge bindweed) detection in sugar beet fields remains a challenging problem due to variation in appearance of plants, illumination changes, foliage occlusions, and different growth stages under field conditions. Current approaches for weed and crop recognition, segmentation and detection rely predominantly on conventional machine-learning techniques that require a large set of hand-crafted features for modelling. These might fail to generalize over different fields and environments. RESULTS: Here, we present an approach that develops a deep convolutional neural network (CNN) based on the tiny YOLOv3 architecture for C. sepium and sugar beet detection. We generated 2271 synthetic images, before combining these images with 452 field images to train the developed model. YOLO anchor box sizes were calculated from the training dataset using a k-means clustering approach. The resulting model was tested on 100 field images, showing that the combination of synthetic and original field images to train the developed model could improve the mean average precision (mAP) metric from 0.751 to 0.829 compared to using collected field images alone. We also compared the performance of the developed model with the YOLOv3 and Tiny YOLO models. The developed model achieved a better trade-off between accuracy and speed. Specifically, the average precisions (APs@IoU0.5) of C. sepium and sugar beet were 0.761 and 0.897 respectively with 6.48 ms inference time per image (800 × 1200) on a NVIDIA Titan X GPU environment. CONCLUSION: The developed model has the potential to be deployed on an embedded mobile platform like the Jetson TX for online weed detection and management due to its high-speed inference. It is recommendable to use synthetic images and empirical field images together in training stage to improve the performance of models.

Active Vision and Surface Reconstruction for 3D Plant Shoot Modelling.

Plant phenotyping is the quantitative description of a plant's physiological, biochemical, and anatomical status which can be used in trait selection and helps to provide mechanisms to link underlying genetics with yield. Here, an active vision- based pipeline is presented which aims to contribute to reducing the bottleneck associated with phenotyping of architectural traits. The pipeline provides a fully automated response to photometric data acquisition and the recovery of three-dimensional (3D) models of plants without the dependency of botanical expertise, whilst ensuring a non-intrusive and non-destructive approach. Access to complete and accurate 3D models of plants supports computation of a wide variety of structural measurements. An Active Vision Cell (AVC) consisting of a camera-mounted robot arm plus combined software interface and a novel surface reconstruction algorithm is proposed. This pipeline provides a robust, flexible, and accurate method for automating the 3D reconstruction of plants. The reconstruction algorithm can reduce noise and provides a promising and extendable framework for high throughput phenotyping, improving current state-of-the-art methods. Furthermore, the pipeline can be applied to any plant species or form due to the application of an active vision framework combined with the automatic selection of key parameters for surface reconstruction.

RootNav 2.0: Deep learning for automatic navigation of complex plant root architectures.

BACKGROUND: In recent years quantitative analysis of root growth has become increasingly important as a way to explore the influence of abiotic stress such as high temperature and drought on a plant's ability to take up water and nutrients. Segmentation and feature extraction of plant roots from images presents a significant computer vision challenge. Root images contain complicated structures, variations in size, background, occlusion, clutter and variation in lighting conditions. We present a new image analysis approach that provides fully automatic extraction of complex root system architectures from a range of plant species in varied imaging set-ups. Driven by modern deep-learning approaches, RootNav 2.0 replaces previously manual and semi-automatic feature extraction with an extremely deep multi-task convolutional neural network architecture. The network also locates seeds, first order and second order root tips to drive a search algorithm seeking optimal paths throughout the image, extracting accurate architectures without user interaction. RESULTS: We develop and train a novel deep network architecture to explicitly combine local pixel information with global scene information in order to accurately segment small root features across high-resolution images. The proposed method was evaluated on images of wheat (Triticum aestivum L.) from a seedling assay. Compared with semi-automatic analysis via the original RootNav tool, the proposed method demonstrated comparable accuracy, with a 10-fold increase in speed. The network was able to adapt to different plant species via transfer learning, offering similar accuracy when transferred to an Arabidopsis thaliana plate assay. A final instance of transfer learning, to images of Brassica napus from a hydroponic assay, still demonstrated good accuracy despite many fewer training images. CONCLUSIONS: We present RootNav 2.0, a new approach to root image analysis driven by a deep neural network. The tool can be adapted to new image domains with a reduced number of images, and offers substantial speed improvements over semi-automatic and manual approaches. The tool outputs root architectures in the widely accepted RSML standard, for which numerous analysis packages exist (http://rootsystemml.github.io/), as well as segmentation masks compatible with other automated measurement tools. The tool will provide researchers with the ability to analyse root systems at larget scales than ever before, at a time when large scale genomic studies have made this more important than ever.

Recovering Wind-Induced Plant Motion in Dense Field Environments via Deep Learning and Multiple Object Tracking.

Understanding the relationships between local environmental conditions and plant structure and function is critical for both fundamental science and for improving the performance of crops in field settings. Wind-induced plant motion is important in most agricultural systems, yet the complexity of the field environment means that it remained understudied. Despite the ready availability of image sequences showing plant motion, the cultivation of crop plants in dense field stands makes it difficult to detect features and characterize their general movement traits. Here, we present a robust method for characterizing motion in field-grown wheat plants (Triticum aestivum) from time-ordered sequences of red, green, and blue images. A series of crops and augmentations was applied to a dataset of 290 collected and annotated images of ear tips to increase variation and resolution when training a convolutional neural network. This approach enables wheat ears to be detected in the field without the need for camera calibration or a fixed imaging position. Videos of wheat plants moving in the wind were also collected and split into their component frames. Ear tips were detected using the trained network, then tracked between frames using a probabilistic tracking algorithm to approximate movement. These data can be used to characterize key movement traits, such as periodicity, and obtain more detailed static plant properties to assess plant structure and function in the field. Automated data extraction may be possible for informing lodging models, breeding programs, and linking movement properties to canopy light distributions and dynamic light fluctuation.

Uncovering the hidden half of plants using new advances in root phenotyping.

Major increases in crop yield are required to keep pace with population growth and climate change. Improvements to the architecture of crop roots promise to deliver increases in water and nutrient use efficiency but profiling the root phenome (i.e. its structure and function) represents a major bottleneck. We describe how advances in imaging and sensor technologies are making root phenomic studies possible. However, methodological advances in acquisition, handling and processing of the resulting 'big-data' is becoming increasingly important. Advances in automated image analysis approaches such as Deep Learning promise to transform the root phenotyping landscape. Collectively, these innovations are helping drive the selection of the next-generation of crops to deliver real world impact for ongoing global food security efforts.

Deep machine learning provides state-of-the-art performance in image-based plant phenotyping.

In plant phenotyping, it has become important to be able to measure many features on large image sets in order to aid genetic discovery. The size of the datasets, now often captured robotically, often precludes manual inspection, hence the motivation for finding a fully automated approach. Deep learning is an emerging field that promises unparalleled results on many data analysis problems. Building on artificial neural networks, deep approaches have many more hidden layers in the network, and hence have greater discriminative and predictive power. We demonstrate the use of such approaches as part of a plant phenotyping pipeline. We show the success offered by such techniques when applied to the challenging problem of image-based plant phenotyping and demonstrate state-of-the-art results (>97% accuracy) for root and shoot feature identification and localization. We use fully automated trait identification using deep learning to identify quantitative trait loci in root architecture datasets. The majority (12 out of 14) of manually identified quantitative trait loci were also discovered using our automated approach based on deep learning detection to locate plant features. We have shown deep learning-based phenotyping to have very good detection and localization accuracy in validation and testing image sets. We have shown that such features can be used to derive meaningful biological traits, which in turn can be used in quantitative trait loci discovery pipelines. This process can be completely automated. We predict a paradigm shift in image-based phenotyping bought about by such deep learning approaches, given sufficient training sets.

The Microphenotron: a robotic miniaturized plant phenotyping platform with diverse applications in chemical biology.

BACKGROUND: Chemical genetics provides a powerful alternative to conventional genetics for understanding gene function. However, its application to plants has been limited by the lack of a technology that allows detailed phenotyping of whole-seedling development in the context of a high-throughput chemical screen. We have therefore sought to develop an automated micro-phenotyping platform that would allow both root and shoot development to be monitored under conditions where the phenotypic effects of large numbers of small molecules can be assessed. RESULTS: The 'Microphenotron' platform uses 96-well microtitre plates to deliver chemical treatments to seedlings of Arabidopsis thaliana L. and is based around four components: (a) the 'Phytostrip', a novel seedling growth device that enables chemical treatments to be combined with the automated capture of images of developing roots and shoots; (b) an illuminated robotic platform that uses a commercially available robotic manipulator to capture images of developing shoots and roots; (c) software to control the sequence of robotic movements and integrate these with the image capture process; (d) purpose-made image analysis software for automated extraction of quantitative phenotypic data. Imaging of each plate (representing 80 separate assays) takes 4 min and can easily be performed daily for time-course studies. As currently configured, the Microphenotron has a capacity of 54 microtitre plates in a growth room footprint of 2.1 m2, giving a potential throughput of up to 4320 chemical treatments in a typical 10 days experiment. The Microphenotron has been validated by using it to screen a collection of 800 natural compounds for qualitative effects on root development and to perform a quantitative analysis of the effects of a range of concentrations of nitrate and ammonium on seedling development. CONCLUSIONS: The Microphenotron is an automated screening platform that for the first time is able to combine large numbers of individual chemical treatments with a detailed analysis of whole-seedling development, and particularly root system development. The Microphenotron should provide a powerful new tool for chemical genetics and for wider chemical biology applications, including the development of natural and synthetic chemical products for improved agricultural sustainability.

AutoRoot: open-source software employing a novel image analysis approach to support fully-automated plant phenotyping.

BACKGROUND: Computer-based phenotyping of plants has risen in importance in recent years. Whilst much software has been written to aid phenotyping using image analysis, to date the vast majority has been only semi-automatic. However, such interaction is not desirable in high throughput approaches. Here, we present a system designed to analyse plant images in a completely automated manner, allowing genuine high throughput measurement of root traits. To do this we introduce a new set of proxy traits. RESULTS: We test the system on a new, automated image capture system, the Microphenotron, which is able to image many 1000s of roots/h. A simple experiment is presented, treating the plants with differing chemical conditions to produce different phenotypes. The automated imaging setup and the new software tool was used to measure proxy traits in each well. A correlation matrix was calculated across automated and manual measures, as a validation. Some particular proxy measures are very highly correlated with the manual measures (e.g. proxy length to manual length, r2 > 0.9). This suggests that while the automated measures are not directly equivalent to classic manual measures, they can be used to indicate phenotypic differences (hence the term, proxy). In addition, the raw discriminative power of the new proxy traits was examined. Principal component analysis was calculated across all proxy measures over two phenotypically-different groups of plants. Many of the proxy traits can be used to separate the data in the two conditions. CONCLUSION: The new proxy traits proposed tend to correlate well with equivalent manual measures, where these exist. Additionally, the new measures display strong discriminative power. It is suggested that for particular phenotypic differences, different traits will be relevant, and not all will have meaningful manual equivalent measures. However, approaches such as PCA can be used to interrogate the resulting data to identify differences between datasets. Select images can then be carefully manually inspected if the nature of the precise differences is required. We suggest such flexible measurement approaches are necessary for fully automated, high throughput systems such as the Microphenotron.

Image-based 3D canopy reconstruction to determine potential productivity in complex multi-species crop systems.

Background and Aims: Intercropping systems contain two or more species simultaneously in close proximity. Due to contrasting features of the component crops, quantification of the light environment and photosynthetic productivity is extremely difficult. However it is an essential component of productivity. Here, a low-tech but high-resolution method is presented that can be applied to single- and multi-species cropping systems to facilitate characterization of the light environment. Different row layouts of an intercrop consisting of Bambara groundnut ( Vigna subterranea ) and proso millet ( Panicum miliaceum ) have been used as an example and the new opportunities presented by this approach have been analysed. Methods: Three-dimensional plant reconstruction, based on stereo cameras, combined with ray tracing was implemented to explore the light environment within the Bambara groundnut-proso millet intercropping system and associated monocrops. Gas exchange data were used to predict the total carbon gain of each component crop. Key Results: The shading influence of the tall proso millet on the shorter Bambara groundnut results in a reduction in total canopy light interception and carbon gain. However, the increased leaf area index (LAI) of proso millet, higher photosynthetic potential due to the C4 pathway and sub-optimal photosynthetic acclimation of Bambara groundnut to shade means that increasing the number of rows of millet will lead to greater light interception and carbon gain per unit ground area, despite Bambara groundnut intercepting more light per unit leaf area. Conclusions: Three-dimensional reconstruction combined with ray tracing provides a novel, accurate method of exploring the light environment within an intercrop that does not require difficult measurements of light interception and data-intensive manual reconstruction, especially for such systems with inherently high spatial possibilities. It provides new opportunities for calculating potential productivity within multi-species cropping systems, enables the quantification of dynamic physiological differences between crops grown as monoculture and those within intercrops, and enables the prediction of new productive combinations of previously untested crops.

The 4-Dimensional Plant: Effects of Wind-Induced Canopy Movement on Light Fluctuations and Photosynthesis.

Physical perturbation of a plant canopy brought about by wind is a ubiquitous phenomenon and yet its biological importance has often been overlooked. This is partly due to the complexity of the issue at hand: wind-induced movement (or mechanical excitation) is a stochastic process which is difficult to measure and quantify; plant motion is dependent upon canopy architectural features which, until recently, were difficult to accurately represent and model in 3-dimensions; light patterning throughout a canopy is difficult to compute at high-resolutions, especially when confounded by other environmental variables. Recent studies have reinforced the expectation that canopy architecture is a strong determinant of productivity and yield; however, links between the architectural properties of the plant and its mechanical properties, particularly its response to wind, are relatively unknown. As a result, biologically relevant data relating canopy architecture, light- dynamics, and short-scale photosynthetic responses in the canopy setting are scarce. Here, we hypothesize that wind-induced movement will have large consequences for the photosynthetic productivity of our crops due to its influence on light patterning. To address this issue, in this study we combined high resolution 3D reconstructions of a plant canopy with a simple representation of canopy perturbation as a result of wind using solid body rotation in order to explore the potential effects on light patterning, interception, and photosynthetic productivity. We looked at two different scenarios: firstly a constant distortion where a rice canopy was subject to a permanent distortion throughout the whole day; and secondly, a dynamic distortion, where the canopy was distorted in incremental steps between two extremes at set time points in the day. We find that mechanical canopy excitation substantially alters light dynamics; light distribution and modeled canopy carbon gain. We then discuss methods required for accurate modeling of mechanical canopy excitation (here coined the 4-dimensional plant) and some associated biological and applied implications of such techniques. We hypothesize that biomechanical plant properties are a specific adaptation to achieve wind-induced photosynthetic enhancement and we outline how traits facilitating canopy excitation could be used as a route for improving crop yield.

High-Resolution Three-Dimensional Structural Data Quantify the Impact of Photoinhibition on Long-Term Carbon Gain in Wheat Canopies in the Field.

Photoinhibition reduces photosynthetic productivity; however, it is difficult to quantify accurately in complex canopies partly because of a lack of high-resolution structural data on plant canopy architecture, which determines complex fluctuations of light in space and time. Here, we evaluate the effects of photoinhibition on long-term carbon gain (over 1 d) in three different wheat (Triticum aestivum) lines, which are architecturally diverse. We use a unique method for accurate digital three-dimensional reconstruction of canopies growing in the field. The reconstruction method captures unique architectural differences between lines, such as leaf angle, curvature, and leaf density, thus providing a sensitive method of evaluating the productivity of actual canopy structures that previously were difficult or impossible to obtain. We show that complex data on light distribution can be automatically obtained without conventional manual measurements. We use a mathematical model of photosynthesis parameterized by field data consisting of chlorophyll fluorescence, light response curves of carbon dioxide assimilation, and manual confirmation of canopy architecture and light attenuation. Model simulations show that photoinhibition alone can result in substantial reduction in carbon gain, but this is highly dependent on exact canopy architecture and the diurnal dynamics of photoinhibition. The use of such highly realistic canopy reconstructions also allows us to conclude that even a moderate change in leaf angle in upper layers of the wheat canopy led to a large increase in the number of leaves in a severely light-limited state.

Phenotyping pipeline reveals major seedling root growth QTL in hexaploid wheat.

Seedling root traits of wheat (Triticum aestivum L.) have been shown to be important for efficient establishment and linked to mature plant traits such as height and yield. A root phenotyping pipeline, consisting of a germination paper-based screen combined with image segmentation and analysis software, was developed and used to characterize seedling traits in 94 doubled haploid progeny derived from a cross between the winter wheat cultivars Rialto and Savannah. Field experiments were conducted to measure mature plant height, grain yield, and nitrogen (N) uptake in three sites over 2 years. In total, 29 quantitative trait loci (QTLs) for seedling root traits were identified. Two QTLs for grain yield and N uptake co-localize with root QTLs on chromosomes 2B and 7D, respectively. Of the 29 root QTLs identified, 11 were found to co-localize on 6D, with four of these achieving highly significant logarithm of odds scores (>20). These results suggest the presence of a major-effect gene regulating seedling root vigour/growth on chromosome 6D.

Root system markup language: toward a unified root architecture description language.

The number of image analysis tools supporting the extraction of architectural features of root systems has increased in recent years. These tools offer a handy set of complementary facilities, yet it is widely accepted that none of these software tools is able to extract in an efficient way the growing array of static and dynamic features for different types of images and species. We describe the Root System Markup Language (RSML), which has been designed to overcome two major challenges: (1) to enable portability of root architecture data between different software tools in an easy and interoperable manner, allowing seamless collaborative work; and (2) to provide a standard format upon which to base central repositories that will soon arise following the expanding worldwide root phenotyping effort. RSML follows the XML standard to store two- or three-dimensional image metadata, plant and root properties and geometries, continuous functions along individual root paths, and a suite of annotations at the image, plant, or root scale at one or several time points. Plant ontologies are used to describe botanical entities that are relevant at the scale of root system architecture. An XML schema describes the features and constraints of RSML, and open-source packages have been developed in several languages (R, Excel, Java, Python, and C#) to enable researchers to integrate RSML files into popular research workflow.

Automated recovery of three-dimensional models of plant shoots from multiple color images.

Increased adoption of the systems approach to biological research has focused attention on the use of quantitative models of biological objects. This includes a need for realistic three-dimensional (3D) representations of plant shoots for quantification and modeling. Previous limitations in single-view or multiple-view stereo algorithms have led to a reliance on volumetric methods or expensive hardware to record plant structure. We present a fully automatic approach to image-based 3D plant reconstruction that can be achieved using a single low-cost camera. The reconstructed plants are represented as a series of small planar sections that together model the more complex architecture of the leaf surfaces. The boundary of each leaf patch is refined using the level-set method, optimizing the model based on image information, curvature constraints, and the position of neighboring surfaces. The reconstruction process makes few assumptions about the nature of the plant material being reconstructed and, as such, is applicable to a wide variety of plant species and topologies and can be extended to canopy-scale imaging. We demonstrate the effectiveness of our approach on data sets of wheat (Triticum aestivum) and rice (Oryza sativa) plants as well as a unique virtual data set that allows us to compute quantitative measures of reconstruction accuracy. The output is a 3D mesh structure that is suitable for modeling applications in a format that can be imported in the majority of 3D graphics and software packages.

Systems analysis of auxin transport in the Arabidopsis root apex.

Auxin is a key regulator of plant growth and development. Within the root tip, auxin distribution plays a crucial role specifying developmental zones and coordinating tropic responses. Determining how the organ-scale auxin pattern is regulated at the cellular scale is essential to understanding how these processes are controlled. In this study, we developed an auxin transport model based on actual root cell geometries and carrier subcellular localizations. We tested model predictions using the DII-VENUS auxin sensor in conjunction with state-of-the-art segmentation tools. Our study revealed that auxin efflux carriers alone cannot create the pattern of auxin distribution at the root tip and that AUX1/LAX influx carriers are also required. We observed that AUX1 in lateral root cap (LRC) and elongating epidermal cells greatly enhance auxin's shootward flux, with this flux being predominantly through the LRC, entering the epidermal cells only as they enter the elongation zone. We conclude that the nonpolar AUX1/LAX influx carriers control which tissues have high auxin levels, whereas the polar PIN carriers control the direction of auxin transport within these tissues.

Integration of hormonal signaling networks and mobile microRNAs is required for vascular patterning in Arabidopsis roots.

As multicellular organisms grow, positional information is continually needed to regulate the pattern in which cells are arranged. In the Arabidopsis root, most cell types are organized in a radially symmetric pattern; however, a symmetry-breaking event generates bisymmetric auxin and cytokinin signaling domains in the stele. Bidirectional cross-talk between the stele and the surrounding tissues involving a mobile transcription factor, SHORT ROOT (SHR), and mobile microRNA species also determines vascular pattern, but it is currently unclear how these signals integrate. We use a multicellular model to determine a minimal set of components necessary for maintaining a stable vascular pattern. Simulations perturbing the signaling network show that, in addition to the mutually inhibitory interaction between auxin and cytokinin, signaling through SHR, microRNA165/6, and PHABULOSA is required to maintain a stable bisymmetric pattern. We have verified this prediction by observing loss of bisymmetry in shr mutants. The model reveals the importance of several features of the network, namely the mutual degradation of microRNA165/6 and PHABULOSA and the existence of an additional negative regulator of cytokinin signaling. These components form a plausible mechanism capable of patterning vascular tissues in the absence of positional inputs provided by the transport of hormones from the shoot.