Reconstruction method and optimum range of camera-shooting angle for 3D plant modeling using a multi-camera photography system.

BACKGROUND: Measurement of plant structure is useful in monitoring plant conditions and understanding the responses of plants to environmental changes. 3D imaging technologies, especially the passive-SfM (Structure from Motion) algorithm combined with a multi-camera photography (MCP) system has been studied to measure plant structure due to its low-cost, close-range, and rapid image capturing ability. However, reconstruction of 3D plant models with complex structure is a time-consuming process and some systems have failed to reconstruct 3D models properly. Therefore, an MCP based SfM system was developed and an appropriate reconstruction method and optimal range of camera-shooting angles were investigated. RESULTS: An MCP system which utilized 10 cameras and a rotary table for plant was developed. The 3D mesh model of a single leaf reconstruction using a set of images taken at each viewing zenith angle (VZA) from 12° (C2 camera) to 60° (C6 camera) by the MCP based SfM system had less undetected or unstable regions in comparison with other VZAs. The 3D mesh model of a whole plant, which merged 3D dense point cloud models built from a set of images taken at each appropriate VZA (Method 1), had high accuracy. The Method 1 error percentages for leaf area, leaf length, leaf width, stem height, and stem width are in the range of 2.6-4.4%, 0.2-2.2%, 1.0-4.9%, 1.9-2.8%, and 2.6-5.7% respectively. Also, the error of the leaf inclination angle was less than 5°. Conversely, the 3D mesh model of a whole plant built directly from a set of images taken at all appropriate VZAs (Method 2) had lower accuracy than that of Method 1. For Method 2, the error percentages of leaf area, leaf length, and leaf width are in the range of 3.1-13.3%, 0.4-3.3%, and 1.6-8.6%, respectively. It was difficult to obtain the error percentages of stem height and stem width because some information was missing in this model. In addition, the calculation time for Method 2 was 1.97 times longer computational time in comparison to Method 1. CONCLUSIONS: In this study, we determined the optimal shooting angles on the MCP based SfM system developed. We found that it is better in terms of computational time and accuracy to merge partial 3D models from images taken at each appropriate VZA, then construct complete 3D model (Method 1), rather than to construct 3D model by using images taken at all appropriate VZAs (Method 2). This is because utilization of incorporation of incomplete images to match feature points could result in reduced accuracy in 3D models and the increase in computational time for 3D model reconstruction.

Automatic method for segmenting leaves by combining 2D and 3D image-processing techniques.

In this study, a method to automatically segment plant leaves from three-dimensional (3D) images using structure from motion is proposed. First, leaves in the 3D images are roughly segmented using a region-growing method in which near points with distances less than 0.2 cm are assigned to the same group. By repeating this process, the leaves not touching each other can be segmented. Then, each segmented leaf is projected onto two-dimensional (2D) images, and the watershed algorithm is executed. This process successfully segments overlapping leaves.

Simple and effective tool for estimating tree trunk diameters and tree species classification.

Tree trunk diameter and tree species are two of the most important parameters in analyzing trees in urban areas and forests. Conventionally, diameters have been measured manually, and the species were determined by sight. An automated tool for these assessments was developed. Tree trunks are automatically detected from captured stereo images. Then, tree trunk diameters are estimated, and the species are determined. The developed graphical user interface tool enables fast and accurate estimation even while one is walking, which reduces the time spent in measuring trees.

Automated tree detection from 3D lidar images using image processing and machine learning.

Trees in 3D images obtained from lidar were automatically extracted in the presence of other objects that were not trees. We proposed a method combining 3D image processing and machine learning techniques for this automatic detection. Consequently, tree detection could be done with 95% accuracy. First, the objects in the 3D images were segmented one by one; then, each of the segmented objects was projected onto 2D images. Finally, the 2D image was classified into "tree" and "not tree" using a one-class support vector machine, and trees in the 3D image were successfully extracted.

Three-Dimensional Monitoring of Plant Structural Parameters and Chlorophyll Distribution.

Image analysis is widely used for accurate and efficient plant monitoring. Plants have complex three-dimensional (3D) structures; hence, 3D image acquisition and analysis is useful for determining the status of plants. Here, 3D images of plants were reconstructed using a photogrammetric approach, called "structure from motion". Chlorophyll content is an important parameter that determines the status of plants. Chlorophyll content was estimated from 3D images of plants with color information. To observe changes in the chlorophyll content and plant structure, a potted plant was kept for five days under a water stress condition and its 3D images were taken once a day. As a result, the normalized Red value and the chlorophyll content were correlated; a high R² value (0.81) was obtained. The absolute error of the chlorophyll content estimation in cross-validation studies was 4.0 × 10-2 µg/mm². At the same time, the structural parameters (i.e., the leaf inclination angle and the azimuthal angle) were calculated by simultaneously monitoring the changes in the plant's status in terms of its chlorophyll content and structural parameters. By combining these parameters related to plant information in plant image analysis, early detection of plant stressors, such as water stress, becomes possible.

Automatic Leaf Segmentation for Estimating Leaf Area and Leaf Inclination Angle in 3D Plant Images.

Automatic and efficient plant monitoring offers accurate plant management. Construction of three-dimensional (3D) models of plants and acquisition of their spatial information is an effective method for obtaining plant structural parameters. Here, 3D images of leaves constructed with multiple scenes taken from different positions were segmented automatically for the automatic retrieval of leaf areas and inclination angles. First, for the initial segmentation, leave images were viewed from the top, then leaves in the top-view images were segmented using distance transform and the watershed algorithm. Next, the images of leaves after the initial segmentation were reduced by 90%, and the seed regions for each leaf were produced. The seed region was re-projected onto the 3D images, and each leaf was segmented by expanding the seed region with the 3D information. After leaf segmentation, the leaf area of each leaf and its inclination angle were estimated accurately via a voxel-based calculation. As a result, leaf area and leaf inclination angle were estimated accurately after automatic leaf segmentation. This method for automatic plant structure analysis allows accurate and efficient plant breeding and growth management.

Estimating 3D Leaf and Stem Shape of Nursery Paprika Plants by a Novel Multi-Camera Photography System.

For plant breeding and growth monitoring, accurate measurements of plant structure parameters are very crucial. We have, therefore, developed a high efficiency Multi-Camera Photography (MCP) system combining Multi-View Stereovision (MVS) with the Structure from Motion (SfM) algorithm. In this paper, we measured six variables of nursery paprika plants and investigated the accuracy of 3D models reconstructed from photos taken by four lens types at four different positions. The results demonstrated that error between the estimated and measured values was small, and the root-mean-square errors (RMSE) for leaf width/length and stem height/diameter were 1.65 mm (R² = 0.98) and 0.57 mm (R² = 0.99), respectively. The accuracies of the 3D model reconstruction of leaf and stem by a 28-mm lens at the first and third camera positions were the highest, and the number of reconstructed fine-scale 3D model shape surfaces of leaf and stem is the most. The results confirmed the practicability of our new method for the reconstruction of fine-scale plant model and accurate estimation of the plant parameters. They also displayed that our system is a good system for capturing high-resolution 3D images of nursery plants with high efficiency.

Estimating chlorophyll content and photochemical yield of photosystem II (ΦPSII) using solar-induced chlorophyll fluorescence measurements at different growing stages of attached leaves.

This paper illustrates the possibility of measuring chlorophyll (Chl) content and Chl fluorescence parameters by the solar-induced Chl fluorescence (SIF) method using the Fraunhofer line depth (FLD) principle, and compares the results with the standard measurement methods. A high-spectral resolution HR2000+ and an ordinary USB4000 spectrometer were used to measure leaf reflectance under solar and artificial light, respectively, to estimate Chl fluorescence. Using leaves of Capsicum annuum cv. 'Sven' (paprika), the relationships between the Chl content and the steady-state Chl fluorescence near oxygen absorption bands of O2B (686nm) and O2A (760nm), measured under artificial and solar light at different growing stages of leaves, were evaluated. The Chl fluorescence yields of ΦF 686nm/ΦF 760nm ratios obtained from both methods correlated well with the Chl content (steady-state solar light: R(2) = 0.73; artificial light: R(2) = 0.94). The SIF method was less accurate for Chl content estimation when Chl content was high. The steady-state solar-induced Chl fluorescence yield ratio correlated very well with the artificial-light-induced one (R(2) = 0.84). A new methodology is then presented to estimate photochemical yield of photosystem II (ΦPSII) from the SIF measurements, which was verified against the standard Chl fluorescence measurement method (pulse-amplitude modulated method). The high coefficient of determination (R(2) = 0.74) between the ΦPSII of the two methods shows that photosynthesis process parameters can be successfully estimated using the presented methodology.

3-D modeling of tomato canopies using a high-resolution portable scanning lidar for extracting structural information.

In the present study, an attempt was made to produce a precise 3D image of a tomato canopy using a portable high-resolution scanning lidar. The tomato canopy was scanned by the lidar from three positions surrounding it. Through the scanning, the point cloud data of the canopy were obtained and they were co-registered. Then, points corresponding to leaves were extracted and converted into polygon images. From the polygon images, leaf areas were accurately estimated with a mean absolute percent error of 4.6%. Vertical profile of leaf area density (LAD) and leaf area index (LAI) could be also estimated by summing up each leaf area derived from the polygon images. Leaf inclination angle could be also estimated from the 3-D polygon image. It was shown that leaf inclination angles had different values at each part of a leaf.

3D Confocal laser scanning microscopy for the analysis of chlorophyll fluorescence parameters of chloroplasts in intact leaf tissues.

We analyzed the chlorophyll fluorescence parameters in a 3D cellular arrangement in vivo by using a modified Nipkow disk-type confocal laser scanning microscope (CLSM). We first defined the 3D values of Phi(PSII) (photochemical yield of PSII) and NPQ (non-photochemical quenching) in mesophyll, epidermal and guard cell chloroplasts from the leaf surface to several tens of microns in depth. We also used this CLSM method to analyze the relationships between actinic light intensity and the chlorophyll fluorescence parameters for Boston fern and broad bean leaf specimens. As the actinic light intensity increased, the mean Phi(PSII) values decreased and the NPQ values increased in all chloroplasts of Boston fern and broad bean leaf. These values differed with cell type and species. The Boston fern chloroplasts had lower Phi(PSII) values than the broad bean chloroplasts, and vice versa for the NPQ values. The Phi(PSII) values of Boston fern chloroplasts decreased in the order mesophyll, epidermal and guard cell chloroplasts. The NPQ values decreased in the order guard cell, mesophyll and epidermal chloroplasts, except at 12 micromol m(-2) s(-1) actinic light, when the mesophyll value was slightly lower than that of the epidermis. The trend in the Phi(PSII) and NPQ values of broad bean mesophyll and guard cell chloroplasts was opposite to that of Boston fern chloroplasts. As 3D CLSM can provide the Phi(PSII) and NPQ values of each chloroplast in a 3D cellular arrangement, this method has potential for investigating differences in the functions of chloroplasts in vivo.

3D monitoring spatio-temporal effects of herbicide on a whole plant using combined range and chlorophyll a fluorescence imaging.

Spatio-temporal effects of herbicide including 3-(3,4 dichlorophenyl)-1,1-dimethylurea (DCMU) on a whole melon (Cucumis melo L.) plant were three-dimensionally monitored using combined range and chlorophyll a fluorescence imaging. The herbicide was treated to soil in a pot and the changes in chlorophyll a fluorescence images of the plant were captured over time. The time series of chlorophyll fluorescence images were combined with 3D polygon model of the whole plant taken by a high-resolution portable scanning lidar. From the produced 3D chlorophyll fluorescence model, it was observed that the increase of chlorophyll fluorescence appeared along veins of leaves and gradually expanded to mesophylls. In addition, it was found by detailed analysis of the images that the invisible herbicide injury on the mature leaves occurred earlier and more severely than on the young and old leaves. The distance from veins, whole leaf area and leaf inclination influenced the extent of the injury within the leaves. These results indicated difference in uptake of herbicide in the plant from soil depends on structural parameters of leaves and the microenvironments as well as leaf age. The findings showed that 3D monitoring using combined range and chlorophyll a fluorescence imaging can be utilised for understanding spatio-temporal changes of herbicide effects on a whole plant.

Detecting seasonal change of broad-leaved woody canopy leaf area density profile using 3D portable LIDAR imaging.

Seasonal change of vertical leaf area density (LAD) profiles of woody canopy broad-leaved trees (Zelkova serrata [Thunberg] Makino) was estimated using 3D portable scanning light detection and ranging (LIDAR) imaging. First, 3D point cloud data for the canopy were collected using a portable LIDAR in spring, summer, autumn and winter. For data collection, the canopy was evenly scanned by the LIDAR from three positions 10 m above the ground. Next, the vertical LAD profile in each season was computed from the LIDAR data using the voxel-based canopy profiling (VCP) method. For the computation, non-photosynthetic tissues were eliminated using the LIDAR data obtained during winter. Influence of leaf inclination angle (LIA) on LAD estimation was corrected by LIA data measured by a high-resolution portable scanning LIDAR. The resultant profiles showed that LAD values tended to increase at the upper canopy from spring to summer and decrease at the middle and lower canopy from summer to autumn. Moreover, LIDAR-derived LIA distributions were compared among different seasons. LIA showed an even distribution in spring but changed to a planophile distribution in summer. In autumn, the angles in the <30° class decreased and those between the 30 and 40°classes increased.

Factors contributing to accuracy in the estimation of the woody canopy leaf area density profile using 3D portable lidar imaging.

Factors that contribute to the accuracy of estimating woody canopy's leaf area density (LAD) using 3D portable lidar imaging were investigated. The 3D point cloud data for a Japanese zelkova canopy [Zelkova serrata (Thunberg) Makino] were collected using a portable scanning lidar from several points established on the ground and at 10 m above the ground. The LAD profiles were computed using voxel-based canopy profiling (VCP). The best LAD results [a root-mean-square error (RMSE) of 0.21 m(2) m(-3)] for the measurement plot (corresponding to an absolute LAI error of 9.5%) were obtained by compositing the ground-level and 10 m measurements. The factors that most strongly affected estimation accuracy included the presence of non-photosynthetic tissues, distribution of leaf inclination angles, number (N) of incident laser beams in each region within the canopy, and G(theta(m)) (the mean projection of a unit leaf area on a plane perpendicular to the direction of the laser beam at the measurement zenith angle of theta(m)). The influences of non-photosynthetic tissues and leaf inclination angle on the estimates amounted to 4.2-32.7% and 7.2-94.2%, respectively. The RMSE of the LAD estimations was expressed using a function of N and G(theta(m)).

3D lidar imaging for detecting and understanding plant responses and canopy structure.

Understanding and diagnosing plant responses to stress will benefit greatly from three-dimensional (3D) measurement and analysis of plant properties because plant responses are strongly related to their 3D structures. Light detection and ranging (lidar) has recently emerged as a powerful tool for direct 3D measurement of plant structure. Here the use of 3D lidar imaging to estimate plant properties such as canopy height, canopy structure, carbon stock, and species is demonstrated, and plant growth and shape responses are assessed by reviewing the development of lidar systems and their applications from the leaf level to canopy remote sensing. In addition, the recent creation of accurate 3D lidar images combined with natural colour, chlorophyll fluorescence, photochemical reflectance index, and leaf temperature images is demonstrated, thereby providing information on responses of pigments, photosynthesis, transpiration, stomatal opening, and shape to environmental stresses; these data can be integrated with 3D images of the plants using computer graphics techniques. Future lidar applications that provide more accurate dynamic estimation of various plant properties should improve our understanding of plant responses to stress and of interactions between plants and their environment. Moreover, combining 3D lidar with other passive and active imaging techniques will potentially improve the accuracy of airborne and satellite remote sensing, and make it possible to analyse 3D information on ecophysiological responses and levels of various substances in agricultural and ecological applications and in observations of the global biosphere.