FF3D: A Rapid and Accurate 3D Fruit Detector for Robotic Harvesting.

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments.

Developing a Flying Explorer for Autonomous Digital Modelling in Wild Unknowns.

Digital modelling stands as a pivotal step in the realm of Digital Twinning. The future trend of Digital Twinning involves automated exploration and environmental modelling in complex scenes. In our study, we propose an innovative solution for robot odometry, path planning, and exploration in unknown outdoor environments, with a focus on Digital modelling. The approach uses a minimum cost formulation with pseudo-randomly generated objectives, integrating multi-path planning and evaluation, with emphasis on full coverage of unknown maps based on feasible boundaries of interest. The approach allows for dynamic changes to expected targets and behaviours. The evaluation is conducted on a robotic platform with a lightweight 3D LiDAR sensor model. The robustness of different types of odometry is compared, and the impact of parameters on motion planning is explored. The consistency and efficiency of exploring completely unknown areas are assessed in both indoor and outdoor scenarios. The experiment shows that the method proposed in this article can complete autonomous exploration and environmental modelling tasks in complex indoor and outdoor scenes. Finally, the study concludes by summarizing the reasons for exploration failures and outlining future focuses in this domain.

A Context-Aware Navigation Framework for Ground Robots in Horticultural Environments.

Environmental mapping and robot navigation are the basis for realizing robot automation in modern agricultural production. This study proposes a new autonomous mapping and navigation method for gardening scene robots. First, a new LiDAR slam-based semantic mapping algorithm is proposed to enable the robots to analyze structural information from point cloud images and generate roadmaps from them. Secondly, a general robot navigation framework is proposed to enable the robot to generate the shortest global path according to the road map, and consider the local terrain information to find the optimal local path to achieve safe and efficient trajectory tracking; this method is equipped in apple orchards. The LiDAR was evaluated on a differential drive robotic platform. Experimental results show that this method can effectively process orchard environmental information. Compared with vnf and pointnet++, the semantic information extraction efficiency and time are greatly improved. The map feature extraction time can be reduced to 0.1681 s, and its MIoU is 0.812. The resulting global path planning achieved a 100% success rate, with an average run time of 4ms. At the same time, the local path planning algorithm can effectively generate safe and smooth trajectories to execute the global path, with an average running time of 36 ms.

Learning-Based Slip Detection for Robotic Fruit Grasping and Manipulation under Leaf Interference.

Robotic harvesting research has seen significant achievements in the past decade, with breakthroughs being made in machine vision, robot manipulation, autonomous navigation and mapping. However, the missing capability of obstacle handling during the grasping process has severely reduced harvest success rate and limited the overall performance of robotic harvesting. This work focuses on leaf interference caused slip detection and handling, where solutions to robotic grasping in an unstructured environment are proposed. Through analysis of the motion and force of fruit grasping under leaf interference, the connection between object slip caused by leaf interference and inadequate harvest performance is identified for the first time in the literature. A learning-based perception and manipulation method is proposed to detect slip that causes problematic grasps of objects, allowing the robot to implement timely reaction. Our results indicate that the proposed algorithm detects grasp slip with an accuracy of 94%. The proposed sensing-based manipulation demonstrated great potential in robotic fruit harvesting, and could be extended to other pick-place applications.

Real-Time Fruit Recognition and Grasping Estimation for Robotic Apple Harvesting.

Robotic harvesting shows a promising aspect in future development of agricultural industry. However, there are many challenges which are still presented in the development of a fully functional robotic harvesting system. Vision is one of the most important keys among these challenges. Traditional vision methods always suffer from defects in accuracy, robustness, and efficiency in real implementation environments. In this work, a fully deep learning-based vision method for autonomous apple harvesting is developed and evaluated. The developed method includes a light-weight one-stage detection and segmentation network for fruit recognition and a PointNet to process the point clouds and estimate a proper approach pose for each fruit before grasping. Fruit recognition network takes raw inputs from RGB-D camera and performs fruit detection and instance segmentation on RGB images. The PointNet grasping network combines depth information and results from the fruit recognition as input and outputs the approach pose of each fruit for robotic arm execution. The developed vision method is evaluated on RGB-D image data which are collected from both laboratory and orchard environments. Robotic harvesting experiments in both indoor and outdoor conditions are also included to validate the performance of the developed harvesting system. Experimental results show that the developed vision method can perform highly efficient and accurate to guide robotic harvesting. Overall, the developed robotic harvesting system achieves 0.8 on harvesting success rate and cycle time is 6.5 seconds.

Fruit Detection and Segmentation for AppleHarvesting Using Visual Sensor in Orchards.

Autonomous harvesting shows a promising prospect in the future development of theagriculture industry, while the vision system is one of the most challenging components in theautonomous harvesting technologies. This work proposes a multi-function network to perform thereal-time detection and semantic segmentation of apples and branches in orchard environments byusing the visual sensor. The developed detection and segmentation network utilises the atrous spatialpyramid pooling and the gate feature pyramid network to enhance feature extraction ability of thenetwork. To improve the real-time computation performance of the network model, a lightweightbackbone network based on the residual network architecture is developed. From the experimentalresults, the detection and segmentation network with ResNet-101 backbone outperformed on thedetection and segmentation tasks, achieving an F1 score of 0.832 on the detection of apples and 87.6%and 77.2% on the semantic segmentation of apples and branches, respectively. The network modelwith lightweight backbone showed the best computation efficiency in the results. It achieved an F1score of 0.827 on the detection of apples and 86.5% and 75.7% on the segmentation of apples andbranches, respectively. The weights size and computation time of the network model with lightweightbackbone were 12.8 M and 32 ms, respectively. The experimental results show that the detection andsegmentation network can effectively perform the real-time detection and segmentation of applesand branches in orchards.

Automatic prostate segmentation based on fusion between deep network and variational methods.

BACKGROUND: Segmentation of prostate from magnetic resonance images (MRI) is a critical process for guiding prostate puncture and biopsy. Currently, the best results are obtained by Convolutional Neural Network (CNN). However, challenges still exist when applying CNN to segment prostate, such as data distribution issue caused by insubstantial and inconsistent intensity levels and vague boundaries in MRI. OBJECTIVE: To segment prostate gland from a MRI dataset including different prostate images with limited resolution and quality. METHODS: We propose and apply a global histogram matching approach to make intensity distribution of the MRI dataset closer to uniformity. To capture the real boundaries and improve segmentation accuracy, we employ a module of variational models to help improve performance. RESULTS: Using seven evaluation metrics to quantify improvements of our proposed fusion approach compared with the state of art V-net model resulted in increase in the Dice Coefficient (11.2%), Jaccard Coefficient (13.7%), Volumetric Similarity (12.3%), Adjusted Rand Index (11.1%), Area under ROC Curve (11.6%), and reduction of the Mean Hausdorff Distance (16.1%) and Mahalanobis Distance (2.8%). The 3D reconstruction also validates the advantages of our proposed framework, especially in terms of smoothness, uniformity, and accuracy. In addition, observations from the selected examples of 2D visualization show that our segmentation results are closer to the real boundaries of the prostate, and better represent the prostate shapes. CONCLUSIONS: Our proposed approach achieves significant performance improvements compared with the existing methods based on the original CNN or pure variational models.

Non-destructive Storage Time Prediction of Newhall Navel Oranges Based on the Characteristics of Rind Oil Glands.

How to non-destructively and quickly estimate the storage time of citrus fruit is necessary and urgent for freshness control in the fruit market. As a feasibility study, we present a non-destructive method for storage time prediction of Newhall navel oranges by investigating the characteristics of the rind oil glands in this paper. Through the observation using a digital microscope, the oil glands were divided into three types and the change of their proportions could indicate the rind status as well as the storage time. Images of the rind of the oranges were taken in intervals of 10 days for 40 days, and they were used to train and test the proposed prediction models based on K-Nearest Neighbors (KNN) and deep learning algorithms, respectively. The KNN-based model demonstrated explicit features for storage time prediction based on the gland characteristics and reached a high accuracy of 93.0%, and the deep learning-based model attained an even higher accuracy of 96.0% due to its strong adaptability and robustness. The workflow presented can be readily replicated to develop non-destructive methods to predict the storage time of other types of citrus fruit with similar oil gland characteristics in different storage conditions featuring high efficiency and accuracy.

A Deep Learning-Based Vision System Combining Detection and Tracking for Fast On-Line Citrus Sorting.

Defective citrus fruits are manually sorted at the moment, which is a time-consuming and cost-expensive process with unsatisfactory accuracy. In this paper, we introduce a deep learning-based vision system implemented on a citrus processing line for fast on-line sorting. For the citrus fruits rotating randomly on the conveyor, a convolutional neural network-based detector was developed to detect and temporarily classify the defective ones, and a SORT algorithm-based tracker was adopted to record the classification information along their paths. The true categories of the citrus fruits were identified through the tracked historical information, resulting in high detection precision of 93.6%. Moreover, the linear Kalman filter model was applied to predict the future path of the fruits, which can be used to guide the robot arms to pick out the defective ones. Ultimately, this research presents a practical solution to realize on-line citrus sorting featuring low costs, high efficiency, and accuracy.

Pore-scale multiple-contact miscibility measurements in a microfluidic chip.

Carbon dioxide enhanced oil recovery is an interim solution as the world transitions to a cleaner energy future, extending oil production from existing fields whilst also sequestering carbon dioxide. To make this process efficient, the gas and oil need to develop miscibility over a period of time through the exchange of chemical components between the two phases, termed multiple-contact miscibility. Currently, measurements to infer the development of multiple-contact miscibility are limited to macroscopic visualization. We present a "rock-on-a-chip" measurement system that offers several potential measurements for different wetting conditions to infer the onset of multiple-contact miscibility. Here, a two-dimensional microfluidic porous medium with a stochastic distribution of pillars was created, and an analogue ternary system was used to mimic the real oil and gas multiple-contact miscibility process. Experiments were performed in two directions, imbibition and drainage, permitting study of different wetting properties of the host rock. The distinct behavior of trapped non-wetting ganglia during imbibition and the evolution of phase interfaces during drainage were observed and analyzed as the system developed miscibility. We show how these observations can be converted into rapid measurements for identifying the development of miscibility.