Respiratory motion prediction based on deep artificial neural networks in CyberKnife system: A comparative study.

BACKGROUND: In external beam radiotherapy, a prediction model is required to compensate for the temporal system latency that affects the accuracy of radiation dose delivery. This study focused on a thorough comparison of seven deep artificial neural networks to propose an accurate and reliable prediction model. METHODS: Seven deep predictor models are trained and tested with 800 breathing signals. In this regard, a nonsequential-correlated hyperparameter optimization algorithm is developed to find the best configuration of parameters for all models. The root mean square error (RMSE), mean absolute error, normalized RMSE, and statistical F-test are also used to evaluate network performance. RESULTS: Overall, tuning the hyperparameters results in a 25%-30% improvement for all models compared to previous studies. The comparison between all models also shows that the gated recurrent unit (GRU) with RMSE = 0.108 ± 0.068 mm predicts respiratory signals with higher accuracy and better performance. CONCLUSION: Overall, tuning the hyperparameters in the GRU model demonstrates a better result than the hybrid predictor model used in the CyberKnife VSI system to compensate for the 115 ms system latency. Additionally, it is demonstrated that the tuned parameters have a significant impact on the prediction accuracy of each model.

Human Motion Prediction via Dual-Attention and Multi-Granularity Temporal Convolutional Networks.

Intelligent devices, which significantly improve the quality of life and work efficiency, are now widely integrated into people's daily lives and work. A precise understanding and analysis of human motion is essential for achieving harmonious coexistence and efficient interaction between intelligent devices and humans. However, existing human motion prediction methods often fail to fully exploit the dynamic spatial correlations and temporal dependencies inherent in motion sequence data, which leads to unsatisfactory prediction results. To address this issue, we proposed a novel human motion prediction method that utilizes dual-attention and multi-granularity temporal convolutional networks (DA-MgTCNs). Firstly, we designed a unique dual-attention (DA) model that combines joint attention and channel attention to extract spatial features from both joint and 3D coordinate dimensions. Next, we designed a multi-granularity temporal convolutional networks (MgTCNs) model with varying receptive fields to flexibly capture complex temporal dependencies. Finally, the experimental results from two benchmark datasets, Human3.6M and CMU-Mocap, demonstrated that our proposed method significantly outperformed other methods in both short-term and long-term prediction, thereby verifying the effectiveness of our algorithm.

Synergy-Space Recurrent Neural Network for Transferable Forearm Motion Prediction from Residual Limb Motion.

Transhumeral amputees experience considerable difficulties with controlling a multifunctional prosthesis (powered hand, wrist, and elbow) due to the lack of available muscles to provide electromyographic (EMG) signals. The residual limb motion strategy has become a popular alternative for transhumeral prosthesis control. It provides an intuitive way to estimate the motion of the prosthesis based on the residual shoulder motion, especially for target reaching tasks. Conventionally, a predictive model, typically an artificial neural network (ANN), is directly trained and relied upon to map the shoulder-elbow kinematics using the data from able-bodied subjects without extracting any prior synergistic information. However, it is essential to explicitly identify effective synergies and make them transferable across amputee users for higher accuracy and robustness. To overcome this limitation of the conventional ANN learning approach, this study explicitly combines the kinematic synergies with a recurrent neural network (RNN) to propose a synergy-space neural network for estimating forearm motions (i.e., elbow joint flexion-extension and pronation-supination angles) based on residual shoulder motions. We tested 36 training strategies for each of the 14 subjects, comparing the proposed synergy-space and conventional neural network learning approaches, and we statistically evaluated the results using Pearson's correlation method and the analysis of variance (ANOVA) test. The offline cross-subject analysis indicates that the synergy-space neural network exhibits superior robustness to inter-individual variability, demonstrating the potential of this approach as a transferable and generalized control strategy for transhumeral prosthesis control.

A Non-Equal Time Interval Incremental Motion Prediction Method for Maritime Autonomous Surface Ships.

Recent technological advancements facilitate the autonomous navigation of maritime surface ships. The accurate data given by a range of various sensors serve as the primary assurance of a voyage's safety. Nevertheless, as sensors have various sample rates, they cannot obtain information at the same time. Fusion decreases the accuracy and reliability of perceptual data if different sensor sample rates are not taken into account. Hence, it is helpful to increase the quality of the fusion information to precisely anticipate the motion status of ships at the sampling time of each sensor. This paper proposes a non-equal time interval incremental prediction method. In this method, the high dimensionality of the estimated state and nonlinearity of the kinematic equation are taken into consideration. First, the cubature Kalman filter is employed to estimate a ship's motion at equal intervals based on the ship's kinematic equation. Next, a ship motion state predictor based on a long short-term memory network structure is created, using the increment and time interval of the historical estimation sequence as the network input and the increment of the motion state at the projected time as the network output. The suggested technique can lessen the effect of the speed difference between the test set and the training set on the prediction accuracy compared with the traditional long short-term memory prediction method. Finally, comparison experiments are carried out to validate the precision and effectiveness of the proposed approach. The experimental results show that the root-mean-square error coefficient of the prediction error is decreased on average by roughly 78% for various modes and speeds when compared with the conventional non-incremental long short-term memory prediction approach. Additionally, the proposed prediction technology and the traditional approach have virtually the same algorithm times, which may fulfill the real engineering requirements.

LiDAR-Based Maintenance of a Safe Distance between a Human and a Robot Arm.

This paper demonstrates the capabilities of three-dimensional (3D) LiDAR scanners in supporting a safe distance maintenance functionality in human-robot collaborative applications. The use of such sensors is severely under-utilised in collaborative work with heavy-duty robots. However, even with a relatively modest proprietary 3D sensor prototype, a respectable level of safety has been achieved, which should encourage the development of such applications in the future. Its associated intelligent control system (ICS) is presented, as well as the sensor's technical characteristics. It acquires the positions of the robot and the human periodically, predicts their positions in the near future optionally, and adjusts the robot's speed to keep its distance from the human above the protective separation distance. The main novelty is the possibility to load an instance of the robot programme into the ICS, which then precomputes the future position and pose of the robot. Higher accuracy and safety are provided, in comparison to traditional predictions from known real-time and near-past positions and poses. The use of a 3D LiDAR scanner in a speed and separation monitoring application and, particularly, its specific placing, are also innovative and advantageous. The system was validated by analysing videos taken by the reference validation camera visually, which confirmed its safe operation in reasonably limited ranges of robot and human speeds.

3D-DIoU: 3D Distance Intersection over Union for Multi-Object Tracking in Point Cloud.

Multi-object tracking (MOT) is a prominent and important study in point cloud processing and computer vision. The main objective of MOT is to predict full tracklets of several objects in point cloud. Occlusion and similar objects are two common problems that reduce the algorithm's performance throughout the tracking phase. The tracking performance of current MOT techniques, which adopt the 'tracking-by-detection' paradigm, is degrading, as evidenced by increasing numbers of identification (ID) switch and tracking drifts because it is difficult to perfectly predict the location of objects in complex scenes that are unable to track. Since the occluded object may have been visible in former frames, we manipulated the speed and location position of the object in the previous frames in order to guess where the occluded object might have been. In this paper, we employed a unique intersection over union (IoU) method in three-dimension (3D) planes, namely a distance IoU non-maximum suppression (DIoU-NMS) to accurately detect objects, and consequently we use 3D-DIoU for an object association process in order to increase tracking robustness and speed. By using a hybrid 3D DIoU-NMS and 3D-DIoU method, the tracking speed improved significantly. Experimental findings on the Waymo Open Dataset and nuScenes dataset, demonstrate that our multistage data association and tracking technique has clear benefits over previously developed algorithms in terms of tracking accuracy. In comparison with other 3D MOT tracking methods, our proposed approach demonstrates significant enhancement in tracking performances.

A New Benchmark for Consumer Visual Tracking and Apparent Demographic Estimation from RGB and Thermal Images.

Visual tracking and attribute estimation related to age or gender information of multiple person entities in a scene are mature research topics with the advent of deep learning techniques. However, when it comes to indoor images such as video sequences of retail consumers, data are not always adequate or accurate enough to essentially train effective models for consumer detection and tracking under various adverse factors. This in turn affects the quality of recognizing age or gender for those detected instances. In this work, we introduce two novel datasets: Consumers comprises 145 video sequences compliant to personal information regulations as far as facial images are concerned and BID is a set of cropped body images from each sequence that can be used for numerous computer vision tasks. We also propose an end-to-end framework which comprises CNNs as object detectors, LSTMs for motion forecasting of the tracklet association component in a sequence, along with a multi-attribute classification model for apparent demographic estimation of the detected outputs, aiming to capture useful metadata of consumer product preferences. Obtained results on tracking and age/gender prediction are promising with respect to reference systems while they indicate the proposed model's potential for practical consumer metadata extraction.

Neural Network-Based Strong Motion Prediction for On-Site Earthquake Early Warning.

Developing on-site earthquake early warning systems has been a challenging problem because of time limitations and the amount of information that can be collected before the warning needs to be issued. A potential solution that could prevent severe disasters is to predict the potential strong motion using the initial P-wave signal and provide warnings before serious ground shaking starts. In practice, the accuracy of prediction is the most critical issue for earthquake early warning systems. Traditional methods use certain criteria, selected through intuition or experience, to make the prediction. However, the criteria thresholds are difficult to select and may significantly affect the prediction accuracy. This paper investigates methods based on artificial intelligence for predicting the greatest earthquake ground motion early, when the P-wave arrives at seismograph stations. A neural network model is built to make the predictions using a small window of the initial P-wave acceleration signal. The model is trained by seismic waves collected from 1991 to 2019 in Taiwan and is evaluated by events in 2020 and 2021. From these evaluations, the proposed scheme significantly outperforms the threshold-based method in terms of its accuracy and average leading time.

Human Arm Motion Prediction for Collision Avoidance in a Shared Workspace.

Industry 4.0 transforms classical industrial systems into more human-centric and digitized systems. Close human-robot collaboration is becoming more frequent, which means security and efficiency issues need to be carefully considered. In this paper, we propose to equip robots with exteroceptive sensors and online motion generation so that the robot is able to perceive and predict human trajectories and react to the motion of the human in order to reduce the occurrence of the collisions. The dataset for training is generated in a real environment in which a human and a robot are sharing their workspace. An Encoder-Decoder based network is proposed to predict the human hand trajectories. A Model Predictive Control (MPC) framework is also proposed, which is able to plan a collision-free trajectory in the shared workspace based on this human motion prediction. The proposed framework is validated in a real environment that ensures collision free collaboration between humans and robots in a shared workspace.

From Perception to Navigation in Environments with Persons: An Indoor Evaluation of the State of the Art.

Research in the field of social robotics is allowing service robots to operate in environments with people. In the aim of realizing the vision of humans and robots coexisting in the same environment, several solutions have been proposed to (1) perceive persons and objects in the immediate environment; (2) predict the movements of humans; as well as (3) plan the navigation in agreement with socially accepted rules. In this work, we discuss the different aspects related to social navigation in the context of our experience in an indoor environment. We describe state-of-the-art approaches and experiment with existing methods to analyze their performance in practice. From this study, we gather first-hand insights into the limitations of current solutions and identify possible research directions to address the open challenges. In particular, this paper focuses on topics related to perception at the hardware and application levels, including 2D and 3D sensors, geometric and mainly semantic mapping, the prediction of people trajectories (physics-, pattern- and planning-based), and social navigation (reactive and predictive) in indoor environments.

Probabilistic Traffic Motion Labeling for Multi-Modal Vehicle Route Prediction.

The prediction of the motion of traffic participants is a crucial aspect for the research and development of Automated Driving Systems (ADSs). Recent approaches are based on multi-modal motion prediction, which requires the assignment of a probability score to each of the multiple predicted motion hypotheses. However, there is a lack of ground truth for this probability score in the existing datasets. This implies that current Machine Learning (ML) models evaluate the multiple predictions by comparing them with the single real trajectory labeled in the dataset. In this work, a novel data-based method named Probabilistic Traffic Motion Labeling (PROMOTING) is introduced in order to (a) generate probable future routes and (b) estimate their probabilities. PROMOTING is presented with the focus on urban intersections. The generation of probable future routes is (a) based on a real traffic dataset and consists of two steps: first, a clustering of intersections with similar road topology, and second, a clustering of similar routes that are driven in each cluster from the first step. The estimation of the route probabilities is (b) based on a frequentist approach that considers how traffic participants will move in the future given their motion history. PROMOTING is evaluated with the publicly available Lyft database. The results show that PROMOTING is an appropriate approach to estimate the probabilities of the future motion of traffic participants in urban intersections. In this regard, PROMOTING can be used as a labeling approach for the generation of a labeled dataset that provides a probability score for probable future routes. Such a labeled dataset currently does not exist and would be highly valuable for ML approaches with the task of multi-modal motion prediction. The code is made open source.

A Human-Following Motion Planning and Control Scheme for Collaborative Robots Based on Human Motion Prediction.

Human-Robot Interaction (HRI) for collaborative robots has become an active research topic recently. Collaborative robots assist human workers in their tasks and improve their efficiency. However, the worker should also feel safe and comfortable while interacting with the robot. In this paper, we propose a human-following motion planning and control scheme for a collaborative robot which supplies the necessary parts and tools to a worker in an assembly process in a factory. In our proposed scheme, a 3-D sensing system is employed to measure the skeletal data of the worker. At each sampling time of the sensing system, an optimal delivery position is estimated using the real-time worker data. At the same time, the future positions of the worker are predicted as probabilistic distributions. A Model Predictive Control (MPC)-based trajectory planner is used to calculate a robot trajectory that supplies the required parts and tools to the worker and follows the predicted future positions of the worker. We have installed our proposed scheme in a collaborative robot system with a 2-DOF planar manipulator. Experimental results show that the proposed scheme enables the robot to provide anytime assistance to a worker who is moving around in the workspace while ensuring the safety and comfort of the worker.

Lateral Motion Prediction of On-Road Preceding Vehicles: A Data-Driven Approach.

Drivers' behaviors and decision making on the road directly affect the safety of themselves, other drivers, and pedestrians. However, as distinct entities, people cannot predict the motions of surrounding vehicles and they have difficulty in performing safe reactionary driving maneuvers in a short time period. To overcome the limitations of making an immediate prediction, in this work, we propose a two-stage data-driven approach: classifying driving patterns of on-road surrounding vehicles using the Gaussian mixture models (GMM); and predicting vehicles' short-term lateral motions (i.e., left/right turn and left/right lane change) based on real-world vehicle mobility data, provided by the U.S. Department of Transportation, with different ensemble decision trees. We considered several important kinetic features and higher order kinematic variables. The research results of our proposed approach demonstrate the effectiveness of pattern classification and on-road lateral motion prediction. This methodology framework has the potential to be incorporated into current data-driven collision warning systems, to enable more practical on-road preprocessing in intelligent vehicles, and to be applied in autopilot-driving scenarios.

Structured Kernel Subspace Learning for Autonomous Robot Navigation.

This paper considers two important problems for autonomous robot navigation in a dynamic environment, where the goal is to predict pedestrian motion and control a robot with the prediction for safe navigation. While there are several methods for predicting the motion of a pedestrian and controlling a robot to avoid incoming pedestrians, it is still difficult to safely navigate in a dynamic environment due to challenges, such as the varying quality and complexity of training data with unwanted noises. This paper addresses these challenges simultaneously by proposing a robust kernel subspace learning algorithm based on the recent advances in nuclear-norm and l 1 -norm minimization. We model the motion of a pedestrian and the robot controller using Gaussian processes. The proposed method efficiently approximates a kernel matrix used in Gaussian process regression by learning low-rank structured matrix (with symmetric positive semi-definiteness) to find an orthogonal basis, which eliminates the effects of erroneous and inconsistent data. Based on structured kernel subspace learning, we propose a robust motion model and motion controller for safe navigation in dynamic environments. We evaluate the proposed robust kernel learning in various tasks, including regression, motion prediction, and motion control problems, and demonstrate that the proposed learning-based systems are robust against outliers and outperform existing regression and navigation methods.

AMHGCN: Adaptive multi-level hypergraph convolution network for human motion prediction.

Human motion prediction is the key technology for many real-life applications, e.g., self-driving and human-robot interaction. The recent approaches adopt the unrestricted full-connection graph representation to capture the relationships inside the human skeleton. However, there are two issues to be solved: (i) these unrestricted full-connection graph representation methods neglect the inherent dependencies across the joints of the human body; (ii) these methods represent human motions using the features extracted from a single level and thus can neither fully exploit the various connection relationships among the human body nor guarantee the human motion prediction results to be reasonable. To tackle the above issues, we propose an adaptive multi-level hypergraph convolution network (AMHGCN), which uses the adaptive multi-level hypergraph representation to capture various dependencies among the human body. Our method has four different levels of hypergraph representations, including (i) the joint-level hypergraph representation to capture inherent kinetic dependencies in the human body, (ii) the part-level hypergraph representation to exploit the kinetic characteristics at a higher level (in comparison to the joint-level) by viewing some part of the human body as an entirety, (iii) the component-level hypergraph representation to model the semantic information, and (iv) the global-level hypergraph representation to extract long-distance dependencies in the human body. In addition, to take full advantage of the knowledge carried in the training data, we propose a reverse loss (i.e., adopting the future human poses to predict the historical poses reversely) to realize data augmentation. Extensive experiments show that our proposed AMHGCN can achieve state-of-the-art performance on three benchmarks, i.e., Human3.6M, CMU-Mocap, and 3DPW.

Real-time motion management in MRI-guided radiotherapy: Current status and AI-enabled prospects.

MRI-guided radiotherapy (MRIgRT) is a highly complex treatment modality, allowing adaptation to anatomical changes occurring from one treatment day to the other (inter-fractional), but also to motion occurring during a treatment fraction (intra-fractional). In this vision paper, we describe the different steps of intra-fractional motion management during MRIgRT, from imaging to beam adaptation, and the solutions currently available both clinically and at a research level. Furthermore, considering the latest developments in the literature, a workflow is foreseen in which motion-induced over- and/or under-dosage is compensated in 3D, with minimal impact to the radiotherapy treatment time. Considering the time constraints of real-time adaptation, a particular focus is put on artificial intelligence (AI) solutions as a fast and accurate alternative to conventional algorithms.

A dynamic spatiotemporal model for fall warning and protection.

Early detection of falls is important for reducing fall injuries. However, existing fall detection strategies mostly focus on reducing impact injuries rather than avoiding falls. This study proposed the concept of identifying "Imbalance Point" to warn the body imbalance, allowing sufficient time to recover balance. And if falling cannot be avoided, an impact sign is released by detecting the "Fall Point" prior to the impact. To achieve this goal, motion prediction model and balance recovery model are integrated into a spatiotemporal framework to analyze dynamic and kinematic features of body motion. Eight healthy young volunteers participated in three sets of experiment: Normal trial, Recovery trial and Fall trial. The body motion in the trials was recorded using Microsoft Azure Kinect. The results show that the developed algorithm for Fall Point detection achieved 100% sensitivity and 98.6% specificity, along with an average lead time of 297 ms. Moreover, Imbalance Point was successfully detected in all Fall trials, and the average time interval between Imbalance Point and Fall Point was 315 ms, longer than reported step reaction time for elderly (approximately 270 ms). The experiment results demonstrate that the developed algorithm have great potential for fall warning and protection in the elderly.

DTDNet: Dynamic Target Driven Network for pedestrian trajectory prediction.

Predicting the trajectories of pedestrians is an important and difficult task for many applications, such as robot navigation and autonomous driving. Most of the existing methods believe that an accurate prediction of the pedestrian intention can improve the prediction quality. These works tend to predict a fixed destination coordinate as the agent intention and predict the future trajectory accordingly. However, in the process of moving, the intention of a pedestrian could be a definite location or a general direction and area, and may change dynamically with the changes of surrounding. Thus, regarding the agent intention as a fixed 2-d coordinate is insufficient to improve the future trajectory prediction. To address this problem, we propose Dynamic Target Driven Network for pedestrian trajectory prediction (DTDNet), which employs a multi-precision pedestrian intention analysis module to capture this dynamic. To ensure that this extracted feature contains comprehensive intention information, we design three sub-tasks: predicting coarse-precision endpoint coordinate, predicting fine-precision endpoint coordinate and scoring scene sub-regions. In addition, we propose a original multi-precision trajectory data extraction method to achieve multi-resolution representation of future intention and make it easier to extract local scene information. We compare our model with previous methods on two publicly available datasets (ETH-UCY and Stanford Drone Dataset). The experimental results show that our DTDNet achieves better trajectory prediction performance, and conducts better pedestrian intention feature representation.

Convolutional LSTM model for cine image prediction of abdominal motion.

Objective.In this study, we tackle the challenge of latency in magnetic resonance linear accelerator (MR-Linac) systems, which compromises target coverage accuracy in gated real-time radiotherapy. Our focus is on enhancing motion prediction precision in abdominal organs to address this issue. We developed a convolutional long short-term memory (convLSTM) model, utilizing 2D cine magnetic resonance (cine-MR) imaging for this purpose.Approach.Our model, featuring a sequence-to-one architecture with six input frames and one output frame, employs structural similarity index measure (SSIM) as loss function. Data was gathered from 17 cine-MRI datasets using the Philips Ingenia MR-sim system and an Elekta Unity MR-Linac equivalent sequence, focusing on regions of interest (ROIs) like the stomach, liver, pancreas, and kidney. The datasets varied in duration from 1 to 10 min.Main results.The study comprised three main phases: hyperparameter optimization, individual training, and transfer learning with or without fine-tuning. Hyperparameters were initially optimized to construct the most effective model. Then, the model was individually applied to each dataset to predict images four frames ahead (1.24-3.28 s). We evaluated the model's performance using metrics such as SSIM, normalized mean square error, normalized correlation coefficient, and peak signal-to-noise ratio, specifically for ROIs with target motion. The average SSIM values achieved were 0.54, 0.64, 0.77, and 0.66 for the stomach, liver, kidney, and pancreas, respectively. In the transfer learning phase with fine-tuning, the model showed improved SSIM values of 0.69 for the liver and 0.78 for the kidney, compared to 0.64 and 0.37 without fine-tuning.Significance. The study's significant contribution is demonstrating the convLSTM model's ability to accurately predict motion for multiple abdominal organs using a Unity-equivalent MR sequence. This advancement is key in mitigating latency issues in MR-Linac radiotherapy, potentially improving the precision and effectiveness of real-time treatment for abdominal cancers.

Validation of a Finite Element Simulation for Predicting Individual Knee Joint Kinematics.

Goal: We introduce an in-vivo validated finite element (FE) simulation approach for predicting individual knee joint kinematics. Our vision is to improve clinicians' understanding of the complex individual anatomy and potential pathologies to improve treatment and restore physiological joint kinematics. Methods: Our 3D FE modeling approach for individual human knee joints is based on segmentation of anatomical structures extracted from routine static magnetic resonance (MR) images. We validate the predictive abilities of our model using static MR images of the knees of eleven healthy volunteers in dedicated knee poses, which are achieved using a customized MR-compatible pneumatic loading device. Results: Our FE simulations reach an average translational accuracy of 2 mm and an average angular accuracy of 1[Formula: see text] compared to the reference knee pose. Conclusions: Reaching high accuracy, our individual FE model can be used in the decision-making process to restore knee joint stability and functionality after various knee injuries.