On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward.

The last five years marked a surge in interest for and use of smart robots, which operate in dynamic and unstructured environments and might interact with humans. We posit that well-validated computer simulation can provide a virtual proving ground that in many cases is instrumental in understanding safely, faster, at lower costs, and more thoroughly how the robots of the future should be designed and controlled for safe operation and improved performance. Against this backdrop, we discuss how simulation can help in robotics, barriers that currently prevent its broad adoption, and potential steps that can eliminate some of these barriers. The points and recommendations made concern the following simulation-in-robotics aspects: simulation of the dynamics of the robot; simulation of the virtual world; simulation of the sensing of this virtual world; simulation of the interaction between the human and the robot; and, in less depth, simulation of the communication between robots. This Perspectives contribution summarizes the points of view that coalesced during a 2018 National Science Foundation/Department of Defense/National Institute for Standards and Technology workshop dedicated to the topic at hand. The meeting brought together participants from a range of organizations, disciplines, and application fields, with expertise at the intersection of robotics, machine learning, and physics-based simulation.

Parkinson's Disease Tremor Detection in the Wild Using Wearable Accelerometers.

Continuous in-home monitoring of Parkinson's Disease (PD) symptoms might allow improvements in assessment of disease progression and treatment effects. As a first step towards this goal, we evaluate the feasibility of a wrist-worn wearable accelerometer system to detect PD tremor in the wild (uncontrolled scenarios). We evaluate the performance of several feature sets and classification algorithms for robust PD tremor detection in laboratory and wild settings. We report results for both laboratory data with accurate labels and wild data with weak labels. The best performance was obtained using a combination of a pre-processing module to extract information from the tremor spectrum (based on non-negative factorization) and a deep neural network for learning relevant features and detecting tremor segments. We show how the proposed method is able to predict patient self-report measures, and we propose a new metric for monitoring PD tremor (i.e., percentage of tremor over long periods of time), which may be easier to estimate the start and end time points of each tremor event while still providing clinically useful information.

A Novel Artificial Intelligence System for Endotracheal Intubation.

OBJECTIVE: Adequate visualization of the glottic opening is a key factor to successful endotracheal intubation (ETI); however, few objective tools exist to help guide providers' ETI attempts toward the glottic opening in real-time. Machine learning/artificial intelligence has helped to automate the detection of other visual structures but its utility with ETI is unknown. We sought to test the accuracy of various computer algorithms in identifying the glottic opening, creating a tool that could aid successful intubation. METHODS: We collected a convenience sample of providers who each performed ETI 10 times on a mannequin using a video laryngoscope (C-MAC, Karl Storz Corp, Tuttlingen, Germany). We recorded each attempt and reviewed one-second time intervals for the presence or absence of the glottic opening. Four different machine learning/artificial intelligence algorithms analyzed each attempt and time point: k-nearest neighbor (KNN), support vector machine (SVM), decision trees, and neural networks (NN). We used half of the videos to train the algorithms and the second half to test the accuracy, sensitivity, and specificity of each algorithm. RESULTS: We enrolled seven providers, three Emergency Medicine attendings, and four paramedic students. From the 70 total recorded laryngoscopic video attempts, we created 2,465 time intervals. The algorithms had the following sensitivity and specificity for detecting the glottic opening: KNN (70%, 90%), SVM (70%, 90%), decision trees (68%, 80%), and NN (72%, 78%). CONCLUSIONS: Initial efforts at computer algorithms using artificial intelligence are able to identify the glottic opening with over 80% accuracy. With further refinements, video laryngoscopy has the potential to provide real-time, direction feedback to the provider to help guide successful ETI.

Next-generation deep learning based on simulators and synthetic data.

Deep learning (DL) is being successfully applied across multiple domains, yet these models learn in a most artificial way: they require large quantities of labeled data to grasp even simple concepts. Thus, the main bottleneck is often access to supervised data. Here, we highlight a trend in a potential solution to this challenge: synthetic data. Synthetic data are becoming accessible due to progress in rendering pipelines, generative adversarial models, and fusion models. Moreover, advancements in domain adaptation techniques help close the statistical gap between synthetic and real data. Paradoxically, this artificial solution is also likely to enable more natural learning, as seen in biological systems, including continual, multimodal, and embodied learning. Complementary to this, simulators and deep neural networks (DNNs) will also have a critical role in providing insight into the cognitive and neural functioning of biological systems. We also review the strengths of, and opportunities and novel challenges associated with, synthetic data.

Exploring the neural correlates of goal-directed action and intention understanding.

Because we are a cooperative species, understanding the goals and intentions of others is critical for human survival. In this fMRI study, participants viewed reaching behaviors in which one of four animated characters moved a hand towards one of two objects and either (a) picked up the object, (b) missed the object, or (c) changed his path halfway to lift the other object. The characters included a human, a humanoid robot, stacked boxes with an arm, and a mechanical claw. The first three moved in an identical, human-like biological pattern. Right posterior superior temporal sulcus (pSTS) activity increased when the human or humanoid robot shifted goals or missed the target relative to obtaining the original goal. This suggests that the pSTS was engaged differentially for figures that appeared more human-like rather than for all human-like motion. Medial frontal areas that are part of a protagonist-monitoring network with the right pSTS (e.g., Mason and Just, 2006) were most engaged for the human character, followed by the robot character. The current data suggest that goal-directed action and intention understanding require this network and it is used similarly for the two processes. Moreover, it is modulated by character identity rather than only the presence of biological motion. We discuss the implications for behavioral theories of goal-directed action and intention understanding.

Sensor-Based Evaluation of Physical Therapy Exercises.

Physical therapy is important for the treatment and prevention of musculoskeletal injuries, as well as recovery from surgery. In this paper, we explore techniques for automatically determining whether an exercise was performed correctly or not, based on camera images and wearable sensors. Classifiers were tested on data collected from 30 patients during normally-scheduled physical therapy appointments. We considered two lower limb exercises, and asked how well classifiers could generalize to the assessment of individuals for whom no prior data were available. We found that our classifiers performed well relative to several metrics (mean accuracy: 0.76, specificity: 0.90), but often returned low sensitivity (mean: 0.34). For one of the two exercises considered, these classifiers compared favorably with human performance.

Comparing laboratory and in-the-wild data for continuous Parkinson's Disease tremor detection.

Passive, continuous monitoring of Parkinson's Disease (PD) symptoms in the wild (i.e., in home environments) could improve disease management, thereby improving a patient's quality of life. We envision a system that uses machine learning to automatically detect PD symptoms from accelerometer data collected in the wild. Building such systems, however, is challenging because it is difficult to obtain labels of symptom occurrences in the wild. Many researchers therefore train machine learning algorithms on laboratory data with the assumption that findings will translate to the wild. This paper assesses how well laboratory data represents wild data by comparing PD symptom (tremor) detection performance of three models on both lab and wild data. Findings indicate that, for this application, laboratory data is not a good representation of wild data. Results also show that training on wild data, even though labels are less precise, leads to better performance on wild data than training on accurate labels from laboratory data.

Sensory adaptation in human balance control: lessons for biomimetic robotic bipeds.

This paper describes mechanisms used by humans to stand on moving platforms, such as a bus or ship, and to combine body orientation and motion information from multiple sensors including vision, vestibular, and proprioception. A simple mechanism, sensory re-weighting, has been proposed to explain how human subjects learn to reduce the effects of inconsistent sensors on balance. Our goal is to replicate this robust balance behavior in bipedal robots. We review results exploring sensory re-weighting in humans and describe implementations of sensory re-weighting in simulation and on a robot.

Weakly-supervised learning for Parkinson's Disease tremor detection.

Continuous, automated monitoring of Parkinsons Disease (PD) symptoms would provide clinicians with more information to understand their patients' disease progression and adjust treatment protocols, thereby improving PD care. Collecting precisely labeled data for Parkinson's symptoms, such as tremor, is difficult. Therefore, algorithms for monitoring should only require weakly-labeled training data. In this paper, we evaluate five standard weakly-supervised algorithms and propose a "stratified" version of three of the algorithms, which take advantage of knowing the approximate amount of tremor within each segment. In particular, we analyze PD tremor detection performance as training segments increase in length from 30 seconds to 10 minutes, and labels thereby become less precise. As segment length increases to 10 minutes, standard algorithms are not able to discriminate tremor from non-tremor. However, our stratified algorithms, which can make use of more nuanced labels, show little decrease in performance as segment length increases.

Predicting Moves-on-Stills for Comic Art Using Viewer Gaze Data.

Comic art consists of a sequence of panels of different shapes and sizes that visually communicate the narrative to the reader. The move-on-stills technique allows such still images to be retargeted for digital displays via camera moves. Today, moves-on-stills can be created by software applications given user-provided parameters for each desired camera move. The proposed algorithm uses viewer gaze as input to computationally predict camera move parameters. The authors demonstrate their algorithm on various comic book panels and evaluate its performance by comparing their results with a professional DVD.

Assessment of Movement Patterns during Intubation between Novice and Experienced Providers Using Mobile Sensors: A Preliminary, Proof of Concept Study.

BACKGROUND: There are likely marked differences in endotracheal intubation (ETI) techniques between novice and experienced providers. We performed a proof of concept study to determine if portable motion technology could identify the motion components of ETI between novice and experienced providers. METHODS: We recruited a sample of novice and experienced providers to perform ETIs on a cadaver. Their movements during ETI were recorded with inertial measurement units (IMUs) on the left wrist. The signals were assessed visually between novice and experienced providers to identify areas of differences at key steps during ETI. We then calculated spectral smoothness (SS), a quantitative measure inversely related to movement variability, for all ETI attempts. RESULTS: We enrolled five novice and five experienced providers. When visually inspecting the data, we noted maximum variability when inserting the blade of the laryngoscope into the mouth and while visualizing the glottic opening. Novice providers also had greater overall variability in their movement patterns (SS novice 6.4 versus SS experienced 26.6). CONCLUSION: Portable IMUs can be used to detect differences in movement patterns between novice and experienced providers in cadavers. Future ETI educational efforts may utilize portable IMUs to help accelerate the learning curve of novice providers.

Anthropomorphic bias found in typically developing children is not found in children with autistic spectrum disorder.

The anthropomorphic bias describes the finding that the perceived naturalness of a biological motion decreases as the human-likeness of a computer-animated agent increases. To investigate the anthropomorphic bias in autistic children, human or cartoon characters were presented with biological and artificial motions side by side on a touchscreen. Children were required to touch one that would grow while the other would disappear, implicitly rewarding their choice. Only typically developing controls depicted the expected preference for biological motion when rendered with human, but not cartoon, characters. Despite performing the task to report a preference, children with autism depicted neither normal nor reversed anthropomorphic bias, suggesting that they are not sensitive to the congruence of form and motion information when observing computer-animated agents' actions.

Are children with autism more responsive to animated characters? A study of interactions with humans and human-controlled avatars.

Few direct comparisons have been made between the responsiveness of children with autism to computer-generated or animated characters and their responsiveness to humans. Twelve 4- to 8-year-old children with autism interacted with a human therapist; a human-controlled, interactive avatar in a theme park; a human actor speaking like the avatar; and cartoon characters who sought social responses. We found superior gestural and verbal responses to the therapist; intermediate response levels to the avatar and the actor; and poorest responses to the cartoon characters, although attention was equivalent across conditions. These results suggest that even avatars that provide live, responsive interactions are not superior to human therapists in eliciting verbal and non-verbal communication from children with autism in this age range.

Hierarchical Aligned Cluster Analysis for Temporal Clustering of Human Motion.

Temporal segmentation of human motion into plausible motion primitives is central to understanding and building computational models of human motion. Several issues contribute to the challenge of discovering motion primitives: the exponential nature of all possible movement combinations, the variability in the temporal scale of human actions, and the complexity of representing articulated motion. We pose the problem of learning motion primitives as one of temporal clustering, and derive an unsupervised hierarchical bottom-up framework called hierarchical aligned cluster analysis (HACA). HACA finds a partition of a given multidimensional time series into m disjoint segments such that each segment belongs to one of k clusters. HACA combines kernel k-means with the generalized dynamic time alignment kernel to cluster time series data. Moreover, it provides a natural framework to find a low-dimensional embedding for time series. HACA is efficiently optimized with a coordinate descent strategy and dynamic programming. Experimental results on motion capture and video data demonstrate the effectiveness of HACA for segmenting complex motions and as a visualization tool. We also compare the performance of HACA to state-of-the-art algorithms for temporal clustering on data of a honey bee dance. The HACA code is available online.

Detecting Parkinsons' symptoms in uncontrolled home environments: a multiple instance learning approach.

In this paper, we propose to use a weakly supervised machine learning framework for automatic detection of Parkinson's Disease motor symptoms in daily living environments. Our primary goal is to develop a monitoring system capable of being used outside of controlled laboratory settings. Such a system would enable us to track medication cycles at home and provide valuable clinical feedback. Most of the relevant prior works involve supervised learning frameworks (e.g., Support Vector Machines). However, in-home monitoring provides only coarse ground truth information about symptom occurrences, making it very hard to adapt and train supervised learning classifiers for symptom detection. We address this challenge by formulating symptom detection under incomplete ground truth information as a multiple instance learning (MIL) problem. MIL is a weakly supervised learning framework that does not require exact instances of symptom occurrences for training; rather, it learns from approximate time intervals within which a symptom might or might not have occurred on a given day. Once trained, the MIL detector was able to spot symptom-prone time windows on other days and approximately localize the symptom instances. We monitored two Parkinson's disease (PD) patients, each for four days with a set of five triaxial accelerometers and utilized a MIL algorithm based on axis parallel rectangle (APR) fitting in the feature space. We were able to detect subject specific symptoms (e.g. dyskinesia) that conformed with a daily log maintained by the patients.

Multi-label classification for the analysis of human motion quality.

Knowing how well an activity is performed is important for home rehabilitation. We would like to not only know if a motion is being performed correctly, but also in what way the motion is incorrect so that we may provide feedback to the user. This paper describes methods for assessing human motion quality using body-worn tri-axial accelerometers and gyroscopes. We use multi-label classifiers to detect subtle errors in exercise performances of eight individuals with knee osteoarthritis, a degenerative disease of the cartilage. We present results obtained using various machine learning methods with decision tree base classifiers. The classifier can detect classes in multi-label data with 75% sensitivity, 90% specificity and 80% accuracy. The methods presented here form the basis for an at-home rehabilitation device that will recognize errors in patient exercise performance, provide appropriate feedback on the performance, and motivate the patient to continue the prescribed regimen.

Motion capture measures variability in laryngoscopic movement during endotracheal intubation: a preliminary report.

INTRODUCTION: Success rates with emergent endotracheal intubation (ETI) improve with increasing provider experience. Few objective metrics exist to quantify differences in ETI technique between providers of various skill levels. We tested the feasibility of using motion capture videography to quantify variability in the motions of the left hand and the laryngoscope in providers with various experience. METHODS: Three providers with varying levels of experience [attending physician (experienced), emergency medicine resident (intermediate), and postdoctoral student with no previous ETI experience (novice)] each performed ETI 4 times on a mannequin. Vicon, a 16-camera system, tracked the 3-dimensional orientation and movement of markers on the providers, handle of the laryngoscope, and mannequin. Attempt duration, path length of the left hand, and the inclination of the plane of the laryngoscope handle (mean square angular deviation from vertical) were calculated for each laryngoscopy attempt. We compared interattempt and interprovider variability of each measure. RESULTS: All ETI attempts were successful. Mean (SD) duration of laryngoscopy attempts differed between experienced [5.50 (0.68) seconds], intermediate [6.32 (1.13) seconds], and novice [12.38 (1.06) seconds] providers (P = 0.021). Mean path length of the left hand did not differ between providers (P = 0.37). Variability of the plane of the laryngoscope differed between providers: 8.3 (experienced), 28.7 (intermediate), and 54.5 (novice) degrees squared. CONCLUSIONS: Motion analysis can detect interprovider differences in hand and laryngoscope movements during ETI, which may be related to provider experience. This technology has potential to objectively measure training and skill in ETI.

Quantitative measurement of motor symptoms in Parkinson's disease: a study with full-body motion capture data.

Recent advancements in the portability and affordability of optical motion capture systems have opened the doors to various clinical applications. In this paper, we look into the potential use of motion capture data for the quantitative analysis of motor symptoms in Parkinson's Disease (PD). The standard of care, human observer-based assessments of the motor symptoms, can be very subjective and are often inadequate for tracking mild symptoms. Motion capture systems, on the other hand, can potentially provide more objective and quantitative assessments. In this pilot study, we perform full-body motion capture of Parkinson's patients with deep brain stimulator off-drugs and with stimulators on and off. Our experimental results indicate that the quantitative measure on spatio-temporal statistics learnt from the motion capture data reveal distinctive differences between mild and severe symptoms. We used a Support Vector Machine (SVM) classifier for discriminating mild vs. severe symptoms with an average accuracy of approximately 90%. Finally, we conclude that motion capture technology could potentially be an accurate, reliable and effective tool for statistical data mining on motor symptoms related to PD. This would enable us to devise more effective ways to track the progression of neurodegenerative movement disorders.