Learning Compliant Box-in-Box Insertion through Haptic-Based Robotic Teleoperation.

In modern logistics, the box-in-box insertion task is representative of a wide range of packaging applications, and automating compliant object insertion is difficult due to challenges in modelling the object deformation during insertion. Using Learning from Demonstration (LfD) paradigms, which are frequently used in robotics to facilitate skill transfer from humans to robots, can be one solution for complex tasks that are difficult to mathematically model. In order to automate the box-in-box insertion task for packaging applications, this study makes use of LfD techniques. The proposed framework has three phases. Firstly, a master-slave teleoperated robot system is used in the initial phase to haptically demonstrate the insertion task. Then, the learning phase involves identifying trends in the demonstrated trajectories using probabilistic methods, in this case, Gaussian Mixture Regression. In the third phase, the insertion task is generalised, and the robot adjusts to any object position using barycentric interpolation. This method is novel because it tackles tight insertion by taking advantage of the boxes' natural compliance, making it possible to complete the task even with a position-controlled robot. To determine whether the strategy is generalisable and repeatable, experimental validation was carried out.

The Design and Development of Instrumented Toys for the Assessment of Infant Cognitive Flexibility.

The first years of an infant's life represent a sensitive period for neurodevelopment where one can see the emergence of nascent forms of executive function (EF), which are required to support complex cognition. Few tests exist for measuring EF during infancy, and the available tests require painstaking manual coding of infant behaviour. In modern clinical and research practice, human coders collect data on EF performance by manually labelling video recordings of infant behaviour during toy or social interaction. Besides being extremely time-consuming, video annotation is known to be rater-dependent and subjective. To address these issues, starting from existing cognitive flexibility research protocols, we developed a set of instrumented toys to serve as a new type of task instrumentation and data collection tool suitable for infant use. A commercially available device comprising a barometer and an inertial measurement unit (IMU) embedded in a 3D-printed lattice structure was used to detect when and how the infant interacts with the toy. The data collected using the instrumented toys provided a rich dataset that described the sequence of toy interaction and individual toy interaction patterns, from which EF-relevant aspects of infant cognition can be inferred. Such a tool could provide an objective, reliable, and scalable method of collecting early developmental data in socially interactive contexts.

Towards Haptic-Based Dual-Arm Manipulation.

Vision is the main component of current robotics systems that is used for manipulating objects. However, solely relying on vision for hand-object pose tracking faces challenges such as occlusions and objects moving out of view during robotic manipulation. In this work, we show that object kinematics can be inferred from local haptic feedback at the robot-object contact points, combined with robot kinematics information given an initial vision estimate of the object pose. A planar, dual-arm, teleoperated robotic setup was built to manipulate an object with hands shaped like circular discs. The robot hands were built with rubber cladding to allow for rolling contact without slipping. During stable grasping by the dual arm robot, under quasi-static conditions, the surface of the robot hand and object at the contact interface is defined by local geometric constraints. This allows one to define a relation between object orientation and robot hand orientation. With rolling contact, the displacement of the contact point on the object surface and the hand surface must be equal and opposite. This information, coupled with robot kinematics, allows one to compute the displacement of the object from its initial location. The mathematical formulation of the geometric constraints between robot hand and object is detailed. This is followed by the methodology in acquiring data from experiments to compute object kinematics. The sensors used in the experiments, along with calibration procedures, are presented before computing the object kinematics from recorded haptic feedback. Results comparing object kinematics obtained purely from vision and from haptics are presented to validate our method, along with the future ideas for perception via haptic manipulation.

Fast Kinematic Re-Calibration for Industrial Robot Arms.

Accurate kinematic modelling is pivotal in the safe and reliable execution of both contact and non-contact robotic applications. The kinematic models provided by robot manufacturers are valid only under ideal conditions and it is necessary to account for the manufacturing errors, particularly the joint offsets introduced during the assembling stages, which is identified as the underlying problem for position inaccuracy in more than 90% of the situations. This work was motivated by a very practical need, namely the discrepancy in terms of end-effector kinematics as computed by factory-calibrated internal controller and the nominal kinematic model as per robot datasheet. Even though the problem of robot calibration is not new, the focus is generally on the deployment of external measurement devices (for open loop calibration) or mechanical fixtures (for closed loop calibration). On the other hand, we use the factory-calibrated controller as an 'oracle' for our fast-recalibration approach. This allows extracting calibrated intrinsic parameters (e.g., link lengths) otherwise not directly available from the 'oracle', for use in ad-hoc control strategies. In this process, we minimize the kinematic mismatch between the ideal and the factory-calibrated robot models for a Kinova Gen3 ultra-lightweight robot by compensating for the joint zero position error and the possible variations in the link lengths. Experimental analysis has been presented to validate the proposed method, followed by the error comparison between the calibrated and un-calibrated models over training and test sets.

Haptic Manipulation of 3D Scans for Geometric Feature Enhancement.

Localisation of geometric features like holes, edges, slots, etc. is vital to robotic planning in industrial automation settings. Low-cost 3D scanners are crucial in terms of improving accessibility, but pose a practical challenge to feature localisation because of poorer resolution and consequently affect robotic planning. In this work, we address the possibility of enhancing the quality of a 3D scan by a manual 'touch-up' of task-relevant features, to ensure their automatic detection prior to automation. We propose a framework whereby the operator (i) has access to both the actual work-piece and its 3D scan; (ii) evaluates the missing salient features from the scan; (iii) uses a haptic stylus to physically interact with the actual work-piece, around such specific features; (iv) interactively updates the scan using the position and force information from the haptic stylus. The contribution of this work is the use of haptic mismatch for geometric update. Specifically, the geometry from the 3D scan is used to predict haptic feedback at a point on the work-piece surface. The haptic mismatch is derived as a measure of error between this prediction and the real interaction forces from physical contact at that point on the work-piece. The geometric update is driven until the haptic mismatch is minimised. Convergence of the proposed algorithm is first numerically verified on an analytical surface with simulated physical interaction. Error analysis of the surface position and orientations were also plotted. Experiments were conducted using a motion capture system providing sub-mm accuracy in position and a 6 axis F/T sensor. Missing features are successfully detected after the update of the scan using the proposed method in an experiment.

Elbow Motion Trajectory Prediction Using a Multi-Modal Wearable System: A Comparative Analysis of Machine Learning Techniques.

Motion intention detection is fundamental in the implementation of human-machine interfaces applied to assistive robots. In this paper, multiple machine learning techniques have been explored for creating upper limb motion prediction models, which generally depend on three factors: the signals collected from the user (such as kinematic or physiological), the extracted features and the selected algorithm. We explore the use of different features extracted from various signals when used to train multiple algorithms for the prediction of elbow flexion angle trajectories. The accuracy of the prediction was evaluated based on the mean velocity and peak amplitude of the trajectory, which are sufficient to fully define it. Results show that prediction accuracy when using solely physiological signals is low, however, when kinematic signals are included, it is largely improved. This suggests kinematic signals provide a reliable source of information for predicting elbow trajectories. Different models were trained using 10 algorithms. Regularization algorithms performed well in all conditions, whereas neural networks performed better when the most important features are selected. The extensive analysis provided in this study can be consulted to aid in the development of accurate upper limb motion intention detection models.

A Complementary Filter Design on SE(3) to IdentifyMicro-Motions during 3D Motion Tracking.

In 3D motion capture, multiple methods have been developed in order to optimize thequality of the captured data. While certain technologies, such as inertial measurement units (IMU),are mostly suitable for 3D orientation estimation at relatively high frequencies, other technologies,such as marker-based motion capture, are more suitable for 3D position estimations at a lower frequencyrange. In this work, we introduce a complementary filter that complements 3D motion capture datawith high-frequency acceleration signals from an IMU. While the local optimization reduces the error ofthe motion tracking, the additional accelerations can help to detect micro-motions that are useful whendealing with high-frequency human motions or robotic applications. The combination of high-frequencyaccelerometers improves the accuracy of the data and helps to overcome limitations in motion capturewhen micro-motions are not traceable with 3D motion tracking system. In our experimental evaluation,we demonstrate the improvements of the motion capture results during translational, rotational,and combined movements.

Estimating Human Wrist Stiffness during a Tooling Task.

In this work, we propose a practical approach to estimate human joint stiffness during tooling tasks for the purpose of programming a robot by demonstration. More specifically, we estimate the stiffness along the wrist radial-ulnar deviation while a human operator performs flexion-extension movements during a polishing task. The joint stiffness information allows to transfer skills from expert human operators to industrial robots. A typical hand-held, abrasive tool used by humans during finishing tasks was instrumented at the handle (through which both robots and humans are attached to the tool) to assess the 3D force/torque interactions between operator and tool during finishing task, as well as the 3D kinematics of the tool itself. Building upon stochastic methods for human arm impedance estimation, the novelty of our approach is that we rely on the natural variability taking place during the multi-passes task itself to estimate (neuro-)mechanical impedance during motion. Our apparatus (hand-held, finishing tool instrumented with motion capture and multi-axis force/torque sensors) and algorithms (for filtering and impedance estimation) were first tested on an impedance-controlled industrial robot carrying out the finishing task of interest, where the impedance could be pre-programmed. We were able to accurately estimate impedance in this case. The same apparatus and algorithms were then applied to the same task performed by a human operators. The stiffness values of the human operator, at different force level, correlated positively with the muscular activity, measured during the same task.

Black-Scholes Theory and Diffusion Processes on the Cotangent Bundle of the Affine Group.

The Black-Scholes partial differential equation (PDE) from mathematical finance has been analysed extensively and it is well known that the equation can be reduced to a heat equation on Euclidean space by a logarithmic transformation of variables. However, an alternative interpretation is proposed in this paper by reframing the PDE as evolving on a Lie group. This equation can be transformed into a diffusion process and solved using mean and covariance propagation techniques developed previously in the context of solving Fokker-Planck equations on Lie groups. An extension of the Black-Scholes theory with coupled asset dynamics produces a diffusion equation on the affine group, which is not a unimodular group. In this paper, we show that the cotangent bundle of a Lie group endowed with a semidirect product group operation, constructed in this paper for the case of groups with trivial centers, is always unimodular and considering PDEs as diffusion processes on the unimodular cotangent bundle group allows a direct application of previously developed mean and covariance propagation techniques, thereby offering an alternative means of solution of the PDEs. Ultimately these results, provided here in the context of PDEs in mathematical finance may be applied to PDEs arising in a variety of different fields and inform new methods of solution.

Analysis of accuracy in pointing with redundant hand-held tools: a geometric approach to the uncontrolled manifold method.

This work introduces a coordinate-independent method to analyse movement variability of tasks performed with hand-held tools, such as a pen or a surgical scalpel. We extend the classical uncontrolled manifold (UCM) approach by exploiting the geometry of rigid body motions, used to describe tool configurations. In particular, we analyse variability during a static pointing task with a hand-held tool, where subjects are asked to keep the tool tip in steady contact with another object. In this case the tool is redundant with respect to the task, as subjects control position/orientation of the tool, i.e. 6 degrees-of-freedom (dof), to maintain the tool tip position (3dof) steady. To test the new method, subjects performed a pointing task with and without arm support. The additional dof introduced in the unsupported condition, injecting more variability into the system, represented a resource to minimise variability in the task space via coordinated motion. The results show that all of the seven subjects channeled more variability along directions not directly affecting the task (UCM), consistent with previous literature but now shown in a coordinate-independent way. Variability in the unsupported condition was only slightly larger at the endpoint but much larger in the UCM.

Tracing curves in the plane: Geometric-invariant learning from human demonstrations.

The empirical laws governing human-curvilinear movements have been studied using various relationships, including minimum jerk, the 2/3 power law, and the piecewise power law. These laws quantify the speed-curvature relationships of human movements during curve tracing using critical speed and curvature as regressors. In this work, we provide a reservoir computing-based framework that can learn and reproduce human-like movements. Specifically, the geometric invariance of the observations, i.e., lateral distance from the closest point on the curve, instantaneous velocity, and curvature, when viewed from the moving frame of reference, are exploited to train the reservoir system. The artificially produced movements are evaluated using the power law to assess whether they are indistinguishable from their human counterparts. The generalisation capabilities of the trained reservoir to curves that have not been used during training are also shown.

How to Train Your Posture: Haptic Feedback Can be Used for Postural Adaptation of the Trunk During Upper-Limb Motor Training.

Poor trunk posture, especially during long periods of sitting, could lead to problems such as Low Back Pain (LBP) and Forward Head Posture (FHP). Typical solutions are based on visual or vibration-based feedback. However, these systems could lead to feedback being ignored by the user and phantom vibration syndrome, respectively. In this study, we propose using haptic feedback for postural adaptation. In this two-part study, twenty-four healthy participants (age 25.87 ± 2.17 years) adapted to three different postural targets in the anterior direction while performing a unimanual reaching task using a robotic device. Results suggest a strong adaptation to the desired postural targets. Mean anterior trunk bending after the intervention is significantly different compared to baseline measurements for all postural targets. Additional analysis of movement straightness and smoothness indicates an absence of any negative interference of posture-based feedback on the performance of reaching movement. Taken together, these results suggest that haptic feedback-based systems could be used for postural adaptation applications. Also, this type of postural adaptation system can be used during the rehabilitation of stroke patients to reduce trunk compensation in lieu of typical physical constraint-based methods.

Design of a Passive Wearable Device Using an Optimized Mechanical Metamaterial for Mirror Therapy.

Mirror Therapy (MT) is an effective therapeutic method used in the rehabilitation of hemiplegics. The effectiveness of this method is improved by employing a bi-modal approach which requires the synchronous movement of the affected and unaffected arm. For this purpose, we describe the design of a wearable device using a Mechanical Metamaterial (MM) that is optimized for the specific user to provide passive assistance of wrist flexion-extension and enable synchronous motion of the affected and unaffected arm during MT.

Embedding inertial-magnetic sensors in everyday objects: assessing spatial cognition in children.

This paper describes an interdisciplinary approach to the assessment of children development of spatial cognition, with a focus on the technology. An instrumented toy (block-box) is presented which embeds magneto-inertial sensors for orientation tracking, specifically developed to assess the ability to insert objects into holes. The functional specifications are derived from experimental protocols devised by neuroscientists to assess spatial cognition skills in children. Technological choices are emphasized with respect to ecological requirements. Ad-hoc calibration procedures are presented which are suitable to unstructured environments. Preliminary results based on experimental trials carried out at a day-care on typically developing children (12-36 months old) show how the instrumented objects can be used effectively in a semi-automatic fashion (i.e., rater-independent) to derive accurate measurements such as orientation errors and insertion time which are relevant to the object insertion task. This study indicates that a technological approach to ecological assessment of spatial cognition in children is indeed feasible and maybe useful for identification and early assessment of developmental delay.

Let the force guide you: a performance-based adaptive algorithm for postural training using haptic feedback.

Motor learning is an essential component of human behavior. Many different factors can influence the process of motor learning, such as the amount of practice and type of feedback. Changes in task difficulty during training can also considerably impact motor learning. Typical motor learning studies include a sequential variation of task difficulty, i.e., easy to challenging, irrespective of user performance. However, many studies have reported the importance of performance-based task difficulty variation for effective motor learning and skill transfer. A performance-based adaptive algorithm for task difficulty variation based on the challenge-point framework is proposed in this study. The algorithm is described for postural adaptation during simultaneous upper-limb training. Ten healthy participants (28 ± 2.44 years) were recruited to validate the algorithm. Participants adapted to a postural target of 20° in the anterior direction from the initial upright posture while performing a unimanual reaching task using a robotic device. Results suggest a significant decrease in postural error after training. The algorithm successfully adapted the task difficulty based on the performance of the user. The proposed algorithm could be modified for different motor skills and can be further evaluated for different applications in order to maximize the potential benefits of rehabilitation sessions.

Haptic feedback system for postural adaptation during robotic rehabilitation of upper limb.

Although trunk compensation during stroke rehabilitation is widely studied, the proposed solutions primarily include a trunk constraint, which has several disadvantages. In this study, we have proposed a haptic feedback-based system for postural training during upper-limb motor rehabilitation. We have tested the proposed system on six healthy people in this preliminary study. Participants performed a simple 1-dimensional reaching task while their posture was being monitored. They received haptic feedback based on their trunk posture. Preliminary results revealed a significant decline in postural error (p<0.05) after the haptic-based training. The reduction in error was maintained even after haptic feedback was turned off. This study shows that haptic feedback could be a viable alternative to the traditional constraint-based methods for postural adaptation. Additional studies need to be conducted to further evaluate the influence of using such feedback strategies.

Pointing with the wrist: a postural model for Donders' law.

The central nervous system uses stereotypical combinations of the three wrist/forearm joint angles to point in a given (2D) direction in space. In this paper, we first confirm and analyze this Donders' law for the wrist as well as the distributions of the joint angles. We find that the quadratic surfaces fitting the experimental wrist configurations during pointing tasks are characterized by a subject-specific Koenderink shape index and by a bias due to the prono-supination angle distribution. We then introduce a simple postural model using only four parameters to explain these characteristics in a pointing task. The model specifies the redundancy of the pointing task by determining the one-dimensional task-equivalent manifold (TEM), parameterized via wrist torsion. For every pointing direction, the torsion is obtained by the concurrent minimization of an extrinsic cost, which guarantees minimal angle rotations (similar to Listing's law for eye movements) and of an intrinsic cost, which penalizes wrist configurations away from comfortable postures. This allows simulating the sequence of wrist orientations to point at eight peripheral targets, from a central one, passing through intermediate points. The simulation first shows that in contrast to eye movements, which can be predicted by only considering the extrinsic cost (i.e., Listing's law), both costs are necessary to account for the wrist/forearm experimental data. Second, fitting the synthetic Donders' law from the simulated task with a quadratic surface yields similar fitting errors compared to experimental data.

A Hessian-based decomposition characterizes how performance in complex motor skills depends on individual strategy and variability.

In complex real-life motor skills such as unconstrained throwing, performance depends on how accurate is on average the outcome of noisy, high-dimensional, and redundant actions. What characteristics of the action distribution relate to performance and how different individuals select specific action distributions are key questions in motor control. Previous computational approaches have highlighted that variability along the directions of first order derivatives of the action-to-outcome mapping affects performance the most, that different mean actions may be associated to regions of the actions space with different sensitivity to noise, and that action covariation in addition to noise magnitude matters. However, a method to relate individual high-dimensional action distribution and performance is still missing. Here we introduce a decomposition of performance into a small set of indicators that compactly and directly characterize the key performance-related features of the distribution of high-dimensional redundant actions. Central to the method is the observation that, if performance is quantified as a mean score, the Hessian (second order derivatives) of the action-to-score function determines how the noise of the action distribution affects performance. We can then approximate the mean score as the sum of the score of the mean action and a tolerance-variability index which depends on both Hessian and action covariance. Such index can be expressed as the product of three terms capturing noise magnitude, noise sensitivity, and alignment of the most variable and most noise sensitive directions. We apply this method to the analysis of unconstrained throwing actions by non-expert participants and show that, consistently across four different throwing targets, each participant shows a specific selection of mean action score and tolerance-variability index as well as specific selection of noise magnitude and alignment indicators. Thus, participants with different strategies may display the same performance because they can trade off suboptimal mean action for better tolerance-variability and higher action variability for better alignment with more tolerant directions in action space.

Neurorehabilitation From a Distance: Can Intelligent Technology Support Decentralized Access to Quality Therapy?

Current neurorehabilitation models primarily rely on extended hospital stays and regular therapy sessions requiring close physical interactions between rehabilitation professionals and patients. The current COVID-19 pandemic has challenged this model, as strict physical distancing rules and a shift in the allocation of hospital resources resulted in many neurological patients not receiving essential therapy. Accordingly, a recent survey revealed that the majority of European healthcare professionals involved in stroke care are concerned that this lack of care will have a noticeable negative impact on functional outcomes. COVID-19 highlights an urgent need to rethink conventional neurorehabilitation and develop alternative approaches to provide high-quality therapy while minimizing hospital stays and visits. Technology-based solutions, such as, robotics bear high potential to enable such a paradigm shift. While robot-assisted therapy is already established in clinics, the future challenge is to enable physically assisted therapy and assessments in a minimally supervized and decentralized manner, ideally at the patient's home. Key enablers are new rehabilitation devices that are portable, scalable and equipped with clinical intelligence, remote monitoring and coaching capabilities. In this perspective article, we discuss clinical and technological requirements for the development and deployment of minimally supervized, robot-assisted neurorehabilitation technologies in patient's homes. We elaborate on key principles to ensure feasibility and acceptance, and on how artificial intelligence can be leveraged for embedding clinical knowledge for safe use and personalized therapy adaptation. Such new models are likely to impact neurorehabilitation beyond COVID-19, by providing broad access to sustained, high-quality and high-dose therapy maximizing long-term functional outcomes.

Robotic Assisted Upper Limb Training Post Stroke: A Randomized Control Trial Using Combinatory Approach Toward Reducing Workforce Demands.

Post stroke upper limb rehabilitation is a challenging problem with poor outcomes as 40% of survivors have functionally useless upper limbs. Robot-aided therapy (RAT) is a potential method to alleviate the effort of intensive, task-specific, repetitive upper limb exercises for both patients and therapists. The present study aims to investigate how a time matched combinatory training scheme that incorporates conventional and RAT, using H-Man, compares with conventional training toward reducing workforce demands. In a randomized control trial (NCT02188628, www.clinicaltrials.gov), 44 subacute to chronic stroke survivors with first-ever clinical stroke and predominant arm motor function deficits were recruited and randomized into two groups of 22 subjects: Robotic Therapy (RT) and Conventional Therapy (CT). Both groups received 18 sessions of 90 min; three sessions per week over 6 weeks. In each session, participants of the CT group received 90 min of 1:1 therapist-supervised conventional therapy while participants of the RT group underwent combinatory training which consisted of 60 min of minimally-supervised H-Man therapy followed by 30 min of conventional therapy. The clinical outcomes [Fugl-Meyer (FMA), Action Research Arm Test and, Grip Strength] and the quantitative measures (smoothness, time efficiency, and task error, derived from two robotic assessment tasks) were independently evaluated prior to therapy intervention (week 0), at mid-training (week 3), at the end of training (week 6), and post therapy (week 12 and 24). Significant differences within group were observed at the end of training for all clinical scales compared with baseline [mean and standard deviation of FMA score changes between baseline and week 6; RT: Δ4.41 (3.46) and CT: Δ3.0 (4.0); p < 0.01]. FMA gains were retained 18 weeks post-training [week 24; RT: Δ5.38 (4.67) and week 24 CT: Δ4.50 (5.35); p < 0.01]. The RT group clinical scores improved similarly when compared to CT group with no significant inter-group at all time points although the conventional therapy time was reduced to one third in RT group. There were no training-related adverse side effects. In conclusion, time matched combinatory training incorporating H-Man RAT produced similar outcomes compared to conventional therapy alone. Hence, this study supports a combinatory approach to improve motor function in post-stroke arm paresis. Clinical Trial Registration: www.ClinicalTrials.gov, identifier: NCT02188628.