Activity, Plan, and Goal Recognition: A Review.

Recognizing the actions, plans, and goals of a person in an unconstrained environment is a key feature that future robotic systems will need in order to achieve a natural human-machine interaction. Indeed, we humans are constantly understanding and predicting the actions and goals of others, which allows us to interact in intuitive and safe ways. While action and plan recognition are tasks that humans perform naturally and with little effort, they are still an unresolved problem from the point of view of artificial intelligence. The immense variety of possible actions and plans that may be encountered in an unconstrained environment makes current approaches be far from human-like performance. In addition, while very different types of algorithms have been proposed to tackle the problem of activity, plan, and goal (intention) recognition, these tend to focus in only one part of the problem (e.g., action recognition), and techniques that address the problem as a whole have been not so thoroughly explored. This review is meant to provide a general view of the problem of activity, plan, and goal recognition as a whole. It presents a description of the problem, both from the human perspective and from the computational perspective, and proposes a classification of the main types of approaches that have been proposed to address it (logic-based, classical machine learning, deep learning, and brain-inspired), together with a description and comparison of the classes. This general view of the problem can help on the identification of research gaps, and may also provide inspiration for the development of new approaches that address the problem in a unified way.

Evaluating the sit-to-stand transfer assistance from a smart walker in older adults with motor impairments.

AIM: To evaluate the effectiveness and user satisfaction with the sit-to-stand (STS) assistance system of a smart walker (SW), and to identify factors associated with them in potential users. METHODS: A total of 33 older adults (29 women, aged ≥65 years) with motor impairments (habitual rollator use) and no severe cognitive impairment (Mini-Mental State Examination ≥17 points) carried out a Five-Chair Stand Test without assistance and five STS transfers with the STS assistance system. Based on the number of successfully completed STS transfers, success rates were calculated for the Five-Chair Stand Test and the SW-assisted STS transfers, and compared using the Wilcoxon signed-rank test. User satisfaction was assessed using the Tele-healthcare Satisfaction Questionnaire-Wearable Technology (0-80 points, higher score = higher satisfaction). Bivariate correlations and multiple linear regression analyses were used to identify participant characteristics associated with the success rate and user satisfaction with the STS assistance system. RESULTS: The success rate for the SW-assisted STS transfers was significantly higher than for the Five-Chair Stand Test (93.3 ± 12.9% vs 54.5 ± 50.6%, P < 0.001). User satisfaction was high (Tele-healthcare Satisfaction Questionnaire-Wearable Technology 62.5 ± 11.2 points). The success rate with the STS assistance system was not significantly associated with any participant characteristics. Higher body mass index was a significant independent predictor of higher user satisfaction. CONCLUSIONS: The SW-integrated STS assistance system can provide effective STS support with high user satisfaction for a wide range of potential users. Our findings suggest the high potential of the STS assistance system for promoting mobility, independence and quality of life for older adults with motor impairments. Geriatr Gerontol Int 2020; 20: 312-316.

A simulation environment for studying transcutaneous electrotactile stimulation.

Transcutaneous electrical nerve stimulation (TENS) allows the artificial excitation of nerve fibres by applying electric-current pulses through electrodes on the skin's surface. This work involves the development of a simulation environment that can be used for studying transcutaneous electrotactile stimulation and its dependence on electrode layout and excitation patterns. Using an eight-electrode array implementation, it is shown how nerves located at different depths and with different orientations respond to specific injected currents, allowing the replication of already reported experimental findings and the creation of new hypotheses about the tactile sensations associated with certain stimulation patterns. The simulation consists of a finite element model of a human finger used to calculate the distribution of the electric potential in the finger tissues neglecting capacitive effects, and a cable model to calculate the excitation/inhibition of action potentials in each nerve.

A systematic review of study results reported for the evaluation of robotic rollators from the perspective of users.

PURPOSE: To evaluate the effectiveness and perception of robotic rollators (RRs) from the perspective of users. METHODS: Studies identified in a previous systematic review published on 2016 on the methodology of studies evaluating RRs by the user perspective were re-screened for eligibility based on the following inclusion criteria: evaluation of the human-robot interaction from the user perspective, use of standardized outcome measurements, and quantitative presentation of study results. RESULTS: Seventeen studies were eligible for inclusion. Due to the clinical and methodological heterogeneity across studies, a narrative synthesis of study results was conducted. We found conflicting results concerning the effectiveness of the robotic functionalities of the RRs. Only a few studies reported superior user performance or reduced physical demands with the RRs compared to unassisted conditions or conventional assistive mobility devices; however, without providing statistical evidence. The user perception of the RRs was found to be generally positive. CONCLUSIONS: There is still no sufficient evidence on the effectiveness of RRs from the user perspective. More well-designed, high-quality studies with adequate study populations, larger sample sizes, appropriate assessment strategies with outcomes specifically tailored to the robotic functionalities, and statistical analyses of results are required to evaluate RRs at a higher level of evidence. Implications for Rehabilitation RRs cover intelligent functionalities that focus on gait assistance, obstacle avoidance, navigation assistance, sit-to-stand transfer, body weight support or fall prevention. The evaluation from the user perspective is essential to ensure that RRs effectively address users' needs, requirements and preferences. The evidence on the effectiveness of RRs is severely hampered by the low methodological quality of most of the available studies. RRs seem generally to be perceived as positive by the users. There is very limited evidence on the effectiveness and benefits of RRs compared to conventional assistive mobility devices. Further research with high methodological quality needs to be conducted to reach more robust conclusions about the effectiveness of RRs.

Invariance and variability in interaction error-related potentials and their consequences for classification.

OBJECTIVE: This paper discusses the invariance and variability in interaction error-related potentials (ErrPs), where a special focus is laid upon the factors of (1) the human mental processing required to assess interface actions (2) time (3) subjects. APPROACH: Three different experiments were designed as to vary primarily with respect to the mental processes that are necessary to assess whether an interface error has occurred or not. The three experiments were carried out with 11 subjects in a repeated-measures experimental design. To study the effect of time, a subset of the recruited subjects additionally performed the same experiments on different days. MAIN RESULTS: The ErrP variability across the different experiments for the same subjects was found largely attributable to the different mental processing required to assess interface actions. Nonetheless, we found that interaction ErrPs are empirically invariant over time (for the same subject and same interface) and to a lesser extent across subjects (for the same interface). SIGNIFICANCE: The obtained results may be used to explain across-study variability of ErrPs, as well as to define guidelines for approaches to the ErrP classifier transferability problem.

Goal-recognition-based adaptive brain-computer interface for navigating immersive robotic systems.

OBJECTIVE: This work proposes principled strategies for self-adaptations in EEG-based Brain-computer interfaces (BCIs) as a way out of the bandwidth bottleneck resulting from the considerable mismatch between the low-bandwidth interface and the bandwidth-hungry application, and a way to enable fluent and intuitive interaction in embodiment systems. The main focus is laid upon inferring the hidden target goals of users while navigating in a remote environment as a basis for possible adaptations. APPROACH: To reason about possible user goals, a general user-agnostic Bayesian update rule is devised to be recursively applied upon the arrival of evidences, i.e. user input and user gaze. Experiments were conducted with healthy subjects within robotic embodiment settings to evaluate the proposed method. These experiments varied along three factors: the type of the robot/environment (simulated and physical), the type of the interface (keyboard or BCI), and the way goal recognition (GR) is used to guide a simple shared control (SC) driving scheme. MAIN RESULTS: Our results show that the proposed GR algorithm is able to track and infer the hidden user goals with relatively high precision and recall. Further, the realized SC driving scheme benefits from the output of the GR system and is able to reduce the user effort needed to accomplish the assigned tasks. Despite the fact that the BCI requires higher effort compared to the keyboard conditions, most subjects were able to complete the assigned tasks, and the proposed GR system is additionally shown able to handle the uncertainty in user input during SSVEP-based interaction. The SC application of the belief vector indicates that the benefits of the GR module are more pronounced for BCIs, compared to the keyboard interface. SIGNIFICANCE: Being based on intuitive heuristics that model the behavior of the general population during the execution of navigation tasks, the proposed GR method can be used without prior tuning for the individual users. The proposed methods can be easily integrated in devising more advanced SC schemes and/or strategies for automatic BCI self-adaptations.

Decision-Making Model for Adaptive Impedance Control of Teleoperation Systems.

This paper presents a haptic assistance strategy for teleoperation that makes a task and situation-specific compromise between improving tracking performance or human-machine interaction in partially structured environments via the scheduling of the parameters of an admittance controller. The proposed assistance strategy builds on decision-making models and combines one of them with impedance control techniques that are standard in bilateral teleoperation systems. Even though several decision-making models have been proposed in cognitive science, their application to assisted teleoperation and assisted robotics has hardly been explored yet. Experimental data supports the Drift-Diffusion model as a suitable scheduling strategy for haptic shared control, in which the assistance mechanism can be adapted via the parameters of reward functions. Guidelines to tune the decision making model are presented. The influence of the reward structure on the realized haptic assistances is evaluated in a user study and results are compared to the no assistance and human assistance case.

Local and Remote Cooperation With Virtual and Robotic Agents: A P300 BCI Study in Healthy and People Living With Spinal Cord Injury.

The development of technological applications that allow people to control and embody external devices within social interaction settings represents a major goal for current and future brain-computer interface (BCI) systems. Prior research has suggested that embodied systems may ameliorate BCI end-user's experience and accuracy in controlling external devices. Along these lines, we developed an immersive P300-based BCI application with a head-mounted display for virtual-local and robotic-remote social interactions and explored in a group of healthy participants the role of proprioceptive feedback in the control of a virtual surrogate (Study 1). Moreover, we compared the performance of a small group of people with spinal cord injury (SCI) to a control group of healthy subjects during virtual and robotic social interactions (Study 2), where both groups received a proprioceptive stimulation. Our attempt to combine immersive environments, BCI technologies and neuroscience of body ownership suggests that providing realistic multisensory feedback still represents a challenge. Results have shown that healthy and people living with SCI used the BCI within the immersive scenarios with good levels of performance (as indexed by task accuracy, optimizations calls and Information Transfer Rate) and perceived control of the surrogates. Proprioceptive feedback did not contribute to alter performance measures and body ownership sensations. Further studies are necessary to test whether sensorimotor experience represents an opportunity to improve the use of future embodied BCI applications.

Advancing the detection of steady-state visual evoked potentials in brain-computer interfaces.

OBJECTIVE: Spatial filtering has proved to be a powerful pre-processing step in detection of steady-state visual evoked potentials and boosted typical detection rates both in offline analysis and online SSVEP-based brain-computer interface applications. State-of-the-art detection methods and the spatial filters used thereby share many common foundations as they all build upon the second order statistics of the acquired Electroencephalographic (EEG) data, that is, its spatial autocovariance and cross-covariance with what is assumed to be a pure SSVEP response. The present study aims at highlighting the similarities and differences between these methods. APPROACH: We consider the canonical correlation analysis (CCA) method as a basis for the theoretical and empirical (with real EEG data) analysis of the state-of-the-art detection methods and the spatial filters used thereby. We build upon the findings of this analysis and prior research and propose a new detection method (CVARS) that combines the power of the canonical variates and that of the autoregressive spectral analysis in estimating the signal and noise power levels. MAIN RESULTS: We found that the multivariate synchronization index method and the maximum contrast combination method are variations of the CCA method. All three methods were found to provide relatively unreliable detections in low signal-to-noise ratio (SNR) regimes. CVARS and the minimum energy combination methods were found to provide better estimates for different SNR levels. SIGNIFICANCE: Our theoretical and empirical results demonstrate that the proposed CVARS method outperforms other state-of-the-art detection methods when used in an unsupervised fashion. Furthermore, when used in a supervised fashion, a linear classifier learned from a short training session is able to estimate the hidden user intention, including the idle state (when the user is not attending to any stimulus), rapidly, accurately and reliably.

Evaluation Studies of Robotic Rollators by the User Perspective: A Systematic Review.

BACKGROUND: Robotic rollators enhance the basic functions of established devices by technically advanced physical, cognitive, or sensory support to increase autonomy in persons with severe impairment. In the evaluation of such ambient assisted living solutions, both the technical and user perspectives are important to prove usability, effectiveness and safety, and to ensure adequate device application. OBJECTIVE: The aim of this systematic review is to summarize the methodology of studies evaluating robotic rollators with focus on the user perspective and to give recommendations for future evaluation studies. METHODS: A systematic literature search up to December 31, 2014, was conducted based on the Cochrane Review methodology using the electronic databases PubMed and IEEE Xplore. Articles were selected according to the following inclusion criteria: evaluation studies of robotic rollators documenting human-robot interaction, no case reports, published in English language. RESULTS: Twenty-eight studies were identified that met the predefined inclusion criteria. Large heterogeneity in the definitions of the target user group, study populations, study designs and assessment methods was found across the included studies. No generic methodology to evaluate robotic rollators could be identified. We found major methodological shortcomings related to insufficient sample descriptions and sample sizes, and lack of appropriate, standardized and validated assessment methods. Long-term use in habitual environment was also not evaluated. CONCLUSIONS: Apart from the heterogeneity, methodological deficits in most of the identified studies became apparent. Recommendations for future evaluation studies include: clear definition of target user group, adequate selection of subjects, inclusion of other assistive mobility devices for comparison, evaluation of the habitual use of advanced prototypes, adequate assessment strategy with established, standardized and validated methods, and statistical analysis of study results. Assessment strategies may additionally focus on specific functionalities of the robotic rollators allowing an individually tailored assessment of innovative features to document their added value.

A key region in the human parietal cortex for processing proprioceptive hand feedback during reaching movements.

Seemingly effortless, we adjust our movements to continuously changing environments. After initiation of a goal-directed movement, the motor command is under constant control of sensory feedback loops. The main sensory signals contributing to movement control are vision and proprioception. Recent neuroimaging studies have focused mainly on identifying the parts of the posterior parietal cortex (PPC) that contribute to visually guided movements. We used event-related TMS and force perturbations of the reaching hand to test whether the same sub-regions of the left PPC contribute to the processing of proprioceptive-only and of multi-sensory information about hand position when reaching for a visual target. TMS over two distinct stimulation sites elicited differential effects: TMS applied over the posterior part of the medial intraparietal sulcus (mIPS) compromised reaching accuracy when proprioception was the only sensory information available for correcting the reaching error. When visual feedback of the hand was available, TMS over the anterior intraparietal sulcus (aIPS) prolonged reaching time. Our results show for the first time the causal involvement of the posterior mIPS in processing proprioceptive feedback for online reaching control, and demonstrate that distinct cortical areas process proprioceptive-only and multi-sensory information for fast feedback corrections.

The role of haptic feedback for the integration of intentions in shared task execution.

Recent developments strive for realizing robotic systems that not only interact, but closely collaborate with humans in performing everyday manipulation tasks. Successful collaboration requires the integration of the individual partner's intentions into a shared action plan, which may involve continuous negotiation of intentions. We focus on collaboration in a kinesthetic task, i.e., joint object manipulation. Here, ways must be found to integrate individual motion and force inputs from the members of the human-robot team, in order to achieve the joint task goal. Before guidelines on how robots should act in this process can be formulated, clarification on whether humans use the haptic channel for communicating their intentions is needed. This paper investigates this question in an experimental setup involving two collaborating humans. We consider physical effort as well as performance as indicators of successful intention integration. Our results strongly suggest that intention integration is enhanced via the haptic channel, i.e., that haptic communication takes place, especially in the case of shared decision situations. This provides a motivation for future investigations to model the process of intention integration itself in order to realize successful haptic human-robot collaboration.

Exploring the design space of haptic assistants: the assistance policy module.

Haptic assistants augment user commands to facilitate manipulation and to increase task performance. The strength of assistance, also referred to as assistance level, is one of the main design factors. While existing implementations mainly realize fixed assistance levels that are selected with respect to one design objective, we introduce an assistance policy module that dynamically changes assistance levels and can incorporate multiple performance measures. The design space of this assistance policy module is systematically analyzed and three design factors, 1) performance criteria, 2) performance reference, and 3) assistance policy, are identified. Different implementations of the assistance policy module are compared for a scenario involving guiding virtual fixtures. A single-user evaluation is used to illustrate the effect of the different implementations on the determined assistance levels, and a multi-user study allows for a statistical comparison of them. Results show that adaptive assistance policies can outperform constant assistance policies, switching assistance policies have advantages over continuously adapting policies, a multi-criteria performance measure should be favored if there is no single criterion that has an outstanding priority, and the selection of the performance reference is highly application dependent.

Beaming: an asymmetric telepresence system.

The Beaming project recreates, virtually, a real environment; using immersive VR, remote participants can visit the virtual model and interact with the people in the real environment. The real environment doesn't need extensive equipment and can be a space such as an office or meeting room, domestic environment, or social space.

Contributions of the PPC to online control of visually guided reaching movements assessed with fMRI-guided TMS.

The posterior parietal cortex (PPC) plays an important role in controlling voluntary movements by continuously integrating sensory information about body state and the environment. We tested which subregions of the PPC contribute to the processing of target- and body-related visual information while reaching for an object, using a reaching paradigm with 2 types of visual perturbation: displacement of the visual target and displacement of the visual feedback about the hand position. Initially, functional magnetic resonance imaging (fMRI) was used to localize putative target areas involved in online corrections of movements in response to perturbations. The causal contribution of these areas to online correction was tested in subsequent neuronavigated transcranial magnetic stimulation (TMS) experiments. Robust TMS effects occurred at distinct anatomical sites along the anterior intraparietal sulcus (aIPS) and the anterior part of the supramarginal gyrus for both perturbations. TMS over neighboring sites did not affect online control. Our results support the hypothesis that the aIPS is more generally involved in visually guided control of movements, independent of body effectors and nature of the visual information. Furthermore, they suggest that the human network of PPC subregions controlling goal-directed visuomotor processes extends more inferiorly than previously thought. Our results also point toward a good spatial specificity of the TMS effects.

Comparison of people's responses to real and virtual handshakes within a virtual environment.

In this paper we present a method for evaluating a haptic device which simulates human handshakes interfaced via a metal rod. We provide an overview of the haptic demonstrator and the control algorithm used for delivering realistic handshakes. For the evaluation of this handshake demonstrator we introduce a 'ground truth' approach, where we compare the robot handshakes with handshakes operated by a human via the same metal rod. For this, an experiment was carried out where the participants entered a virtual environment, i.e. a virtual cocktail party, and were asked to perform a number of handshakes, either with the robot operating with one of two control algorithms operating the metal rod - a basic one for comparison or the proposed new advanced one, or with a human operating the metal rod. The virtual environment was represented only through audio and haptics, without any visual representation, i.e. the subjects participated blindfolded. The evaluation of each handshake was achieved through the subjective scoring of each of the handshakes. The results of the study show that the demonstrator operating with the proposed new control scheme was evaluated significantly more human-like than with the demonstrator operating with the basic algorithm, and also that the real human handshake was evaluated more like a real human handshake than both types of robot handshakes. Although the difference between the advanced robot and human handshake was significant, the effect sizes are not very different, indicating substantial confusion of participants between the advanced robot and human operated handshakes.

Plugfest 2009: Global Interoperability in Telerobotics and Telemedicine.

Despite the great diversity of teleoperator designs and applications, their underlying control systems have many similarities. These similarities can be exploited to enable inter-operability between heterogeneous systems. We have developed a network data specification, the Interoperable Telerobotics Protocol, that can be used for Internet based control of a wide range of teleoperators. In this work we test interoperable telerobotics on the global Internet, focusing on the telesurgery application domain. Fourteen globally dispersed telerobotic master and slave systems were connected in thirty trials in one twenty four hour period. Users performed common manipulation tasks to demonstrate effective master-slave operation. With twenty eight (93%) successful, unique connections the results show a high potential for standardizing telerobotic operation. Furthermore, new paradigms for telesurgical operation and training are presented, including a networked surgery trainer and upper-limb exoskeleton control of micro-manipulators.

Seeing the hand while reaching speeds up on-line responses to a sudden change in target position.

Goal-directed movements are executed under the permanent supervision of the central nervous system, which continuously processes sensory afferents and triggers on-line corrections if movement accuracy seems to be compromised. For arm reaching movements, visual information about the hand plays an important role in this supervision, notably improving reaching accuracy. Here, we tested whether visual feedback of the hand affects the latency of on-line responses to an external perturbation when reaching for a visual target. Two types of perturbation were used: visual perturbation consisted in changing the spatial location of the target and kinesthetic perturbation in applying a force step to the reaching arm. For both types of perturbation, the hand trajectory and the electromyographic (EMG) activity of shoulder muscles were analysed to assess whether visual feedback of the hand speeds up on-line corrections. Without visual feedback of the hand, on-line responses to visual perturbation exhibited the longest latency. This latency was reduced by about 10% when visual feedback of the hand was provided. On the other hand, the latency of on-line responses to kinesthetic perturbation was independent of the availability of visual feedback of the hand. In a control experiment, we tested the effect of visual feedback of the hand on visual and kinesthetic two-choice reaction times--for which coordinate transformation is not critical. Two-choice reaction times were never facilitated by visual feedback of the hand. Taken together, our results suggest that visual feedback of the hand speeds up on-line corrections when the position of the visual target with respect to the body must be re-computed during movement execution. This facilitation probably results from the possibility to map hand- and target-related information in a common visual reference frame.