Pivotal Response Treatment with and without robot-assistance for children with autism: a randomized controlled trial.

Pivotal response treatment (PRT) is a promising intervention focused on improving social communication skills in children with autism spectrum disorder (ASD). Since robots potentially appeal to children with ASD and may contribute to their motivation for social interaction, this exploratory randomized controlled trial (RCT) was conducted comparing PRT (PRT and robot-assisted PRT) with treatment-as-usual (TAU). Seventy-three children (PRT: n = 25; PRT + robot: n = 25; TAU: n = 23) with ASD, aged 3-8 years were assessed at baseline, after 10 and 20 weeks of intervention, and at 3-month follow-up. There were no significant group differences on parent- and teacher-rated general social-communicative skills and blindly rated global functioning directly after treatment. However, at follow-up largest gains were observed in robot-assisted PRT compared to other groups. These results suggest that robot-assistance may contribute to intervention efficacy for children with ASD when using game scenarios for robot-child interaction during multiple sessions combined with motivational components of PRT. This trial is registered at https://www.trialregister.nl/trial/4487 ; NL4487/NTR4712 (2014-08-01).

Robot-mediated therapy to reduce worrying in persons with visual and intellectual disabilities.

BACKGROUND: The study explored the use of a robot-mediated therapeutic intervention in persons with visual and intellectual disabilities. METHOD: Three robot-mediated intervention sessions were developed to teach three coping skills for worrying. Effectiveness was examined using a multiple-baseline case study design (N = 7). Baseline, pre-intervention and post-intervention assessments included social validity, severity of worrying (PSWQ-C-NL), and observations by caregivers (SDQ). Short checklists on worrying were repeated throughout baseline and intervention stages. Transcripts of the sessions were analysed for participants' emotional openness. RESULTS: Social validity was equally high before and after the intervention. The intervention did not impact the severity of worrying, although mentor caregivers reported a lower impact of personal difficulties for participants. We found no change in self-disclosure towards the robot over sessions. CONCLUSIONS: The participants' positive responses warrant further exploration of using robot-mediated therapy for persons with visual and intellectual disabilities. Recommendations for additional adaptations are discussed.

Crowd of Oz: A Crowd-Powered Social Robotics System for Stress Management.

Coping with stress is crucial for a healthy lifestyle. In the past, a great deal of research has been conducted to use socially assistive robots as a therapy to alleviate stress and anxiety related problems. However, building a fully autonomous social robot which can deliver psycho-therapeutic solutions is a very challenging endeavor due to limitations in artificial intelligence (AI). To overcome AI's limitations, researchers have previously introduced crowdsourcing-based teleoperation methods, which summon the crowd's input to control a robot's functions. However, in the context of robotics, such methods have only been used to support the object manipulation, navigational, and training tasks. It is not yet known how to leverage real-time crowdsourcing (RTC) to process complex therapeutic conversational tasks for social robotics. To fill this gap, we developed Crowd of Oz (CoZ), an open-source system that allows Softbank's Pepper robot to support such conversational tasks. To demonstrate the potential implications of this crowd-powered approach, we investigated how effectively, crowd workers recruited in real-time can teleoperate the robot's speech, in situations when the robot needs to act as a life coach. We systematically varied the number of workers who simultaneously handle the speech of the robot (N = 1, 2, 4, 8) and investigated the concomitant effects for enabling RTC for social robotics. Additionally, we present Pavilion, a novel and open-source algorithm for managing the workers' queue so that a required number of workers are engaged or waiting. Based on our findings, we discuss salient parameters that such crowd-powered systems must adhere to, so as to enhance their performance in response latency and dialogue quality.

Effects of the Level of Interactivity of a Social Robot and the Response of the Augmented Reality Display in Contextual Interactions of People with Dementia.

The well-being of people with dementia (PWD) living in long-term care facilities is hindered due to disengagement and social isolation. Animal-like social robots are increasingly used in dementia care as they can provide companionship and engage PWD in meaningful activities. While most previous human-robot interaction (HRI) research studied engagement independent from the context, recent findings indicate that the context of HRI sessions has an impact on user engagement. This study aims to explore the effects of contextual interactions between PWD and a social robot embedded in the augmented responsive environment. Three experimental conditions were compared: reactive context-enhanced robot interaction; dynamic context-enhanced interaction with a static robot; a control condition with only the dynamic context presented. Effectiveness evaluations were performed with 16 participants using four observational rating scales on observed engagement, affective states, and apathy related behaviors. Findings suggested that the higher level of interactivity of a social robot and the interactive contextualized feedback helped capture and maintain users' attention during engagement; however, it did not significantly improve their positive affective states. Additionally, the presence of either a static or a proactive robot reduced apathy-related behaviors by facilitating purposeful activities, thus, motivating behavioral engagement.

The effects of a bioresponse system on the joint attention behaviour of adults with visual and severe or profound intellectual disabilities and their affective mutuality with their caregivers.

BACKGROUND: The subtle communicative behaviour of individuals with visual and severe/profound intellectual disabilities hinders the success of their interaction with professional caregivers. The bioresponse system, a tool to raise caregivers' awareness of the client's communicative behaviour, may improve the client's joint attention behaviour and the dyad's affective mutuality. METHOD: Four client-caregiver dyads participated in a randomized multiple baseline study with repeated baseline, intervention and follow-up observations. The bioresponse system's effect was evaluated with measures of joint attention and affective mutuality. RESULTS: Two clients showed a significant difference on one or two joint attention subscales (including one significant decrease), and for all clients, at least one joint attention subscale revealed a positive trend. Positive trends in affective mutuality scores were observed in two dyads. CONCLUSIONS: The results stress the importance of further research to the effects of using the bioresponse system's in daily care for persons with severe/profound intellectual disabilities.

Development of an AI-Enabled System for Pain Monitoring Using Skin Conductance Sensoring in Socks.

Background: Where self-report is unfeasible or observations are difficult, physiological estimates of pain are needed. Methods: Pain-data from 30 healthy adults were gathered to create a database of physiological pain responses. A model was then developed, to analyze pain-data and visualize the AI-estimated level of pain on a mobile app. Results: The initial low precision and F1-score of the pain classification algorithm were resolved by interpolating a percentage of similar data. Discussion: This system presents a novel approach to assess pain in noncommunicative people with the use of a sensor sock, AI predictor and mobile app. Performance analysis and the limitations of the AI algorithm are discussed.

Personalizing HRI in Musical Instrument Practicing: The Influence of Robot Roles (Evaluative Versus Nonevaluative) on the Child's Motivation for Children in Different Learning Stages.

Learning to play a musical instrument involves skill learning and requires long-term practicing to reach expert levels. Research has already proven that the assistance of a robot can improve children's motivation and performance during practice. In an earlier study, we showed that the specific role (evaluative role versus nonevaluative role) the robot plays can determine children's motivation and performance. In the current study, we argue that the role of the robot has to be different for children in different learning stages (musical instrument expertise levels). Therefore, this study investigated whether children in different learning stages would have higher motivation when assisted by a robot in different supporting roles (i.e., evaluative role versus nonevaluative role). We conducted an empirical study in a real practice room of a music school with 31 children who were at different learning stages (i.e., beginners, developing players, and advanced players). In this study, every child practiced for three sessions: practicing alone, assisted by the evaluative robot, or assisted by the nonevaluative robot (in a random order). We measured motivation by using a questionnaire and analyzing video data. Results showed a significant interaction between condition (i.e., alone, evaluative robot, and nonevaluative robot) and learning stage groups indicating that children in different learning stage groups had different levels of motivation when practicing alone or with an evaluative or nonevaluative robot. More specifically, beginners had higher persistence when practicing with the nonevaluative robot, while advanced players expressed higher motivation after practicing with a robot than alone, but no difference was found between the two robot roles. Exploratory results also indicated that gender might have an interaction effect with the robot roles on child's motivation in music practice with social robots. This study offers more insight into the child-robot interaction and robot role design in musical instrument learning. Specifically, our findings shed light on personalization in HRI, that is, from adapting the role of the robot to the characteristics and the development level of the user.

Identifying Interaction Patterns of Tangible Co-Adaptations in Human-Robot Team Behaviors.

As robots become more ubiquitous, they will increasingly need to behave as our team partners and smoothly adapt to the (adaptive) human team behaviors to establish successful patterns of collaboration over time. A substantial amount of adaptations present themselves through subtle and unconscious interactions, which are difficult to observe. Our research aims to bring about awareness of co-adaptation that enables team learning. This paper presents an experimental paradigm that uses a physical human-robot collaborative task environment to explore emergent human-robot co-adaptions and derive the interaction patterns (i.e., the targeted awareness of co-adaptation). The paradigm provides a tangible human-robot interaction (i.e., a leash) that facilitates the expression of unconscious adaptations, such as "leading" (e.g., pulling the leash) and "following" (e.g., letting go of the leash) in a search-and-navigation task. The task was executed by 18 participants, after which we systematically annotated videos of their behavior. We discovered that their interactions could be described by four types of adaptive interactions: stable situations, sudden adaptations, gradual adaptations and active negotiations. From these types of interactions we have created a language of interaction patterns that can be used to describe tacit co-adaptation in human-robot collaborative contexts. This language can be used to enable communication between collaborating humans and robots in future studies, to let them share what they learned and support them in becoming aware of their implicit adaptations.

You Were Always on My Mind: Introducing Chef's Hat and COPPER for Personalized Reinforcement Learning.

Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef's Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent's actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef's Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two "dummy" baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef's Hat competitive game and the implementation of the Chef's Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning.

Self-initiations in young children with autism during Pivotal Response Treatment with and without robot assistance.

LAY ABSTRACT: The initiation of social interaction is often defined as a core deficit of autism spectrum disorder. Optimizing these self-initiations is therefore a key component of Pivotal Response Treatment, an established intervention for children with autism spectrum disorder. However, little is known about the development of self-initiations during intervention and whether this development can be facilitated by robot assistance within Pivotal Response Treatment. The aim of this study was to (1) investigate the effect of Pivotal Response Treatment and robot-assisted Pivotal Response Treatment on self-initiations (functional and social) of young children with autism spectrum disorder over the course of intervention and (2) explore the relation between development in self-initiations and additional gains in general social-communicative skills. Forty-four children with autism spectrum disorder (aged 3-8 years) were included in this study. Self-initiations were assessed during parent-child interaction videos of therapy sessions and coded by raters who did not know which treatment (Pivotal Response Treatment or robot-assisted Pivotal Response Treatment) the child received. General social-communicative skills were assessed before start of the treatment, after 10 and 20 weeks of intervention and 3 months after the treatment was finalized. Results showed that self-initiations increased in both treatment groups, with the largest improvements in functional self-initiations in the group that received robot-assisted Pivotal Response Treatment. Increased self-initiations were related to higher parent-rated social awareness 3 months after finalizing the treatment.

Adherence and acceptability of a robot-assisted Pivotal Response Treatment protocol for children with autism spectrum disorder.

The aim of this study is to present a robot-assisted therapy protocol for children with ASD based on the current state-of-the-art in both ASD intervention research and robotics research, and critically evaluate its adherence and acceptability based on child as well as parent ratings. The robot-assisted therapy was designed based on motivational components of Pivotal Response Treatment (PRT), a highly promising and feasible intervention focused at training "pivotal" (key) areas such as motivation for social interaction and self-initiations, with the goal of establishing collateral gains in untargeted areas of functioning and development, affected by autism spectrum disorders. Overall, children (3-8 y) could adhere to the robot-assisted therapy protocol (Mean percentage of treatment adherence 85.5%), showed positive affect ratings after therapy sessions (positive in 86.6% of sessions) and high robot likability scores (high in 79.4% of sessions). Positive likability ratings were mainly given by school-aged children (H(1) = 7.91, p = .005) and related to the movements, speech and game scenarios of the robot. Parent ratings on the added value of the robot were mainly positive (Mean of 84.8 on 0-100 scale), while lower parent ratings were related to inflexibility of robot behaviour.

Simulated trust: a cheap social learning strategy.

Animals use heuristic strategies to determine from which conspecifics to learn socially. This leads to directed social learning. Directed social learning protects them from copying non-adaptive information. So far, the strategies of animals, leading to directed social learning, are assumed to rely on (possibly indirect) inferences about the demonstrator's success. As an alternative to this assumption, we propose a strategy that only uses self-established estimates of the pay-offs of behavior. We evaluate the strategy in a number of agent-based simulations. Critically, the strategy's success is warranted by the inclusion of an incremental learning mechanism. Our findings point out new theoretical opportunities to regulate social learning for animals. More broadly, our simulations emphasize the need to include a realistic learning mechanism in game-theoretic studies of social learning strategies, and call for re-evaluation of previous findings.

Social training of autistic children with interactive intelligent agents.

The ability of autistic children to learn by applying logical rules has been used widely in behavioral therapies for social training. We propose to teach social skills to autistic children through games that simultaneously stimulate social behavior and include recognition of elements of social interaction. For this purpose we created a multi-agent platform of interactive blocks, and we created appropriate games that require shared activities leading to a common goal. The games included perceiving and understanding elements of social behavior that non-autistic children can recognize. We argue that the importance of elements of social interaction such as perceiving interaction behaviors and assigning metaphoric meanings has been overlooked, and that they are very important in the social training of autistic children. Two games were compared by testing them with users. The first game focused only on the interaction between the agents and the other combined interaction between the agents and metaphoric meanings that are assigned to them. The results show that most of the children recognized the patterns of interaction as well as the metaphors when they were demonstrated through embodied agents and were included within games having features that engage the interest of this user group. The results also show the potential of the platform and the games to influence the social behavior of the children positively.

Effects of Robot Facial Characteristics and Gender in Persuasive Human-Robot Interaction.

The growing interest in social robotics makes it relevant to examine the potential of robots as persuasive agents and, more specifically, to examine how robot characteristics influence the way people experience such interactions and comply with the persuasive attempts by robots. The purpose of this research is to identify how the (ostensible) gender and the facial characteristics of a robot influence the extent to which people trust it and the psychological reactance they experience from its persuasive attempts. This paper reports a laboratory study where SociBot™, a robot capable of displaying different faces and dynamic social cues, delivered persuasive messages to participants while playing a game. In-game choice behavior was logged, and trust and reactance toward the advisor were measured using questionnaires. Results show that a robotic advisor with upturned eyebrows and lips (features that people tend to trust more in humans) is more persuasive, evokes more trust, and less psychological reactance compared to one displaying eyebrows pointing down and lips curled downwards at the edges (facial characteristics typically not trusted in humans). Gender of the robot did not affect trust, but participants experienced higher psychological reactance when interacting with a robot of the opposite gender. Remarkably, mediation analysis showed that liking of the robot fully mediates the influence of facial characteristics on trusting beliefs and psychological reactance. Also, psychological reactance was a strong and reliable predictor of trusting beliefs but not of trusting behavior. These results suggest robots that are intended to influence human behavior should be designed to have facial characteristics we trust in humans and could be personalized to have the same gender as the user. Furthermore, personalization and adaptation techniques designed to make people like the robot more may help ensure they will also trust the robot.

Directing Attention Through Gaze Hints Improves Task Solving in Human-Humanoid Interaction.

In this paper, we report an experimental study designed to examine how participants perceive and interpret social hints from gaze exhibited by either a robot or a human tutor when carrying out a matching task. The underlying notion is that knowing where an agent is looking at provides cues that can direct attention to an object of interest during the activity. In this regard, we asked human participants to play a card matching game in the presence of either a human or a robotic tutor under two conditions. In one case, the tutor gave hints to help the participant find the matching cards by gazing toward the correct match, in the other case, the tutor only looked at the participants and did not give them any help. The performance was measured based on the time and the number of tries taken to complete the game. Results show that gaze hints (helping tutor) made the matching task significantly easier (fewer tries) with the robot tutor. Furthermore, we found out that the robots' gaze hints were recognized significantly more often than the human tutor gaze hints, and consequently, the participants performed significantly better with the robot tutor. The reported study provides new findings towards the use of non-verbal gaze hints in human-robot interaction, and lays out new design implications, especially for robot-based educative interventions.

Quantity of Movement as a Measure of Engagement for Dementia: The Influence of Motivational Disorders.

Engagement in activities is crucial to improve quality of life in dementia. Yet, its measurement relies exclusively on behavior observation and the influence that behavioral and psychological symptoms of dementia (BPSD) have on it is overlooked. This study investigated whether quantity of movement, gauged with a wrist-worn accelerometer, could be a sound measure of engagement and whether apathy and depression negatively affected engagement. Fourteen participants with dementia took part in 6 sessions of activities: 3 of cognitive games (eg, jigsaw puzzles) and 3 of robot play (Pleo). Results highlighted significant correlations between quantity of movement and observational scales of engagement and a strong negative influence of apathy and depression on engagement. Overall, these findings suggest that quantity of movement could be used as an ancillary measure of engagement and underline the need to profile people with dementia according to their concurrent BPSD to better understand their engagement in activities.

Understanding Engagement in Dementia Through Behavior. The Ethographic and Laban-Inspired Coding System of Engagement (ELICSE) and the Evidence-Based Model of Engagement-Related Behavior (EMODEB).

Engagement in activities is of crucial importance for people with dementia. State of the art assessment techniques rely exclusively on behavior observation to measure engagement in dementia. These techniques are either too general to grasp how engagement is naturally expressed through behavior or too complex to be traced back to an overall engagement state. We carried out a longitudinal study to develop a coding system of engagement-related behavior that could tackle these issues and to create an evidence-based model of engagement to make meaning of such a coding system. Fourteen elderlies with mild to moderate dementia took part in the study. They were involved in two activities: a game-based cognitive stimulation and a robot-based free play. The coding system was developed with a mixed approach: ethographic and Laban-inspired. First, we developed two ethograms to describe the behavior of participants in the two activities in detail. Then, we used Laban Movement Analysis (LMA) to identify a common structure to the behaviors in the two ethograms and unify them in a unique coding system. The inter-rater reliability (IRR) of the coding system proved to be excellent for cognitive games (kappa = 0.78) and very good for robot play (kappa = 0.74). From the scoring of the videos, we developed an evidence-based model of engagement. This was based on the most frequent patterns of body part organization (i.e., the way body parts are connected in movement) observed during activities. Each pattern was given a meaning in terms of engagement by making reference to the literature. The model was tested using structural equation modeling (SEM). It achieved an excellent goodness of fit and all the hypothesized relations between variables were significant. We called the coding system that we developed the Ethographic and Laban-Inspired Coding System of Engagement (ELICSE) and the model the Evidence-based Model of Engagement-related Behavior (EMODEB). To the best of our knowledge, the ELICSE and the EMODEB constitute the first formalization of engagement-related behavior for dementia that describes how behavior unfolds over time and what it means in terms of engagement.

Modelling engagement in dementia through behaviour. Contribution for socially interactive robotics.

In this paper, we present a novel tool to measure engagement in people with dementia playing board games and interacting with a social robot, Pleo. We carried out two studies to reach a comprehensive inventory of behaviours accounting for engagement in dementia. The first one is an exploratory study aimed at modelling engagement in cognitive board games. The second one is a longitudinal study to investigate how people with dementia express engagement in cognitive games and in interactions with social robots. We adopted a technique coming from Ethology to mould behaviour, the ethogram. Ethogram is founded on low level behaviours, and allows hierarchical structuring. Herein, we present preliminary results consisting in the description of two ethograms and in their structuring obtained through thematic analysis. Such results show that an underlying structure of engagement exists across activities, and that different activities trigger different behavioural displays of engagement that adhere to such a structure.

A Hierarchical Bayesian Model for Crowd Emotions.

Estimation of emotions is an essential aspect in developing intelligent systems intended for crowded environments. However, emotion estimation in crowds remains a challenging problem due to the complexity in which human emotions are manifested and the capability of a system to perceive them in such conditions. This paper proposes a hierarchical Bayesian model to learn in unsupervised manner the behavior of individuals and of the crowd as a single entity, and explore the relation between behavior and emotions to infer emotional states. Information about the motion patterns of individuals are described using a self-organizing map, and a hierarchical Bayesian network builds probabilistic models to identify behaviors and infer the emotional state of individuals and the crowd. This model is trained and tested using data produced from simulated scenarios that resemble real-life environments. The conducted experiments tested the efficiency of our method to learn, detect and associate behaviors with emotional states yielding accuracy levels of 74% for individuals and 81% for the crowd, similar in performance with existing methods for pedestrian behavior detection but with novel concepts regarding the analysis of crowds.

Event based self-supervised temporal integration for multimodal sensor data.

A method for synergistic integration of multimodal sensor data is proposed in this paper. This method is based on two aspects of the integration process: (1) achieving synergistic integration of two or more sensory modalities, and (2) fusing the various information streams at particular moments during processing. Inspired by psychophysical experiments, we propose a self-supervised learning method for achieving synergy with combined representations. Evidence from temporal registration and binding experiments indicates that different cues are processed individually at specific time intervals. Therefore, an event-based temporal co-occurrence principle is proposed for the integration process. This integration method was applied to a mobile robot exploring unfamiliar environments. Simulations showed that integration enhanced route recognition with many perceptual similarities; moreover, they indicate that a perceptual hierarchy of knowledge about instant movement contributes significantly to short-term navigation, but that visual perceptions have bigger impact over longer intervals.