Interacting with autistic virtual characters: intrapersonal synchrony of nonverbal behavior affects participants' perception.

Temporal coordination of communicative behavior is not only located between but also within interaction partners (e.g., gaze and gestures). This intrapersonal synchrony (IaPS) is assumed to constitute interpersonal alignment. Studies show systematic variations in IaPS in individuals with autism, which may affect the degree of interpersonal temporal coordination. In the current study, we reversed the approach and mapped the measured nonverbal behavior of interactants with and without ASD from a previous study onto virtual characters to study the effects of the differential IaPS on observers (N = 68), both with and without ASD (crossed design). During a communication task with both characters, who indicated targets with gaze and delayed pointing gestures, we measured response times, gaze behavior, and post hoc impression formation. Results show that character behavior indicative of ASD resulted in overall enlarged decoding times in observers and this effect was even pronounced in observers with ASD. A classification of observer's gaze types indicated differentiated decoding strategies. Whereas non-autistic observers presented with a rather consistent eyes-focused strategy associated with efficient and fast responses, observers with ASD presented with highly variable decoding strategies. In contrast to communication efficiency, impression formation was not influenced by IaPS. The results underline the importance of timing differences in both production and perception processes during multimodal nonverbal communication in interactants with and without ASD. In essence, the current findings locate the manifestation of reduced reciprocity in autism not merely in the person, but in the interactional dynamics of dyads.

Temporal binding and sense of agency in major depression.

Background: Alterations in the experience of controlling oneself and one's environment are of high relevance to understanding the psychopathology of depression. This study investigated the relationship between Temporal Binding for action-event sequences, sense of agency, self-efficacy and symptom severity in Major Depressive Disorder. Method: We employed the Sense of Agency Scale (SoAS) and the General Self-Efficacy Scale (GSE) to assess explicit Sense of Agency and self-efficacy in a group of 42 persons diagnosed with Major Depressive Disorder (MDD) [20 identifying as female, 19 as male; mean age 37.8 years (± 13.3)] and 40 control persons without a psychiatric diagnosis (CG) [22 identifying as female, 20 as male; mean age 38.0 years ( ± 13.3)]. Depressive symptom severity was measured using the BDI-II. We additionally performed a temporal binding paradigm as a potential correlate to Sense of Agency. Participants partook in a time estimation task judging three intervals (250ms, 450ms, 650ms) while either observing or causing stimulus presentations. The underestimation of intervals following intentional actions causing stimulus presentations (compared to merely observing the stimulus presentation) is interpreted as temporal binding. Results: SoAS scores demonstrated an inverse correlation with depressive symptoms (CG: p=.032, R2=.113; MDD: p<.001, R2=.260) and a positive correlation with GSE scores (CG: p<.001, R2=.379; MDD: p<.001, R2=.254). We found distinct differences in temporal binding between healthy participants and the Major Depressive Disorder group without significant correlation between temporal binding and the SoAS or GSE scores. The data suggest group differences in time estimation particular pertaining to time intervals involving intentional action and increasingly complex multisensory stimuli. Discussion: We investigated parameters of subjective control, namely Sense of Agency and Self Efficacy. Here, we were able to reveal their inverse relationship with depressive symptoms in patients with major depressive disorder, highlighting a profound experience of loss of control with increasing symptom load. Deficits in experiencing control, particularly involving intentional motor actions (and more complex multisensory stimuli), appear to be more pronounced in Major Depressive Disorder, involving not only negative self-efficacy expectations but also an altered Sense of Agency and temporal binding. Temporal binding and SoAS scores did not correlate, adding to the growing evidence that the two measures may not be directly related. We propose that future research be directed at this contiguous relationship between Sense of Agency and Self Efficacy in Major Depressive Disorder.

Impaired recognition of interactive intentions in adults with autism spectrum disorder not attributable to differences in visual attention or coordination via eye contact and joint attention.

Altered nonverbal communication patterns especially with regard to gaze interactions are commonly reported for persons with autism spectrum disorder (ASD). In this study we investigate and differentiate for the first time the interplay of attention allocation, the establishment of shared focus (eye contact and joint attention) and the recognition of intentions in gaze interactions in adults with ASD compared to control persons. Participants interacted via gaze with a virtual character (VC), who they believed was controlled by another person. Participants were instructed to ascertain whether their partner was trying to interact with them. In fact, the VC was fully algorithm-controlled and showed either interactive or non-interactive gaze behavior. Participants with ASD were specifically impaired in ascertaining whether their partner was trying to interact with them or not as compared to participants without ASD whereas neither the allocation of attention nor the ability to establish a shared focus were affected. Thus, perception and production of gaze cues seem preserved while the evaluation of gaze cues appeared to be impaired. An additional exploratory analysis suggests that especially the interpretation of contingencies between the interactants' actions are altered in ASD and should be investigated more closely.

Differentiated, rather than shared, strategies for time-coordinated action in social and non-social domains in autistic individuals.

Autism spectrum disorder (ASD) is a neurodevelopmental condition with a highly heterogeneous adult phenotype that includes social and non-social behavioral characteristics. The link between the characteristics assignable to the different domains remains unresolved. One possibility is that social and non-social behaviors in autism are modulated by a common underlying deficit. However, here we report evidence supporting an alternative concept that is individual-centered rather than deficit-centered. Individuals are assumed to have a distinctive style in the strategies they adopt to perform social and non-social tasks with these styles presumably being structured differently between autistic individuals and typically-developed (TD) individuals. We tested this hypothesis for the execution of time-coordinated (synchronized) actions. Participants performed (i) a social task that required synchronized gaze and pointing actions to interact with another person, and (ii) a non-social task that required finger-tapping actions synchronized to periodic stimuli at different time-scales and sensory modalities. In both tasks, synchronization behavior differed between ASD and TD groups. However, a principal component analysis of individual behaviors across tasks revealed associations between social and non-social features for the TD persons but such cross-domain associations were strikingly absent for autistic individuals. The highly differentiated strategies between domains in ASD are inconsistent with a general synchronization deficit and instead highlight the individualized developmental heterogeneity in the acquisition of domain-specific behaviors. We propose a cognitive model to help disentangle individual-centered from deficit-centered effects in other domains. Our findings reinforce the importance to identify individually differentiated phenotypes to personalize autism therapies.

Dissociating passage and duration of time experiences through the intensity of ongoing visual change.

The experience of passage of time is assumed to be a constitutive component of our subjective phenomenal experience and our everyday life that is detached from the estimation of time durations. However, our understanding of the factors contributing to passage of time experience has been mostly restricted to associated emotional and cognitive experiences in temporally extended situations. Here, we tested the influence of low-level visual stimuli on the experience of passage and duration of time in 10-30 s intervals. We introduce a new paradigm in a starfield environment that allows to study the effects of basic visual aspects of a scene (velocity and density of stars in the starfield) and the duration of the situation, both embedded in a color tracking task. Results from two experiments show that velocity and density of stars in the starfield affect passage of time experience independent from duration estimation and the color tracking task: the experienced passage of time is accelerated with higher rates of moment-to-moment changes in the starfield while duration estimations are comparably unaffected. The results strongly suggest differential psychological processes underlying the experience of time passing by and the ability to estimate time durations. Potential mechanisms behind these results and the prospects of experimental approaches towards passage of time experience in psychological and neuroscientific research are discussed.

Intrapersonal synchrony analysis reveals a weaker temporal coherence between gaze and gestures in adults with autism spectrum disorder.

The temporal encoding of nonverbal signals within individuals, referred to as intrapersonal synchrony (IaPS), is an implicit process and essential feature of human communication. Based on existing evidence, IaPS is thought to be a marker of nonverbal behavior characteristics in autism spectrum disorders (ASD), but there is a lack of empirical evidence. The aim of this study was to quantify IaPS in adults during an experimentally controlled real-life interaction task. A sample of adults with a confirmed ASD diagnosis and a matched sample of typically-developed adults were tested (N = 48). Participants were required to indicate the appearance of a target invisible to their interaction partner nonverbally through gaze and pointing gestures. Special eye-tracking software allowed automated extraction of temporal delays between nonverbal signals and their intrapersonal variability with millisecond temporal resolution as indices for IaPS. Likelihood ratio tests of multilevel models showed enlarged delays between nonverbal signals in ASD. Larger delays were associated with greater intrapersonal variability in delays. The results provide a quantitative constraint on nonverbal temporality in typically-developed adults and suggest weaker temporal coherence between nonverbal signals in adults with ASD. The results provide a potential diagnostic marker and inspire predictive coding theories about the role of IaPS in interpersonal synchronization processes.

Temporal binding of social events less pronounced in individuals with Autism Spectrum Disorder.

Differences in predictive processing are considered amongst the prime candidates for mechanisms underlying different symptoms of autism spectrum disorder (ASD). A particularly valuable paradigm to investigate these processes is temporal binding (TB) assessed through time estimation tasks. In this study, we report on two separate experiments using a TB task designed to assess the influence of top-down social information on action event related TB. Both experiments were performed with a group of individuals diagnosed with ASD and a matched group without ASD. The results replicate earlier findings on a pronounced social hyperbinding for social action-event sequences and extend them to persons with ASD. Hyperbinding however, is less pronounced in the group with ASD as compared to the group without ASD. We interpret our results as indicative of a reduced predictive processing during social interaction. This reduction most likely results from differences in the integration of top-down social information into action-event monitoring. We speculate that this corresponds to differences in mentalizing processes in ASD.

Temporal Behavioral Parameters of On-Going Gaze Encounters in a Virtual Environment.

To navigate the social world, humans heavily rely on gaze for non-verbal communication as it conveys information in a highly dynamic and complex, yet concise manner: For instance, humans utilize gaze effortlessly to direct and infer the attention of a possible interaction partner. Many traditional paradigms in social gaze research though rely on static ways of assessing gaze interaction, e.g., by using images or prerecorded videos as stimulus material. Emerging gaze contingent paradigms, in which algorithmically controlled virtual characters can respond flexibly to the gaze behavior of humans, provide high ecological validity. Ideally, these are based on models of human behavior which allow for precise, parameterized characterization of behavior, and should include variable interactive settings and different communicative states of the interacting agents. The present study provides a complete definition and empirical description of a behavioral parameter space of human gaze behavior in extended gaze encounters. To this end, we (i) modeled a shared 2D virtual environment on a computer screen in which a human could interact via gaze with an agent and simultaneously presented objects to create instances of joint attention and (ii) determined quantitatively the free model parameters (temporal and probabilistic) of behavior within this environment to provide a first complete, detailed description of the behavioral parameter space governing joint attention. This knowledge is essential to enable the modeling of interacting agents with a high degree of ecological validity, be it for cognitive studies or applications in human-robot interaction.

Temporal binding is enhanced in social contexts.

Temporal binding (TB) refers to an underestimation of time intervals between two events, most commonly for actions and their effects. This temporal contraction is measurable for both perceived changes in social stimuli such as faces, as well as for interactions with a partner. We investigated TB in two separate experiments to uncover the individual influences of (i) participants' belief in an interaction with a human partner (as compared to a computer), and (ii) a face-like stimulus versus an abstract stimulus mediating the interaction. The results show that TB is more pronounced when self-initiated actions result in a personal event as opposed to a mere physical effect, being suggestive of a "social hyperbinding." The social hyperbinding effect appeared to be driven both by the belief in interacting with another person and by a face-like stimulus. However, there seemed to be no further enhancing effect when combining the top-down processes ("beliefs") with the bottom-up processes ("perceptions"). These findings suggest a prioritization of social information for TB regardless of whether this information is introduced by top-down (beliefs) or bottom-up information (stimuli). Our results add to existing literature demonstrating an increase in action-event monitoring for social cues.

The Temporality of Situated Cognition.

Situated cognition embeds perceptions, thoughts, and behavior within the contextual framework of so-called "4E cognition" understanding cognition to be embodied, enactive, extended, and embedded. Whereas this definition is primarily based on the spatial properties of a situation, it neglects a fundamental constituent: the cognitive situation as enduring. On a subpersonal level, situated cognition requires the integration of information processing within a minimal temporal extension generating the basic building blocks of perception and action ("microlayer" of time). On a personal level, lived situations and experienced narratives leading to our biography can be defined by their broader temporal horizons ("macrolayer" of time). The macrolayer of time is based on and emerges from information processing on the microlayer of time. Whereas the constraints on the microlayer are primarily defined by the integrity of neurobiological processes within an individual cognitive system, the temporal horizons and subsequently the situational context on the macrolayer are defined by the complex affordances of a situation on a personal or interpersonal level. On both time layers, cognition can be defined as a continuous dynamic process, reflecting the transition from one situated state to another. Taken together, the events forming the delimiting horizons of these situations correspond to the temporal structure of the cognitive process along which it continuously proceeds. The dynamic driving and enabling this transition from state to state is synonymous with the inherent flow of time. Just as the layers of time, flow and structure, are inseparably connected. The integration of temporal flow and temporal structure into the continuous dynamic process constitutes the enduring situatedness of cognition. By providing everyday examples and examples from psychopathology, we highlight the benefits of understanding cognitive processes as part of enduring situations.

Distinguishing Social From Private Intentions Through the Passive Observation of Gaze Cues.

Observing others' gaze is most informative during social encounters between humans: We can learn about potentially salient objects in the shared environment, infer others' mental states and detect their communicative intentions. We almost automatically follow the gaze of others in order to check the relevance of the target of the other's attention. This phenomenon called gaze cueing can be conceptualized as a triadic interaction involving a gaze initiator, a gaze follower and a gaze target, i.e., an object or person of interest in the environment. Gaze cueing can occur as "gaze pointing" with a communicative or "social" intention by the initiator, telling the observer that she/he is meant to follow, or as an incidental event, in which the observer follows spontaneously without any intention of the observed person. Here, we investigate which gaze cues let an observer ascribe a social intention to the observed person's gaze and whether and to which degree previous eye contact in combination with an object fixation contributes to this ascription. We varied the orientation of the starting position of gaze toward the observer and the orientation of the end position of a lateral gaze shift. In two experiments participants had to infer from the gaze behavior either mere approach ("the person looked at me") vs. a social ("the person wanted to show me something") or a social vs. a private motivation ("the person was interested in something"). Participants differentially attributed either approach behavior, a social, or a private intention to the agent solely based on the passive observation of the two specific gaze cues of start and end position. While for the attribution of privately motivated behavior, participants relied solely on the end position of the gaze shift, the social interpretation of the observed behavior depended additionally upon initial eye contact. Implications of these results for future social gaze and social cognition research in general are discussed.

Introducing ACASS: An Annotated Character Animation Stimulus Set for Controlled (e)Motion Perception Studies.

Others' movements inform us about their current activities as well as their intentions and emotions. Research on the distinct mechanisms underlying action recognition and emotion inferences has been limited due to a lack of suitable comparative stimulus material. Problematic confounds can derive from low-level physical features (e.g., luminance), as well as from higher-level psychological features (e.g., stimulus difficulty). Here we present a standardized stimulus dataset, which allows to address both action and emotion recognition with identical stimuli. The stimulus set consists of 792 computer animations with a neutral avatar based on full body motion capture protocols. Motion capture was performed on 22 human volunteers, instructed to perform six everyday activities (mopping, sweeping, painting with a roller, painting with a brush, wiping, sanding) in three different moods (angry, happy, sad). Five-second clips of each motion protocol were rendered into AVI-files using two virtual camera perspectives for each clip. In contrast to video stimuli, the computer animations allowed to standardize the physical appearance of the avatar and to control lighting and coloring conditions, thus reducing the stimulus variation to mere movement. To control for low level optical features of the stimuli, we developed and applied a set of MATLAB routines extracting basic physical features of the stimuli, including average background-foreground proportion and frame-by-frame pixel change dynamics. This information was used to identify outliers and to homogenize the stimuli across action and emotion categories. This led to a smaller stimulus subset (n = 83 animations within the 792 clip database) which only contained two different actions (mopping, sweeping) and two different moods (angry, happy). To further homogenize this stimulus subset with regard to psychological criteria we conducted an online observer study (N = 112 participants) to assess the recognition rates for actions and moods, which led to a final sub-selection of 32 clips (eight per category) within the database. The ACASS database and its subsets provide unique opportunities for research applications in social psychology, social neuroscience, and applied clinical studies on communication disorders. All 792 AVI-files, selected subsets, MATLAB code, annotations, and motion capture data (FBX-files) are available online.

Inferring Interactivity From Gaze Patterns During Triadic Person-Object-Agent Interactions.

Observing others' gaze informs us about relevant matters in the environment. Humans' sensitivity to gaze cues and our ability to use this information to focus our own attention is crucial to learning, social coordination, and survival. Gaze can also be a deliberate social signal which captures and directs the gaze of others toward an object of interest. In the current study, we investigated whether the intention to actively communicate one's own attentional focus can be inferred from the dynamics of gaze alone. We used a triadic gaze interaction paradigm based on the recently proposed classification of attentional states and respective gaze patterns in person-object-person interactions, the so-called "social gaze space (SGS)." Twenty-eight participants interacted with a computer controlled virtual agent while they assumed to interact with a real human. During the experiment, the virtual agent engaged in various gaze patterns which were determined by the agent's attentional communicative state, as described by the concept of SGS. After each interaction, participants were asked to judge whether the other person was trying to deliberately interact with them. Results show that participants were able to infer the communicative intention solely from the agent's gaze behavior. The results substantiate claims about the pivotal role of gaze in social coordination and relationship formation. Our results further reveal that social expectations are reflected in differential responses to the displayed gaze patterns and may be crucial for impression formation during gaze-based interaction. To the best of our knowledge, this is the first study to document the experience of interactivity in continuous and contingent triadic gaze interactions.

The "Social Gaze Space": A Taxonomy for Gaze-Based Communication in Triadic Interactions.

Humans substantially rely on non-verbal cues in their communication and interaction with others. The eyes represent a "simultaneous input-output device": While we observe others and obtain information about their mental states (including feelings, thoughts, and intentions-to-act), our gaze simultaneously provides information about our own attention and inner experiences. This substantiates its pivotal role for the coordination of communication. The communicative and coordinative capacities - and their phylogenetic and ontogenetic impacts - become fully apparent in triadic interactions constituted in its simplest form by two persons and an object. Technological advances have sparked renewed interest in social gaze and provide new methodological approaches. Here we introduce the 'Social Gaze Space' as a new conceptual framework for the systematic study of gaze behavior during social information processing. It covers all possible categorical states, namely 'partner-oriented,' 'object-oriented,' 'introspective,' 'initiating joint attention,' and 'responding joint attention.' Different combinations of these states explain several interpersonal phenomena. We argue that this taxonomy distinguishes the most relevant interactional states along their distinctive features, and will showcase the implications for prominent social gaze phenomena. The taxonomy allows to identify research desiderates that have been neglected so far. We argue for a systematic investigation of these phenomena and discuss some related methodological issues.

Eyes on the mind: investigating the influence of gaze dynamics on the perception of others in real-time social interaction.

Social gaze provides a window into the interests and intentions of others and allows us to actively point out our own. It enables us to engage in triadic interactions involving human actors and physical objects and to build an indispensable basis for coordinated action and collaborative efforts. The object-related aspect of gaze in combination with the fact that any motor act of looking encompasses both input and output of the minds involved makes this non-verbal cue system particularly interesting for research in embodied social cognition. Social gaze comprises several core components, such as gaze-following or gaze aversion. Gaze-following can result in situations of either "joint attention" or "shared attention." The former describes situations in which the gaze-follower is aware of sharing a joint visual focus with the gazer. The latter refers to a situation in which gazer and gaze-follower focus on the same object and both are aware of their reciprocal awareness of this joint focus. Here, a novel interactive eye-tracking paradigm suited for studying triadic interactions was used to explore two aspects of social gaze. Experiments 1a and 1b assessed how the latency of another person's gaze reactions (i.e., gaze-following or gaze version) affected participants' sense of agency, which was measured by their experience of relatedness of these reactions. Results demonstrate that both timing and congruency of a gaze reaction as well as the other's action options influence the sense of agency. Experiment 2 explored differences in gaze dynamics when participants were asked to establish either joint or shared attention. Findings indicate that establishing shared attention takes longer and requires a larger number of gaze shifts as compared to joint attention, which more closely seems to resemble simple visual detection. Taken together, novel insights into the sense of agency and the awareness of others in gaze-based interaction are provided.