Human social motor solutions for human-machine interaction in dynamical task contexts.

Multiagent activity is commonplace in everyday life and can improve the behavioral efficiency of task performance and learning. Thus, augmenting social contexts with the use of interactive virtual and robotic agents is of great interest across health, sport, and industry domains. However, the effectiveness of human-machine interaction (HMI) to effectively train humans for future social encounters depends on the ability of artificial agents to respond to human coactors in a natural, human-like manner. One way to achieve effective HMI is by developing dynamical models utilizing dynamical motor primitives (DMPs) of human multiagent coordination that not only capture the behavioral dynamics of successful human performance but also, provide a tractable control architecture for computerized agents. Previous research has demonstrated how DMPs can successfully capture human-like dynamics of simple nonsocial, single-actor movements. However, it is unclear whether DMPs can be used to model more complex multiagent task scenarios. This study tested this human-centered approach to HMI using a complex dyadic shepherding task, in which pairs of coacting agents had to work together to corral and contain small herds of virtual sheep. Human-human and human-artificial agent dyads were tested across two different task contexts. The results revealed (i) that the performance of human-human dyads was equivalent to those composed of a human and the artificial agent and (ii) that, using a "Turing-like" methodology, most participants in the HMI condition were unaware that they were working alongside an artificial agent, further validating the isomorphism of human and artificial agent behavior.

Effects of Task Constraint on Action Dynamics.

The actualization of action possibilities (i.e., affordances) can often be accomplished in numerous ways. For instance, an individual could walk over to a rubbish bin to drop an item in or throw the piece of rubbish into the bin from some distance away. The aim of the current study was to investigate the action dynamics that emerge from such under-constrained task or action spaces using an object transportation task. Participants were instructed to transport balls between a starting location and a large wooden box located 9 meters away. The temporal interval between the sequential presentation of balls was manipulated as a control parameter and was expected to influence the distance participants moved prior to throwing or dropping the ball into the target box. A two-parameter state space derived from the Cusp Catastrophe Model was employed to illustrate how behavioral variability emerged as a consequence of the under-constrained task context. Two follow-up experiments demonstrated direct correspondence between model predictions and observed action dynamics as a function of increasing task constraints. Implications for modelling, the theory of affordances, and empirical studies more generally are discussed.

Herd Those Sheep: Emergent Multiagent Coordination and Behavioral-Mode Switching.

Effectively coordinating one's behaviors with those of others is essential for successful multiagent activity. In recent years, increased attention has been given to understanding the dynamical principles that underlie such coordination because of a growing interest in behavioral synchrony and complex-systems phenomena. Here, we examined the behavioral dynamics of a novel, multiagent shepherding task, in which pairs of individuals had to corral small herds of virtual sheep in the center of a virtual game field. Initially, all pairs adopted a complementary, search-and-recover mode of behavioral coordination, in which both members corralled sheep predominantly on their own sides of the field. Over the course of game play, however, a significant number of pairs spontaneously discovered a more effective mode of behavior: coupled oscillatory containment, in which both members synchronously oscillated around the sheep. Analysis and modeling revealed that both modes were defined by the task's underlying dynamics and, moreover, reflected context-specific realizations of the lawful dynamics that define functional shepherding behavior more generally.

Predicting and understanding human action decisions during skillful joint-action using supervised machine learning and explainable-AI.

This study investigated the utility of supervised machine learning (SML) and explainable artificial intelligence (AI) techniques for modeling and understanding human decision-making during multiagent task performance. Long short-term memory (LSTM) networks were trained to predict the target selection decisions of expert and novice players completing a multiagent herding task. The results revealed that the trained LSTM models could not only accurately predict the target selection decisions of expert and novice players but that these predictions could be made at timescales that preceded a player's conscious intent. Importantly, the models were also expertise specific, in that models trained to predict the target selection decisions of experts could not accurately predict the target selection decisions of novices (and vice versa). To understand what differentiated expert and novice target selection decisions, we employed the explainable-AI technique, SHapley Additive explanation (SHAP), to identify what informational features (variables) most influenced modelpredictions. The SHAP analysis revealed that experts were more reliant on information about target direction of heading and the location of coherders (i.e., other players) compared to novices. The implications and assumptions underlying the use of SML and explainable-AI techniques for investigating and understanding human decision-making are discussed.

Behavioral dynamics of conversation, (mis)communication and coordination in noisy environments.

During conversations people coordinate simultaneous channels of verbal and nonverbal information to hear and be heard. But the presence of background noise levels such as those found in cafes and restaurants can be a barrier to conversational success. Here, we used speech and motion-tracking to reveal the reciprocal processes people use to communicate in noisy environments. Conversations between twenty-two pairs of typical-hearing adults were elicited under different conditions of background noise, while standing or sitting around a table. With the onset of background noise, pairs rapidly adjusted their interpersonal distance and speech level, with the degree of initial change dependent on noise level and talker configuration. Following this transient phase, pairs settled into a sustaining phase in which reciprocal speech and movement-based coordination processes synergistically maintained effective communication, again with the magnitude of stability of these coordination processes covarying with noise level and talker configuration. Finally, as communication breakdowns increased at high noise levels, pairs exhibited resetting behaviors to help restore communication-decreasing interpersonal distance and/or increasing speech levels in response to communication breakdowns. Approximately 78 dB SPL defined a threshold where behavioral processes were no longer sufficient for maintaining effective conversation and communication breakdowns rapidly increased.

Dynamical biomarkers in teams and other multiagent systems.

Effective team behavior in high-performance environments such as in sport and the military requires individual team members to efficiently perceive the unfolding task events, predict the actions and action intents of the other team members, and plan and execute their own actions to simultaneously accomplish individual and collective goals. To enhance team performance through effective cooperation, it is crucial to measure the situation awareness and dynamics of each team member and how they collectively impact the team's functioning. Further, to be practically useful for real-life settings, such measures must be easily obtainable from existing sensors. This paper presents several methodologies that can be used on positional and movement acceleration data of team members to quantify and/or predict team performance, assess situation awareness, and to help identify task-relevant information to support individual decision-making. Given the limited reporting of these methods within military cohorts, these methodologies are described using examples from team sports and teams training in virtual environments, with discussion as to how they can be applied to real-world military teams.

Interview with an avatar: Comparing online and virtual reality perspective taking for gender bias in STEM hiring decisions.

Virtual perspective taking can reduce unconscious bias and increase empathy and prosocial behavior toward individuals who are marginalized based on group stereotypes such as age, race, or socioeconomic status. However, the question remains whether this approach might reduce implicit gender bias, and the degree to which virtual immersion contributes to behavioral modulation following perspective taking tasks is unknown. Accordingly, we investigate the role of virtual perspective taking for binary gender using an online platform (Study 1) and immersive virtual reality (Study 2). Female and male undergraduates performed a simulated interview while virtually represented by an avatar that was either congruent or incongruent with their own gender. All participants rated a male and a female candidate on competence, hireability, likeability, empathy, and interpersonal closeness and then chose one of these two equivalently qualified candidates to hire for a laboratory assistant position in the male dominated industry of information technology. Online perspective taking did not reveal a significant influence of avatar gender on candidate ratings or candidate choice, whereas virtual reality perspective taking resulted in significant changes to participant behavior following exposure to a gender-incongruent avatar (e.g., male embodied as female), such that men showed preference for the female candidate and women showed preference for the male candidate. Although between-group differences in candidate ratings were subtle, rating trends were consistent with substantial differences in candidate choice, and this effect was greater for men. Compared to an online approach, virtual reality perspective taking appears to exert greater influence on acute behavioral modulation for gender bias due to its ability to fully immerse participants in the experience of (temporarily) becoming someone else, with empathy as a potential mechanism underlying this phenomenon.

Conversation dynamics in a multiplayer video game with knowledge asymmetry.

Despite the challenges associated with virtually mediated communication, remote collaboration is a defining characteristic of online multiplayer gaming communities. Inspired by the teamwork exhibited by players in first-person shooter games, this study investigated the verbal and behavioral coordination of four-player teams playing a cooperative online video game. The game, Desert Herding, involved teams consisting of three ground players and one drone operator tasked to locate, corral, and contain evasive robot agents scattered across a large desert environment. Ground players could move throughout the environment, while the drone operator's role was akin to that of a "spectator" with a bird's-eye view, with access to veridical information of the locations of teammates and the to-be-corralled agents. Categorical recurrence quantification analysis (catRQA) was used to measure the communication dynamics of teams as they completed the task. Demands on coordination were manipulated by varying the ground players' ability to observe the environment with the use of game "fog." Results show that catRQA was sensitive to changes to task visibility, with reductions in task visibility reorganizing how participants conversed during the game to maintain team situation awareness. The results are discussed in the context of future work that can address how team coordination can be augmented with the inclusion of artificial agents, as synthetic teammates.

Assessing Team Effectiveness by How Players Structure Their Search in a First-Person Multiplayer Video Game.

People working as a team can achieve more than when working alone due to a team's ability to parallelize the completion of tasks. In collaborative search tasks, this necessitates the formation of effective division of labor strategies to minimize redundancies in search. For such strategies to be developed, team members need to perceive the task's relevant components and how they evolve over time, as well as an understanding of what others will do so that they can structure their own behavior to contribute to the team's goal. This study explored whether the capacity for team members to coordinate effectively can be related to how participants structure their search behaviors in an online multiplayer collaborative search task. Our results demonstrated that the structure of search behavior, quantified using detrended fluctuation analysis, was sensitive to contextual factors that limit a participant's ability to gather information. Further, increases in the persistence of movement fluctuations during search behavior were found as teams developed more effective coordinative strategies and were associated with better task performance.

Development and Initial Validation of the Relaxation Sensitivity Index.

Relaxation sensitivity indexes the fear of relaxation-related events. The purpose of this study was to develop and provide initial validation of a self-report measure of relaxation sensitivity, the Relaxation Sensitivity Index (RSI). Three independent samples of undergraduate students (n=300 unselected, n=349 non-clinical, and n=197 clinical analogs with elevated anxiety/depression symptoms) completed self-report measures to examine the factor structure, reliability, and validity of the RSI. Results of exploratory and confirmatory factor analyses supported a three-factor structure (correlated Physical, Cognitive, and Social Concerns). The RSI demonstrated good internal consistency and construct validity as evidenced by expected correlations with measures of anxiety and depression symptoms. The RSI showed good predictive validity in terms of a history of fearful responding to relaxation. RSI scores were significantly higher in the symptomatic compared to non-clinical sample. Results suggest the RSI is a valid and reliable measure that may be useful in clinical and research settings.

Navigational Behavior of Humans and Deep Reinforcement Learning Agents.

Rapid advances in the field of Deep Reinforcement Learning (DRL) over the past several years have led to artificial agents (AAs) capable of producing behavior that meets or exceeds human-level performance in a wide variety of tasks. However, research on DRL frequently lacks adequate discussion of the low-level dynamics of the behavior itself and instead focuses on meta-level or global-level performance metrics. In doing so, the current literature lacks perspective on the qualitative nature of AA behavior, leaving questions regarding the spatiotemporal patterning of their behavior largely unanswered. The current study explored the degree to which the navigation and route selection trajectories of DRL agents (i.e., AAs trained using DRL) through simple obstacle ridden virtual environments were equivalent (and/or different) from those produced by human agents. The second and related aim was to determine whether a task-dynamical model of human route navigation could not only be used to capture both human and DRL navigational behavior, but also to help identify whether any observed differences in the navigational trajectories of humans and DRL agents were a function of differences in the dynamical environmental couplings.

Task dynamics define the contextual emergence of human corralling behaviors.

Social animals have the remarkable ability to organize into collectives to achieve goals unobtainable to individual members. Equally striking is the observation that despite differences in perceptual-motor capabilities, different animals often exhibit qualitatively similar collective states of organization and coordination. Such qualitative similarities can be seen in corralling behaviors involving the encirclement of prey that are observed, for example, during collaborative hunting amongst several apex predator species living in disparate environments. Similar encirclement behaviors are also displayed by human participants in a collaborative problem-solving task involving the herding and containment of evasive artificial agents. Inspired by the functional similarities in this behavior across humans and non-human systems, this paper investigated whether the containment strategies displayed by humans emerge as a function of the task's underlying dynamics, which shape patterns of goal-directed corralling more generally. This hypothesis was tested by comparing the strategies naïve human dyads adopt during the containment of a set of evasive artificial agents across two disparate task contexts. Despite the different movement types (manual manipulation or locomotion) required in the different task contexts, the behaviors that humans display can be predicted as emergent properties of the same underlying task-dynamic model.

Synchronous vs. non-synchronous imitation: Using dance to explore interpersonal coordination during observational learning.

Observational learning can enhance the acquisition and performance quality of complex motor skills. While an extensive body of research has focused on the benefits of synchronous (i.e., concurrent physical practice) and non-synchronous (i.e., delayed physical practice) observational learning strategies, the question remains as to whether these approaches differentially influence performance outcomes. Accordingly, we investigate the differential outcomes of synchronous and non-synchronous observational training contexts using a novel dance sequence. Using multidimensional cross-recurrence quantification analysis, movement time-series were recorded for novice dancers who either synchronised with (n = 22) or observed and then imitated (n = 20) an expert dancer. Participants performed a 16-count choreographed dance sequence for 20 trials assisted by the expert, followed by one final, unassisted performance trial. Although end-state performance did not significantly differ between synchronous and non-synchronous learners, a significant decline in performance quality from imitation to independent replication was shown for synchronous learners. A non-significant positive trend in performance accuracy was shown for non-synchronous learners. For all participants, better imitative performance across training trials led to better end-state performance, but only for the accuracy (and not timing) of movement reproduction. Collectively, the results suggest that synchronous learners came to rely on a real-time mapping process between visual input from the expert and their own visual and proprioceptive intrinsic feedback, to the detriment of learning. Thus, the act of synchronising alone does not ensure an appropriate training context for advanced sequence learning.

Hopf Bifurcations in Complex Multiagent Activity: The Signature of Discrete to Rhythmic Behavioral Transitions.

Most human actions are composed of two fundamental movement types, discrete and rhythmic movements. These movement types, or primitives, are analogous to the two elemental behaviors of nonlinear dynamical systems, namely, fixed-point and limit cycle behavior, respectively. Furthermore, there is now a growing body of research demonstrating how various human actions and behaviors can be effectively modeled and understood using a small set of low-dimensional, fixed-point and limit cycle dynamical systems (differential equations). Here, we provide an overview of these dynamical motorprimitives and detail recent research demonstrating how these dynamical primitives can be used to model the task dynamics of complex multiagent behavior. More specifically, we review how a task-dynamic model of multiagent shepherding behavior, composed of rudimentary fixed-point and limit cycle dynamical primitives, can not only effectively model the behavior of cooperating human co-actors, but also reveals how the discovery and intentional use of optimal behavioral coordination during task learning is marked by a spontaneous, self-organized transition between fixed-point and limit cycle dynamics (i.e., via a Hopf bifurcation).

Feedback delays can enhance anticipatory synchronization in human-machine interaction.

Research investigating the dynamics of coupled physical systems has demonstrated that small feedback delays can allow a dynamic response system to anticipate chaotic behavior. This counterintuitive phenomenon, termed anticipatory synchronization, has been observed in coupled electrical circuits, laser semi-conductors, and artificial neurons. Recent research indicates that the same process might also support the ability of humans to anticipate the occurrence of chaotic behavior in other individuals. Motivated by this latter work, the current study examined whether the process of feedback delay induced anticipatory synchronization could be employed to develop an interactive artificial agent capable of anticipating chaotic human movement. Results revealed that incorporating such delays within the movement-control dynamics of an artificial agent not only enhances an artificial agent's ability to anticipate chaotic human behavior, but to synchronize with such behavior in a manner similar to natural human-human anticipatory synchronization. The implication of these findings for the development of human-machine interaction systems is discussed.

Exploring complexity matching and asynchrony dynamics in synchronized and syncopated task performances.

When two people synchronize their rhythmic behaviors (e.g., finger tapping; walking) they match one another not only at a local scale of beat-to-beat intervals, but also at a global scale of the complex (fractal) patterns of variation in their interval series. This "complexity matching" had been demonstrated in a variety of timing behaviors, but the current study was designed to address two important gaps in previous research. First, very little was known about complexity matching outside of synchronization tasks. This was important because different modes are associated with differences in the strength of coordination and the fractal scaling of the task performance. Second, very little was known about the dynamics of the asynchrony series. This was important because asynchrony is a variable directly quantifying the coordination between the two timing behaviors and the task goal. So, the current study explored complexity matching in both synchronized and syncopated finger tapping tasks, and included analyses of the fractal scaling of the asynchrony series. Participants completed an interpersonal finger tapping task, in both synchronization and syncopation conditions. The magnitude of variation and the exact power law scaling of the tapping intervals were manipulated by having one participant tap in time with a metronome. Complexity matching was most stable when there was sufficient variation in the task behavior and when a persistent scaling dynamic was presented. There were, however, several interesting differences between the two coordination modes, in terms of the heterogeneity of the complexity matching effect and the scaling of the asynchronies. These findings raised a number of important points concerning how to approach and understand the interaction of inherently complex systems.

To Pass or Not to Pass: Modeling the Movement and Affordance Dynamics of a Pick and Place Task.

Humans commonly engage in tasks that require or are made more efficient by coordinating with other humans. In this paper we introduce a task dynamics approach for modeling multi-agent interaction and decision making in a pick and place task where an agent must move an object from one location to another and decide whether to act alone or with a partner. Our aims were to identify and model (1) the affordance related dynamics that define an actor's choice to move an object alone or to pass it to their co-actor and (2) the trajectory dynamics of an actor's hand movements when moving to grasp, relocate, or pass the object. Using a virtual reality pick and place task, we demonstrate that both the decision to pass or not pass an object and the movement trajectories of the participants can be characterized in terms of a behavioral dynamics model. Simulations suggest that the proposed behavioral dynamics model exhibits features observed in human participants including hysteresis in decision making, non-straight line trajectories, and non-constant velocity profiles. The proposed model highlights how the same low-dimensional behavioral dynamics can operate to constrain multiple (and often nested) levels of human activity and suggests that knowledge of what, when, where and how to move or act during pick and place behavior may be defined by these low dimensional task dynamics and, thus, can emerge spontaneously and in real-time with little a priori planning.

Harmony from chaos? Perceptual-motor delays enhance behavioral anticipation in social interaction.

Effective interpersonal coordination is fundamental to robust social interaction, and the ability to anticipate a coactor's behavior is essential for achieving this coordination. However, coordination research has focused on the behavioral synchrony that occurs between the simple periodic movements of coactors and, thus, little is known about the anticipation that occurs during complex, everyday interaction. Research on the dynamics of coupled neurons, human motor control, electrical circuits, and laser semiconductors universally demonstrates that small temporal feedback delays are necessary for the anticipation of chaotic events. We therefore investigated whether similar feedback delays would promote anticipatory behavior during social interaction. Results revealed that coactors were not only able to anticipate others' chaotic movements when experiencing small perceptual-motor delays, but also exhibited movement patterns of equivalent complexity. This suggests that such delays, including those within the human nervous system, may enhance, rather than hinder, the anticipatory processes that underlie successful social interaction.

Self-organized complementary joint action: Behavioral dynamics of an interpersonal collision-avoidance task.

Understanding stable patterns of interpersonal movement coordination is essential to understanding successful social interaction and activity (i.e., joint action). Previous research investigating such coordination has primarily focused on the synchronization of simple rhythmic movements (e.g., finger/forearm oscillations or pendulum swinging). Very few studies, however, have explored the stable patterns of coordination that emerge during task-directed complementary coordination tasks. Thus, the aim of the current study was to investigate and model the behavioral dynamics of a complementary collision-avoidance task. Participant pairs performed a repetitive targeting task in which they moved computer stimuli back and forth between sets of target locations without colliding into each other. The results revealed that pairs quickly converged onto a stable, asymmetric pattern of movement coordination that reflected differential control across participants, with 1 participant adopting a more straight-line movement trajectory between targets, and the other participant adopting a more elliptical trajectory between targets. This asymmetric movement pattern was also characterized by a phase lag between participants and was essential to task success. Coupling directionality analysis and dynamical modeling revealed that this dynamic regime was due to participant-specific differences in the coupling functions that defined the task-dynamics of participant pairs. Collectively, the current findings provide evidence that the dynamical coordination processes previously identified to underlie simple motor synchronization can also support more complex, goal-directed, joint action behavior, and can participate the spontaneous emergence of complementary joint action roles.