Human social motor solutions for human-machine interaction in dynamical task contexts.

Multiagent activity is commonplace in everyday life and can improve the behavioral efficiency of task performance and learning. Thus, augmenting social contexts with the use of interactive virtual and robotic agents is of great interest across health, sport, and industry domains. However, the effectiveness of human-machine interaction (HMI) to effectively train humans for future social encounters depends on the ability of artificial agents to respond to human coactors in a natural, human-like manner. One way to achieve effective HMI is by developing dynamical models utilizing dynamical motor primitives (DMPs) of human multiagent coordination that not only capture the behavioral dynamics of successful human performance but also, provide a tractable control architecture for computerized agents. Previous research has demonstrated how DMPs can successfully capture human-like dynamics of simple nonsocial, single-actor movements. However, it is unclear whether DMPs can be used to model more complex multiagent task scenarios. This study tested this human-centered approach to HMI using a complex dyadic shepherding task, in which pairs of coacting agents had to work together to corral and contain small herds of virtual sheep. Human-human and human-artificial agent dyads were tested across two different task contexts. The results revealed (i) that the performance of human-human dyads was equivalent to those composed of a human and the artificial agent and (ii) that, using a "Turing-like" methodology, most participants in the HMI condition were unaware that they were working alongside an artificial agent, further validating the isomorphism of human and artificial agent behavior.

Conversational distance adaptation in noise and its effect on signal-to-noise ratio in realistic listening environments.

Everyday environments impose acoustical conditions on speech communication that require interlocutors to adapt their behavior to be able to hear and to be heard. Past research has focused mainly on the adaptation of speech level, while few studies investigated how interlocutors adapt their conversational distance as a function of noise level. Similarly, no study tested the interaction between distance and speech level adaptation in noise. In the present study, participant pairs held natural conversations while binaurally listening to identical noise recordings of different realistic environments (range of 53-92 dB sound pressure level), using acoustically transparent headphones. Conversations were in standing or sitting (at a table) conditions. Interlocutor distances were tracked using wireless motion-capture equipment, which allowed subjects to move closer or farther from each other. The results show that talkers adapt their voices mainly according to the noise conditions and much less according to distance. Distance adaptation was highest in the standing condition. Consequently, mainly in the loudest environments, listeners were able to improve the signal-to-noise ratio (SNR) at the receiver location in the standing condition compared to the sitting condition, which became less negative. Analytical approximations are provided for the conversational distance as well as the receiver-related speech and SNR.

Effects of Task Constraint on Action Dynamics.

The actualization of action possibilities (i.e., affordances) can often be accomplished in numerous ways. For instance, an individual could walk over to a rubbish bin to drop an item in or throw the piece of rubbish into the bin from some distance away. The aim of the current study was to investigate the action dynamics that emerge from such under-constrained task or action spaces using an object transportation task. Participants were instructed to transport balls between a starting location and a large wooden box located 9 meters away. The temporal interval between the sequential presentation of balls was manipulated as a control parameter and was expected to influence the distance participants moved prior to throwing or dropping the ball into the target box. A two-parameter state space derived from the Cusp Catastrophe Model was employed to illustrate how behavioral variability emerged as a consequence of the under-constrained task context. Two follow-up experiments demonstrated direct correspondence between model predictions and observed action dynamics as a function of increasing task constraints. Implications for modelling, the theory of affordances, and empirical studies more generally are discussed.

Herd Those Sheep: Emergent Multiagent Coordination and Behavioral-Mode Switching.

Effectively coordinating one's behaviors with those of others is essential for successful multiagent activity. In recent years, increased attention has been given to understanding the dynamical principles that underlie such coordination because of a growing interest in behavioral synchrony and complex-systems phenomena. Here, we examined the behavioral dynamics of a novel, multiagent shepherding task, in which pairs of individuals had to corral small herds of virtual sheep in the center of a virtual game field. Initially, all pairs adopted a complementary, search-and-recover mode of behavioral coordination, in which both members corralled sheep predominantly on their own sides of the field. Over the course of game play, however, a significant number of pairs spontaneously discovered a more effective mode of behavior: coupled oscillatory containment, in which both members synchronously oscillated around the sheep. Analysis and modeling revealed that both modes were defined by the task's underlying dynamics and, moreover, reflected context-specific realizations of the lawful dynamics that define functional shepherding behavior more generally.

Influence of stimulus velocity profile on unintentional visuomotor entrainment depends on eye movements.

Humans spontaneously entrain their movements to visual rhythms in the environment. Previous research has shown that the strength of such unintentional visuomotor entrainment is enhanced when observing rhythms characterized by the nonlinear, Rayleigh kinematics typical of human movements; such movements are characterized by greater slowness towards the trajectory turning points compared to sinusoidal movements. However, the enhanced unintentional entrainment to rhythms exhibiting Rayleigh kinematics has only been shown to occur when participants tracked stimulus movements with their eyes, which might have facilitated access to important information for enhanced entrainment. The current study compared the strength of unintentional visuomotor entrainment with both Rayleigh and sinusoidal kinematics when participants were either tracking (eye following the oscillating stimulus) or non-tracking (eye fixed at the centre of the stimulus trajectory) stimulus movements. The results showed that enhanced unintentional entrainment with Rayleigh stimuli only occurred with eye-tracking, supporting that slowness of rhythmic movements towards turning points facilitate entrainment and that access to this information depends on eye movements.

Can low-cost motion-tracking systems substitute a Polhemus system when researching social motor coordination in children?

Functionally stable and robust interpersonal motor coordination has been found to play an integral role in the effectiveness of social interactions. However, the motion-tracking equipment required to record and objectively measure the dynamic limb and body movements during social interaction has been very costly, cumbersome, and impractical within a non-clinical or non-laboratory setting. Here we examined whether three low-cost motion-tracking options (Microsoft Kinect skeletal tracking of either one limb or whole body and a video-based pixel change method) can be employed to investigate social motor coordination. Of particular interest was the degree to which these low-cost methods of motion tracking could be used to capture and index the coordination dynamics that occurred between a child and an experimenter for three simple social motor coordination tasks in comparison to a more expensive, laboratory-grade motion-tracking system (i.e., a Polhemus Latus system). Overall, the results demonstrated that these low-cost systems cannot substitute the Polhemus system in some tasks. However, the lower-cost Microsoft Kinect skeletal tracking and video pixel change methods were successfully able to index differences in social motor coordination in tasks that involved larger-scale, naturalistic whole body movements, which can be cumbersome and expensive to record with a Polhemus. However, we found the Kinect to be particularly vulnerable to occlusion and the pixel change method to movements that cross the video frame midline. Therefore, particular care needs to be taken in choosing the motion-tracking system that is best suited for the particular research.

Terrestrial support of aquatic food webs depends on light inputs: a geographically-replicated test using tank bromeliads.

Food webs of freshwater ecosystems can be subsidized by allochthonous resources. However, it is still unknown which environmental factors regulate the relative consumption of allochthonous resources in relation to autochthonous resources. Here, we evaluated the importance of allochthonous resources (litterfall) for the aquatic food webs in Neotropical tank bromeliads, a naturally replicated aquatic microcosm. Aquatic invertebrates were sampled in more than 100 bromeliads within either open or shaded habitats and within five geographically distinct sites located in four different countries. Using stable isotope analyses, we determined that allochthonous sources comprised 74% (±17%) of the food resources of aquatic invertebrates. However, the allochthonous contribution to aquatic invertebrates strongly decreased from shaded to open habitats, as light incidence increased in the tanks. The density of detritus in the tanks had no impact on the importance of allochthonous sources to aquatic invertebrates. This overall pattern held for all invertebrates, irrespective of the taxonomic or functional group to which they belonged. We concluded that, over a broad geographic range, aquatic food webs of tank bromeliads are mostly allochthonous-based, but the relative importance of allochthonous subsidies decreases when light incidence favors autochthonous primary production. These results suggest that, for other freshwater systems, some of the between-study variation in the importance of allochthonous subsidies may similarly be driven by the relative availability of autochthonous resources.

Dominant predators mediate the impact of habitat size on trophic structure in bromeliad invertebrate communities.

Local habitat size has been shown to influence colonization and extinction processes of species in patchy environments. However, species differ in body size, mobility, and trophic level, and may not respond in the same way to habitat size. Thus far, we have a limited understanding of how habitat size influences the structure of multitrophic communities and to what extent the effects may be generalizable over a broad geographic range. Here, we used water-filled bromeliads of different sizes as a natural model system to examine the effects of habitat size on the trophic structure of their inhabiting invertebrate communities. We collected composition and biomass data from 651 bromeliad communities from eight sites across Central and South America differing in environmental conditions, species pools, and the presence of large-bodied odonate predators. We found that trophic structure in the communities changed dramatically with changes in habitat (bromeliad) size. Detritivore : resource ratios showed a consistent negative relationship with habitat size across sites. In contrast, changes in predator: detritivore (prey) ratios depended on the presence of odonates as dominant predators in the regional pool. At sites without odonates, predator: detritivore biomass ratios decreased with increasing habitat size. At sites with odonates, we found odonates to be more frequently present in large than in small bromeliads, and predator: detritivore biomass ratios increased with increasing habitat size to the point where some trophic pyramids became inverted. Our results show that the distribution of biomass amongst food-web levels depends strongly on habitat size, largely irrespective of geographic differences in environmental conditions or detritivore species compositions. However, the presence of large-bodied predators in the regional species pool may fundamentally alter this relationship between habitat size and trophic structure. We conclude that taking into account the response and multitrophic effects of dominant, mobile species may be critical when predicting changes in community structure along a habitat-size gradient.

Interaction between intention and environmental constraints on the fractal dynamics of human performance.

The current study investigated whether the influence of available task constraints on power-law scaling might be moderated by a participant's task intention. Participants performed a simple rhythmic movement task with the intention of controlling either movement period or amplitude, either with or without an experimental stimulus designed to constrain period. In the absence of the stimulus, differences in intention did not produce any changes in power-law scaling. When the stimulus was present, however, a shift toward more random fluctuations occurred in the corresponding task dimension, regardless of participants' intentions. More importantly, participants' intentions interacted with available task constraints to produce an even greater shift toward random variation when the task dimension constrained by the stimulus was also the dimension the participant intended to control. Together, the results suggest that intentions serve to more tightly constrain behavior to existing environmental constraints, evidenced by changes in the fractal scaling of task performance.

Fractal coordination in adults' attention to hierarchical visual patterns.

A display that contains hierarchically nested levels of order requires the perceiver to selectively attend to one of the levels. We investigate the degree to which such selective attention is sustained by a soft-assembled emergent coordinative process, one that does not require designated executive control. In the case of emergent soft-assembly, performance from one trial to the next should show characteristic interdependence, visible in the fractal structure of reaction time. To test this hypothesis, we asked participants across three experiments to decide whether two displays matched in a certain way (e.g., in a local element). In order to gauge this coordinative process, task constraints were experimentally manipulated (e.g., familiarity, predictability, and task instruction). Obtained reaction-time data were subjected to a spectral analysis to measure the degree of interdependence among trials. As predicted, results show correlated structure across trials, significantly different from what would be predicted by an independent-process view of selective attention. Results also show that the obtained spectral scaling exponents track the degree of coupling in the task as a function of the degree of task constraints. Findings are discussed in terms of the relative organism-environment coupling to sustain an adaptive behavior.

Strategic communication and behavioral coupling in asymmetric joint action.

How is coordination achieved in asymmetric joint actions where co-actors have unequal access to task information? Pairs of participants performed a non-verbal tapping task with the goal of synchronizing taps to different targets. We tested whether 'Leaders' knowing the target locations would support 'Followers' without this information. Experiment 1 showed that Leaders tapped with higher amplitude that also scaled with specific target distance, thereby emphasizing differences between correct targets and possible alternatives. This strategic communication only occurred when Leaders' movements were fully visible, but not when they were partially occluded. Full visual information between co-actors also resulted in higher and more stable behavioral coordination than partial vision. Experiment 2 showed that Leaders' amplitude adaptation facilitated target prediction by independent Observers. We conclude that fully understanding joint action coordination requires both representational (i.e., strategic adaptation) and dynamical systems (i.e., behavioral coupling) accounts.

Initial phase and frequency modulations of pumping a playground swing.

The playground swing is a dynamic, coupled oscillator system consisting of the swing as an object and a human as the swinger. Here, we propose a model for capturing the effect of the initial phase of natural upper body motion on the continuous pumping of a swing and validate this model from the motion data of ten participants pumping swings of three different swing chain lengths. Our model predicts that the swing pumps the most if the phase of maximum lean back, which we call the initial phase, occurs when the swing is at a vertical (midpoint) position and moving forward when the amplitude is small. As the amplitude grows, the optimal initial phase gradually shifts towards an earlier phase of the cycle, the back extreme of the swing's trajectory. As predicted by our model, all participants shifted the initial phase of their upper body movements earlier as swing amplitude increased. This indicated that swingers adjust both the frequency and initial phase of their upper body movements to successfully pump a playground swing.

Predicting and understanding human action decisions during skillful joint-action using supervised machine learning and explainable-AI.

This study investigated the utility of supervised machine learning (SML) and explainable artificial intelligence (AI) techniques for modeling and understanding human decision-making during multiagent task performance. Long short-term memory (LSTM) networks were trained to predict the target selection decisions of expert and novice players completing a multiagent herding task. The results revealed that the trained LSTM models could not only accurately predict the target selection decisions of expert and novice players but that these predictions could be made at timescales that preceded a player's conscious intent. Importantly, the models were also expertise specific, in that models trained to predict the target selection decisions of experts could not accurately predict the target selection decisions of novices (and vice versa). To understand what differentiated expert and novice target selection decisions, we employed the explainable-AI technique, SHapley Additive explanation (SHAP), to identify what informational features (variables) most influenced modelpredictions. The SHAP analysis revealed that experts were more reliant on information about target direction of heading and the location of coherders (i.e., other players) compared to novices. The implications and assumptions underlying the use of SML and explainable-AI techniques for investigating and understanding human decision-making are discussed.

Behavioral dynamics of conversation, (mis)communication and coordination in noisy environments.

During conversations people coordinate simultaneous channels of verbal and nonverbal information to hear and be heard. But the presence of background noise levels such as those found in cafes and restaurants can be a barrier to conversational success. Here, we used speech and motion-tracking to reveal the reciprocal processes people use to communicate in noisy environments. Conversations between twenty-two pairs of typical-hearing adults were elicited under different conditions of background noise, while standing or sitting around a table. With the onset of background noise, pairs rapidly adjusted their interpersonal distance and speech level, with the degree of initial change dependent on noise level and talker configuration. Following this transient phase, pairs settled into a sustaining phase in which reciprocal speech and movement-based coordination processes synergistically maintained effective communication, again with the magnitude of stability of these coordination processes covarying with noise level and talker configuration. Finally, as communication breakdowns increased at high noise levels, pairs exhibited resetting behaviors to help restore communication-decreasing interpersonal distance and/or increasing speech levels in response to communication breakdowns. Approximately 78 dB SPL defined a threshold where behavioral processes were no longer sufficient for maintaining effective conversation and communication breakdowns rapidly increased.

Autistic young people adaptively use gaze to facilitate joint attention during multi-gestural dyadic interactions.

LAY ABSTRACT: Autistic people have been said to have 'problems' with joint attention, that is, looking where someone else is looking. Past studies of joint attention have used tasks that require autistic people to continuously look at and respond to eye-gaze cues. But joint attention can also be done using other social cues, like pointing. This study looked at whether autistic and non-autistic young people use another person's eye gaze during joint attention in a task that did not require them to look at their partner's face. In the task, each participant worked together with their partner to find a computer-generated object in virtual reality. Sometimes the participant had to help guide their partner to the object, and other times, they followed their partner's lead. Participants were told to point to guide one another but were not told to use eye gaze. Both autistic and non-autistic participants often looked at their partner's face during joint attention interactions and were faster to respond to their partner's hand-pointing when the partner also looked at the object before pointing. This shows that autistic people can and do use information from another person's eyes, even when they don't have to. It is possible that, by not forcing autistic young people to look at their partner's face and eyes, they were better able to gather information from their partner's face when needed, without being overwhelmed. This shows how important it is to design tasks that provide autistic people with opportunities to show what they can do.

Dynamical biomarkers in teams and other multiagent systems.

Effective team behavior in high-performance environments such as in sport and the military requires individual team members to efficiently perceive the unfolding task events, predict the actions and action intents of the other team members, and plan and execute their own actions to simultaneously accomplish individual and collective goals. To enhance team performance through effective cooperation, it is crucial to measure the situation awareness and dynamics of each team member and how they collectively impact the team's functioning. Further, to be practically useful for real-life settings, such measures must be easily obtainable from existing sensors. This paper presents several methodologies that can be used on positional and movement acceleration data of team members to quantify and/or predict team performance, assess situation awareness, and to help identify task-relevant information to support individual decision-making. Given the limited reporting of these methods within military cohorts, these methodologies are described using examples from team sports and teams training in virtual environments, with discussion as to how they can be applied to real-world military teams.

Location but not amount of stimulus occlusion influences the stability of visuomotor coordination.

The current study examined whether the amount and location of available movement information influenced the stability of visuomotor coordination. Participants coordinated a handheld pendulum with an oscillating visual stimulus in an inphase and antiphase manner. The effects of occluding different amounts of phase at different phase locations were examined. Occluding the 0°/180° phase locations (end-points) significantly increased the variability of the visuomotor coordination. The amount of occlusion had little or no affect on the stability of the coordination. We concluded that the end-points of a visual rhythm are privileged and provide access to movement information that ensures stable coordination. The results are discussed with respect to the proposal of Bingham (Ecol Psychol 16:45­43, 2004) and Wilson et al. (Exp Brain Res 165:351­361, 2005) that the relevant information for rhythmic visual coordination is relative direction information.

Validation of deep learning-based markerless 3D pose estimation.

Deep learning-based approaches to markerless 3D pose estimation are being adopted by researchers in psychology and neuroscience at an unprecedented rate. Yet many of these tools remain unvalidated. Here, we report on the validation of one increasingly popular tool (DeepLabCut) against simultaneous measurements obtained from a reference measurement system (Fastrak) with well-known performance characteristics. Our results confirm close (mm range) agreement between the two, indicating that under specific circumstances deep learning-based approaches can match more traditional motion tracking methods. Although more work needs to be done to determine their specific performance characteristics and limitations, this study should help build confidence within the research community using these new tools.

Conversation dynamics in a multiplayer video game with knowledge asymmetry.

Despite the challenges associated with virtually mediated communication, remote collaboration is a defining characteristic of online multiplayer gaming communities. Inspired by the teamwork exhibited by players in first-person shooter games, this study investigated the verbal and behavioral coordination of four-player teams playing a cooperative online video game. The game, Desert Herding, involved teams consisting of three ground players and one drone operator tasked to locate, corral, and contain evasive robot agents scattered across a large desert environment. Ground players could move throughout the environment, while the drone operator's role was akin to that of a "spectator" with a bird's-eye view, with access to veridical information of the locations of teammates and the to-be-corralled agents. Categorical recurrence quantification analysis (catRQA) was used to measure the communication dynamics of teams as they completed the task. Demands on coordination were manipulated by varying the ground players' ability to observe the environment with the use of game "fog." Results show that catRQA was sensitive to changes to task visibility, with reductions in task visibility reorganizing how participants conversed during the game to maintain team situation awareness. The results are discussed in the context of future work that can address how team coordination can be augmented with the inclusion of artificial agents, as synthetic teammates.

Assessing Team Effectiveness by How Players Structure Their Search in a First-Person Multiplayer Video Game.

People working as a team can achieve more than when working alone due to a team's ability to parallelize the completion of tasks. In collaborative search tasks, this necessitates the formation of effective division of labor strategies to minimize redundancies in search. For such strategies to be developed, team members need to perceive the task's relevant components and how they evolve over time, as well as an understanding of what others will do so that they can structure their own behavior to contribute to the team's goal. This study explored whether the capacity for team members to coordinate effectively can be related to how participants structure their search behaviors in an online multiplayer collaborative search task. Our results demonstrated that the structure of search behavior, quantified using detrended fluctuation analysis, was sensitive to contextual factors that limit a participant's ability to gather information. Further, increases in the persistence of movement fluctuations during search behavior were found as teams developed more effective coordinative strategies and were associated with better task performance.