(Mis)align: a simple dynamic framework for modeling interpersonal coordination.

As people coordinate in daily interactions, they engage in different patterns of behavior to achieve successful outcomes. This includes both synchrony-the temporal coordination of the same behaviors at the same time-and complementarity-the coordination of the same or different behaviors that may occur at different relative times. Using computational methods, we develop a simple framework to describe the interpersonal dynamics of behavioral synchrony and complementarity over time, and explore their task-dependence. A key feature of this framework is the inclusion of a task context that mediates interactions, and consists of active, inactive, and inhibitory constraints on communication. Initial simulation results show that these task constraints can be a robust predictor of simulated agents' behaviors over time. We also show that the framework can reproduce some general patterns observed in human interaction data. We describe preliminary theoretical implications from these results, and relate them to broader proposals of synergistic self-organization in communication.

What is retained about common ground? Distinct effects of linguistic and visual co-presence.

Common ground can be mutually established between conversational partners in several ways. We examined whether the modality (visual or linguistic) with which speakers share information with their conversational partners results in memory traces that affect subsequent references addressed to a particular partner. In 32 triads, directors arranged a set of tangram cards with one matcher and then with another, but in different modalities, sharing some cards only linguistically (by describing cards the matcher couldn't see), some only visually (by silently showing them), some both linguistically and visually, and others not at all. Then directors arranged the cards again in separate rounds with each matcher. The modality with which they previously established common ground about a particular card with a particular matcher (e.g., linguistically with one partner and visually with the other) affected subsequent referring: References to cards previously shared only visually included more idea units, words, and reconceptualizations than those shared only linguistically, which in turn included more idea units, words, and reconceptualizations than those shared both linguistically and visually. Moreover, speakers were able to tailor references to the same card appropriately to the distinct modality shared with each addressee. Such gradient, partner-specific adaptation during re-referring suggests that memory encodes rich-enough representations of multimodal shared experiences to effectively cue relevant constraints about the perceptual conditions under which speakers and addressees establish common ground.

Exploring the SenseMaking Process through Interactions and fNIRS in Immersive Visualization.

Theories of cognition inform our decisions when designing human-computer interfaces, and immersive systems enable us to examine these theories. This work explores the sensemaking process in an immersive environment through studying both internal and external user behaviors with a classical visualization problem: a visual comparison and clustering task. We developed an immersive system to perform a user study, collecting user behavior data from different channels: AR HMD for capturing external user interactions, functional near-infrared spectroscopy (fNIRS) for capturing internal neural sequences, and video for references. To examine sensemaking, we assessed how the layout of the interface (planar 2D vs. cylindrical 3D layout) and the challenge level of the task (low vs. high cognitive load) influenced the users' interactions, how these interactions changed over time, and how they influenced task performance. We also developed a visualization system to explore joint patterns among all the data channels. We found that increased interactions and cerebral hemodynamic responses were associated with more accurate performance, especially on cognitively demanding trials. The layout types did not reliably influence interactions or task performance. We discuss how these findings inform the design and evaluation of immersive systems, predict user performance and interaction, and offer theoretical insights about sensemaking from the perspective of embodied and distributed cognition.

Are Spatial Memories for Familiar Environments Orientation Dependent?

In one experiment we examined the organizational structure of spatial memories for familiar environments, comparing it directly with that for unfamiliar environments. Participants in the familiar condition pointed from imagined perspectives towards objects in their own rooms and their performance was compared to that of matched controls in an unfamiliar condition who carried out the same task after studying the same rooms in immersive Virtual Reality. In both conditions, participants were faster and more accurate in pointing from imagined perspectives that were aligned with the geometry of the room (vs. not aligned), suggesting the presence of orientation-dependent representations. Whereas in the unfamiliar condition pointing performance was best along a single axis, performance in the familiar condition was about equal across all 4 orientations that were aligned with the geometric structure of the room. Moreover, performance in the familiar condition was influenced by the orientation from which participants started to preview the room prior to testing; in contrast, in the unfamiliar condition performance was not influenced by the orientation from which encoding started. This finding suggests that post-encoding situational factors (e.g., the starting orientation from which an environment is previewed) can prime the accessibility of information in well-established long-term spatial memories.

Complex Communication Dynamics: Exploring the Structure of an Academic Talk.

Communication is a multimodal phenomenon. The cognitive mechanisms supporting it are still understudied. We explored a natural dataset of academic lectures to determine how communication modalities are used and coordinated during the presentation of complex information. Using automated and semi-automated techniques, we extracted and analyzed, from the videos of 30 speakers, measures capturing the dynamics of their body movement, their slide change rate, and various aspects of their speech (speech rate, articulation rate, fundamental frequency, and intensity). There were consistent but statistically subtle patterns in the use of speech rate, articulation rate, intensity, and body motion across the presentation. Principal component analysis also revealed patterns of system-like covariation among modalities. These findings, although tentative, do suggest that the cognitive system is integrating body, slides, and speech in a coordinated manner during natural language use. Further research is needed to clarify the specific coordination patterns that occur between the different modalities.

Interacting Timescales in Perspective-Taking.

Through theoretical discussion, literature review, and a computational model, this paper poses a challenge to the notion that perspective-taking involves a fixed architecture in which particular processes have priority. For example, some research suggests that egocentric perspectives can arise more quickly, with other perspectives (such as of task partners) emerging only secondarily. This theoretical dichotomy-between fast egocentric and slow other-centric processes-is challenged here. We propose a general view of perspective-taking as an emergent phenomenon governed by the interplay among cognitive mechanisms that accumulate information at different timescales. We first describe the pervasive relevance of perspective-taking to cognitive science. A dynamical systems model is then introduced that explicitly formulates the timescale interaction proposed. This model illustrates that, rather than having a rigid time course, perspective-taking can be fast or slow depending on factors such as task context. Implications are discussed, with ideas for future empirical research.

Toward Integrative Dynamic Models for Adaptive Perspective Taking.

In a matter of mere milliseconds, conversational partners can transform their expectations about the world in a way that accords with another person's perspective. At the same time, in similar situations, the exact opposite also appears to be true. Rather than being at odds, these findings suggest that there are multiple contextual and processing constraints that may guide when and how people consider perspective. These constraints are shaped by a host of factors, including the availability of social and environmental cues, and intrinsic biases and cognitive abilities. To explain how these might be integrated in a new way forward, we turn to an adaptive account of interpersonal interaction. This account draws from basic principles of dynamical systems, principles that we argue are already expressed, both implicitly and explicitly, within a broad landscape of existing research. We then showcase an initial attempt to develop a computational framework to instantiate some of these principles. This framework, consisting of what we argue to be important mechanistic insights rendered by neural network models, is based on a promising and long-standing approach that has yet to take hold in the current domain. We argue that by bridging this gap, new insights into other theoretical accounts, such as the connections between memory and common ground information, might be revealed.

Updating spatial relations to remote locations described in narratives.

In four experiments we examined whether sensorimotor encoding influences readers' reasoning about spatial scenes acquired through narratives. Participants read a narrative that described the geometry of a store and then pointed to the memorized locations of described objects from imagined perspectives. In Experiment 1, participants walked during learning towards the direction of every described object and then visualized these objects as being in the immediate environment. In Experiment 2 they rotated their body to the direction of the described objects instead of walking to them, while in Experiment 3 they only turned their heads towards the objects. In Experiment 4, we eliminated the instructions to visualize the objects altogether. Results from the first three experiments revealed a performance benefit for responding from the perspective that participants physically occupied at testing. However, results from Experiment 4 showed that only participants who, in a post-task questionnaire, indicated that they had linked the described environment to their immediate environment exhibited such a benefit. Findings indicate that (1) the physical change in orientation influences reasoning about described environments if the remote environments are linked to participants' sensorimotor framework and, (2) visualization instructions are sufficient to produce such a link.

The protagonist's first perspective influences the encoding of spatial information in narratives.

Three experiments examined the first-perspective alignment effect that is observed when retrieving spatial information from memory about described environments. Participants read narratives that described the viewpoint of a protagonist in fictitious environments and then pointed to the memorized locations of described objects from imagined perspectives. Results from Experiments 1 and 2 showed that performance was best when participants responded from the protagonist's first perspective even though object locations were described from a different perspective. In Experiment 3, in which participants were physically oriented with the perspective used to describe object locations, performance from that description perspective was better than that from the protagonist's first perspective, which was, in turn, better than performance from other perspectives. These findings suggest that when reading narratives, people default to using a reference frame that is aligned with their own facing direction, although physical movement may facilitate retrieval from other perspectives.

What's so difficult with adopting imagined perspectives?

Research on spatial cognition suggests that transformation processes and/or spatial conflicts may influence performance on mental perspective-taking tasks. However, conflicting findings have complicated our understanding about the processes involved in perspective-taking, particularly those giving rise to angular disparity effects, whereby performance worsens as the imagined perspective adopted deviates from one's actual perspective. Based on data from experiments involving mental perspective-taking in immediate and remote spatial layouts, we propose here a novel account for the difficulty with perspective-taking. According to this account, the main difficulty lies in maintaining an imagined perspective in working memory, especially in the presence of salient sensorimotor information.

Social and representational cues jointly influence spatial perspective-taking.

We examined how social cues (the conversational partner's viewpoint) and representational ones (the intrinsic structure of a spatial layout) jointly shape people's spatial memory representations and their subsequent descriptions. In 24 pairs, Directors studied an array with a symmetrical structure while either knowing their Matcher's subsequent viewpoint or not. During the subsequent description of the array, the array's intrinsic structure was aligned with the Director, the Matcher, or neither partner. According to memory tests preceding descriptions, Directors who had studied the array while aligned with its structure were more likely to use its orientation as an organizing direction. Directors who had studied the array while misaligned with its structure used its orientation more frequently as an organizing orientation when knowing that the Matcher would be aligned with it, but used their own viewpoint more frequently as an organizing direction when not knowing the Matcher's viewpoint. Directors also adapted their descriptions strategically, using more egocentric expressions when aligned with the intrinsic structure and more partner-centered expressions when their Matchers were the ones aligned with the structure, even when this information wasn't available in advance. These findings suggest that speakers are guided by converging social and representational cues to adapt flexibly the organization of their memories and the perspectives of their descriptions.

Flexible spatial perspective-taking: conversational partners weigh multiple cues in collaborative tasks.

Research on spatial perspective-taking often focuses on the cognitive processes of isolated individuals as they adopt or maintain imagined perspectives. Collaborative studies of spatial perspective-taking typically examine speakers' linguistic choices, while overlooking their underlying processes and representations. We review evidence from two collaborative experiments that examine the contribution of social and representational cues to spatial perspective choices in both language and the organization of spatial memory. Across experiments, speakers organized their memory representations according to the convergence of various cues. When layouts were randomly configured and did not afford intrinsic cues, speakers encoded their partner's viewpoint in memory, if available, but did not use it as an organizing direction. On the other hand, when the layout afforded an intrinsic structure, speakers organized their spatial memories according to the person-centered perspective reinforced by the layout's structure. Similarly, in descriptions, speakers considered multiple cues whether available a priori or at the interaction. They used partner-centered expressions more frequently (e.g., "to your right") when the partner's viewpoint was misaligned by a small offset or coincided with the layout's structure. Conversely, they used egocentric expressions more frequently when their own viewpoint coincided with the intrinsic structure or when the partner was misaligned by a computationally difficult, oblique offset. Based on these findings we advocate for a framework for flexible perspective-taking: people weigh multiple cues (including social ones) to make attributions about the relative difficulty of perspective-taking for each partner, and adapt behavior to minimize their collective effort. This framework is not specialized for spatial reasoning but instead emerges from the same principles and memory-depended processes that govern perspective-taking in non-spatial tasks.

Collaborating in spatial tasks: how partners coordinate their spatial memories and descriptions.

We summarize findings from a study examining whether the availability of the conversational partner's spatial viewpoint influences the speaker's spatial memories, description strategies, their joint efficiency and accuracy on the task, as well as the partner's resulting spatial memories. In 18 pairs, Directors described to a misaligned Matcher arrays that they learned while either knowing their Matcher's viewpoint or not. Memory tests preceding descriptions revealed that Directors represented their Matcher's viewpoint when known in advance. Moreover, Directors adapted the perspective of their descriptions according to each other's cognitive demands, given their misalignment. The number of conversational turns pairs took to coordinate suggested that pairs' strategies were effective at minimizing their collective effort. Nonetheless, in terms of accuracy on the task, pairs reconstructed more distorted arrays the more partner-centered descriptions Directors used. The Directors' descriptions also predicted Matchers' facilitation for their own perspective in memory tests following the description. Together, these findings demonstrate that partners in collaborative spatial tasks adapt their respective memory representations and descriptions contingently with the aim of optimizing coordination.

Egocentric updating of remote locations.

The study examined whether people update remote spatial locations in unfamiliar environments during physical movement. Participants learned a layout of objects from one perspective and carried out perspective-taking trials after physically rotating to a new perspective in either the same room as learning or in an adjacent room. Prior to rotation in the adjacent room participants were instructed to visualize the objects as being around them. Responses to perspective-taking trials involved either pointing or verbal labeling. In both testing environments, participants pointed more efficiently from imagined perspectives aligned with either the initial learning perspective or their current facing orientation than from a novel imagined perspective; this indicates that they had updated the encoded spatial relations during the physical rotation and treated remote objects as immediate. Differences in performance among perspectives were less pronounced for verbal labeling in both environments, suggesting that this response mode is more flexibly used from imagined perspectives.

Encoding and updating spatial information presented in narratives.

Four experiments investigated whether directional spatial relations encoded by reading narratives are updated following described protagonist rotations. Participants memorized locations of objects described in short stories that placed them, as the protagonist, in remote settings. After reading a description that the protagonist rotated to the left or the right of the initial orientation, participants made judgements about object relations in the described environment (Experiment 1). Before making these judgements, participants were instructed to physically rotate to match (Experiment 2) or mismatch (Experiment 4) the protagonist's described rotation and in Experiments 3 and 4 to also visualize the changed relations following rotation. Participants' performance suggested that they relied on the initial representation they constructed during encoding rather than on the updated protagonist-to-object relations. Participants' physical movement to match the described rotation and additional visualization instructions did not facilitate updating through a sensorimotor process. In these respects, updating spatial relations in situation models constructed from narratives differs from updating in perceptually experienced environments.