Putting it Together, Together.

People are not as fast or as strong as many other creatures that evolved around us. What gives us an evolutionary advantage is working together to achieve common aims. Coordinating joint action begins at a tender age with such cooperative activities as alternating babbling and clapping games. Adult joint activities are far more complex and use multiple means of coordination. Joint action has attracted qualitative analyses by sociolinguists, cognitive scientists, and philosophers as well as empirical analyses and theories by cognitive scientists. Here, we analyze how joint action is spontaneously coordinated from start to finish in a novel complex real-life joint activity, assembling a piece of furniture, a task that captures the essentials of joint action, collaborators, things in the world, and communicative devices. Pairs of strangers assembled a TV cart from a stack of parts and a photo of the completed cart. Coordination prior to each assembly action was coded as explicit, using speech or gesture, or implicit, actions that both advanced the task and communicated the next step. Initial planning relied on explicit communication about structure, but not action nor division of labor, which were improvised. That served to establish a joint representation of the goal that informed actions and monitored progress. As assembly progressed, coordination was increasingly implicit, through actions alone. Joint action is a dynamic interplay of explicit and implicit signaling with respect to things in the world to coordinate ongoing progress, guided by a shared representation of the goal.

Precueing Object Placement and Orientation for Manual Tasks in Augmented Reality.

When a user is performing a manual task, AR or VR can provide information about the current subtask (cueing) and upcoming subtasks (precueing) that makes them easier and faster to complete. Previous research on cueing and precueing in AR and VR has focused on path-following tasks requiring simple actions at each of a series of locations, such as pushing a button or just visiting. We consider a more complex task, whose subtasks involve moving to and picking up an item, moving that item to a designated place while rotating it to a specific angle, and depositing it. We conducted two user studies to examine how people accomplish this task while wearing an AR headset, guided by different visualizations that cue and precue movement and rotation. Participants performed best when given movement information for two successive subtasks and rotation information for a single subtask. In addition, participants performed best when the rotation visualization was split across the manipulated object and its destination.

Using Multi-Level Precueing to Improve Performance in Path-Following Tasks in Virtual Reality.

Work on VR and AR task interaction and visualization paradigms has typically focused on providing information about the current step (a cue) immediately before or during its performance. Some research has also shown benefits to simultaneously providing information about the next step (a precue). We explore whether it would be possible to improve efficiency by precueing information about multiple upcoming steps before completing the current step. To accomplish this, we developed a remote VR user study comparing task completion time and subjective metrics for different levels and styles of precueing in a path-following task. Our visualizations vary the precueing level (number of steps precued in advance) and style (whether the path to a target is communicated through a line to the target, and whether the place of a target is communicated through graphics at the target). Participants in our study performed best when given two to three precues for visualizations using lines to show the path to targets. However, performance degraded when four precues were used. On the other hand, participants performed best with only one precue for visualizations without lines, showing only the places of targets, and performance degraded when a second precue was given. In addition, participants performed better using visualizations with lines than ones without lines.

Thinking Tools: Gestures Change Thought About Time.

Our earliest tools are our bodies. Our hands raise and turn and toss and carry and push and pull, our legs walk and climb and kick allowing us to move and act in the world and to create the multitude of artifacts that improve our lives. The list of actions made by our hands and feet and other parts of our bodies is long. What is more remarkable is we turn those actions in the world into actions on thought through gestures, language, and graphics, thereby creating cognitive tools that expand the mind. The focus here is gesture; gestures transform actions on perceptible objects to actions on imagined thoughts, carrying meaning with them rapidly, precisely, and directly. We review evidence showing that gestures enhance our own thinking and change the thought of others. We illustrate the power of gestures in studies showing that gestures uniquely change conceptions of time, from sequential to simultaneous, from sequential to cyclical, and from a perspective embedded in a timeline to an external perspective looking on a timeline, and by so doing obviate the ambiguities of an embedded perspective. We draw parallels between representations in gesture and in graphics; both use marks or actions arrayed in space to communicate more immediately than symbolic language.

Anne Marie Treisman (1937-2018).

Memorializes Anne Marie Treisman (1937-2018). Her 1962 dissertation at Oxford University included a remarkable 14 experiments and yielded 11 publications. This impressive body of work convinced Donald Broadbent and others to accept Anne's revised theory of attention in which unattended information was attenuated but not blocked. Many of the experiments rapidly made their way into textbooks and into the classroom. Although her work on attention and feature integration is best known, Anne's other contributions were prolific, insightful, and broad. Throughout her career, Anne was showered with awards, among them awards from the Royal Society, the National Academy of Sciences, and the British Academy, as well as the National Medal of Science, presented in 2013 by President Barack Obama. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Changing Perspective: Building Creative Mindsets.

The search for new ideas often frustratingly cycles back to old ones, a phenomenon known as fixation. Recent research has shown ways to kick-start finding new uses for familiar objects, a prototypical creativity task: wandering in the mind or the world or working on a messy desk. Those techniques seem to succeed by helping break fixation, but do not guide the search for new ideas. The perspective-taking or human-centric or empathic mindset championed by many in HCI and in design firms does provide a search strategy. We compared the mind-wandering mindset to a perspective-taking mindset, the latter priming thinking of ways that people in different roles (gardener, artist, etc.) might use the objects. In two studies, the Perspective-Taking mindset yielded more ideas and more original ideas than Mind-Wandering, which did not differ from a No-Mindset control. Original ideas came late, rewarding persistence. The perspective-taking mindset is productive for problem-solving, forecasting, and social interactions as well as innovation.

The kinematics that you do not expect: Integrating prior information and kinematics to understand intentions.

Expectations facilitate perception of expected stimuli but may hinder perception of unexpected alternatives. Here, we consider how prior expectations about others' intentions are integrated with visual kinematics over time in detecting the intention of an observed motor act (grasp-to-pour vs. grasp-to-drink). Using rigorous psychophysics methods, we find that the processes of ascribing intentions to others are well described by drift diffusion models in which evidence from observed movements is accumulated over time until a decision threshold is reached. Testing of competing models revealed that when kinematics contained no discriminative intention information, prior expectations predicted the intention choice of the observer. When kinematics contained intention information, kinematics predicted the intention choice. These findings provide evidence for a diffusion process in which the influence of expectations is modulated by movement informativeness and informative kinematics can override initial expectations.

Gestures can create diagrams (that are neither imagistic nor analog).

The claim that gesture is primarily imagistic, analog, and holistic is challenged by the presence of abstract diagrammatic gestures, here points and lines, that represent point-like and line-like concepts and are integrated into larger constituents.

When Far Becomes Near.

On many occasions, people spontaneously or deliberately take the perspective of a person facing them rather than their own perspective. How is this done? Using a spatial perspective task in which participants were asked to identify objects at specific locations, we found that self-perspective judgments were faster for objects presented to the right, rather than the left, and for objects presented closer to the participants' own bodies. Strikingly, taking the opposing perspective of another person led to a reversal (i.e., remapping) of these effects, with reference to the other person's position (Experiment 1). A remapping of spatial relations was also observed when an empty chair replaced the other person (Experiment 2), but not when access to the other viewpoint was blocked (Experiment 3). Thus, when the spatial scene allows a physically feasible but opposing point of view, people respond as if their own bodies were in that place. Imagination can thus overcome perception.

From hands to minds: Gestures promote understanding.

Gestures serve many roles in communication, learning and understanding both for those who view them and those who create them. Gestures are especially effective when they bear resemblance to the thought they represent, an advantage they have over words. Here, we examine the role of conceptually congruent gestures in deepening understanding of dynamic systems. Understanding the structure of dynamic systems is relatively easy, but understanding the actions of dynamic systems can be challenging. We found that seeing gestures representing actions enhanced understanding of the dynamics of a complex system as revealed in invented language, gestures and visual explanations. Gestures can map many meanings more directly than language, representing many concepts congruently. Designing and using gestures congruent with meaning can augment comprehension and learning.

Creating visual explanations improves learning.

Many topics in science are notoriously difficult for students to learn. Mechanisms and processes outside student experience present particular challenges. While instruction typically involves visualizations, students usually explain in words. Because visual explanations can show parts and processes of complex systems directly, creating them should have benefits beyond creating verbal explanations. We compared learning from creating visual or verbal explanations for two STEM domains, a mechanical system (bicycle pump) and a chemical system (bonding). Both kinds of explanations were analyzed for content and learning assess by a post-test. For the mechanical system, creating a visual explanation increased understanding particularly for participants of low spatial ability. For the chemical system, creating both visual and verbal explanations improved learning without new teaching. Creating a visual explanation was superior and benefitted participants of both high and low spatial ability. Visual explanations often included crucial yet invisible features. The greater effectiveness of visual explanations appears attributable to the checks they provide for completeness and coherence as well as to their roles as platforms for inference. The benefits should generalize to other domains like the social sciences, history, and archeology where important information can be visualized. Together, the findings provide support for the use of learner-generated visual explanations as a powerful learning tool.

Through your eyes: incongruence of gaze and action increases spontaneous perspective taking.

What makes people spontaneously adopt the perspective of others? Previous work suggested that perspective taking can serve understanding the actions of others. Two studies corroborate and extend that interpretation. The first study varied cues to intentionality of eye gaze and action, and found that the more the actor was perceived as potentially interacting with the objects, the stronger the tendency to take his perspective. The second study investigated how manipulations of gaze affect the tendency to adopt the perspective of another reaching for an object. Eliminating gaze cues by blurring the actor's face did not reduce perspective-taking, suggesting that in the absence of gaze information, observers rely entirely on the action. Intriguingly, perspective-taking was higher when gaze and action did not signal the same intention, suggesting that in presence of ambiguous behavioral intention, people are more likely take the other's perspective to try to understand the action.

Cognitive tools shape thought: diagrams in design.

Thinking often entails interacting with cognitive tools. In many cases, notably design, the predominant tool is the page. The page allows externalizing, organizing, and reorganizing thought. Yet, the page has its own properties that by expressing thought affect it: path, proximity, place, and permanence. The effects of these properties were evident in designs of information systems created by students Paths were interpreted as routes through components. Proximity was used to group subsystems. Horizontal position on the page was used to express temporal sequence and vertical position to reflect real-world spatial position. The permanence of designs on the page guided but also constrained generation of alternative designs. Cognitive tools both reflect and affect thought.

How to put things together.

Instructions for putting things together or understanding how things work are notoriously frustrating. Performance relies on constructing mental models of the object and the actions of the object from text or diagrams or both. Here, we show that instructions can be improved by turning users into designers and deriving design principles from their designs. People first assembled an object and then crafted assembly instructions, using text alone or text and diagrams. Some were required to be brief and to include only the most essential information. Users' instructions had a narrative structure with an introduction, a middle, and an end. The essential middle described or depicted the step-by-step sequence of actions on parts. Diagrams were regarded as fundamental, and redundancy of depictions and descriptions desirable. These design principles have wide applicability to many kinds of explanations.

The shape of action.

How do people understand the everyday, yet intricate, behaviors that unfold around them? In the present research, we explored this by presenting viewers with self-paced slideshows of everyday activities and recording looking times, subjective segmentation (breakpoints) into action units, and slide-to-slide physical change. A detailed comparison of the joint time courses of these variables showed that looking time and physical change were locally maximal at breakpoints and greater for higher level action units than for lower level units. Even when slideshows were scrambled, breakpoints were regarded longer and were more physically different from ordinary moments, showing that breakpoints are distinct even out of context. Breakpoints are bridges: from one action to another, from one level to another, and from perception to conception.

Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks.

Degree of pupil dilation has been shown to be a valid and reliable measure of cognitive load, but the effect of aural versus visual task presentation on pupil dilation is unknown. To evaluate effects of presentation mode, pupil dilation was measured in three tasks spanning a range of cognitive activities: mental multiplication, digit sequence recall, and vigilance. Stimuli were presented both aurally and visually, controlling for all known visual influences on pupil diameter. The patterns of dilation were similar for both aural and visual presentation for all three tasks, but the magnitudes of pupil response were greater for aural presentation. Accuracy was higher for visual presentation for mental arithmetic and digit recall. The findings can be accounted for in terms of dual codes in working memory and suggest that cognitive load is lower for visual than for aural presentation.

Visualizing thought.

Depictive expressions of thought predate written language by thousands of years. They have evolved in communities through a kind of informal user testing that has refined them. Analyzing common visual communications reveals consistencies that illuminate how people think as well as guide design; the process can be brought into the laboratory and accelerated. Like language, visual communications abstract and schematize; unlike language, they use properties of the page (e.g., proximity and place: center, horizontal/up-down, vertical/left-right) and the marks on it (e.g., dots, lines, arrows, boxes, blobs, likenesses, symbols) to convey meanings. The visual expressions of these meanings (e.g., individual, category, order, relation, correspondence, continuum, hierarchy) have analogs in language, gesture, and especially in the patterns that are created when people design the world around them, arranging things into piles and rows and hierarchies and arrays, spatial-abstraction-action interconnections termed spractions. The designed world is a diagram.

Visualizing space, time, and agents: production, performance, and preference.

Visualizations of space, time, and agents (or objects) are ubiquitous in science, business, and everyday life, from weather maps to scheduling meetings. Effective communications, including visual ones, emerge from use in the field, but no conventional visualization form has yet emerged for this confluence of information. The real-world spiral of production, comprehension, and use that fine-tunes communications can be accelerated in the laboratory. Here, we do so in search of effective visualizations of space, time, and agents. Users' production, preference, and performance aligned to favor matrix representations with time as rows or columns and space and agents as entries. Overall, performance and preference were greater for matrices with discrete dots representing cell entries than for matrices with lines, but lines connecting cells may provide an advantage when evaluating temporal sequence. Both the diagram type and the technique have broader applications.