A metacognitive confidence calibration (MCC) tool to help medical students scaffold diagnostic reasoning in decision-making during high-fidelity patient simulations.

The purpose of this study was to 1) help novice students scaffold problem-solving and engage safely in the deliberate practice of diagnostic reasoning and medical decision-making in real time; 2) assess how accurately students gather and apply data in medical reasoning and treatment during high-fidelity patient simulations (HFPSs); 3) identify students' scientific misconceptions related to the case; 4) promote student metacognitive processing, self-assessment, and self-efficacy; and 5) facilitate the explicit calibration of student confidence in deliberate reasoning with patient outcomes. In a mixed-method design, a metacognitive calibration self-assessing (MCC) survey tool was applied to HFPS (n = 80, 20 teams of 6 medical students) and semistructured interviews were conducted with faculty (n = 5). When scored by faculty with a rubric, the mean student accuracy ranged from 23% to 74%, whereas their self-assessment of confidence ranged from 71% to 86%. This result revealed overconfidence bias in novice students regarding the correctness of their wrong responses. The most common misconception identified was inverting cause and effect: metabolic acidosis was pointed to as the cause of the patient's problems rather than a consequence of untreated diabetes mellitus. The most common treatment error was overtreatment, with unnecessary added medication. Interviews with faculty suggested that the MCC tool improved the team process by slowing students down, requiring them to think through their answers, and that overall the tool improved their critical thinking. This study demonstrated the feasibility of using a metacognitive confidence calibration tool to assist novice students in learning safely to make deliberate diagnostic reasoning and decisions on patient care in real time during complex simulations while observing objectively their levels of psychological confidence against patient outcomes.NEW & NOTEWORTHY This study demonstrates the feasibility of a metacognitive confidence calibration tool (MCC) to assess and promote novices in the learning of diagnostic reasoning and treatment decisions on patient care in real time during high-fidelity patient simulations while comparing confidence and accuracy data and identifying students' scientific misconceptions. Results revealed the presence of overconfidence bias, overtreatment, and the misconception of metabolic acidosis as the cause of the patient's problems rather than a consequence of untreated diabetes mellitus.

Complex collaborative problem-solving processes in mission control.

INTRODUCTION: NASA's Mission Control Center (MCC) is responsible for control of the International Space Station (ISS), which includes responding to problems that obstruct the functioning of the ISS and that may pose a threat to the health and well-being of the flight crew. These problems are often complex, requiring individuals, teams, and multiteam systems, to work collaboratively. Research is warranted to examine individual and collaborative problem-solving processes in this context. Specifically, focus is placed on how Mission Control personnel-each with their own skills and responsibilities-exchange information to gain a shared understanding of the problem. The Macrocognition in Teams Model describes the processes that individuals and teams undertake in order to solve problems and may be applicable to Mission Control teams. METHOD: Semistructured interviews centering on a recent complex problem were conducted with seven MCC professionals. In order to assess collaborative problem-solving processes in MCC with those predicted by the Macrocognition in Teams Model, a coding scheme was developed to analyze the interview transcriptions. RESULTS: Findings are supported with excerpts from participant transcriptions and suggest that team knowledge-building processes accounted for approximately 50% of all coded data and are essential for successful collaborative problem solving in mission control. Support for the internalized and externalized team knowledge was also found (19% and 20%, respectively). DISCUSSION: The Macrocognition in Teams Model was shown to be a useful depiction of collaborative problem solving in mission control and further research with this as a guiding framework is warranted.

Individual and team profiling to support theory of mind in artificial social intelligence.

We describe an approach aimed at helping artificial intelligence develop theory of mind of their human teammates to support team interactions. We show how this can be supported through the provision of quantifiable, machine-readable, a priori information about the human team members to an agent. We first show how our profiling approach can capture individual team member characteristic profiles that can be constructed from sparse data and provided to agents to support the development of artificial theory of mind. We then show how it captures features of team composition that may influence team performance. We document this through an experiment examining factors influencing the performance of ad-hoc teams executing a complex team coordination task when paired with an artificial social intelligence (ASI) teammate. We report the relationship between the individual and team characteristics and measures related to task performance and self-reported perceptions of the ASI. The results show that individual and emergent team profiles were able to characterize features of the team that predicted behavior and explain differences in perceptions of ASI. Further, the features of these profiles may interact differently when teams work with human versus ASI advisors. Most strikingly, our analyses showed that ASI advisors had a strong positive impact on low potential teams such that they improved the performance of those teams across mission outcome measures. We discuss these findings in the context of developing intelligent technologies capable of social cognition and engage in collaborative behaviors that improve team effectiveness.

Perceptual training for visual search.

People are better at visual search than the best fully automated methods. Despite this, visual search remains a difficult perceptual task. The goal of this investigation was to experimentally test the ways in which visual search performance could be improved through two categories of training interventions: perceptual training and conceptual training. To determine the effects of each training on a later performance task, the two types of trainings were manipulated using a between-subjects design (conceptual vs. perceptual × training present vs. training absent). Perceptual training led to speed and accuracy improvements in visual search. Issues with the design and administration of the conceptual training limited conclusions on its effectiveness but provided useful lessons for conceptual training design. The results suggest that when the visual search task involves detecting heterogeneous or otherwise unpredictable stimuli, perceptual training can improve visual search performance. Similarly, careful consideration of the performance task and training design is required to evaluate the effectiveness of conceptual training. PRACTITIONER SUMMARY: Visual search is a difficult, yet critical, task in industries such as baggage screening and radiology. This study investigated the effectiveness of perceptual training for visual search. The results suggest that when visual search involves detecting heterogeneous or otherwise unpredictable stimuli, perceptual training may improve the speed and accuracy of visual search.

Supporting Artificial Social Intelligence With Theory of Mind.

In this paper, we discuss the development of artificial theory of mind as foundational to an agent's ability to collaborate with human team members. Agents imbued with artificial social intelligence will require various capabilities to gather the social data needed to inform an artificial theory of mind of their human counterparts. We draw from social signals theorizing and discuss a framework to guide consideration of core features of artificial social intelligence. We discuss how human social intelligence, and the development of theory of mind, can contribute to the development of artificial social intelligence by forming a foundation on which to help agents model, interpret and predict the behaviors and mental states of humans to support human-agent interaction. Artificial social intelligence will need the processing capabilities to perceive, interpret, and generate combinations of social cues to operate within a human-agent team. Artificial Theory of Mind affords a structure by which a socially intelligent agent could be imbued with the ability to model their human counterparts and engage in effective human-agent interaction. Further, modeling Artificial Theory of Mind can be used by an ASI to support transparent communication with humans, improving trust in agents, so that they may better predict future system behavior based on their understanding of and support trust in artificial socially intelligent agents.

EveryBOTy Counts: Examining Human-Machine Teams in Open Source Software Development.

In this study, we explore the future of work by examining differences in productivity when teams are composed of only humans or both humans and machine agents. Our objective was to characterize the similarities and differences between human and human-machine teams as they work to coordinate across their specialized roles. This form of research is increasingly important given that machine agents are becoming commonplace in sociotechnical systems and playing a more active role in collaborative work. One particular class of machine agents, bots, is being introduced to these systems to facilitate both taskwork and teamwork. We investigated the association between bots and productivity outcomes in open source software development through an analysis of hundreds of project teams. The presence of bots in teams was associated with higher levels of productivity and higher work centralization in addition to greater amounts of observed communication. The adoption of bots in software teams may have tradeoffs, in that doing so may increase productivity, but could also increase workload. We discuss the theoretical and practical implications of these findings for advancing human-machine teaming research.

Building a transdisciplinary expert consensus on the cognitive drivers of performance under pressure: An international multi-panel Delphi study.

Introduction: The ability to perform optimally under pressure is critical across many occupations, including the military, first responders, and competitive sport. Despite recognition that such performance depends on a range of cognitive factors, how common these factors are across performance domains remains unclear. The current study sought to integrate existing knowledge in the performance field in the form of a transdisciplinary expert consensus on the cognitive mechanisms that underlie performance under pressure. Methods: International experts were recruited from four performance domains [(i) Defense; (ii) Competitive Sport; (iii) Civilian High-stakes; and (iv) Performance Neuroscience]. Experts rated constructs from the Research Domain Criteria (RDoC) framework (and several expert-suggested constructs) across successive rounds, until all constructs reached consensus for inclusion or were eliminated. Finally, included constructs were ranked for their relative importance. Results: Sixty-eight experts completed the first Delphi round, with 94% of experts retained by the end of the Delphi process. The following 10 constructs reached consensus across all four panels (in order of overall ranking): (1) Attention; (2) Cognitive Control-Performance Monitoring; (3) Arousal and Regulatory Systems-Arousal; (4) Cognitive Control-Goal Selection, Updating, Representation, and Maintenance; (5) Cognitive Control-Response Selection and Inhibition/Suppression; (6) Working memory-Flexible Updating; (7) Working memory-Active Maintenance; (8) Perception and Understanding of Self-Self-knowledge; (9) Working memory-Interference Control, and (10) Expert-suggested-Shifting. Discussion: Our results identify a set of transdisciplinary neuroscience-informed constructs, validated through expert consensus. This expert consensus is critical to standardizing cognitive assessment and informing mechanism-targeted interventions in the broader field of human performance optimization.

Multiscale movement coordination dynamics in collaborative team problem solving.

During collaborative problem solving (CPS), coordination occurs at different spatial and temporal scales. This multiscale coordination should play a functional role in facilitating effective collaboration. To evaluate this, we conducted a study of computer-based CPS with 42 dyadic teams. We used cross-wavelet coherence to examine movement coordination, extracted from videos, at several scales, and tested whether the observed coordination was greater than expected due to chance and due to task demands. We found that coordination at scales less than 2s was greater than chance and at most scales (except 16s, 1m, and 2m), was greater than expected due to task demands. Lastly, we observed that coherence at .25s and 1s scales was predictive of performance. However, when including relative phase, our results suggest that higher in-phase movement coordination at the 1s scale was the strongest predictor of CPS performance. Further, we used growth curve modeling to examine how movement coordination changes across the duration of the task and whether this is moderated by CPS performance. We found that coordination over the duration of the CPS task is quadratic (a U shape) and that better performing teams have higher coordination with a shallower curve. We discuss these findings and their relevance to understanding how low-level movement coordination facilitates CPS.

Problem-Solving Phase Transitions During Team Collaboration.

Multiple theories of problem-solving hypothesize that there are distinct qualitative phases exhibited during effective problem-solving. However, limited research has attempted to identify when transitions between phases occur. We integrate theory on collaborative problem-solving (CPS) with dynamical systems theory suggesting that when a system is undergoing a phase transition it should exhibit a peak in entropy and that entropy levels should also relate to team performance. Communications from 40 teams that collaborated on a complex problem were coded for occurrence of problem-solving processes. We applied a sliding window entropy technique to each team's communications and specified criteria for (a) identifying data points that qualify as peaks and (b) determining which peaks were robust. We used multilevel modeling, and provide a qualitative example, to evaluate whether phases exhibit distinct distributions of communication processes. We also tested whether there was a relationship between entropy values at transition points and CPS performance. We found that a proportion of entropy peaks was robust and that the relative occurrence of communication codes varied significantly across phases. Peaks in entropy thus corresponded to qualitative shifts in teams' CPS communications, providing empirical evidence that teams exhibit phase transitions during CPS. Also, lower average levels of entropy at the phase transition points predicted better CPS performance. We specify future directions to improve understanding of phase transitions during CPS, and collaborative cognition, more broadly.

Interdisciplinary approaches for uncovering the impacts of architecture on collective behaviour.

Built structures, such as animal nests or buildings that humans occupy, serve two overarching purposes: shelter and a space where individuals interact. The former has dominated much of the discussion in the literature. But, as the study of collective behaviour expands, it is time to elucidate the role of the built environment in shaping collective outcomes. Collective behaviour in social animals emerges from interactions, and collective cognition in humans emerges from communication and coordination. These collective actions have vast economic implications in human societies and critical fitness consequences in animal systems. Despite the obvious influence of space on interactions, because spatial proximity is necessary for an interaction to occur, spatial constraints are rarely considered in studies of collective behaviour or collective cognition. An interdisciplinary exchange between behavioural ecologists, evolutionary biologists, cognitive scientists, social scientists, architects and engineers can facilitate a productive exchange of ideas, methods and theory that could lead us to uncover unifying principles and novel research approaches and questions in studies of animal and human collective behaviour. This article, along with those in this theme issue aims to formalize and catalyse this interdisciplinary exchange.This article is part of the theme issue 'Interdisciplinary approaches for uncovering the impacts of architecture on collective behaviour'.

Towards a computational model of social norms.

We describe a computational model of social norms based on identifying values that a certain culture finds desirable such as dignity, generosity and politeness. The model quantifies these values in the form of Culture-Sanctioned Social Metrics (CSSMs) and treats social norms as the requirement to maximize these metrics from the perspective of the self, peers and public. This model can be used to create realistic social simulations, to explain or predict human behavior in specific scenarios, or as a component of robots or agents that need to interact with humans in specific social-cultural settings. We validate the model by using it to represent a complex deception scenario and showing that it can yield non-trivial insights such as the explanation of apparently irrational human behavior.

Advancing the Science of Collaborative Problem Solving.

Collaborative problem solving (CPS) has been receiving increasing international attention because much of the complex work in the modern world is performed by teams. However, systematic education and training on CPS is lacking for those entering and participating in the workforce. In 2015, the Programme for International Student Assessment (PISA), a global test of educational progress, documented the low levels of proficiency in CPS. This result not only underscores a significant societal need but also presents an important opportunity for psychological scientists to develop, adopt, and implement theory and empirical research on CPS and to work with educators and policy experts to improve training in CPS. This article offers some directions for psychological science to participate in the growing attention to CPS throughout the world. First, it identifies the existing theoretical frameworks and empirical research that focus on CPS. Second, it provides examples of how recent technologies can automate analyses of CPS processes and assessments so that substantially larger data sets can be analyzed and so students can receive immediate feedback on their CPS performance. Third, it identifies some challenges, debates, and uncertainties in creating an infrastructure for research, education, and training in CPS. CPS education and assessment are expected to improve when supported by larger data sets and theoretical frameworks that are informed by psychological science. This will require interdisciplinary efforts that include expertise in psychological science, education, assessment, intelligent digital technologies, and policy.

The science of team science: A review of the empirical evidence and research gaps on collaboration in science.

Collaborations among researchers and across disciplinary, organizational, and cultural boundaries are vital to address increasingly complex challenges and opportunities in science and society. In addition, unprecedented technological advances create new opportunities to capitalize on a broader range of expertise and information in scientific collaborations. Yet rapid increases in the demand for scientific collaborations have outpaced changes in the factors needed to support teams in science, such as institutional structures and policies, scientific culture, and funding opportunities. The Science of Team Science (SciTS) field arose with the goal of empirically addressing questions from funding agencies, administrators, and scientists regarding the value of team science (TS) and strategies for successfully leading, engaging in, facilitating, and supporting science teams. Closely related fields have rich histories studying teams, groups, organizations, and management and have built a body of evidence for effective teaming in contexts such as industry and the military. Yet few studies had focused on science teams. Unique contextual factors within the scientific enterprise create an imperative to study these teams in context, and provide opportunities to advance understanding of other complex forms of collaboration. This review summarizes the empirical findings from the SciTS literature, which center around five key themes: the value of TS, team composition and its influence on TS performance, formation of science teams, team processes central to effective team functioning, and institutional influences on TS. Cross-cutting issues are discussed in the context of new research opportunities to further advance SciTS evidence and better inform policies and practices for effective TS. (PsycINFO Database Record

Augmenting team cognition in human-automation teams performing in complex operational environments.

There is a growing reliance on automation (e.g., intelligent agents, semi-autonomous robotic systems) to effectively execute increasingly cognitively complex tasks. Successful team performance for such tasks has become even more dependent on team cognition, addressing both human-human and human-automation teams. Team cognition can be viewed as the binding mechanism that produces coordinated behavior within experienced teams, emerging from the interplay between each team member's individual cognition and team process behaviors (e.g., coordination, communication). In order to better understand team cognition in human-automation teams, team performance models need to address issues surrounding the effect of human-agent and human-robot interaction on critical team processes such as coordination and communication. Toward this end, we present a preliminary theoretical framework illustrating how the design and implementation of automation technology may influence team cognition and team coordination in complex operational environments. Integrating constructs from organizational and cognitive science, our proposed framework outlines how information exchange and updating between humans and automation technology may affect lower-level (e.g., working memory) and higher-level (e.g., sense making) cognitive processes as well as teams' higher-order "metacognitive" processes (e.g., performance monitoring). Issues surrounding human-automation interaction are discussed and implications are presented within the context of designing automation technology to improve task performance in human-automation teams.

Organizational and training factors that promote team science: A qualitative analysis and application of theory to the National Institutes of Health's BIRCWH career development program.

INTRODUCTION: Research organizations face challenges in creating infrastructures that cultivates and sustains interdisciplinary team science. The objective of this paper is to identify structural elements of organizations and training that promote team science. METHODS: We qualitatively analyzed the National Institutes of Health's Building Interdisciplinary Research Careers in Women's Health, K12 using organizational psychology and team science theories to identify organizational design factors for successful team science and training. PRINCIPAL RESULTS: Seven key design elements support team science: (1) semiformal meta-organizational structure, (2) shared context and goals, (3) formal evaluation processes, (4) meetings to promote communication, (5) role clarity in mentoring, (6) building interpersonal competencies among faculty and trainees, and (7) designing promotion and tenure and other organizational processes to support interdisciplinary team science. CONCLUSION: This application of theory to a long-standing and successful program provides important foundational elements for programs and institutions to consider in promoting team science.

Technology as Teammate: Examining the Role of External Cognition in Support of Team Cognitive Processes.

In this paper we advance team theory by describing how cognition occurs across the distribution of members and the artifacts and technology that support their efforts. We draw from complementary theorizing coming out of cognitive engineering and cognitive science that views forms of cognition as external and extended and integrate this with theorizing on macrocognition in teams. Two frameworks are described that provide the groundwork for advancing theory and aid in the development of more precise measures for understanding team cognition via focus on artifacts and the technologies supporting their development and use. This includes distinctions between teamwork and taskwork and the notion of general and specific competencies from the organizational sciences along with the concepts of offloading and scaffolding from the cognitive sciences. This paper contributes to the team cognition literature along multiple lines. First, it aids theory development by synthesizing a broad set of perspectives on the varied forms of cognition emerging in complex collaborative contexts. Second, it supports research by providing diagnostic guidelines to study how artifacts are related to team cognition. Finally, it supports information systems designers by more precisely describing how to conceptualize team-supporting technology and artifacts. As such, it provides a means to more richly understand process and performance as it occurs within sociotechnical systems. Our overarching objective is to show how team cognition can both be more clearly conceptualized and more precisely measured by integrating theory from cognitive engineering and the cognitive and organizational sciences.

Prospects for direct social perception: a multi-theoretical integration to further the science of social cognition.

In this paper we suggest that differing approaches to the science of social cognition mirror the arguments between radical embodied and traditional approaches to cognition. We contrast the use in social cognition of theoretical inference and mental simulation mechanisms with approaches emphasizing a direct perception of others' mental states. We build from a recent integrative framework unifying these divergent perspectives through the use of dual-process theory and supporting social neuroscience research. Our elaboration considers two complementary notions of direct perception: one primarily stemming from ecological psychology and the other from enactive cognition theory. We use this as the foundation from which to offer an account of the informational basis for social information and assert a set of research propositions to further the science of social cognition. In doing so, we point out how perception of the minds of others can be supported in some cases by lawful information, supporting direct perception of social affordances and perhaps, mental states, and in other cases by cues that support indirect perceptual inference. Our goal is to extend accounts of social cognition by integrating advances across disciplines to provide a multi-level and multi-theoretic description that can advance this field and offer a means through which to reconcile radical embodied and traditional approaches to cognitive neuroscience.

Shifting the paradigm of music instruction: implications of embodiment stemming from an augmented reality guitar learning system.

Musical instruction often includes materials that can act as a barrier to learning. New technologies using augmented reality may aid in reducing the initial difficulties involved in learning music by lowering these barriers characteristic of traditional instructional materials. Therefore, this set of studies examined a novel augmented reality guitar learning system (i.e., the Fretlight® guitar) in regards to current theories of embodied music cognition. Specifically, we examined the effects of using this system in comparison to a standard instructional material (i.e., diagrams). First, we review major theories related to musical embodiment and specify a niche within this research space we call embodied music technology for learning. Following, we explicate two parallel experiments that were conducted to address the learning effects of this system. Experiment 1 examined short-term learning effects within one experimental session, while Experiment 2 examined both short-term and long-term effects across two sessions spaced at a 2-week interval. Analyses demonstrated that, for many of our dependent variables, all participants increased in performance across time. Further, the Fretlight® condition consistently led to significantly better outcomes via interactive effects, including significantly better long term retention for the learned information across a 2 week time interval. These results are discussed in the context of embodied cognition theory as it relates to music. Potential limitations and avenues for future research are described.