Human-AI Teaming in Critical Care: A Comparative Analysis of Data Scientists' and Clinicians' Perspectives on AI Augmentation and Automation.

BACKGROUND: Artificial intelligence (AI) holds immense potential for enhancing clinical and administrative health care tasks. However, slow adoption and implementation challenges highlight the need to consider how humans can effectively collaborate with AI within broader socio-technical systems in health care. OBJECTIVE: In the example of intensive care units (ICUs), we compare data scientists' and clinicians' assessments of the optimal utilization of human and AI capabilities by determining suitable levels of human-AI teaming for safely and meaningfully augmenting or automating 6 core tasks. The goal is to provide actionable recommendations for policy makers and health care practitioners regarding AI design and implementation. METHODS: In this multimethod study, we combine a systematic task analysis across 6 ICUs with an international Delphi survey involving 19 health data scientists from the industry and academia and 61 ICU clinicians (25 physicians and 36 nurses) to define and assess optimal levels of human-AI teaming (level 1=no performance benefits; level 2=AI augments human performance; level 3=humans augment AI performance; level 4=AI performs without human input). Stakeholder groups also considered ethical and social implications. RESULTS: Both stakeholder groups chose level 2 and 3 human-AI teaming for 4 out of 6 core tasks in the ICU. For one task (monitoring), level 4 was the preferred design choice. For the task of patient interactions, both data scientists and clinicians agreed that AI should not be used regardless of technological feasibility due to the importance of the physician-patient and nurse-patient relationship and ethical concerns. Human-AI design choices rely on interpretability, predictability, and control over AI systems. If these conditions are not met and AI performs below human-level reliability, a reduction to level 1 or shifting accountability away from human end users is advised. If AI performs at or beyond human-level reliability and these conditions are not met, shifting to level 4 automation should be considered to ensure safe and efficient human-AI teaming. CONCLUSIONS: By considering the sociotechnical system and determining appropriate levels of human-AI teaming, our study showcases the potential for improving the safety and effectiveness of AI usage in ICUs and broader health care settings. Regulatory measures should prioritize interpretability, predictability, and control if clinicians hold full accountability. Ethical and social implications must be carefully evaluated to ensure effective collaboration between humans and AI, particularly considering the most recent advancements in generative AI.

Effects of interacting with a large language model compared with a human coach on the clinical diagnostic process and outcomes among fourth-year medical students: study protocol for a prospective, randomised experiment using patient vignettes.

INTRODUCTION: Versatile large language models (LLMs) have the potential to augment diagnostic decision-making by assisting diagnosticians, thanks to their ability to engage in open-ended, natural conversations and their comprehensive knowledge access. Yet the novelty of LLMs in diagnostic decision-making introduces uncertainties regarding their impact. Clinicians unfamiliar with the use of LLMs in their professional context may rely on general attitudes towards LLMs more broadly, potentially hindering thoughtful use and critical evaluation of their input, leading to either over-reliance and lack of critical thinking or an unwillingness to use LLMs as diagnostic aids. To address these concerns, this study examines the influence on the diagnostic process and outcomes of interacting with an LLM compared with a human coach, and of prior training vs no training for interacting with either of these 'coaches'. Our findings aim to illuminate the potential benefits and risks of employing artificial intelligence (AI) in diagnostic decision-making. METHODS AND ANALYSIS: We are conducting a prospective, randomised experiment with N=158 fourth-year medical students from Charité Medical School, Berlin, Germany. Participants are asked to diagnose patient vignettes after being assigned to either a human coach or ChatGPT and after either training or no training (both between-subject factors). We are specifically collecting data on the effects of using either of these 'coaches' and of additional training on information search, number of hypotheses entertained, diagnostic accuracy and confidence. Statistical methods will include linear mixed effects models. Exploratory analyses of the interaction patterns and attitudes towards AI will also generate more generalisable knowledge about the role of AI in medicine. ETHICS AND DISSEMINATION: The Bern Cantonal Ethics Committee considered the study exempt from full ethical review (BASEC No: Req-2023-01396). All methods will be conducted in accordance with relevant guidelines and regulations. Participation is voluntary and informed consent will be obtained. Results will be published in peer-reviewed scientific medical journals. Authorship will be determined according to the International Committee of Medical Journal Editors guidelines.

Choosing human over AI doctors? How comparative trust associations and knowledge relate to risk and benefit perceptions of AI in healthcare.

The development of artificial intelligence (AI) in healthcare is accelerating rapidly. Beyond the urge for technological optimization, public perceptions and preferences regarding the application of such technologies remain poorly understood. Risk and benefit perceptions of novel technologies are key drivers for successful implementation. Therefore, it is crucial to understand the factors that condition these perceptions. In this study, we draw on the risk perception and human-AI interaction literature to examine how explicit (i.e., deliberate) and implicit (i.e., automatic) comparative trust associations with AI versus physicians, and knowledge about AI, relate to likelihood perceptions of risks and benefits of AI in healthcare and preferences for the integration of AI in healthcare. We use survey data (N = 378) to specify a path model. Results reveal that the path for implicit comparative trust associations on relative preferences for AI over physicians is only significant through risk, but not through benefit perceptions. This finding is reversed for AI knowledge. Explicit comparative trust associations relate to AI preference through risk and benefit perceptions. These findings indicate that risk perceptions of AI in healthcare might be driven more strongly by affect-laden factors than benefit perceptions, which in turn might depend more on reflective cognition. Implications of our findings and directions for future research are discussed considering the conceptualization of trust as heuristic and dual-process theories of judgment and decision-making. Regarding the design and implementation of AI-based healthcare technologies, our findings suggest that a holistic integration of public viewpoints is warranted.

Human-AI teaming: leveraging transactive memory and speaking up for enhanced team effectiveness.

In this prospective observational study, we investigate the role of transactive memory and speaking up in human-AI teams comprising 180 intensive care (ICU) physicians and nurses working with AI in a simulated clinical environment. Our findings indicate that interactions with AI agents differ significantly from human interactions, as accessing information from AI agents is positively linked to a team's ability to generate novel hypotheses and demonstrate speaking-up behavior, but only in higher-performing teams. Conversely, accessing information from human team members is negatively associated with these aspects, regardless of team performance. This study is a valuable contribution to the expanding field of research on human-AI teams and team science in general, as it emphasizes the necessity of incorporating AI agents as knowledge sources in a team's transactive memory system, as well as highlighting their role as catalysts for speaking up. Practical implications include suggestions for the design of future AI systems and human-AI team training in healthcare and beyond.

Solving the explainable AI conundrum by bridging clinicians' needs and developers' goals.

Explainable artificial intelligence (XAI) has emerged as a promising solution for addressing the implementation challenges of AI/ML in healthcare. However, little is known about how developers and clinicians interpret XAI and what conflicting goals and requirements they may have. This paper presents the findings of a longitudinal multi-method study involving 112 developers and clinicians co-designing an XAI solution for a clinical decision support system. Our study identifies three key differences between developer and clinician mental models of XAI, including opposing goals (model interpretability vs. clinical plausibility), different sources of truth (data vs. patient), and the role of exploring new vs. exploiting old knowledge. Based on our findings, we propose design solutions that can help address the XAI conundrum in healthcare, including the use of causal inference models, personalized explanations, and ambidexterity between exploration and exploitation mindsets. Our study highlights the importance of considering the perspectives of both developers and clinicians in the design of XAI systems and provides practical recommendations for improving the effectiveness and usability of XAI in healthcare.

Exceeding the Ordinary: A Framework for Examining Teams Across the Extremeness Continuum and Its Impact on Future Research.

Work teams increasingly face unprecedented challenges in volatile, uncertain, complex, and often ambiguous environments. In response, team researchers have begun to focus more on teams whose work revolves around mitigating risks in these dynamic environments. Some highly insightful contributions to team research and organizational studies have originated from investigating teams that face unconventional or extreme events. Despite this increased attention to extreme teams, however, a comprehensive theoretical framework is missing. We introduce such a framework that envisions team extremeness as a continuous, multidimensional variable consisting of environmental extremeness (i.e., external team context) and task extremeness (i.e., internal team context). The proposed framework allows every team to be placed on the team extremeness continuum, bridging the gap between literature on extreme and more traditional teams. Furthermore, we present six propositions addressing how team extremeness may interact with team processes, emergent states, and outcomes using core variables for team effectiveness and the well-established input-mediator-output-input model to structure our theorizing. Finally, we outline some potential directions for future research by elaborating on temporal considerations (i.e., patterns and trajectories), measurement approaches, and consideration of multilevel relationships involving team extremeness. We hope that our theoretical framework and theorizing can create a path forward, stimulating future research within the organizational team literature to further examine the impact of team extremeness on team dynamics and effectiveness.

Differing Visual Behavior Between Inexperienced and Experienced Critical Care Nurses While Using a Closed-Loop Ventilation System-A Prospective Observational Study.

Introduction: Closed-loop ventilation modes are increasingly being used in intensive care units to ensure more automaticity. Little is known about the visual behavior of health professionals using these ventilation modes. The aim of this study was to analyze gaze patterns of intensive care nurses while ventilating a patient in the closed-loop mode with Intellivent adaptive support ventilation® (I-ASV) and to compare inexperienced with experienced nurses. Materials and Methods: Intensive care nurses underwent eye-tracking during daily care of a patient ventilated in the closed-loop ventilation mode. Five specific areas of interest were predefined (ventilator settings, ventilation curves, numeric values, oxygenation Intellivent, ventilation Intellivent). The main independent variable and primary outcome was dwell time. Secondary outcomes were revisits, average fixation time, first fixation and fixation count on areas of interest in a targeted tracking-time of 60 min. Gaze patterns were compared between I-ASV inexperienced (n = 12) and experienced (n = 16) nurses. Results: In total, 28 participants were included. Overall, dwell time was longer for ventilator settings and numeric values compared to the other areas of interest. Similar results could be obtained for the secondary outcomes. Visual fixation of oxygenation Intellivent and ventilation Intellivent was low. However, dwell time, average fixation time and first fixation on oxygenation Intellivent were longer in experienced compared to inexperienced intensive care nurses. Discussion: Gaze patterns of intensive care nurses were mainly focused on numeric values and settings. Areas of interest related to traditional mechanical ventilation retain high significance for intensive care nurses, despite use of closed-loop mode. More visual attention to oxygenation Intellivent and ventilation Intellivent in experienced nurses implies more routine and familiarity with closed-loop modes in this group. The findings imply the need for constant training and education with new tools in critical care, especially for inexperienced professionals.

Shared leadership in multiteam systems: how cockpit and cabin crews lead each other to safety.

OBJECTIVE: In this study, we aimed to examine the effect of shared leadership within and across teams in multiteam systems (MTS) on team goal attainment and MTS success. BACKGROUND: Due to different and sometimes competing goals in MTS, leadership is required within and across teams. Shared leadership, the effectiveness of which has been proven in single teams, may be an effective strategy to cope with these challenges. METHOD: We observed leadership in 84 cockpit and cabin crews that collaborated in the form of six-member MTS aircrews (N = 504) during standardized simulations of an in-flight emergency. Leadership was coded by three trained observers using a structured observation system. Team goal attainment was assessed by two subject matter experts using a checklist-based rating tool. MTS goal attainment was measured objectively on the basis of the outcome of the simulated flights. RESULTS: In successful MTS aircrews, formal leaders and team members displayed significantly more leadership behaviors, shared leadership by pursers and flight attendants predicted team goal attainment, and pursers' shared leadership across team boundaries predicted cross-team goal attainment. In cockpit crews, leadership was not shared and captains' vertical leadership predicted team goal attainment regardless of MTS success. CONCLUSION: The results indicate that in general, shared leadership positively relates to team goal attainment and MTS success,whereby boundary spanners' dual leadership role is key. APPLICATION: Leadership training in MTS should address shared rather than merely vertical forms of leadership, and component teams in MTS should be trained together with emphasis on boundary spanners' dual leadership role. Furthermore, team members should be empowered to engage in leadership processes when required.