Artificial Intelligence for Anesthesiology Board-Style Examination Questions: Role of Large Language Models.

New artificial intelligence tools have been developed that have implications for medical usage. Large language models (LLMs), such as the widely used ChatGPT developed by OpenAI, have not been explored in the context of anesthesiology education. Understanding the reliability of various publicly available LLMs for medical specialties could offer insight into their understanding of the physiology, pharmacology, and practical applications of anesthesiology. An exploratory prospective review was conducted using 3 commercially available LLMs--OpenAI's ChatGPT GPT-3.5 version (GPT-3.5), OpenAI's ChatGPT GPT-4 (GPT-4), and Google's Bard--on questions from a widely used anesthesia board examination review book. Of the 884 eligible questions, the overall correct answer rates were 47.9% for GPT-3.5, 69.4% for GPT-4, and 45.2% for Bard. GPT-4 exhibited significantly higher performance than both GPT-3.5 and Bard (p = 0.001 and p < 0.001, respectively). None of the LLMs met the criteria required to secure American Board of Anesthesiology certification, according to the 70% passing score approximation. GPT-4 significantly outperformed GPT-3.5 and Bard in terms of overall performance, but lacked consistency in providing explanations that aligned with scientific and medical consensus. Although GPT-4 shows promise, current LLMs are not sufficiently advanced to answer anesthesiology board examination questions with passing success. Further iterations and domain-specific training may enhance their utility in medical education.

Validation of a Virtual Simulator With Haptic Feedback for Endotracheal Intubation Training.

INTRODUCTION: Endotracheal intubation (ETI) is a procedure that varies in difficulty because of patient characteristics and clinical conditions. Existing physical simulators do not encompass these variations. The Virtual Airway Skills Trainer for Endotracheal Intubation (VAST-ETI) was developed to provide different patient characteristics and high-fidelity haptic feedback to improve training. METHODS: We demonstrate the effectiveness of VAST-ETI as a training and evaluation tool for ETI. Construct validation was evaluated by scoring the performance of experts ( N = 15) and novices ( N = 15) on the simulator to ensure its ability to distinguish technical proficiency. Convergent and predictive validity were evaluated by performing a learning curve study, in which a group of novices ( N = 7) were trained for 2 weeks using VAST-ETI and then compared with a control group ( N = 9). RESULTS: The VAST-ETI was able to distinguish between expert and novice based on mean simulator scores ( t [88] = -6.61, P < 0.0005). When used during repeated practice, individuals demonstrated a significant increase in their score on VAST-ETI over the learning period ( F [11,220] = 7206, P < 0.001); however when compared with a control group, there was not a significant interaction effect on the simulator score. There was a significant difference between the simulator-trained and control groups ( t [12.85] = -2.258, P = 0.042) when tested in the operating room. CONCLUSIONS: Our results demonstrate the effectiveness of virtual simulation with haptic feedback for assessing performance and training of ETI. The simulator was not able to differentiate performance between more experienced trainees and experts because of limits in simulator difficulty.

Intelligent Virtual Operating Room for Enhancing Nontechnical Skills.

This article discusses an intelligent immersive virtual operating room to enable teams to train in a distributed fashion wearing head-mounted displays.

Measuring Team Hierarchy During High-Stakes Clinical Decision Making: Development and Validation of a New Behavioral Observation Method.

PURPOSE: Hierarchy is often cited as a cause of health care team failure; however, there are no validated measures of team hierarchy. Research on group processes in sociology provides a theoretical framework-status characteristics and expectation states (SCES)-that explains the mechanisms that produce the observable power and prestige order (status hierarchy) of the team. The authors use this formal theoretical framework to gather evidence of validity by adapting the method to measure the status hierarchy of medical teams. METHOD: In this retrospective, secondary analysis, the authors analyzed archived videorecorded training exercises conducted between 2007 and 2010 of mixed-gender health care teams of first-year residents and nurses engaged in simulated, complex decision-making scenarios. Analyses were conducted in 2013 with data reanalyzed in July 2022. By adapting the SCES framework for the unique features of academic health care, they developed and refined a coding method from videos and transcripts. To examine validity, they consider the content, response process, internal structure, relation to other variables, and consequences of the framework. RESULTS: Having established an acceptable level of coding reliability for key variables for videos and transcripts, the authors demonstrate relation to other variables, specifically detailing how the coding scheme delineates 2 status characteristics-occupation and gender. The mean numbers of statement types by gender and occupation were largely as predicted. Directives, question directives, patient work, and knowledge claims were more likely to be coded during video than transcript coding, whereas questions, statements of fact, and compliance were more likely to be coded during transcript than video coding. However, the relative rates of each statement type by status remained largely consistent among the coding methods. CONCLUSIONS: This study provides important insight into the mechanisms by which hierarchy impacts team decision making and develops the necessary framework and measurement tool to perform larger studies.

Distractions in the operating room: a survey of the healthcare team.

BACKGROUND: Distractions during surgical procedures are associated with team inefficiency and medical error. Little is published about the healthcare provider's perception of distraction and its adverse impact in the operating room. We aim to explore the perception of the operating room team on multiple distractions during surgical procedures. METHODS: A 26-question survey was administered to surgeons, anesthesia team members, nurses, and scrub technicians at our institution. Respondents were asked to identify and rank multiple distractions and indicate how each distraction might affect the flow of surgery. RESULTS: There was 160 responders for a response rate of 19.18% (160/834), of which 71 (44.1%) male and 82 (50.9%) female, 48 (29.8%) surgeons, 59 (36.6%) anesthesiologists, Certified Registered Nurse Anesthetists (CRNA), and 53 (32.9%) OR nurses and scrub technicians. Responders were classified into a junior group (< 10 years of experience) and a senior group (≥ 10 years). Auditory distraction followed by equipment were the most distracting factors in the operating room. All potential auditory distractions in this survey were associated with higher percentage of certain level of negative impact on the flow of surgery except for music. The top 5 distractors belonged to equipment and environment categories. Phone calls/ pagers/ beepers and case relevant communications were consistently among the top 5 most common distractors. Case relevant communications, music, teaching, and consultation were the top 4 most perceived positive impact on the flow of surgery. Distractors with higher levels of "bothersome" rating appeared to associate with a higher level of perceived negative impact on the flow of surgery. Vision was the least distracting factor and appeared to cause minimal positive impact on the flow of surgery. CONCLUSIONS: To our knowledge, this is the first survey studying perception of surgery, anesthesia, and OR staff on various distractions in the operating room. Fewer unnecessary distractions might improve the flow of surgery, improve OR teamwork, and potentially improve patient outcomes.

Modeling Haptic Interactions in Endoscopic Submucosal Dissection (ESD).

The ability to provide realistic haptic feedback is indispensable for virtual-reality (VR) based endoscopic colorectal surgery simulators. Despite its importance, force feedback is commonly simulated by simplified approximations with parameters manually tuned in preliminary evaluations due to the complexity of the dynamics of haptic interaction in colonoscopy interventions. Endoscopic submucosal dissection (ESD) is a particularly challenging intervention that requires advanced manual skills for endoscopic control. This work proposes a mechanical impedance model for haptic interactions in ESD formulated via an experimental methodology applied to endoscopic colorectal interventions in general. The developed model is shown to capture the variations in the interaction force during two operations performed at distinct locations on a porcine sample. Salient cues in the recorded haptic interaction data are presented, and changes in the impedance characteristics of the tool-tissue interaction between the steps of the operation are analyzed.

Disruptions to shared mental models from poor quality of service in collaborative virtual environments.

Collaborative virtual environments are being used in various applications ranging from online games to complex team training scenarios. The key to the success of such environments is the ability of the participants to form a shared mental model of the collaborative task being performed. Poor quality of service can deteriorate user performance and quality of experience, leading to a disruption of this mental model. While the effects of quality of service have been analyzed for traditional desktop environments, these effects remain unclear in collaborative virtual environments during user-to-user interactions. Here, we analyze the role of latency and packet bursts, two common problems in collaborative applications, on both simulator perception and actual task performance in a collaborative fire-fighting simulator. This exploratory study indicates that large latencies have a significant (p < 0.05) impact on the quality of experience, but not task performance. In contrast, packet bursts have a much larger impact on both the quality of experience and performance. Additionally, the network role, such as whether a user is a client or server, showed a significant (p < 0.05) impact on task performance in conditions impaired by packet bursts.

A Virtual Reality Memory Palace Variant Aids Knowledge Retrieval from Scholarly Articles.

We present exploratory research of virtual reality techniques and mnemonic devices to assist in retrieving knowledge from scholarly articles. We used abstracts of scientific publications to represent knowledge in scholarly articles; participants were asked to read, remember, and retrieve knowledge from a set of abstracts. We conducted an experiment to compare participants' recall and recognition performance in three different conditions: a control condition without a pre-specified strategy to test baseline individual memory ability, a condition using an image-based variant of a mnemonic called a "memory palace," and a condition using a virtual reality-based variant of a memory palace. Our analyses show that using a virtual reality-based memory palace variant greatly increased the amount of knowledge retrieved and retained over the baseline, and it shows a moderate improvement over the other image-based memory palace variant. Anecdotal feedback from participants suggested that personalizing a memory palace variant would be appreciated. Our results support the value of virtual reality for some high-level cognitive tasks and help improve future applications of virtual reality and visualization.

Endoscopic submucosal dissection: a cognitive task analysis framework toward training design.

BACKGROUND: One of the major impediments to the proliferation of endoscopic submucosal dissection (ESD) training in Western countries is the lack of sufficient experts as instructors. One way to address this gap is to develop didactic systems, such as surgical simulators, to support the role of trainers. Cognitive task analysis (CTA) has been used in healthcare for the design and improvement of surgical training programs, and therefore can potentially be used for design of similar systems for ESD. OBJECTIVE: The aim of the study was to apply a CTA-based approach to identify the cognitive aspects of performing ESD, and to generate qualitative insights for training. MATERIALS AND METHODS: Semi-structured interviews were designed based on the CTA framework to elicit knowledge of ESD practitioners relating to the various tasks involved in the procedure. Three observations were conducted of expert ESD trainers either while they performed actual ESD procedures or at a training workshop. Interviews were either conducted over the phone or in person. Interview participants included four experts and four novices. The observation notes and interviews were analyzed for emergent qualitative themes and relationships. RESULTS: The qualitative analysis yielded thematic insights related to four main cognition-related categories: learning goals/principles, challenges/concerns, strategies, and decision-making. The specific insights under each of these categories were systematically mapped to the various tasks inherent to the ESD procedure. CONCLUSIONS: The CTA approach was applied to identify cognitive themes related to ESD procedural tasks. Insights developed based on the qualitative analysis of interviews and observations of ESD practitioners can be used to inform the design of ESD training systems, such as virtual reality-based simulators.

Qualitative findings from a pilot stage implementation of a novel organizational learning tool toward operationalizing the Safety-II paradigm in health care.

'Safety-II' is a new approach to safety, which emphasizes learning proactively about how safety and efficacy are achieved in everyday frontline work. Previous research developed a new lesson-sharing tool designed based on the Safety-II approach: Resilience Engineering Tool to Improve Patient Safety (RETIPS). The tool comprises questions designed to elicit narratives of adaptations that have contributed to effectiveness in care delivery. The purpose of this study is to revise and validate the design of RETIPS. The tool was revised based on feedback of clinicians at a large multi-specialty hospital, resulting in a version customized for anesthesia residents, RETIPS-AnRes. RETIPS-AnRes was administered on a pilot-basis to anesthesia resident groups for a limited period of time. A review of the reports obtained shows a strong alignment of responses with the conceptual basis of the tool, i.e. learning about how things go well in everyday work. The exemplars include both, specific instances of successful patient care, as well as generic routines that contribute to safe and/or effective care delivery. These findings support RETIPS as a tool to operationalize the Safety-II paradigm in healthcare. Lessons and implications for implementation on a wider scale are discussed.

A task and performance analysis of endoscopic submucosal dissection (ESD) surgery.

BACKGROUND: ESD is an endoscopic technique for en bloc resection of gastrointestinal lesions. ESD is a widely-used in Japan and throughout Asia, but not as prevalent in Europe or the US. The procedure is technically challenging and has higher adverse events (bleeding, perforation) compared to endoscopic mucosal resection. Inadequate training platforms and lack of established training curricula have restricted its wide acceptance in the US. Thus, we aim to develop a Virtual Endoluminal Surgery Simulator (VESS) for objective ESD training and assessment. In this work, we performed task and performance analysis of ESD surgeries. METHODS: We performed a detailed colorectal ESD task analysis and identified the critical ESD steps for lesion identification, marking, injection, circumferential cutting, dissection, intraprocedural complication management, and post-procedure examination. We constructed a hierarchical task tree that elaborates the order of tasks in these steps. Furthermore, we developed quantitative ESD performance metrics. We measured task times and scores of 16 ESD surgeries performed by four different endoscopic surgeons. RESULTS: The average time of the marking, injection, and circumferential cutting phases are 203.4 (σ: 205.46), 83.5 (σ: 49.92), 908.4 s. (σ: 584.53), respectively. Cutting the submucosal layer takes most of the time of overall ESD procedure time with an average of 1394.7 s (σ: 908.43). We also performed correlation analysis (Pearson's test) among the performance scores of the tasks. There is a moderate positive correlation (R = 0.528, p = 0.0355) between marking scores and total scores, a strong positive correlation (R = 0.7879, p = 0.0003) between circumferential cutting and submucosal dissection and total scores. Similarly, we noted a strong positive correlation (R = 0.7095, p = 0.0021) between circumferential cutting and submucosal dissection and marking scores. CONCLUSIONS: We elaborated ESD tasks and developed quantitative performance metrics used in analysis of actual surgery performance. These ESD metrics will be used in future validation studies of our VESS simulator.

Using visual design experts in critique-based evaluation of 2D vector visualization methods.

We describe an experiment in which art and illustration experts evaluated six 2D vector visualization methods. We found that these expert critiques mirrored previously recorded experimental results; these findings support that using artists, visual designers and illustrators to critique scientific visualizations can be faster and more productive than quantitative user studies. Our participants successfully evaluated how well the given methods would let users complete a given set of tasks. Our results show a statistically significant correlation with a previous objective study: designers' subjective predictions of user performance by these methods match users measured performance. The experts improved the evaluation by providing insights into the reasons for the effectiveness of each visualization method and suggesting specific improvements.

CAVE and fishtank virtual-reality displays: a qualitative and quantitative comparison.

We present the results from a qualitative and quantitative user study comparing fishtank virtual-reality (VR) and CAVE displays. The results of the qualitative study show that users preferred the fishtank VR display to the CAVE system for our scientific visualization application because of perceived higher resolution, brightness and crispness of imagery, and comfort of use. The results of the quantitative study show that users performed an abstract visual search task significantly more quickly and more accurately on the fishtank VR display system than in the CAVE. The same study also showed that visual context had no significant effect on task performance for either of the platforms. We suggest that fishtank VR displays are more effective than CAVEs for applications in which the task occurs outside the user's reference frame, the user views and manipulates the virtual world from the outside in, and the size of the virtual object that the user interacts with is smaller than the user's body and fits into the fishtank VR display. The results of both studies support this proposition.

Comparing 2D vector field visualization methods: a user study.

We present results from a user study that compared six visualization methods for two-dimensional vector data. Users performed three simple but representative tasks using visualizations from each method: 1) locating all critical points in an image, 2) identifying critical point types, and 3) advecting a particle. Visualization methods included two that used different spatial distributions of short arrow icons, two that used different distributions of integral curves, one that used wedges located to suggest flow lines, and line-integral convolution (LIC). Results show different strengths and weaknesses for each method. We found that users performed these tasks better with methods that: 1) showed the sign of vectors within the vector field, 2) visually represented integral curves, and 3) visually represented the locations of critical points. Expert user performance was not statistically different from nonexpert user performance. We used several methods to analyze the data including omnibus analysis of variance, pairwise t-tests, and graphical analysis using inferential confidence intervals. We concluded that using the inferential confidence intervals for displaying the overall pattern of results for each task measure and for performing subsequent pairwise comparisons of the condition means was the best method for analyzing the data in this study. These results provide quantitative support for some of the anecdotal evidence concerning visualization methods. The tasks and testing framework also provide a basis for comparing other visualization methods, for creating more effective methods and for defining additional tasks to further understand the tradeoffs among the methods. In the future, we also envision extending this work to more ambitious comparisons, such as evaluating two-dimensional vectors on two-dimensional surfaces embedded in three-dimensional space and defining analogous tasks for three-dimensional visualization methods.