Generative Artificial Intelligence for Chest Radiograph Interpretation in the Emergency Department.

Importance: Multimodal generative artificial intelligence (AI) methodologies have the potential to optimize emergency department care by producing draft radiology reports from input images. Objective: To evaluate the accuracy and quality of AI-generated chest radiograph interpretations in the emergency department setting. Design, Setting, and Participants: This was a retrospective diagnostic study of 500 randomly sampled emergency department encounters at a tertiary care institution including chest radiographs interpreted by both a teleradiology service and on-site attending radiologist from January 2022 to January 2023. An AI interpretation was generated for each radiograph. The 3 radiograph interpretations were each rated in duplicate by 6 emergency department physicians using a 5-point Likert scale. Main Outcomes and Measures: The primary outcome was any difference in Likert scores between radiologist, AI, and teleradiology reports, using a cumulative link mixed model. Secondary analyses compared the probability of each report type containing no clinically significant discrepancy with further stratification by finding presence, using a logistic mixed-effects model. Physician comments on discrepancies were recorded. Results: A total of 500 ED studies were included from 500 unique patients with a mean (SD) age of 53.3 (21.6) years; 282 patients (56.4%) were female. There was a significant association of report type with ratings, with post hoc tests revealing significantly greater scores for AI (mean [SE] score, 3.22 [0.34]; P < .001) and radiologist (mean [SE] score, 3.34 [0.34]; P < .001) reports compared with teleradiology (mean [SE] score, 2.74 [0.34]) reports. AI and radiologist reports were not significantly different. On secondary analysis, there was no difference in the probability of no clinically significant discrepancy between the 3 report types. Further stratification of reports by presence of cardiomegaly, pulmonary edema, pleural effusion, infiltrate, pneumothorax, and support devices also yielded no difference in the probability of containing no clinically significant discrepancy between the report types. Conclusions and Relevance: In a representative sample of emergency department chest radiographs, results suggest that the generative AI model produced reports of similar clinical accuracy and textual quality to radiologist reports while providing higher textual quality than teleradiologist reports. Implementation of the model in the clinical workflow could enable timely alerts to life-threatening pathology while aiding imaging interpretation and documentation.

Comparison of artificial intelligence versus real-time physician assessment of pulmonary edema with lung ultrasound.

BACKGROUND: Lung ultrasound can evaluate for pulmonary edema, but data suggest moderate inter-rater reliability among users. Artificial intelligence (AI) has been proposed as a model to increase the accuracy of B line interpretation. Early data suggest a benefit among more novice users, but data are limited among average residency-trained physicians. The objective of this study was to compare the accuracy of AI versus real-time physician assessment for B lines. METHODS: This was a prospective, observational study of adult Emergency Department patients presenting with suspected pulmonary edema. We excluded patients with active COVID-19 or interstitial lung disease. A physician performed thoracic ultrasound using the 12-zone technique. The physician recorded a video clip in each zone and provided an interpretation of positive (≥3 B lines or a wide, dense B line) or negative (<3 B lines and the absence of a wide, dense B line) for pulmonary edema based upon the real-time assessment. A research assistant then utilized the AI program to analyze the same saved clip to determine if it was positive versus negative for pulmonary edema. The physician sonographer was blinded to this assessment. The video clips were then reviewed independently by two expert physician sonographers (ultrasound leaders with >10,000 prior ultrasound image reviews) who were blinded to the AI and initial determinations. The experts reviewed all discordant values and reached consensus on whether the field (i.e., the area of lung between two adjacent ribs) was positive or negative using the same criteria as defined above, which served as the gold standard. RESULTS: 71 patients were included in the study (56.3% female; mean BMI: 33.4 [95% CI 30.6-36.2]), with 88.3% (752/852) of lung fields being of adequate quality for assessment. Overall, 36.1% of lung fields were positive for pulmonary edema. The physician was 96.7% (95% CI 93.8%-98.5%) sensitive and 79.1% (95% CI 75.1%-82.6%) specific. The AI software was 95.6% (95% CI 92.4%-97.7%) sensitive and 64.1% (95% CI 59.8%-68.5%) specific. CONCLUSION: Both the physician and AI software were highly sensitive, though the physician was more specific. Future research should identify which factors are associated with increased diagnostic accuracy.

Education Value Units in Medical Education: A Scoping Review.

PURPOSE: On the basis of the tripartite mission of patient care, research, and education, a need has arisen to better support faculty in non-revenue-generating activities, such as education. As a result, some programs have developed education value unit (EVU) systems to incentivize these activities. The purpose of this scoping review is to analyze the existing literature on EVUs to identify current structures and future directions for research. METHOD: The authors conducted a literature search of 5 databases without restrictions, searching for any articles on EVU systems published from database inception to January 12, 2022. Two authors independently screened articles for inclusion. Two authors independently extracted data and all authors performed quantitative and qualitative synthesis, consistent with best practice recommendations for scoping reviews. RESULTS: Fifty-eight articles were included. The most common rationale was to incentivize activities prioritized by the department or institution. Of those reporting funding, departmental revenue was most common. The majority of EVU systems were created using a dedicated committee, although composition of the committees varied. Stakeholder engagement was a key component for EVU system development. Most EVU systems also included noneducational activities, such as clinical activities, scholarship activities, administrative or leadership activities, and citizenship. Incentive models varied widely but typically involved numeric- or time-based quantification. EVUs were generally seen as positive, having increased equity and transparency as well as a positive impact on departmental metrics. CONCLUSIONS: This scoping review summarizes the existing literature on EVU systems, providing valuable insights for application to practice and areas for future research.

Educator's blueprint: A how-to guide for developing high-quality multiple-choice questions.

Multiple-choice questions are commonly used for assessing learners' knowledge, as part of educational programs and scholarly endeavors. To ensure that questions accurately assess the learners and provide meaningful data, it is important to understand best practices in multiple-choice question design. This Educator's Blueprint paper provides 10 strategies for developing high-quality multiple-choice questions. These strategies include determining the purpose, objectives, and scope of the question; assembling a writing team; writing succinctly; asking questions that assess knowledge and comprehension rather than test-taking ability; ensuring consistent and independent answer choices; using plausible foils; avoiding grouped options; selecting the ideal response number and order; writing high-quality explanations; and gathering validity evidence before and evaluating the questions after use.

The test developer's dilemma: Evaluating the balance of feasibility and empiric performance of test development techniques for repeated written assessments.

PURPOSE: Written assessments face challenges when administered repeatedly, including resource-intensive item development and the potential for performance improvement secondary to item recall as opposed to understanding. This study examines the efficacy of three-item development techniques in addressing these challenges. METHODS: Learners at five training programs completed two 60-item repeated assessments. Items from the first test were randomized to one of three treatments for the second assessment: (1) Verbatim repetition, (2) Isomorphic changes, or (3) Total revisions. Primary outcomes were the stability of item psychometrics across test versions and evidence of item recall influencing performance as measured by the rate of items answered correctly and then incorrectly (correct-to-incorrect rate), which suggests guessing. RESULTS: Forty-six learners completed both tests. Item psychometrics were comparable across test versions. Correct-to-incorrect rates differed significantly between groups with the highest guessing rate (lowest recall effect) in the Total Revision group (0.15) and the lowest guessing rate (highest recall effect) in the Verbatim group (0.05), p = 0.01. CONCLUSIONS: Isomorphic and total revisions demonstrated superior performance in mitigating the effect of recall on repeated assessments. Given the high costs of total item revisions, there is promise in exploring isomorphic items as an efficient and effective approach to repeated written assessments.[Box: see text].

A consensus list of ultrasound competencies for graduating emergency medicine residents.

Objectives: Emergency ultrasound (EUS) is a critical component of emergency medicine (EM) resident education. Currently, there is no consensus list of competencies for EUS training, and graduating residents have varying levels of skill and comfort. The objective of this study was to define a widely accepted comprehensive list of EUS competencies for graduating EM residents through a modified Delphi method. Methods: We developed a list of EUS applications through a comprehensive literature search, the American College of Emergency Physicians list of core EUS benchmarks, and the Council of Emergency Medicine Residency-Academy of Emergency Ultrasound consensus document. We assembled a multi-institutional expert panel including 15 faculty members from diverse practice environments and geographical regions. The panel voted on the list of competencies through two rounds of a modified Delphi process using a modified Likert scale (1 = not at all important, 5 = very important) to determine levels of agreement for each application-with revisions occurring between the two rounds. High agreement for consensus was set at >80%. Results: Fifteen of 15 panelists completed the first-round survey (100%) that included 359 topics related to EUS. After the first round, 195 applications achieved high agreement, four applications achieved medium agreement, and 164 applications achieved low agreement. After the discussion, we removed three questions and added 13 questions. Fifteen of 15 panelists completed the second round of the survey (100%) with 209 of the 369 applications achieving consensus. Conclusion: Our final list represents expert opinion on EUS competencies for graduating EM residents. We hope to use this consensus list to implement a more consistent EUS curriculum for graduating EM residents and to standardize EUS training across EM residency programs.

A novel emergency medicine cultural competency curriculum addressing health care disparities.

Background: Effective cultural competency (CC) training for future health professionals is an important first step towards improving healthcare disparities (HCD). The Accreditation Council for Graduate Medical Education (ACGME) now requires that institutions train residents and faculty members in CC relevant to the patient population they serve. Methods: Using Kern's Model, we created and implemented a novel CC curriculum tailored to specific program needs in an emergency medicine residency program. Results: At the end of the curriculum, respondents reported having a better understanding of the importance of CC for their practice (p = 0.004) and of how a patient's personal and historical context affects treatment (p = 0.002). They also reported an increase in the frequency of practicing strategies to reduce bias in themselves (p < 0.001) and others (p < 0.001), as well as comfort interacting with and treating patients from different backgrounds (p < 0.001). Lastly, they reported improved preparedness to collaborate with communities to address HCD (p = 0.004) and to identify community leaders to do so (p < 0.001). Conclusions: The challenges of CC training demonstrate the need for a standard yet adaptable framework. We have designed, implemented, and evaluated a novel curriculum tailored to the specific needs of our EM residency program. The curriculum improved participants' attitudes, preparedness, and self-reported behaviors regarding CC and HCD. This framework represents an example of a successful model to meet ACGME requirements.

Development of an expert consensus checklist for emergency ultrasound.

Background: Within today's competency-based medical education, traditional set number proficiency benchmarks have been called into question. Checklists may help guide individualized training and standardized outcomes. However, multicenter expert consensus checklists based on established guidelines with supporting validity evidence have not been published for specific emergency ultrasound (EUS) applications. We describe a robust national EUS expert consensus methodology for developing a checklist for the extended focused assessment with sonography in trauma (eFAST examination). Methods: Utilizing the ACEP imaging compendium as a primary reference, 10 national EUS experts iteratively refined and agreed upon a final checklist. To obtain initial reliability and validity evidence, two different EUS experts blinded to resident experience then assessed 24 residents performing an eFAST in a simulated environment. Inter-rater reliability of the checklist was assessed using Cohen's kappa coefficient. Validity was assessed by comparing mean performance with the Student's t-test and discriminant ability of individual checklist items using item-total correlation. Results: The 10 EUS experts agreed on the final checklist items after two rounds of iterations. When evaluating 24 emergency medicine (EM) PGY-1 to -4 residents, the kappa correlation between two blinded EUS faculty raters was moderate at 0.670. Kappa and agreement were near-perfect or perfect in right and left chest image optimization, right upper quadrant (RUQ) probe placement, RUQ anatomy identification, and pelvic first-view anatomy identification. The difference in checklist performance between junior and senior EM residents was significant at -8.1% (p = 0.004). Identification of pelvic structures and placement of the probe for pelvic views were found to have an excellent item-total correlation with values of >0.4. Conclusions: We have described a robust national EUS expert consensus methodology for developing an eFAST checklist based on the ACEP imaging guidelines. Based on this encouraging initial reliability and validity evidence, further research and checklist development is warranted for additional EUS applications.

Beyond the emergency department: Effects of COVID-19 on emergency medicine resident education.

BACKGROUND: In December 2019, a novel coronavirus (COVID-19) caused widespread clinical disease, triggering limited in-person gatherings and social-distancing guidelines to minimize transmission. These regulations led most emergency medicine (EM) residency training programs to rapidly transition to virtual didactics. We sought to evaluate EM resident perceptions of the effects of COVID-19 on their didactic and clinical education. METHODS: We performed a cross-sectional survey study at seven EM residency programs using a mixed-methods approach designed to understand resident perceptions regarding the impact of COVID-19 on their educational experience. Quantitative data were presented as percentages with comparison of subgroups, while open-ended responses were analyzed using qualitative methodology. RESULTS: We achieved a 59% response rate (187/313). The majority of respondents (119/182, 65.4%) reported that the COVID-19 pandemic had a negative impact on their residency education with junior residents disproportionately affected. A total of 81 of 182 (44.5%) participants reported that one or more of their clinical rotations were partially or completely canceled due to the pandemic. Additionally, we identified four themes and 34 subthemes highlighting the contextual effects of the pandemic, which were then divided into positive and negative influences on the residency experience. The four themes include systems experience, clinical experience, didactic experience, and wellness. CONCLUSION: Our study examined the impact of COVID-19 on residents' educational experiences. We found overall mixed responses with a slightly negative impact on residency education, wellness, and clinical rotations, while satisfaction with EM as a career choice was increased. Factors influencing this included systems, clinical, and didactic experiences as well as overall wellness.

Rethinking Residency Conferences in the Era of COVID-19.

The COVID-19 pandemic requires a substantial change to the traditional approach to conference didactics. Switching to a virtual medium for conference sessions presents several challenges, particularly with regard to aspects that rely heavily on in-person components (e.g., simulation, ultrasound). This paper will discuss the challenges and strategies to address them for conference planning in the era of COVID-19.

Clinical Teaching: An Evidence-based Guide to Best Practices from the Council of Emergency Medicine Residency Directors.

Clinical teaching is the primary educational tool use to train learners from day one of medical school all the way to the completion of fellowship. However, concerns over time constraints and patient census have led to a decline in bedside teaching. This paper provides a critical review of the literature on clinical teaching with a focus on instructor teaching strategies, clinical teaching models, and suggestions for incorporating technology. Recommendations for instructor-related teaching factors include adequate preparation, awareness of effective teacher attributes, using evidence-based-knowledge dissemination strategies, ensuring good communication, and consideration of environmental factors. Proposed recommendations for potential teaching strategies include the Socratic method, the One-Minute Preceptor model, SNAPPS, ED STAT, teaching scripts, and bedside presentation rounds. Additionally, this article will suggest approaches to incorporating technology into clinical teaching, including just-in-time training, simulation, and telemedical teaching. This paper provides readers with strategies and techniques for improving clinical teaching effectiveness.

Evidence-Based Interventions that Promote Resident Wellness from the Council of Emergency Residency Directors.

Initiatives for addressing resident wellness are a recent requirement of the Accreditation Council for Graduate Medical Education in response to high rates of resident burnout nationally. We review the literature on wellness and burnout in residency education with a focus on assessment, individual-level interventions, and systemic or organizational interventions.

Tools for Measuring Clinical Ultrasound Competency: Recommendations From the Ultrasound Competency Work Group.

Competency in clinical ultrasound is essential to ensuring safe patient care. Competency in clinical ultrasound includes identifying when to perform a clinical ultrasound, performing the technical skills required for ultrasound image acquisition, accurately interpreting ultrasound images, and incorporating sonographic findings into clinical practice. In this concept paper, we discuss the advantages and limitations of existing tools to measure ultrasound competency. We propose strategies and future directions for assessing competency in clinical ultrasound.

Ultrasound for airway management: An evidence-based review for the emergency clinician.

BACKGROUND: Airway management is a common procedure performed in the Emergency Department with significant potential for complications. Many of the traditional physical examination maneuvers have limitations in the assessment and management of difficult airways. Point-of-care ultrasound (POCUS) has been increasingly studied for the evaluation and management of the airway in a variety of settings. OBJECTIVE: This article summarizes the current literature on POCUS for airway assessment, intubation confirmation, endotracheal tube (ETT) depth assessment, and performing cricothyroidotomy with an emphasis on those components most relevant for the Emergency Medicine clinician. DISCUSSION: POCUS can be a useful tool for identifying difficult airways by measuring the distance from the skin to the thyrohyoid membrane, hyoid bone, or epiglottis. It can also predict ETT size better than age-based formulae. POCUS is highly accurate for confirming ETT placement in adult and pediatric patients. The typical approach involves transtracheal visualization but can also include lung sliding and diaphragmatic elevation. ETT depth can be assessed by visualizing the ETT cuff in the trachea, as well as using lung sliding and the lung pulse sign. Finally, POCUS can identify the cricothyroid membrane more quickly and accurately than the landmark-based approach. CONCLUSION: Airway management is a core skill in the Emergency Department. POCUS can be a valuable tool with applications ranging from airway assessment to dynamic cricothyroidotomy. This paper summarizes the key literature on POCUS for airway management.

Accuracy of ultrasound for endotracheal intubation between different transducer types.

INTRODUCTION: Ultrasound has been increasingly utilized for the identification of endotracheal tube (ETT) location after an intubation attempt, particularly among patients in cardiac arrest. However, prior studies have varied with respect to the choice of transducer and no studies have directly compared the accuracy between transducer types. Our study is the first to directly compare the accuracy of ETT confirmation between the linear and curvilinear transducer. METHODS: This study was performed in a cadaver lab using three different cadavers chosen to represent varying neck circumferences. Cadavers were randomized to tracheal or esophageal intubation. Blinded sonographers assessed the location of the ETT using either a linear or curvilinear transducer in an alternating sequence. Accuracy of sonographer identification, time to identification, and operator confidence were assessed. RESULTS: Four hundred and five assessments were performed with 198 (48.9%) tracheal and 207 (51.1%) esophageal intubations. The linear transducer was 98% (95% CI 95.1% to 99.2%) accurate. The curvilinear transducer was 95% (95% CI 91.1% to 97.3%) accurate. The mean time to identification was significantly lower with the linear transducer [7.46 s (95% CI 6.23 to 8.7 s)] as compared with the curvilinear transducer [11.63 s (95% CI 9.05 to 14.2 s)]. The mean operator confidence was significantly higher with the linear transducer [4.84/5.0 (95% CI 4.76 to 4.91)] than with the curvilinear transducer [4.44/5.0 (95% CI 4.3 to 4.57)]. All operators preferred the linear transducer over the curvilinear transducer. CONCLUSION: The diagnostic accuracy of ultrasound for ETT confirmation did not significantly differ between ultrasound transducer types, but the curvilinear transducer was associated with a longer time to confirmation and lower operator confidence. Further studies are needed to determine if the accuracy would change with more novice providers or in specific patient populations.