Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study.

Importance: Diagnostic errors are common and cause significant morbidity. Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves diagnostic reasoning. Objective: To assess the impact of the GPT-4 LLM on physicians' diagnostic reasoning compared to conventional resources. Design: Multi-center, randomized clinical vignette study. Setting: The study was conducted using remote video conferencing with physicians across the country and in-person participation across multiple academic medical institutions. Participants: Resident and attending physicians with training in family medicine, internal medicine, or emergency medicine. Interventions: Participants were randomized to access GPT-4 in addition to conventional diagnostic resources or to just conventional resources. They were allocated 60 minutes to review up to six clinical vignettes adapted from established diagnostic reasoning exams. Main Outcomes and Measures: The primary outcome was diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps. Secondary outcomes included time spent per case and final diagnosis. Results: 50 physicians (26 attendings, 24 residents) participated, with an average of 5.2 cases completed per participant. The median diagnostic reasoning score per case was 76.3 percent (IQR 65.8 to 86.8) for the GPT-4 group and 73.7 percent (IQR 63.2 to 84.2) for the conventional resources group, with an adjusted difference of 1.6 percentage points (95% CI -4.4 to 7.6; p=0.60). The median time spent on cases for the GPT-4 group was 519 seconds (IQR 371 to 668 seconds), compared to 565 seconds (IQR 456 to 788 seconds) for the conventional resources group, with a time difference of -82 seconds (95% CI -195 to 31; p=0.20). GPT-4 alone scored 15.5 percentage points (95% CI 1.5 to 29, p=0.03) higher than the conventional resources group. Conclusions and Relevance: In a clinical vignette-based study, the availability of GPT-4 to physicians as a diagnostic aid did not significantly improve clinical reasoning compared to conventional resources, although it may improve components of clinical reasoning such as efficiency. GPT-4 alone demonstrated higher performance than both physician groups, suggesting opportunities for further improvement in physician-AI collaboration in clinical practice.

Characteristics differentiating problem representation synthesis between novices and experts.

BACKGROUND: Formulating a thoughtful problem representation (PR) is fundamental to sound clinical reasoning and an essential component of medical education. Aside from basic structural recommendations, little consensus exists on what characterizes high-quality PRs. OBJECTIVES: To elucidate characteristics that distinguish PRs created by experts and novices. METHODS: Early internal medicine residents (novices) and inpatient teaching faculty (experts) from two academic medical centers were given two written clinical vignettes and were instructed to write a PR and three-item differential diagnosis for each. Deductive content analysis described the characteristics comprising PRs. An initial codebook of characteristics was refined iteratively. The primary outcome was differences in characteristic frequencies between groups. The secondary outcome was characteristics correlating with diagnostic accuracy. Mixed-effects regression with random effects modeling compared case-level outcomes by group. RESULTS: Overall, 167 PRs were analyzed from 30 novices and 54 experts. Experts included 0.8 fewer comorbidities (p < .01) and 0.6 more examination findings (p = .01) than novices on average. Experts were less likely to include irrelevant comorbidities (odds ratio [OR] = 0.4, 95% confidence interval [CI] = 0.2-0.8) or a diagnosis (OR = 0.3, 95% CI = 0.1-0.8) compared with novices. Experts encapsulated clinical data into higher-order terms (e.g., sepsis) than novices (p < .01) while including similar numbers of semantic qualifiers (SQs). Regardless of expertise level, PRs following a three-part structure (e.g., demographics, temporal course, and clinical syndrome) and including temporal SQs were associated with diagnostic accuracy (p < .01). CONCLUSIONS: Compared with novices, expert PRs include less irrelevant data and synthesize information into higher-order concepts. Future studies should determine whether targeted educational interventions for PRs improve diagnostic accuracy.

Consensus-derived Recommended Skills for Transition to Residency Courses.

PURPOSE: Transition to residency (TTR) courses facilitate the medical student-residency transition and are an integral part of senior medical student training. The authors established a common set of skills for TTR courses and an expected level of entrustment students should demonstrate in each skill on TTR course completion. METHOD: A modified Delphi approach was used with 3 survey iterations between 2020 and 2022 to establish skills to be included in a TTR course. Nine TTR experts suggested general candidate skills and conducted a literature search to ensure no vital skills were missed. A stakeholder panel was solicited from email lists of TTR educators, residency program directors, and residents at the panelists' institutions. Consensus was defined as more than 75% of participants selecting a positive inclusion response. An entrustment questionnaire asked panelists to assign a level of expected entrustment to each skill, with 1 indicating observation only and 6 indicating perform independently. RESULTS: The stakeholder panel initially consisted of 118 respondents with representation across educational contexts and clinical specialties. Response rates were 54% in iteration 2, 42% in iteration 3, and 33% on the entrustment questionnaire. After 3 iterations, 54 skills met consensus and were consolidated into 37 final skills categorized into 18 clinical skills (e.g., assessment and management of inpatient concerns), 14 communication skills (e.g., delivering serious news or having difficult conversations), 4 personal and professional skills (e.g., prioritization of clinical tasks), and 1 procedural skill (mask ventilation). Median entrustment levels were reported for all skills, with 19 skills having a level of expected entrustment of 4 (perform independently and have all findings double-checked). CONCLUSIONS: These consensus skills can serve as the foundation of a standardized national TTR curriculum framework. Entrustment guidance may help educational leaders optimize training and allocation of resources for TTR curriculum development and implementation.

Framework and Schema are False Synonyms: Defining Terms to Improve Learning.

Clinical reasoning is an essential expertise of health care professionals that includes the complex cognitive processes that lead to diagnosis and management decisions. In order to optimally teach, learn, and assess clinical reasoning, it is imperative for teachers and learners to have a shared understanding of the language. Currently, educators use the terms schema and framework interchangeably but they are distinct concepts. In this paper, we offer definitions for schema and framework and use the high-stakes field of aviation to demonstrate the interplay of these concepts. We offer examples of framework and schema in the medical education field and discuss how a clear understanding of these concepts allows for greater intentionality when teaching and assessing clinical reasoning.

Factors Associated With Health Care Professionals' Choice to Practice in Rural Minnesota.

Importance: Rural health inequities are due in part to a shortage of health care professionals in these areas. Objective: To determine the factors associated with health care professionals' decisions about where to practice. Design, Setting, and Participants: This prospective, cross-sectional survey study of health care professionals in Minnesota was administered by the Minnesota Department of Health from October 18, 2021, to July 25, 2022. Advanced practice registered nurses (APRNs), physicians, physician assistants (PAs), and registered nurses (RNs) renewing their professional licenses were eligible. Exposures: Individuals' ratings on survey items related to their choice of practice location. Main Outcomes and Measures: Rural or urban practice location as defined by the US Department of Agriculture's Rural-Urban Commuting Area typology. Results: A total of 32 086 respondents were included in the analysis (mean [SD] age, 44.4 [12.2] years; 22 728 identified as female [70.8%]). Response rates were 60.2% for APRNs (n = 2174), 97.7% for PAs (n = 2210), 95.1% for physicians (n = 11 019), and 61.6% for RNs (n = 16 663). The mean (SD) age of APRNs was 45.0 (10.3) years (1833 [84.3%] female); PAs, 39.0 (9.4) years (1648 [74.6%] female); physicians, 48.0 (11.9) years (4455 [40.4%] female); and RNs, 42.6 (12.3) years (14 792 [88.8%] female). Most respondents worked in urban (29 456 [91.8%]) vs rural (2630 [8.2%]) areas. Bivariate analysis suggested that family considerations are the most important determinant of practice location. Multivariate analysis revealed that having grown up in a rural area was the strongest factor associated with rural practice (odds ratio [OR] for APRNs, 3.44 [95% CI, 2.68-4.42]; OR for PAs, 3.75 [95% CI, 2.81-5.00]; OR for physicians, 2.44 [95% CI, 2.18-2.73]; OR for RNs, 3.77 [95% CI, 3.44-4.15]). When controlling for rural background, other associated factors included the availability of loan forgiveness (OR for APRNs, 1.42 [95% CI, 1.19-1.69]; OR for PAs, 1.60 [95% CI, 1.31-1.94]; OR for physicians, 1.54 [95% CI, 1.38-1.71]; OR for RNs, 1.20 [95% CI, 1.12-1.28]) and an educational program that prepared for rural practice (OR for APRNs, 1.44 [95% CI, 1.18-1.76]; OR for PAs. 1.70 [95% CI, 1.34-2.15]; OR for physicians, 1.31 [95% CI, 1.17-1.47]; OR for RNs, 1.23 [95% CI, 1.15-1.31]). Autonomy in one's work (OR for APRNs, 1.42 [95% CI, 1.08-1.86]; OR for PAs, 1.18 [95% CI, 0.89-1.58]; OR for physicians, 1.53 [95% CI, 1.31-1.78]; OR for RNs, 1.16 [95% CI, 1.07-1.25]) and a broad scope of practice (OR for APRNs, 1.46 [95% CI, 1.15-1.86]; OR for PAs, 0.96 [95% CI, 0.74-1.24]; OR for physicians, 1.62 [95% CI, 1.40-1.87]; OR for RNs, 0.96 [95% CI, 0.89-1.03]) were important factors associated with rural practice. Lifestyle and area considerations were not associated with rural practice; family considerations were associated with rural practice for RNs only (OR for APRNs, 0.97 [95% CI, 0.90-1.06]; OR for PAs, 0.95 [95% CI, 0.87-1.04]; OR for physicians, 0.92 [95% CI, 0.88-0.96]; OR for RNs, 1.05 [95% CI, 1.02-1.07]). Conclusions and Relevance: Understanding the interconnected factors involved in rural practice requires modeling relevant factors. The findings of this survey study suggest that loan forgiveness, rural training, autonomy, and a broad scope of practice are factors associated with rural practice for most health care professionals. Other factors associated with rural practice vary by profession, suggesting that there may not be a one-size-fits-all approach to recruitment of rural health care professionals.

Exploring relationships between physician stress, burnout, and diagnostic elements in clinician notes.

OBJECTIVES: To understand the relationship between stressful work environments and patient care by assessing work conditions, burnout, and elements of the diagnostic process. METHODS: Notes and transcripts of audiotaped encounters were assessed for verbal and written documentation related to psychosocial data, differential diagnosis, acknowledgement of uncertainty, and other diagnosis-relevant contextual elements using 5-point Likert scales in seven primary care physicians (PCPs) and 28 patients in urgent care settings. Encounter time spent vs time needed (time pressure) was collected from time stamps and clinician surveys. Study physicians completed surveys on stress, burnout, and work conditions using the Mini-Z survey. RESULTS: Physicians with high stress or burnout were less likely to record psychosocial information in transcripts and notes (psychosocial information noted in 0% of encounters in 4 high stress/burned-out physicians), whereas low stress physicians (n=3) recorded psychosocial information consistently in 67% of encounters. Burned-out physicians discussed a differential diagnosis in only 31% of encounters (low counts concentrated in two physicians) vs. in 73% of non-burned-out doctors' encounters. Burned-out and non-burned-out doctors spent comparable amounts of time with patients (about 25 min). CONCLUSIONS: Key diagnostic elements were seen less often in encounter transcripts and notes in burned-out urgent care physicians.

Imagining the future of diagnostic performance feedback.

Diagnostic performance is uniquely challenging to measure, and providing feedback on diagnostic performance to catalyze diagnostic recalibration remains the exception to the rule in healthcare. Diagnostic accuracy, timeliness, and explanation to the patient are essential dimensions of diagnostic performance that each intersect with a variety of technical, contextual, cultural, and policy barriers. Setting aside assumptions about current constraints, we explore the future of diagnostic performance feedback by describing the "minimum viable products" and the "ideal state" solutions that can be envisioned for each of several important barriers. Only through deliberate and iterative approaches to breaking down these barriers can we improve recalibration and continuously drive the healthcare ecosystem towards diagnostic excellence.

Improving diagnosis: adding context to cognition.

BACKGROUND: The environment in which clinicians provide care and think about their patients is a crucial and undervalued component of the diagnostic process. CONTENT: In this paper, we propose a new conceptual model that links work conditions to clinician responses such as stress and burnout, which in turn impacts the quality of the diagnostic process and finally patient diagnostic outcomes. The mechanism for these interactions critically depends on the relationship between working memory (WM) and long-term memory (LTM), and ways WM and LTM interactions are affected by working conditions. SUMMARY: We propose a conceptual model to guide interventions to improve work conditions, clinician reactions and ultimately diagnostic process, accuracy and outcomes. OUTLOOK: Improving diagnosis can be accomplished if we are able to understand, measure and increase our knowledge of the context of care.

Research Methods: How to Perform an Effective Peer Review.

Scientific peer review has existed for centuries and is a cornerstone of the scientific publication process. Because the number of scientific publications has rapidly increased over the past decades, so has the number of peer reviews and peer reviewers. In this paper, drawing on the relevant medical literature and our collective experience as peer reviewers, we provide a user guide to the peer review process, including discussion of the purpose and limitations of peer review, the qualities of a good peer reviewer, and a step-by-step process of how to conduct an effective peer review.