Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial.

Importance: Large language model (LLM) artificial intelligence (AI) systems have shown promise in diagnostic reasoning, but their utility in management reasoning with no clear right answers is unknown. Objective: To determine whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources. Design: Prospective, randomized controlled trial conducted from 30 November 2023 to 21 April 2024. Setting: Multi-institutional study from Stanford University, Beth Israel Deaconess Medical Center, and the University of Virginia involving physicians from across the United States. Participants: 92 practicing attending physicians and residents with training in internal medicine, family medicine, or emergency medicine. Intervention: Five expert-developed clinical case vignettes were presented with multiple open-ended management questions and scoring rubrics created through a Delphi process. Physicians were randomized to use either GPT-4 via ChatGPT Plus in addition to conventional resources (e.g., UpToDate, Google), or conventional resources alone. Main Outcomes and Measures: The primary outcome was difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case. Results: Physicians using the LLM scored higher compared to those using conventional resources (mean difference 6.5 %, 95% CI 2.7-10.2, p<0.001). Significant improvements were seen in management decisions (6.1%, 95% CI 2.5-9.7, p=0.001), diagnostic decisions (12.1%, 95% CI 3.1-21.0, p=0.009), and case-specific (6.2%, 95% CI 2.4-9.9, p=0.002) domains. GPT-4 users spent more time per case (mean difference 119.3 seconds, 95% CI 17.4-221.2, p=0.02). There was no significant difference between GPT-4-augmented physicians and GPT-4 alone (-0.9%, 95% CI -9.0 to 7.2, p=0.8). Conclusions and Relevance: LLM assistance improved physician management reasoning compared to conventional resources, with particular gains in contextual and patient-specific decision-making. These findings indicate that LLMs can augment management decision-making in complex cases. Trial Registration: ClinicalTrials.gov Identifier: NCT06208423 ; https://classic.clinicaltrials.gov/ct2/show/NCT06208423. Key Points: Question: Does large language model (LLM) assistance improve physician performance on complex management reasoning tasks compared to conventional resources?Findings: In this randomized controlled trial of 92 physicians, participants using GPT-4 achieved higher scores on management reasoning compared to those using conventional resources (e.g., UpToDate).Meaning: LLM assistance enhances physician management reasoning performance in complex cases with no clear right answers.

Decisions to Practice in Rural Areas Among Mental Health Care Professionals.

This survey study assesses factors associated with mental health care professionals choosing to practice in rural locations.

Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research.

From dual process to a family of theories known collectively as situativity, both micro and macro theories of cognition inform our current understanding of clinical reasoning (CR) and error. CR is a complex process that occurs in a complex environment, and a nuanced, expansive, integrated model of these theories is necessary to fully understand how CR is performed in the present day and in the future. In this perspective, we present these individual theories along with figures and descriptive cases for purposes of comparison before exploring the implications of a transtheoretical model of these theories for teaching, assessment, and research in CR and error.

Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study.

Importance: Diagnostic errors are common and cause significant morbidity. Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves diagnostic reasoning. Objective: To assess the impact of the GPT-4 LLM on physicians' diagnostic reasoning compared to conventional resources. Design: Multi-center, randomized clinical vignette study. Setting: The study was conducted using remote video conferencing with physicians across the country and in-person participation across multiple academic medical institutions. Participants: Resident and attending physicians with training in family medicine, internal medicine, or emergency medicine. Interventions: Participants were randomized to access GPT-4 in addition to conventional diagnostic resources or to just conventional resources. They were allocated 60 minutes to review up to six clinical vignettes adapted from established diagnostic reasoning exams. Main Outcomes and Measures: The primary outcome was diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps. Secondary outcomes included time spent per case and final diagnosis. Results: 50 physicians (26 attendings, 24 residents) participated, with an average of 5.2 cases completed per participant. The median diagnostic reasoning score per case was 76.3 percent (IQR 65.8 to 86.8) for the GPT-4 group and 73.7 percent (IQR 63.2 to 84.2) for the conventional resources group, with an adjusted difference of 1.6 percentage points (95% CI -4.4 to 7.6; p=0.60). The median time spent on cases for the GPT-4 group was 519 seconds (IQR 371 to 668 seconds), compared to 565 seconds (IQR 456 to 788 seconds) for the conventional resources group, with a time difference of -82 seconds (95% CI -195 to 31; p=0.20). GPT-4 alone scored 15.5 percentage points (95% CI 1.5 to 29, p=0.03) higher than the conventional resources group. Conclusions and Relevance: In a clinical vignette-based study, the availability of GPT-4 to physicians as a diagnostic aid did not significantly improve clinical reasoning compared to conventional resources, although it may improve components of clinical reasoning such as efficiency. GPT-4 alone demonstrated higher performance than both physician groups, suggesting opportunities for further improvement in physician-AI collaboration in clinical practice.

Characteristics differentiating problem representation synthesis between novices and experts.

BACKGROUND: Formulating a thoughtful problem representation (PR) is fundamental to sound clinical reasoning and an essential component of medical education. Aside from basic structural recommendations, little consensus exists on what characterizes high-quality PRs. OBJECTIVES: To elucidate characteristics that distinguish PRs created by experts and novices. METHODS: Early internal medicine residents (novices) and inpatient teaching faculty (experts) from two academic medical centers were given two written clinical vignettes and were instructed to write a PR and three-item differential diagnosis for each. Deductive content analysis described the characteristics comprising PRs. An initial codebook of characteristics was refined iteratively. The primary outcome was differences in characteristic frequencies between groups. The secondary outcome was characteristics correlating with diagnostic accuracy. Mixed-effects regression with random effects modeling compared case-level outcomes by group. RESULTS: Overall, 167 PRs were analyzed from 30 novices and 54 experts. Experts included 0.8 fewer comorbidities (p < .01) and 0.6 more examination findings (p = .01) than novices on average. Experts were less likely to include irrelevant comorbidities (odds ratio [OR] = 0.4, 95% confidence interval [CI] = 0.2-0.8) or a diagnosis (OR = 0.3, 95% CI = 0.1-0.8) compared with novices. Experts encapsulated clinical data into higher-order terms (e.g., sepsis) than novices (p < .01) while including similar numbers of semantic qualifiers (SQs). Regardless of expertise level, PRs following a three-part structure (e.g., demographics, temporal course, and clinical syndrome) and including temporal SQs were associated with diagnostic accuracy (p < .01). CONCLUSIONS: Compared with novices, expert PRs include less irrelevant data and synthesize information into higher-order concepts. Future studies should determine whether targeted educational interventions for PRs improve diagnostic accuracy.

Consensus-Derived Recommended Skills for Transition to Residency Courses.

PURPOSE: Transition to residency (TTR) courses facilitate the medical student-residency transition and are an integral part of senior medical student training. The authors established a common set of skills for TTR courses, and an expected level of entrustment students should demonstrate in each skill on TTR course completion. METHOD: A modified Delphi approach was used with 3 survey iterations between 2020 and 2022 to establish skills to be included in a TTR course. Nine TTR experts suggested general candidate skills and conducted a literature search to ensure no vital skills were missed. A stakeholder panel was solicited from email lists of TTR educators, residency program directors, and residents at the panelists' institutions. Consensus was defined as more than 75% of participants selecting a positive inclusion response. An entrustment questionnaire asked panelists to assign a level of expected entrustment to each skill, with 1 indicating observation only and 6 indicating perform independently. RESULTS: The stakeholder panel initially consisted of 118 respondents with representation across educational contexts and clinical specialties. Response rates were 54% in iteration 2, 42% in iteration 3, and 33% on the entrustment questionnaire. After 3 iterations, 54 skills met consensus and were consolidated into 37 final skills categorized into 18 clinical skills (e.g., assessment and management of inpatient concerns), 14 communication skills (e.g., delivering serious news or having difficult conversations), 4 personal and professional skills (e.g., prioritization of clinical tasks), and 1 procedural skill (mask ventilation). Median entrustment levels were reported for all skills, with 19 skills having a level of expected entrustment of 4 (perform independently and have all findings double-checked). CONCLUSIONS: These consensus skills can serve as the foundation of a standardized national TTR curriculum framework. Entrustment guidance may help educational leaders optimize training and allocation of resources for TTR curriculum development and implementation.

Framework and Schema are False Synonyms: Defining Terms to Improve Learning.

Clinical reasoning is an essential expertise of health care professionals that includes the complex cognitive processes that lead to diagnosis and management decisions. In order to optimally teach, learn, and assess clinical reasoning, it is imperative for teachers and learners to have a shared understanding of the language. Currently, educators use the terms schema and framework interchangeably but they are distinct concepts. In this paper, we offer definitions for schema and framework and use the high-stakes field of aviation to demonstrate the interplay of these concepts. We offer examples of framework and schema in the medical education field and discuss how a clear understanding of these concepts allows for greater intentionality when teaching and assessing clinical reasoning.

Factors Associated With Health Care Professionals' Choice to Practice in Rural Minnesota.

Importance: Rural health inequities are due in part to a shortage of health care professionals in these areas. Objective: To determine the factors associated with health care professionals' decisions about where to practice. Design, Setting, and Participants: This prospective, cross-sectional survey study of health care professionals in Minnesota was administered by the Minnesota Department of Health from October 18, 2021, to July 25, 2022. Advanced practice registered nurses (APRNs), physicians, physician assistants (PAs), and registered nurses (RNs) renewing their professional licenses were eligible. Exposures: Individuals' ratings on survey items related to their choice of practice location. Main Outcomes and Measures: Rural or urban practice location as defined by the US Department of Agriculture's Rural-Urban Commuting Area typology. Results: A total of 32 086 respondents were included in the analysis (mean [SD] age, 44.4 [12.2] years; 22 728 identified as female [70.8%]). Response rates were 60.2% for APRNs (n = 2174), 97.7% for PAs (n = 2210), 95.1% for physicians (n = 11 019), and 61.6% for RNs (n = 16 663). The mean (SD) age of APRNs was 45.0 (10.3) years (1833 [84.3%] female); PAs, 39.0 (9.4) years (1648 [74.6%] female); physicians, 48.0 (11.9) years (4455 [40.4%] female); and RNs, 42.6 (12.3) years (14 792 [88.8%] female). Most respondents worked in urban (29 456 [91.8%]) vs rural (2630 [8.2%]) areas. Bivariate analysis suggested that family considerations are the most important determinant of practice location. Multivariate analysis revealed that having grown up in a rural area was the strongest factor associated with rural practice (odds ratio [OR] for APRNs, 3.44 [95% CI, 2.68-4.42]; OR for PAs, 3.75 [95% CI, 2.81-5.00]; OR for physicians, 2.44 [95% CI, 2.18-2.73]; OR for RNs, 3.77 [95% CI, 3.44-4.15]). When controlling for rural background, other associated factors included the availability of loan forgiveness (OR for APRNs, 1.42 [95% CI, 1.19-1.69]; OR for PAs, 1.60 [95% CI, 1.31-1.94]; OR for physicians, 1.54 [95% CI, 1.38-1.71]; OR for RNs, 1.20 [95% CI, 1.12-1.28]) and an educational program that prepared for rural practice (OR for APRNs, 1.44 [95% CI, 1.18-1.76]; OR for PAs. 1.70 [95% CI, 1.34-2.15]; OR for physicians, 1.31 [95% CI, 1.17-1.47]; OR for RNs, 1.23 [95% CI, 1.15-1.31]). Autonomy in one's work (OR for APRNs, 1.42 [95% CI, 1.08-1.86]; OR for PAs, 1.18 [95% CI, 0.89-1.58]; OR for physicians, 1.53 [95% CI, 1.31-1.78]; OR for RNs, 1.16 [95% CI, 1.07-1.25]) and a broad scope of practice (OR for APRNs, 1.46 [95% CI, 1.15-1.86]; OR for PAs, 0.96 [95% CI, 0.74-1.24]; OR for physicians, 1.62 [95% CI, 1.40-1.87]; OR for RNs, 0.96 [95% CI, 0.89-1.03]) were important factors associated with rural practice. Lifestyle and area considerations were not associated with rural practice; family considerations were associated with rural practice for RNs only (OR for APRNs, 0.97 [95% CI, 0.90-1.06]; OR for PAs, 0.95 [95% CI, 0.87-1.04]; OR for physicians, 0.92 [95% CI, 0.88-0.96]; OR for RNs, 1.05 [95% CI, 1.02-1.07]). Conclusions and Relevance: Understanding the interconnected factors involved in rural practice requires modeling relevant factors. The findings of this survey study suggest that loan forgiveness, rural training, autonomy, and a broad scope of practice are factors associated with rural practice for most health care professionals. Other factors associated with rural practice vary by profession, suggesting that there may not be a one-size-fits-all approach to recruitment of rural health care professionals.

Exploring relationships between physician stress, burnout, and diagnostic elements in clinician notes.

OBJECTIVES: To understand the relationship between stressful work environments and patient care by assessing work conditions, burnout, and elements of the diagnostic process. METHODS: Notes and transcripts of audiotaped encounters were assessed for verbal and written documentation related to psychosocial data, differential diagnosis, acknowledgement of uncertainty, and other diagnosis-relevant contextual elements using 5-point Likert scales in seven primary care physicians (PCPs) and 28 patients in urgent care settings. Encounter time spent vs time needed (time pressure) was collected from time stamps and clinician surveys. Study physicians completed surveys on stress, burnout, and work conditions using the Mini-Z survey. RESULTS: Physicians with high stress or burnout were less likely to record psychosocial information in transcripts and notes (psychosocial information noted in 0% of encounters in 4 high stress/burned-out physicians), whereas low stress physicians (n=3) recorded psychosocial information consistently in 67% of encounters. Burned-out physicians discussed a differential diagnosis in only 31% of encounters (low counts concentrated in two physicians) vs. in 73% of non-burned-out doctors' encounters. Burned-out and non-burned-out doctors spent comparable amounts of time with patients (about 25 min). CONCLUSIONS: Key diagnostic elements were seen less often in encounter transcripts and notes in burned-out urgent care physicians.