Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study.

Importance: Diagnostic errors are common and cause significant morbidity. Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves diagnostic reasoning. Objective: To assess the impact of the GPT-4 LLM on physicians' diagnostic reasoning compared to conventional resources. Design: Multi-center, randomized clinical vignette study. Setting: The study was conducted using remote video conferencing with physicians across the country and in-person participation across multiple academic medical institutions. Participants: Resident and attending physicians with training in family medicine, internal medicine, or emergency medicine. Interventions: Participants were randomized to access GPT-4 in addition to conventional diagnostic resources or to just conventional resources. They were allocated 60 minutes to review up to six clinical vignettes adapted from established diagnostic reasoning exams. Main Outcomes and Measures: The primary outcome was diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps. Secondary outcomes included time spent per case and final diagnosis. Results: 50 physicians (26 attendings, 24 residents) participated, with an average of 5.2 cases completed per participant. The median diagnostic reasoning score per case was 76.3 percent (IQR 65.8 to 86.8) for the GPT-4 group and 73.7 percent (IQR 63.2 to 84.2) for the conventional resources group, with an adjusted difference of 1.6 percentage points (95% CI -4.4 to 7.6; p=0.60). The median time spent on cases for the GPT-4 group was 519 seconds (IQR 371 to 668 seconds), compared to 565 seconds (IQR 456 to 788 seconds) for the conventional resources group, with a time difference of -82 seconds (95% CI -195 to 31; p=0.20). GPT-4 alone scored 15.5 percentage points (95% CI 1.5 to 29, p=0.03) higher than the conventional resources group. Conclusions and Relevance: In a clinical vignette-based study, the availability of GPT-4 to physicians as a diagnostic aid did not significantly improve clinical reasoning compared to conventional resources, although it may improve components of clinical reasoning such as efficiency. GPT-4 alone demonstrated higher performance than both physician groups, suggesting opportunities for further improvement in physician-AI collaboration in clinical practice.

Rethinking the consultation paradigm: Validity evidence for a new framework, a multimethods study.

BACKGROUND: In-hospital consultation is essential for patient care. We previously proposed a framework of seven specific consultation types to classify consult requests to improve communication, workflow, and provider satisfaction. METHODS: This multimethods study's aim was to evaluate the applicability of the consult classification framework to real internal medicine (IM) consults. We sought validity evidence using Kane's validity model with focus groups and classifying consult requests from five IM specialties. Participants attended five 1 h semi-structured focus groups that were recorded, transcribed, and coded for thematic saturation. For each specialty, three specialists and three hospitalists categorized 100 (total 500) random anonymized consult requests. The primary outcome was concordance in the classification of consult requests, defined as the sum of partial concordance and perfect concordance, where respectively 4-5/6 and 6/6 participants classified a consult in the same category. We used χ2 tests to compare concordance rates across specialties and between specialists and hospitalists. RESULTS: Five major themes were identified in the qualitative analysis of the focus groups: (1) consult question, (2) interpersonal interactions, (3) value, (4) miscommunication, (5) consult framework application, barriers, and iterative development. In the quantitative analysis, the overall concordance rate was 88.8% (95% confidence interval [CI]: 85.7-91.4), and perfect concordance was 46.6% (95% CI: 42.2-51.1). Concordance differed significantly between hospitalists and specialists overall (p = .01), with a higher proportion of hospitalists having perfect concordance compared to specialists (67.2% vs. 57.8%, p = .002). CONCLUSIONS: The consult classification framework was found to be applicable to consults from five different IM specialties, and could improve communication and education.

Validation of an Educational Tool for Skin Abscess Incision and Drainage by Delphi and Angoff Methods.

BACKGROUND: Bedside incision and drainage (I&D) of skin abscesses is a common medical procedure performed in a variety of medical settings. Yet, there is a paucity of published validated educational tools to teach and assess competency for this procedure. OBJECTIVE: To validate an educational tool to teach and assess competency for bedside I&D of skin abscesses via the Delphi consensus and Angoff standard setting methods. DESIGN: Expert consensus on the importance of each procedural step in the educational tool was obtained using the Delphi method, consisting of four rounds of iterative revisions based on input from a panel of experts. The passing cut-off score for a proficient provider was determined using the modified dichotomous Angoff method. PARTICIPANTS: All participants met the minimum criteria of active involvement in resident education and performance of at least 20 skin abscess I&D's within the past 5 years. Participant specialties included general surgery, emergency medicine, and internal medicine. MAIN MEASURES: The primary outcome was consensus on procedural steps and errors, defined as an interquartile range ≤ 2 on a 9-point Likert scale. A cut-off score was determined by the average across all respondents for the anticipated number of errors that would be committed by a provider with the level of proficiency defined in the survey. Qualitative input was incorporated into the educational tool. KEY RESULTS: At the end of four rounds of review via the Delphi process, participants achieved consensus on 93% of items on the clinical checklist and 85% of errors on the assessment checklist. Via the modified dichotomous Angoff method, the determined passing cut-off for competency was 6 out of 22 errors. CONCLUSION: An educational and evaluation tool for bedside I&D of skin abscesses was validated via the Delphi and Angoff methods.

Procedural Competency Among Hospitalists: A Literature Review and Future Considerations.

BACKGROUND: As general internists practicing in the inpatient setting, hospitalists at many institutions are expected to perform invasive bedside procedures, as defined by professional standards. In reality, hospitalists are doing fewer procedures and increasingly are referring to specialists, which threatens their ability to maintain procedural skills. The discrepancy between expectations and reality, especially when hospitalists may be fully credentialed to perform procedures, poses significant risks to patients because of morbidity and mortality associated with complications, some of which derive from practitioner inexperience. METHODS: We performed a structured search of the peer-reviewed literature to identify articles focused on hospitalists performing procedures. RESULTS: Our synthesis of the literature characterizes contributors to hospitalists' procedural competency and discusses: (1) temporal trends for procedures performed by hospitalists and their associated referral patterns, (2) data comparing use and clinical outcomes of procedures performed by hospitalists compared with specialists, (3) the lack of nationwide standardization of hospitalist procedural training and credentialing, and (4) the role of medical procedure services, although limited in supportive evidence, in concentrating procedural skill and mitigating risk in the hands of a few well-trained hospitalists. CONCLUSION: We conclude with recommendations for hospital medicine groups to ensure the safety of hospitalized patients undergoing bedside procedures.

Portal vein thrombosis prevalence and associated mortality in cirrhosis in a nationally representative inpatient cohort.

BACKGROUND AND AIM: Portal vein thrombosis (PVT) is increasingly common in cirrhotics, but its impact on mortality and outcomes is unclear. Studies evaluating PVT have been limited by small sample size. This study analyzes the trend of the prevalence of PVT and its associated mortality in hospitalized decompensated cirrhotics. METHODS: The Nationwide Inpatient Sample, the largest nationally representative database of hospital discharges, was queried from 1998 to 2014. Inpatients older than 18 years with decompensated cirrhosis were included, while those who received liver transplantation or had hepatocellular carcinoma were excluded. The primary outcomes were the trend in prevalence and associated mortality with PVT. Secondary outcomes included identifying risk factors of PVT and the effect of PVT on complications of portal hypertension. Multivariable logistic regression evaluated the outcomes. RESULTS: A total of 3 045 098 discharges were included, of which 1.5% had PVT. PVT prevalence increased from 0.7% to 2.4%, annual percent change of 9%. Mortality associated with PVT declined from 11.9% to 9.1%, annual percent change of -3.0%. In multivariable analysis controlling for factors associated with mortality in cirrhotics, PVT was associated with an increased risk of mortality (OR 1.12, P < 0.001). Multivariable logistic regression also demonstrated that PVT significantly increased the risk of acute kidney injury (OR 1.75, P < 0.001) and hepatorenal syndrome (OR 1.62, P < 0.001). CONCLUSIONS: The prevalence of PVT is increasing while its associated mortality is decreasing. However, PVT still is associated with risk of mortality and kidney injury, implying a significant impact on cirrhotic outcomes.

Subarachnoid Hemorrhage and Long-Term Stroke Risk After Traumatic Brain Injury.

BACKGROUND: Recent studies suggest that traumatic brain injury (TBI) is a risk factor for subsequent ischemic stroke, even years after the initial insult. The mechanisms of the association remain unclear. The presence of traumatic subarachnoid hemorrhage (tSAH) may mediate the effect of TBI on long-term stroke risk, as it has previously been linked to short-term vasospasm and delayed cerebral ischemia. METHODS: Using administrative claims data, we conducted a retrospective cohort study of acute care hospitalizations. Patients discharged with a first-recorded diagnosis of tSAH were followed for a primary diagnosis of stroke. They were matched to patients with TBI but not tSAH. Cox proportional hazards modeling was used to assess the association between tSAH and stroke while adjusting for covariates. RESULTS: We identified 40 908 patients with TBI (20 454 patients with tSAH) who were followed for a mean of 4.3 + 1.8 years. A total of 531 had an ischemic stroke after discharge. There was no significant difference in stroke risk between those with tSAH (1.79%; 95% confidence interval [CI] 1.54%-2.08%) versus without tSAH (2.12%; 95% CI 1.83%-2.44%). The same pattern was found in adjusted analyses even when the group was stratified by age-group or by proxies of TBI severity. CONCLUSIONS: Our findings do not support a role of tSAH in mediating the association between TBI and protracted stroke risk. Further study is required to elucidate the mechanisms of long-term increased stroke risk after TBI.

Stroke Risk and Mortality in Patients With Ventricular Assist Devices.

BACKGROUND AND PURPOSE: Ventricular assist devices (VADs) have advanced the management of end-stage heart failure. However, these devices are associated with hemorrhagic and thrombotic complications, including stroke. We assessed the incidence, risk factors, and outcomes of ischemic and hemorrhagic stroke after VAD placement. METHODS: Using administrative claims data from acute care hospitals in California, Florida, and New York from 2005 to 2013, we identified patients who underwent VAD placement, defined by the International Classification of Diseases, Ninth Revision, Clinical Modification code 37.66. Ischemic and hemorrhagic strokes were identified by previously validated coding algorithms. We used survival statistics to determine the incidence rates and Cox proportional hazard analyses to examine the associations. RESULTS: Among 1813 patients, we identified 201 ischemic strokes and 116 hemorrhagic strokes during 3.4 (±2.0) years of follow-up after implantation of a VAD. The incidence of stroke was 8.7% per year (95% confidence interval [CI], 7.7-9.7). The annual incidence of ischemic stroke (5.5%; 95% CI, 4.8-6.4) was nearly double that of hemorrhagic stroke (3.1%; 95% CI, 2.6-3.8). Women faced a higher hazard of stroke than men (hazard ratio, 1.6; 95% CI, 1.2-2.1), particularly hemorrhagic stroke (hazard ratio, 2.2; 95% CI, 1.4-3.4). Stroke was strongly associated with subsequent in-hospital mortality (hazard ratio, 6.1; 95% CI, 4.6-7.9). CONCLUSIONS: The incidence of stroke after VAD implantation was 8.7% per year, and incident stroke was strongly associated with subsequent in-hospital mortality. Notably, ischemic stroke occurred at nearly twice the rate of hemorrhagic stroke. Women seemed to face a higher risk for hemorrhagic stroke than men.