Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study.

BACKGROUND: Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients. OBJECTIVE: The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews. METHODS: We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated "unsafe" or "too cautious." RESULTS: Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%). CONCLUSIONS: ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation.

Improving the online presence of residency programs to ameliorate COVID-19's impact on residency applications.

Objectives: The coronavirus disease 2019 (COVID-19) pandemic has created several challenges for residency programs and prospective interns alike during the upcoming application cycle, including the cancellation of away sub-internships and in-person interviews. Given prior research documenting that applicants' application and ranking decisions are significantly influenced by residency webpages, a potential solution to the loss of in-person experiences during the pandemic is the expansion of residency programs' online presence through their program websites, provision of virtual grand rounds and pseudo-away rotations, and enhancement of virtual interviews. This study seeks to summarize the existing literature on these areas and provide concrete suggestions for improving programs' virtual presence.Methods: The authors summarize earlier literature querying the content of program websites across 14 medical specialties, which documented significant gaps in the content of interest to applicants.Results: Among 14 analyzed specialties, the majority of programs had a functional website (>90%), with the exception of interventional radiology (73.9%). However, significant gaps in content were documented, with the percentage of content variables contained on websites ranging from 33.3% to 70.5% (median = 47.0%, interquartile range = 37.8-52.6%). Program websites were also limited by underrepresentation of content most valued by applicants as well as potential areas of inaccurate or outdated information.Conclusions: There are several interventions programs can undertake to address existing gaps in online presence. During an application cycle facing unprecedented resource strain, bolstering the online presence of programs may facilitate an improved fit between programs and future residents.

Enforcing the "4T": An In-Line Calculator for HIT Antibody Ordering in the Electronic Medical Record.

Heparin-induced thrombocytopenia (HIT) remains a difficult clinical diagnosis, even with the under-utilized standardized scoring systems, like the '4T' score, to aid in clinical decision-making. Our quality improvement study sought to assess the use of '4T' score, improve the use of HIT antibody (HITA) testing and improvement management of possible HIT by implementing an in-line calculator with guidance within our electronic medical record (EMR) at our institution. We retrospectively reviewed patient charts between October 2017 and October 2018, assessing practices before and after implementation of the '4T' in-line calculator in April 2018. HITA were ordered inappropriately (for 4T <4) in 141 (67%) of 210 instances (75 before and 66 after). We found no statistically significant difference in positive predictive value (PPV) or 4T documentation in provider notes after its implementation. We were able to identify problematic areas in HIT management, such as the ordering of non-heparin anticoagulants, and implement additional changes addressing these problems.

Are specific elements of electronic health record use associated with clinician burnout more than others?

OBJECTIVE: The study sought to examine the association between clinician burnout and measures of electronic health record (EHR) workload and efficiency, using vendor-derived EHR action log data. MATERIALS AND METHODS: We combined data from a statewide clinician survey on burnout with Epic EHR data from the ambulatory sites of 2 large health systems; the combined dataset included 422 clinicians. We examined whether specific EHR workload and efficiency measures were independently associated with burnout symptoms, using multivariable logistic regression and controlling for clinician characteristics. RESULTS: Clinicians with the highest volume of patient call messages had almost 4 times the odds of burnout compared with clinicians with the fewest (adjusted odds ratio, 3.81; 95% confidence interval, 1.44-10.14; P = .007). No other workload measures were significantly associated with burnout. No efficiency variables were significantly associated with burnout in the main analysis; however, in a subset of clinicians for whom note entry data were available, clinicians in the top quartile of copy and paste use were significantly less likely to report burnout, with an adjusted odds ratio of 0.22 (95% confidence interval, 0.05-0.93; P = .039). DISCUSSION: High volumes of patient call messages were significantly associated with clinician burnout, even when accounting for other measures of workload and efficiency. In the EHR, "patient calls" encompass many of the inbox tasks occurring outside of face-to-face visits and likely represent an important target for improving clinician well-being. CONCLUSIONS: Our results suggest that increased workload is associated with burnout and that EHR efficiency tools are not likely to reduce burnout symptoms, with the exception of copy and paste.

Prescription Opioid Use Among Patients With Acute Gout Discharged From the Emergency Department.

OBJECTIVE: Acute gout is among the most painful inflammatory arthritides and a frequent cause of emergency department (ED) visits. Prescription opioids are the leading contributor to the ongoing opioid epidemic; EDs are often the source of the index prescription. Our aim was to assess the burden of opioid use and factors associated with its use among gout patients discharged from the ED. METHODS: In the electronic health records system of Lifespan Healthcare System (currently contains 2.2 million records), adult gout patients discharged from the ED or hospital were identified using International Classification of Diseases, Ninth Revision or Tenth Revision diagnostic codes. The study period was March 2015 to September 2017, and only patients with a primary diagnosis of gout were included. If a patient was seen multiple times, only the first encounter was included. For these patients, we estimated the frequency, dose, and duration of opioids prescribed. Using multivariable logistic regression, we ascertained the factors associated with increased odds of opioid prescription at discharge among patients with acute gout. RESULTS: Of the 456 patients, 129 (28.3%) received opioids at discharge (~80% were new patients). The average dose of prescription was mean ± SD 37.9 ± 17.2 mg of morphine equivalent for a median duration of 8 days (interquartile range 5-14). We noted that patients with polyarticular gout attack and diabetes mellitus and those taking opioids prior to admission had higher odds of receiving opioids at discharge. CONCLUSION: Despite the availability of effective treatments, opioids are commonly used for the management of acute gout. This study highlights an opportunity to curb the opioid epidemic among gout patients.