Evaluating prompt engineering on GPT-3.5's performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4.

This study was designed to assess how different prompt engineering techniques, specifically direct prompts, Chain of Thought (CoT), and a modified CoT approach, influence the ability of GPT-3.5 to answer clinical and calculation-based medical questions, particularly those styled like the USMLE Step 1 exams. To achieve this, we analyzed the responses of GPT-3.5 to two distinct sets of questions: a batch of 1000 questions generated by GPT-4, and another set comprising 95 real USMLE Step 1 questions. These questions spanned a range of medical calculations and clinical scenarios across various fields and difficulty levels. Our analysis revealed that there were no significant differences in the accuracy of GPT-3.5's responses when using direct prompts, CoT, or modified CoT methods. For instance, in the USMLE sample, the success rates were 61.7% for direct prompts, 62.8% for CoT, and 57.4% for modified CoT, with a p-value of 0.734. Similar trends were observed in the responses to GPT-4 generated questions, both clinical and calculation-based, with p-values above 0.05 indicating no significant difference between the prompt types. The conclusion drawn from this study is that the use of CoT prompt engineering does not significantly alter GPT-3.5's effectiveness in handling medical calculations or clinical scenario questions styled like those in USMLE exams. This finding is crucial as it suggests that performance of ChatGPT remains consistent regardless of whether a CoT technique is used instead of direct prompts. This consistency could be instrumental in simplifying the integration of AI tools like ChatGPT into medical education, enabling healthcare professionals to utilize these tools with ease, without the necessity for complex prompt engineering.

Association of red blood cell distribution width with hospital admission and in-hospital mortality across all-cause adult emergency department visits.

Objectives: To test the association between the initial red blood cell distribution width (RDW) value in the emergency department (ED) and hospital admission and, among those admitted, in-hospital mortality. Materials and Methods: We perform a retrospective analysis of 210 930 adult ED visits with complete blood count results from March 2013 to February 2022. Primary outcomes were hospital admission and in-hospital mortality. Variables for each visit included demographics, comorbidities, vital signs, basic metabolic panel, complete blood count, and final diagnosis. The association of each outcome with the initial RDW value was calculated across 3 age groups (<45, 45-65, and >65) as well as across 374 diagnosis categories. Logistic regression (LR) and XGBoost models using all variables excluding final diagnoses were built to test whether RDW was a highly weighted and informative predictor for each outcome. Finally, simplified models using only age, sex, and vital signs were built to test whether RDW had additive predictive value. Results: Compared to that of discharged visits (mean [SD]: 13.8 [2.03]), RDW was significantly elevated in visits that resulted in admission (15.1 [2.72]) and, among admissions, those resulting in intensive care unit stay (15.3 [2.88]) and/or death (16.8 [3.25]). This relationship held across age groups as well as across various diagnosis categories. An RDW >16 achieved 90% specificity for hospital admission, while an RDW >18.5 achieved 90% specificity for in-hospital mortality. LR achieved a test area under the curve (AUC) of 0.77 (95% confidence interval [CI] 0.77-0.78) for hospital admission and 0.85 (95% CI 0.81-0.88) for in-hospital mortality, while XGBoost achieved a test AUC of 0.90 (95% CI 0.89-0.90) for hospital admission and 0.96 (95% CI 0.94-0.97) for in-hospital mortality. RDW had high scaled weights and information gain for both outcomes and had additive value in simplified models predicting hospital admission. Discussion: Elevated RDW, previously associated with mortality in myocardial infarction, pulmonary embolism, heart failure, sepsis, and COVID-19, is associated with hospital admission and in-hospital mortality across all-cause adult ED visits. Used alone, elevated RDW may be a specific, but not sensitive, test for both outcomes, with multivariate LR and XGBoost models showing significantly improved test characteristics. Conclusions: RDW, a component of the complete blood count panel routinely ordered as the initial workup for the undifferentiated patient, may be a generalizable biomarker for acuity in the ED.

How Patients Use a Patient Portal: An Institutional Case Study of Demographics and Usage Patterns.

BACKGROUND: Given the widespread electronic health record adoption, there is increasing interest to leverage patient portals to improve care. OBJECTIVE: To determine characteristics of patient portal users and the activities they accessed in the patient portal. METHODS: We performed a retrospective analysis of patient portal usage at University of California, Los Angeles, Health from July 2014 to May 2015. A total dataset of 505,503 patients was compiled with 396,303 patients who did not register for the patient portal and 109,200 patients who registered for a patient portal account. We compared patients who did not register for the online portal to the top 75th percentile of users based on number of logins, which was done to exclude those who only logged in to register. Finally, to avoid doing statistical analysis on too large of a sample and overpower the analysis, we performed statistical tests on a random sample of 300 patients in each of the two groups. RESULTS: Patient portal users tended to be older (49.45 vs. 46.22 years in the entire sample, p = 0.008 in the random sample) and more likely female (62.59 vs. 54.91% in the entire sample, p = 0.035 in the random sample). Nonusers had more monthly emergency room (ER) visits on average (0.047 vs. 0.014, p < 0.001). The most frequently accessed activity on the portal was viewing laboratory results (79.7% of users looked at laboratory results). CONCLUSION: There are differences between patient portal users and nonusers, and further understanding of these differences can serve as foundation for further investigation and possible interventions to drive patient engagement and health outcomes.

Characteristics of the National Applicant Pool for Clinical Informatics Fellowships (2016-2017).

We conducted a national study to assess the numbers and diversity of applicants for 2016 and 2017 clinical informatics fellowship positions. In each year, we collected data on the number of applications that programs received from candidates who were ultimately successful vs. unsuccessful. In 2017, we also conducted an anonymous applicant survey. Successful candidates applied to an average of 4.2 and 5.5 programs for 2016 and 2017, respectively. In the survey, unsuccessful candidates reported applying to fewer programs. Assuming unsuccessful candidates submitted between 2-5 applications each, the total applicant pool numbered 42-69 for 2016 (competing for 24 positions) and 52-85 for 2017 (competing for 30 positions). Among survey respondents (n=33), 24% were female, 1 was black and none were Hispanic. We conclude that greater efforts are needed to enhance interest in clinical informatics among medical students and residents, particularly among women and members of underrepresented minority groups.