Foundations of Emergency Medicine: Impact of a Standardized, Open-access, Core Content Curriculum on In-Training Exam Scores.

Introduction: Learners frequently benefit from modalities such as small-group, case-based teaching and interactive didactic experiences rather than passive learning methods. These contemporary techniques are features of Foundations of Emergency Medicine (FoEM) curricula, and particularly the Foundations I (F1) course, which targets first-year resident (PGY-1) learners. The American Board of Emergency Medicine administers the in-training exam (ITE) that provides an annual assessment of EM-specific medical knowledge. We sought to assess the effect of F1 implementation on ITE scores. Methods: We retrospectively analyzed data from interns at four EM residency programs accredited by the Accreditation Council for Graduate Medical Education. We collected data in 2021. Participating sites were geographically diverse and included three- and four-year training formats. We collected data from interns two years before (control group) and two years after (intervention group) implementation of F1 at each site. Year of F1 implementation ranged from 2015-2018 at participating sites. We abstracted data using a standard form including program, ITE raw score, year of ITE administration, US Medical Licensing Exam Step 1 score, Step 2 Clinical Knowledge (CK) score, and gender. We performed univariable and multivariable linear regression to explore differences between intervention and control groups. Results: We collected data for 180 PGY-1s. Step 1 and Step 2 CK scores were significant predictors of ITE in univariable analyses (both with P < 0.001). After accounting for Step 1 and Step 2 CK scores, we did not find F1 implementation to be a significant predictor of ITE score, P = 0.83. Conclusion: Implementation of F1 curricula did not show significant changes in performance on the ITE after controlling for important variables.

Individualized empirical null estimation for exact tests of healthcare quality.

United States federal agencies evaluate healthcare providers to identify, flag, and potentially penalize those that deliver low-quality care compared to national expectations. In practice, evaluation metrics are inevitably impacted by unobserved confounding factors, which reduce flagging accuracy and cause the statistics to be overdispersed relative to the theoretical null distributions. In response to this issue, several authors have proposed individualized empirical null (IEN) methods to estimate an appropriate null distribution for each provider's evaluation statistic while taking into account the provider's effective size. However, existing IEN methods require that the statistics asymptotically follow normal distributions, which often does not hold in applications with small providers or misspecified models. In this article, we develop an IEN framework for exact hypothesis tests that accounts for the impact of unobserved confounding without making any asymptotic assumptions. Simulations show that the proposed IEN method has greater flagging accuracy compared to conventional approaches. We apply these methods to evaluate dialysis facilities and transplant centers that are monitored by the Centers for Medicare and Medicaid Services.

The more things change the more they stay the same: Factors influencing emergency medicine residency selection in the virtual era.

Background: Interviews for emergency medicine (EM) residency positions largely transitioned to a virtual-only format in 2020-2021. The impact of virtual interview factors on applicants' rank of programs is unknown. Objective: We sought to assess the impact of modifiable factors in virtual interviews on applicants' rank of EM residency programs. Methods: We conducted a cross-sectional mixed-methods survey of students applying to at least one of seven study authors' EM residency programs in the United States during the 2020-2021 application cycle. The survey was developed using an interactive Delphi process and piloted prior to implementation. The survey was administered from May to June 2021 with up to four email reminders. Quantitative analysis included descriptive statistics. Three authors performed a thematic qualitative analysis of free-text responses. Results: A total of 664 of 2281 (29.1%) students completed the survey, including 335 (50.5%) male, 316 (47.7%) female, and six (0.9%) nonbinary. A total of 143 (21.6%) respondents identified as underrepresented in medicine and 84 (12.7%) identified as LGBTQIA+. Respondents participated in a median of 14 interviews and ranked a median of 14 programs. Most respondents (335, 50.6%) preferred a choice of in-person or virtual, while 183 (27.6%) preferred all in-person, and 144 (21.8%) preferred all virtual. The program website and interview social were the most important factors influencing respondent ranking. Qualitative analysis revealed several positive aspects of virtual interviews including logistical ease and comfort. Negative aspects include technical issues, perceived interview hoarding, and barriers to applicant assessment and performance. Demonstrated effort by the program, effective information delivery, communication of resident culture, and a well-implemented interview day positively influenced respondents' rank of programs. Conclusions: This study identified characteristics of the virtual interview format that impact applicants' rank of programs. These results can inform future recruitment practices.

Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Reconceptualizing the emergency medicine resident scholarly requirement: Proposed framework and rubric.

Background: The completion of a scholarly project is a common program requirement by the Accreditation Council for Graduate Medical Education (ACGME) for all residency training programs. However, the implementation can vary significantly between programs. Lack of generalizable standards for scholarly projects required of all trainees within ACGME-accredited residencies has led to a large range of quality and effort put forth to complete these projects. Our goal is to introduce a framework and propose a corresponding rubric for application to resident scholarship to quantify and qualify the components of scholarship to better measure resident scholarly output across the graduate medical education (GME) continuum. Methods: Eight experienced educators and members of the Society for Academic Emergency Medicine Education Committee were selected to explore the current scholarly project guidelines and propose a definition that can be universally applied to diverse training programs. Following a review of the current literature, the authors engaged in iterative, divergent, and convergent discussions via meetings and asynchronous dialogue to develop a framework and associated rubric. Results: The group proposes that emergency medicine (EM) resident scholarship should (1) involve a structured process, (2) generate outcomes, (3) be disseminated, and (4) be peer reviewed. These components of resident scholarly activity are achieved whether this is a single project encompassing all four domains, or multiple smaller projects that sum to the whole. To assist residency programs in assessing a given individual resident's achievement of the standards set forth, a rubric is proposed. Conclusion: Based on current literature and consensus, we propose a framework and rubric for tracking of resident scholarly project achievement in an effort to elevate and advance EM scholarship. Future work should explore the optimal application of this framework and define minimal scholarship goals for EM resident scholarship.

Pitfalls of the concordance index for survival outcomes.

Prognostic models are useful tools for assessing a patient's risk of experiencing adverse health events. In practice, these models must be validated before implementation to ensure that they are clinically useful. The concordance index (C-Index) is a popular statistic that is used for model validation, and it is often applied to models with binary or survival outcome variables. In this paper, we summarize existing criticism of the C-Index and show that many limitations are accentuated when applied to survival outcomes, and to continuous outcomes more generally. We present several examples that show the challenges in achieving high concordance with survival outcomes, and we argue that the C-Index is often not clinically meaningful in this setting. We derive a relationship between the concordance probability and the coefficient of determination under an ordinary least squares model with normally distributed predictors, which highlights the limitations of the C-Index for continuous outcomes. Finally, we recommend existing alternatives that more closely align with common uses of survival models.

Concordance indices with left-truncated and right-censored data.

In the context of time-to-event analysis, a primary objective is to model the risk of experiencing a particular event in relation to a set of observed predictors. The Concordance Index (C-Index) is a statistic frequently used in practice to assess how well such models discriminate between various risk levels in a population. However, the properties of conventional C-Index estimators when applied to left-truncated time-to-event data have not been well studied, despite the fact that left-truncation is commonly encountered in observational studies. We show that the limiting values of the conventional C-Index estimators depend on the underlying distribution of truncation times, which is similar to the situation with right-censoring as discussed in Uno et al. (2011) [On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine 30(10), 1105-1117]. We develop a new C-Index estimator based on inverse probability weighting (IPW) that corrects for this limitation, and we generalize this estimator to settings with left-truncated and right-censored data. The proposed IPW estimators are highly robust to the underlying truncation distribution and often outperform the conventional methods in terms of bias, mean squared error, and coverage probability. We apply these estimators to evaluate a predictive survival model for mortality among patients with end-stage renal disease.

Quality assessment of optic nerve sheath diameter ultrasonography: Scoping literature review and Delphi protocol.

BACKGROUND AND PURPOSE: The optic nerve is surrounded by the extension of meningeal coverings of the brain. When the pressure in the cerebrospinal fluid increases, it causes a distention of the optic nerve sheath diameter (ONSD), which allows the use of this measurement by ultrasonography (US) as a noninvasive surrogate of elevated intracranial pressure. However, ONSD measurements in the literature have exhibited significant heterogeneity, suggesting a need for consensus on ONSD image acquisition and measurement. We aim to establish a consensus for an ONSD US Quality Criteria Checklist (ONSD US QCC). METHODS: A scoping systematic review of published ultrasound ONSD imaging and measurement criteria was performed to guide the development of a preliminary ONSD US QCC that will undergo a modified Delphi study to reach expert consensus on ONSD quality criteria. The protocol of this modified Delphi study is presented in this manuscript. RESULTS: A total of 357 ultrasound studies were included in the review. Quality criteria were evaluated under five categories: probe selection, safety, positioning, image acquisition, and measurement. CONCLUSIONS: This review and Delphi protocol aim to establish ONSD US QCC. A broad consensus from this process may reduce the variability of ONSD measurements in future studies, which would ultimately translate into improved ONSD clinical applications. This protocol was reviewed and endorsed by the German Society of Ultrasound in Medicine.

Effect of Perceived Level of Interaction on Faculty Evaluations of 3rd Year Medical Students.

INTRODUCTION: Several factors are known to affect the way clinical performance evaluations (CPEs) of medical students are completed by supervising physicians. We sought to explore the effect of faculty perceived "level of interaction" (LOI) on these evaluations. METHODS: Our third-year CPE requires evaluators to identify perceived LOI with each student as low, moderate, or high. We examined CPEs completed during the academic year 2018-2019 for differences in (1) clinical and professionalism ratings, (2) quality of narrative comments, (3) quantity of narrative comments, and (4) percentage of evaluation questions left unrated. RESULTS: A total of 3682 CPEs were included in the analysis. ANOVA revealed statistically significant differences between LOI and clinical ratings (p ≤ .001), with mean ratings from faculty with a high LOI significantly higher than from faculty with a moderate or low LOI (p ≤ .001). Chi-squared analysis demonstrated differences based on faculty LOI and whether questions were left unrated (p ≤ .001), quantity of narrative comments (p ≤ .001), and specificity of narrative comments (p ≤ .001). CONCLUSIONS: Faculty who perceive higher LOI were more likely to assign that student higher ratings, complete more of the clinical evaluation and were more likely to provide narrative feedback with more specific, higher-quality comments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01307-w.

The Impact of Shortened Clinical Clerkships on Medical Student Performance and Clerkship Assessment.

INTRO: Medical schools sometimes need to adjust the length of third-year clinical clerkships. The literature surrounding the effects of shortened clerkships on student experience and performance is mixed. METHODS: Our medical school shortened the third year by an average of 20% per clerkship to accommodate a curricular re-design in 2018-2019. We examined test scores and measures of clinical performance as well as student experience in order to understand the impact of this change. RESULTS: Two hundred and eight students were included in the analysis, 104 in each cohort. No statistically significant differences were noted between cohorts on NBME subject examination results. There were no significant differences on Step 2 CK scores between the traditional curriculum cohort (M = 249.4, SD = 13.7) and shortened curriculum cohort (M = 248.7, SD = 15.8). Student performance on OSCE cases was similar. Similar percentages of students rated each clerkship either "good" or "excellent" in the traditional (77%) and shortened (78%) curriculum. CONCLUSION: There was no significant impact on student test scores after shortening the curriculum. Measures of student satisfaction and experience also remained stable, likely related to emphasis on retaining patient care experiences and streamlining of didactics. Curricular shortening during the third year of medical school was feasible and safe from the student perspective in our experience.

A Model Research Curriculum for Emergency Medicine Residency: A Modified Delphi Consensus.

BACKGROUND: Research training is important for all resident physicians to be able to interpret and critically appraise scientific literature. It is particularly important for those desiring future careers in academics or research. However, there is limited literature on research training in residency. The ideal scope and content of a model curriculum for research in emergency medicine (EM) residency programs has not been recently defined. METHODS: We used a modified Delphi method to achieve expert consensus as to what content should be included in a model research curriculum for EM residents as well as for those who choose to undertake an elective in EM research. RESULTS: Eight experts in EM clinical and education research participated in the modified Delphi process with two rounds of responses. Cronbach's alpha was 0.82 for round 1 and 0.95 for round 2. A curricular outline of 44 items was produced by this consensus process. CONCLUSIONS: A model curriculum for EM research is proposed here, along with references to assist residencies in building curricular components.

Academic Emergency Medicine Faculty Experiences with Racial and Sexual Orientation Discrimination.

INTRODUCTION: Despite the increasing diversity of individuals entering medicine, physicians from racial and sexual minority groups continue to experience bias and discrimination in the workplace. The objective of this study was to determine the current experiences and perceptions of discrimination on the basis of race and sexual orientation among academic emergency medicine (EM) faculty. METHODS: We conducted a cross-sectional survey of a convenience sample of EM faculty across six programs. Survey items included the Overt Gender Discrimination at Work (OGDW) Scale adapted for race and sexual orientation, and the frequency and source of experienced and observed discrimination. Group comparisons were made using t-tests or chi-square analyses, and relationships between race or sexual orientation, and we evaluated physicians' experiences using correlation analyses. RESULTS: A total of 141 out of 352 (40.1%) subjects completed at least a portion of the survey. Non-White physicians reported higher mean racial OGDW scores than their White counterparts (13.4 vs 8.6; 95% confidence interval (CI) for difference, -7.7 - -2.9). Non-White EM faculty were also more likely to report having experienced discriminatory treatment based on race than were White EM faculty (48.0% vs 12.6%; CI for difference, 16.6% - 54.2%), although both groups were equally likely to report having observed race-based discrimination of another physician. EM faculty who identified as sexual minorities reported higher mean sexual minority OGDW scores than their heterosexual counterparts (11.1 vs 7.1; 95% CI for difference, -7.3 - -0.6). There were no significant differences between sexual minority and heterosexual faculty in their reports of experiencing or observing discrimination based on sexual orientation. CONCLUSION: EM faculty from racial and sexual minority groups perceived more discrimination based on race or sexual orientation in their workplace than their majority counterparts. EM faculty regardless of race or sexual orientation were similar in their observations of discriminatory treatment of another physician based on race or sexual orientation.

The Implementation of a National Multifaceted Emergency Medicine Resident Wellness Curriculum Is Not Associated With Changes in Burnout.

BACKGROUND: The Accreditation Council for Graduate Medical Education Common Program Requirements effective 2017 state that programs and sponsoring institutions have the same responsibility to address well-being as they do other aspects of resident competence. OBJECTIVES: The authors sought to determine if the implementation of a multifaceted wellness curriculum improved resident burnout as measured by the Maslach Burnout Inventory (MBI). METHODS: We performed a multicenter educational interventional trial at 10 emergency medicine (EM) residencies. In February 2017, we administered the MBI at all sites. A year-long wellness curriculum was then introduced at five intervention sites while five control sites agreed not to introduce new wellness initiatives during the study period. The MBI was readministered in August 2017 and February 2018. RESULTS: Of 523 potential respondents, 437 (83.5%) completed at least one MBI assessment. When burnout was assessed as a continuous variable, there was a statistically significant difference in the depersonalization component favoring the control sites at the baseline and final survey administrations. There was also a higher mean personal accomplishment score at the control sites at the second survey administration. However, when assessed as a dichotomous variable, there were no differences in global burnout between the groups at any survey administration and burnout scores did not change over time for either control or intervention sites. CONCLUSIONS: In this national study of EM residents, MBI scores remained stable over time and the introduction of a multifaceted wellness curriculum was not associated with changes in global burnout scores.

#MeToo in EM: A Multicenter Survey of Academic Emergency Medicine Faculty on Their Experiences with Gender Discrimination and Sexual Harassment.

INTRODUCTION: Gender-based discrimination and sexual harassment of female physicians are well documented. The #MeToo movement has brought renewed attention to these problems. This study examined academic emergency physicians' experiences with workplace gender discrimination and sexual harassment. METHODS: We conducted a cross-sectional survey of a convenience sample of emergency medicine (EM) faculty across six programs. Survey items included the following: the Overt Gender Discrimination at Work (OGDW) Scale; the frequency and source of experienced and observed discrimination; and whether subjects had encountered unwanted sexual behaviors by a work superior or colleague in their careers. For the latter question, we asked subjects to characterize the behaviors and whether those experiences had a negative effect on their self-confidence and career advancement. We made group comparisons using t-tests or chi-square analyses, and evaluated relationships between gender and physicians' experiences using correlation analyses. RESULTS: A total of 141 out of 352 (40.1%) subjects completed at least a portion of the survey. Women reported higher mean OGDW scores than men (15.4 vs 10.2; 95% confidence interval [CI], 3.6-6.8). Female faculty were also more likely to report having experienced gender-based discriminatory treatment than male faculty (62.7% vs 12.5%; 95% CI, 35.1%-65.4%), although male and female faculty were equally likely to report having observed gender-based discriminatory treatment of another physician (64.7% vs 56.3%; 95% CI, 8.6%-25.5%). The three most frequent sources of experienced or observed gender-based discriminatory treatment were patients, consulting or admitting physicians, and nursing staff. The majority of women reported having encountered unwanted sexual behaviors in their careers, with a significantly greater proportion of women reporting them compared to men (52.9% vs 26.2%, 95% CI, 9.9%-43.4%). The majority of unwanted behaviors were sexist remarks and sexual advances. Of those respondents who encountered these unwanted behaviors, 22.9% and 12.5% reported at least somewhat negative effects on their self-confidence and career advancement. CONCLUSION: Female EM faculty perceived more gender-based discrimination in their workplaces than their male counterparts. The majority of female and approximately a quarter of male EM faculty encountered unwanted sexual behaviors in their careers.

The Correlation Between Emergency Medicine Residents' Grit and Achievement.

BACKGROUND: Early identification of emergency medicine (EM) residents who struggle with educational attainment is difficult. In-training examination (ITE) scores predict success on the American Board of Emergency Medicine (ABEM) Qualifying Examination; however, results are not available until late in the academic year. The noncognitive trait "grit," defined as "perseverance and passion for long-term goals," predicts achievement in high school graduation rates, undergraduate GPA, and gross anatomy. Grit-S is a validated eight-question scale scored 1 to 5; the average of responses represents a person's grit. Our objective was to determine the correlation between EM resident Grit-S scores and achievement, as measured by MCAT percentiles, ITE scores, and remediation rates. STUDY DESIGN AND METHODS: This was a 1-year prospective, multicenter trial involving ten EM residencies from 2017 to 2018. Subjects were PGY-1 to -4 EM residents. Grit-S scores, MCAT percentile, remediation rates, ITE scores, and the ITE score's prediction of passing the ABEM Qualifying Examination were collected. Correlation coefficients were computed to assess the relationship between residents' grit and achievement. RESULTS: A total 385 of 434 (88.7%) residents participated who completed the Grit-S as part of a larger study. The mean Grit-S score was 3.62. Grit positively correlated with the predicted likelihood of passing the ABEM Qualifying Examination (r = 0.134, n = 382, p = 0.025). There was no correlation between grit and remediation (r = -0.04, n = 378, p = 0.46) or grit and MCAT percentiles (r =- 0.08, n = 262, p = 0.22). CONCLUSIONS: The positive correlation between Grit-S scores and percent likelihood of passing the ABEM Qualifying Examination demonstrates grit's potential to assist residency leadership in early identification of residents who may attain a lower ITE score.

Practical Application of Assessment Principles in the Development of an Innovative Clinical Performance Evaluation in the Entrustable Professional Activity Era.

Educators have been challenged to create assessments that are competency-based and grounded in accepted standards such as the entrustable professional activities (EPAs). The clinical performance evaluation (CPE) is a commonly utilized assessment modality, which allows multiple evaluators to provide feedback on a learner's performance in the clinical workplace. In this paper, we describe the relevant principles that served as a guide as we developed a new CPE for medical students that fully incorporate EPAs. This may help ease the transition for other institutions looking to introduce a new student CPE.

Examining the Effects of Narrative Commentary on Evaluators' Summative Assessments of Resident Performance.

Anchor-based, end-of-shift ratings are commonly used to conduct performance assessments of resident physicians. These performance evaluations often include narrative assessments, such as solicited or "free-text" commentary. Although narrative commentary can help to create a more detailed and specific assessment of performance, there are limited data describing the effects of narrative commentary on the global assessment process. This single-group, observational study examined the effect of narrative comments on global performance assessments. A subgroup of the clinical competency committee, blinded to resident identity, assigned a single, consensus-based performance score (1-6) to each resident based solely on end-of-shift milestone scores. De-identified narrative comments from end-of-shift evaluations were then included and the process was repeated. We compared milestone-only scores to milestone plus narrative commentary scores using a nonparametric sign test. During the study period, 953 end-of-shift evaluations were submitted on 41 residents. Of these, 535 evaluations included free-text narrative comments. In 17 of the 41 observations, performance scores changed after the addition of narrative comments. In two cases, scores decreased with the addition of free-text commentary. In 15 cases, scores increased. The frequency of net positive change was significant (p = .0023). The addition of narrative commentary to anchor-based ratings significantly influenced the global performance assessment of Emergency Medicine residents by a committee of educators. Descriptive commentary collected at the end of shift may inform more meaningful appraisal of a resident's progress in a milestone-based paradigm. The authors recommend clinical training programs collect unstructured narrative impressions of residents' performance from supervising faculty.

Determinants of medical specialty competitiveness.

BACKGROUND: Although the concept of medical specialty competitiveness may seem intuitive, there are very little existing empirical data on the determinants of specialty competitiveness in USA. An understanding of the determinants of specialty competitiveness may inform career choices among students and their advisors. Specialty competitiveness correlates with availability and appeal. METHODS: This narrative review examines 2019 National Resident Matching Program (NRMP) data and the existing literature to define the determinants of specialty competitiveness. A statistical analysis of key elements of the 2019 NRMP data was performed. RESULTS: Using US senior applicant fill rate as a measure of competitiveness, medical specialty competitiveness follows general principles of supply and demand. The demand, or appeal, of a specialty correlates with several factors, including salary, prestige and lifestyle. Salary correlates strongly with US senior fill rate (r=0.78, p=0.001). Relatively few positions are available for the most competitive specialties in the NRMP match. The negative correlation between US senior fill rate and position availability is also strong (r=-0.85; p<0.0001). CONCLUSION: A 'competitive specialty' correlates strongly with high earnings potential and limited position availability. In an ideal world, a student's pursuit of a medical specialty should be guided by interest, qualifications and ability to succeed in that field. However, students must contend with the realities of competition created by the residency matching system.