Initial Validity Evidence for the American Board of Emergency Medicine Emergency Medical Services Certification Examination.

OBJECTIVES: The American Board of Emergency Medicine (ABEM) Emergency Medical Services Medicine (EMS) subspecialty was approved by the American Board of Medical Specialties on September 23, 2010. Subspecialty certification in EMS was contingent on two key elements-completing Accreditation Council for Graduate Medical Education (ACGME)-accredited EMS training and passing the subspecialty certification examination developed by ABEM. The first EMS certification examination was offered in October 2013. Meaningful certification requires rigorous assessment. In this instance, the EMS certification examination sought to embrace the tenets of validity, reliability, and fairness. For the purposes of this report, the sources of validity evidence were anchored on the EMS core content, the examination development process, and the association between fellowship training and passing the certification examination. METHODS: We chose to use validity evidence that included: 1) content validity (based on the EMS core content); 2) response processes (test items required intended cognitive processes); 3) internal structure supported by the internal relationships among items; 4) relations to other variables, specifically the association between examination performance and ACGME-accredited fellowship training; and 5) the consequences of testing. RESULTS: There is strong content validity evidence for the EMS examination based on the core content and its detailed development process. The core content and supporting job-task analysis was also used to define the examination blueprint. Internal structure support was evidenced by Cronbach's coefficient alpha, which ranged from 0.82 to 0.92. Physicians who completed ACGME-accredited EMS fellowship training were more likely to pass the EMS certification examination (chi square, p < 0.0001; Cramér's, V = 0.24). Finally, there were two sources of consequential validity evidence-use of test results to determine certification and use of the resulting certificate. CONCLUSIONS: There is substantial and varied validity evidence to support the use of the EMS certifying examination in making summative decisions to award certification in EMS. Of note, there was a statistically significant association between ACGME-accredited fellowship training and passing the examination.

Emergency Medicine Residency Does Not Reduce a Racial Minority-Based Test Performance Gap.

BACKGROUND: Health care inequity is corrected more readily when safe, high-quality care is provided by physicians who reflect the gender, race, and ethnicity of patient communities. It is important to train and evaluate racially diverse physicians involved in residency training. OBJECTIVE: This study sought to determine any test-taking differences for black Emergency Medicine (EM) residents and whether any such differences would narrow as residency progressed. METHODS: This was an observational, cross-sectional study that reviewed performance (scaled scores) on the American Board of Emergency Medicine (ABEM) In-Training Examination (ITE) for 2018, 2019, and 2020. The study included EM residents in 3-year programs who took the ITE. A linear regression model was used for the variables of race, which included black physicians and white physicians (reference group), and level of training (EM resident year 1 [EM1] as the reference group). RESULTS: There were 9591 residents included; 539 were black and 9052 were white. Mean scaled scores were higher as a function of training level. Regression showed a scaled score intercept of 73.51. The ITE score increased for all groups as a function of training level (EM2 ß = +5.45, p < 0.0001; EM3 ß = +8.09, p < 0.0001). The regression coefficient for black residents was -5.87 (p < 0.0001). There was relative improvement by training level compared with improvement in the reference group, but this difference was not materially or statistically significant. CONCLUSION: In this study of the ABEM ITE, a test-taking performance gap identified early in residency for black physicians persisted into late residency.

The 2019 Core Content of Emergency Medical Services Medicine.

On March 13, 2019 the EMS Examination Committee of the American Board of Emergency Medicine (ABEM) approved modifications to the Core Content of EMS Medicine. The Core Content is used to define the subspecialty of EMS Medicine, provides the basis for questions to be used during written examinations, and leads to development of a certification examination blueprint. The Core Content defines the universe of knowledge for the treatment of prehospital patients that is necessary to practice EMS Medicine. It informs fellowship directors and candidates for certification of the full range of content that might appear on certification examinations.

American Board of Emergency Medicine Report on Residency and Fellowship Training Information (2018-2019).

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine-sponsored residency and fellowship programs, residents and fellows training in those programs, and all fellows for whom ABEM issues subspecialty certifications. We present the 2019 annual report on the status of US emergency medicine training programs.

American Board of Emergency Medicine Report on Residency and Fellowship Training Information (2017-2018).

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine-sponsored residency and fellowship programs, as well as the residents and fellows training in those programs. We present the 2018 annual report on the status of US emergency medicine training programs.

American Board of Emergency Medicine Report on Residency and Fellowship Training Information (2016-2017).

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine residency programs and the residents training in those programs. We present the 2017 annual report on the status of US emergency medicine training programs.

American Board of Emergency Medicine Report on Residency Training Information (2015-2016).

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine residency programs and the residents training in those programs. We present the 2016 annual report on the status of US emergency medicine training programs.

American Board of Emergency Medicine report on residency training information (2014-2015).

The American Board of Emergency Medicine (ABEM) gathers extensive background information on emergency medicine residency programs and the residents in those programs. We present the 2015 annual report on the status of US emergency medicine training programs.

Comparison of performance data between emergency medicine 1-3 and 1-4 program formats.

Objective: This study compares performance data from physicians completing 3-year versus 4-year emergency medicine residency training programs. Currently, there are 2 training formats and little is known about objective performance differences. Methods: This was a retrospective cross-sectional analysis of emergency residents and physicians. Multiple analyses were conducted comparing physicians' performances, including Accreditation Council of Graduate Medical Education Milestones and American Board of Emergency Medicine In-training Examination (ITE), Qualifying Examination (QE), Oral Certification Examination (OCE), and program extensions from 3-year and 4-year residency programs. Some confounding variables were not or could not be considered, such as rationale for medical students to choose one format over another, as well as application and final match rates. Results: Milestone scores are higher for emergency medicine 3 residents in 1-3 programs (3.51) versus emergency medicine 3 residents in 1-4 programs (3.07; P < 0.001, d = 1.47) and highest for emergency medicine 4 residents (3.67). There was no significant difference in program extension rates (emergency medicine 1-3, 8.1%; emergency medicine 1-4, 9.6%; P = 0.05, ω = 0.02). ITE scores were higher for emergency medicine 1, 2, and 3 residents from 1-3 programs and emergency medicine 4 residents from 1-4 programs scored highest. Mean QE score was slightly higher for emergency 1-3 physicians (83.55 vs 83.00; P < 0.01, d = 0.10). QE pass rate was higher for emergency 1-3 physicians (93.1% vs 90.8%; P < 0.001, ω = 0.08). Mean OCE score was slightly higher for emergency 1-4 physicians (5.67 vs 5.65; P = 0.03, d = -0.07) but did not reach a priori statistical significance (α < 0.01). OCE pass rate was also slightly higher for emergency 1-4 physicians (96.9% vs 95.5%; P = 0.06, ω = -0.07) but also non-significant. Conclusions: These results suggest that although performance measures demonstrate small differences between physicians from emergency medicine 1-3 and 1-4 programs, these differences are limited in their ability to make causal claims about performance on the basis of program format alone.

Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination.

Background: The American Board of Emergency Medicine (ABEM) in-person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID-19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020. Objectives: The purpose of this investigation was to determine whether there was sufficient validity and reliability evidence to support the continued use of the ABEM virtual Oral Examination (VOE) for certification decisions. Methods: This retrospective, descriptive study used multiple data sources to provide validity evidence and reliability data. Validity evidence focused on test content, response processes, internal structure (e.g., internal consistency and item response theory), and the consequences of testing. A multifaceted Rasch reliability coefficient was used to measure reliability. Study data were from two 2019 in-person OCEs and the first four VOE administrations. Results: There were 2279 physicians who took the 2019 in-person OCE examination and 2153 physicians who took the VOE during the study period. Among the OCE group, 92.0% agreed or strongly agreed that the cases on the examination were cases that an emergency physician should be expected to see; 91.1% of the VOE group agreed or strongly agreed. A similar pattern of responses given to a question about whether the cases on the examination were cases that they had seen. Additional evidence of validity was obtained by the use of the EM Model, the process for case development, the use of think-aloud protocols, and similar test performance patterns (e.g., pass rates). For reliability, the Rasch reliability coefficients for the OCE and the VOE during the study period were all >0.90. Conclusions: There was substantial validity evidence and reliability to support ongoing use of the ABEM VOE to make confident and defensible certification decisions.

The 2021 Core Content of Medical Toxicology.

The Medical Toxicology Subboard approved modifications to the Core Content of Medical Toxicology in March 2021. The document outlines the areas of knowledge considered essential for the practice of medical toxicology. The Core Content provides the organizational framework for the development of the Medical Toxicology Certification and Cognitive Expertise Examinations and serves as a template for the development of curricula for medical toxicology fellowship training programs.

Emergency Medicine Resident Burnout and Examination Performance.

OBJECTIVES: Burnout afflicts emergency physicians (EPs) to a significant degree. The impact of burnout spans from decreased clinical efficiency to increased medical errors to heightened risk of physician suicide. This large-scale study captures responses from emergency medicine (EM) residents regarding two burnout items and examines the correlation between in-training examination (ITE) scores and burnout risk as well as that between residency year and burnout risk. METHODS: This was a prospective, mixed-methods, cross-sectional cohort study. All residents in U.S. categorical EM residents who took the 2019 ITE were included. At the end of the ITE, residents were invited to complete a voluntary survey that included two items from the Maslach Burnout Inventory (MBI) that have been found to be strongly indicative of burnout: one about self-perception of being burned out and one about feelings of callousness. Responses were on a 7-level Likert scale (1-7), ranging from very low frequency (1) to very high frequency (7). Measurements included the number of residents in each year-level of training (EM1-EM4), the MBI item ratings, and the ABEM ITE score. Performance, as measured by the scaled, equated score, was compared to the MBI item responses. A corrected Spearman's correlation coefficient (ρ) was used to compare continuous data (score) against a discrete ordinal variable (MBI Likert response). RESULTS: There were 2,501 EM1 residents, 2,389 EM2 residents, 2,206 EM3 residents, and 616 EM4 residents in the study group. There were 7,206 (93.4%) physicians who completed the first MBI question about burnout; 7,172 (93%) completed the second MBI question about callousness. There was no statistically significant association between the burnout item response and ITE performance (ρ = -0.03; p = 0.015). There was a positive, statistically significant association between the callousness item response and higher ITE performance (ρ = 0.07; p < 0.001). There was a statistically significant association between the response to the burnout item and training level (ρ = 0.07; p <0.001). There was also a statistically significant association between the response to the callousness item and training level (ρ = 0.15; p < 0.001). The overall prevalence of burnout risk in various training levels were EM1, 28.2%; EM2, 39%; EM3, 41.1%; and EM4, 43.3%. CONCLUSIONS: Our study found no significant correlation between ITE score and burnout risk. There was a weakly positive correlation between ITE scores and callousness. Based on our study results, ITE scores may not be useful in prognosticating burnout risk for EM residents and, interestingly, higher ITE scores correlated to stronger feelings of callousness. Our study indicates that EM residents at higher levels of training reported stronger self-perceptions of burnout and callousness. Further investigation into why residents at higher levels of training may experience greater burnout risk is warranted.

Physician Age and Performance on the American Board of Emergency Medicine ConCert Examination.

OBJECTIVE: This study was undertaken to expand on results from a 2014 study on the association between physician age and performance on the American Board of Emergency Medicine (ABEM) ConCert examination. METHODS: This was a retrospective, longitudinal growth study comparing performance on the ConCert examination and physicians' ages at the time of examination. All examination attempts from 1990 to 2016 made by residency-trained physicians were eligible for inclusion. Multilevel growth models were constructed to examine the relationship between age at time of examination and performance, controlling for physician characteristics. RESULTS: The study group included 15,533 examination attempts by 12,786 physicians. The mean (±SD) age of the physicians across all examination administrations was 45.02 (±5.18) years (range = 35 to 72 years). The mean (±SD) ConCert examination score across all administrations was 85.39 (±5.71; range = 51 to 100). Among first-time ConCert examination takers, older age was associated with lower examination scores (r = -0.25, p < 0.0001). Across all examination attempts, age was negatively correlated to examination scores (r = -0.24; p < 0.0001). CONCLUSIONS: After physician characteristics were controlled for, there was an association between advancing age and declining performance on the ABEM ConCert examination. This information may be important to the individual physician to develop targeted competency assessment and professional development.

Initial Validity Analysis of the American Board of Emergency Medicine Enhanced Oral Examination.

OBJECTIVES: The American Board of Emergency Medicine (ABEM) has introduced a new testing format for the oral certification examination (OCE): the enhanced oral or "eOral" format. The purpose of this study was to perform initial validity analyses of the eOral format. The two hypotheses were: 1) the case content in the eOral format was sufficiently similar to clinical practice and 2) the eOral case materials were sufficiently similar to clinical practice. The eOral and traditional formats were compared for these characteristics. METHODS: This was a prospective survey study. The survey was administered as a voluntary postexamination activity at the end of the 2015 spring (April 25-27) and fall (October 10-13) ABEM OCEs. The survey is a routine part of the ABEM oral examination experience. For 2015, two additional questions were added to gauge the similarity of the eOral format to clinical practice. Validity was defined by content and substantive elements within Messick's model of construct validity as well as portions of Kane's validity model. RESULTS: Of the 1,746 physicians who took the oral examination, 1,380 physicians (79.0%) completed all or part of the study survey questions. The majority of respondents agreed the patient presentations in the cases were similar (strongly agreed or agreed) to cases seen in clinical practice, in both the traditional cases (95.1%) and the eOral cases (90.1%). Likewise, the majority of respondents answered that the case materials (e.g., laboratory, radiographs) were similar (strongly agreed or agreed) to what they encounter in clinical practice, both in the traditional format (85.8%) and in the eOral cases (93.7%). CONCLUSIONS: Most emergency physicians reported that the types of cases tested in the traditional and eOral formats were similar to cases encountered in clinical practice. In addition, most physicians found the case materials to be similar to what is seen in clinical practice. This study provides early validity evidence for the eOral format.

The Relationship Between ACGME Duty Hour Requirements and Performance on the American Board of Emergency Medicine Qualifying Examination.

BACKGROUND: In 2003, the Accreditation Council for Graduate Medical Education (ACGME) instituted requirements that limited the number of hours residents could spend on duty, and in 2011, it revised these requirements. OBJECTIVE: This study explored whether the implementation of the 2003 and 2011 duty hour limits was associated with a change in emergency medicine residents' performance on the American Board of Emergency Medicine (ABEM) Qualifying Examination (QE). METHODS: Beginning with the 1999 QE and ending with the 2014 QE, candidates for whom all training occurred without duty hour requirements (Group A), candidates under the first set of duty hour requirements (Group C), and candidates under the second set of duty hour requirements (Group E) were compared. Comparisons included mean scores and pass rates. RESULTS: In Group A, 5690 candidates completed the examination, with a mean score of 82.8 and a 90.2% pass rate. In Group C, 8333 candidates had a mean score of 82.4 and a 90.5% pass rate. In Group E, there were 1269 candidates, with a mean score of 82.5 and an 89.4% pass rate. There was a small but statistically significant decrease in the mean scores (0.04, P < .001) after implementation of the first duty hour requirements, but this difference did not occur after implementation of the 2011 standards. There was no difference among pass rates for any of the study groups (χ2 = 1.68, P = .43). CONCLUSIONS: We did not identify an association between the 2003 and 2011 ACGME duty hour requirements and performance of test takers on the ABEM QE.

The American Board of Emergency Medicine ConCert Examination: Emergency Physicians' Perceptions of Learning and Career Benefits.

OBJECTIVE: As part of the American Board of Emergency Medicine (ABEM) Maintenance of Certification (MOC) program, ABEM-certified physicians are required to pass the Continuous Certification (ConCert) examination at least every 10 years. With the 2015 ConCert examination, ABEM sought to better understand emergency physicians' perceptions of the benefits of preparing for and taking the examination and the career benefits of staying ABEM-certified. METHODS: This was a prospective survey study. A voluntary postexamination survey was administered at the end of the 2015 ABEM ConCert examination (September 21-26, 2015). Physicians were asked about the benefits of preparing for the examination and maintaining ABEM certification. Examination performance was compared to perceptions of learning and career benefits. RESULTS: Of the 2,601 on-time test takers, 2,511 respondents participated (96.5% participation rate). The majority of participants (92.0%) identified a benefit to preparing for the ConCert examination, which included reinforced medical knowledge (73.9%), increased knowledge (66.8%), and making them a better clinician (39.4%). The majority of respondents (90.8%) identified a career benefit of maintaining ABEM certification, which included more employment options (73.8%), more positively viewed by other physicians (56.8%), and better financial outcomes (29.8%). There was a statistically significant association between the perception of knowledge reinforcement and examination performance (p < 0.001). There was also a statistically significant association between the perception that staying certified created more career opportunities and examination performance (p < 0.001). CONCLUSIONS: Most emergency physicians identified benefits of preparing for and taking the ABEM ConCert examination, which included reinforcing or adding medical knowledge and making them better clinicians. Most physicians also found career benefits to remaining ABEM-certified, which included greater employment choices, higher financial compensation, and higher esteem from other physicians. The belief that preparing for and taking the examination reinforced medical knowledge was associated with better examination performance.

Physician Preparation for the American Board of Emergency Medicine ConCert Examination.

OBJECTIVES: To maintain certification by the American Board of Emergency Medicine (ABEM), physicians are required to pass the Continuous Certification (ConCert) examination at least every 10 years. On the 2014 ConCert postexamination survey, ABEM sought to understand the manner in which ABEM diplomates prepared for the test and to identify associations between test preparation approaches and performance on the ConCert examination. METHODS: This was a cross-sectional survey study. The survey was administered at the end of the 2014 ConCert examination. Analyses included chi-square and linear regression to determine the association of preparation methods with performance. RESULTS: Of the 2,431 on-time test-takers, 2,338 (96.2%) were included. The most commonly used study approach was the review of written materials designed for test preparation (1,585; 67.8%), followed by an online training course (1,006; 43.0%). There were 758 (32.4%) physicians who took a single onsite board review course, while 41 (1.8%) took two or more onsite courses. Most physicians (1,611; 68.9%) spent over 35 hours preparing for the ConCert examination. The study method that was most associated with favorable test scores was the review of written materials designed for test preparation (p < 0.001). Attending an onsite preparation course was associated with poorer performance (p < 0.001). There was a significant association between no additional preparation and failing the examination (chi-square with Yates correction; p = 0.001). CONCLUSIONS: A substantial majority (97.8%) of physicians taking the 2014 ABEM ConCert examination prepared for it. The majority of physicians used written materials specifically designed for test preparation. Reviewing written materials designed for test preparation was associated with the highest performance.