Initial Validity Evidence for the American Board of Emergency Medicine Emergency Medical Services Certification Examination.

OBJECTIVES: The American Board of Emergency Medicine (ABEM) Emergency Medical Services Medicine (EMS) subspecialty was approved by the American Board of Medical Specialties on September 23, 2010. Subspecialty certification in EMS was contingent on two key elements-completing Accreditation Council for Graduate Medical Education (ACGME)-accredited EMS training and passing the subspecialty certification examination developed by ABEM. The first EMS certification examination was offered in October 2013. Meaningful certification requires rigorous assessment. In this instance, the EMS certification examination sought to embrace the tenets of validity, reliability, and fairness. For the purposes of this report, the sources of validity evidence were anchored on the EMS core content, the examination development process, and the association between fellowship training and passing the certification examination. METHODS: We chose to use validity evidence that included: 1) content validity (based on the EMS core content); 2) response processes (test items required intended cognitive processes); 3) internal structure supported by the internal relationships among items; 4) relations to other variables, specifically the association between examination performance and ACGME-accredited fellowship training; and 5) the consequences of testing. RESULTS: There is strong content validity evidence for the EMS examination based on the core content and its detailed development process. The core content and supporting job-task analysis was also used to define the examination blueprint. Internal structure support was evidenced by Cronbach's coefficient alpha, which ranged from 0.82 to 0.92. Physicians who completed ACGME-accredited EMS fellowship training were more likely to pass the EMS certification examination (chi square, p < 0.0001; Cramér's, V = 0.24). Finally, there were two sources of consequential validity evidence-use of test results to determine certification and use of the resulting certificate. CONCLUSIONS: There is substantial and varied validity evidence to support the use of the EMS certifying examination in making summative decisions to award certification in EMS. Of note, there was a statistically significant association between ACGME-accredited fellowship training and passing the examination.

Workplace Mistreatment, Career Choice Regret, and Burnout in Emergency Medicine Residency Training in the United States.

STUDY OBJECTIVE: The influence of workplace mistreatment on the well-being and career satisfaction of emergency medicine residents is unknown. This study examined the relationships between burnout, career choice regret, and workplace mistreatment in a national sample of emergency medicine residents. METHODS: This was a secondary analysis of a survey study on the prevalence of workplace mistreatment among emergency residents. Residents who reported emotional exhaustion or depersonalization at least once per week were considered to have burnout. Residents who reported dissatisfaction with their decision to become an emergency physician were considered to have career choice regret. Respondents also reported the type (discrimination, abuse, sexual harassment) and frequency of mistreatment over the academic year. Multivariable logistic regression, adjusting for program characteristics, was used to examine resident characteristics associated with burnout and career choice regret, with the frequency of mistreatment as a covariate. RESULTS: Of the 8,162 eligible residents, 7,680 (94.1 %) participated. About a third of respondents reported burnout (2,188 of 6,902, 31.7%), whereas a minority (224 of 6,923, 3.2%) reported career choice regret. Of the 7,087 responses on mistreatment frequency, 2,117 (29.9%) reported "a few times per year," and 1,296 (18.3%) reported "a few times per month or more." Compared with residents who never experienced mistreatment, residents who reported increasing frequencies of mistreatment were associated with having burnout-from mistreatment a few times per year (OR [odds ratio],1.6; 99% CI [confidence interval], 1.3 to 1.9) to a few times per month or more (OR, 3.3; 99% CI, 2.7 to 4.1). Compared with residents without burnout, residents who reported burnout were associated with having career choice regret (OR, 11.3; 99% CI, 7.0 to 18.1). After adjusting for burnout, there were no significant relationships between the frequency of mistreatment and career choice regret. CONCLUSIONS: Workplace mistreatment is associated with burnout, but not career choice regret, among emergency medicine residents. Efforts to address workplace mistreatment may improve emergency medicine residents' professional well-being.

Emergency Medicine Milestones Final Ratings Are Often Subpar.

Background: The emergency medicine (EM) milestones are objective behaviors that are categorized into thematic domains called "subcompetencies" (eg, emergency stabilization). The scale for rating milestones is predicated on the assumption that a rating (level) of 1.0 corresponds to an incoming EM-1 resident and a rating of 4.0 is the "target rating" (albeit not an expectation) for a graduating resident. Our aim in this study was to determine the frequency with which graduating residents received the target milestone ratings. Methods: This retrospective, cross-sectional study was a secondary analysis of a dataset used in a prior study but was not reported previously. We analyzed milestone subcompetency ratings from April 25-June 24, 2022 for categorical EM residents in their final year of training. Ratings were dichotomized as meeting the expected level at the time of program completion (ratings of ≥3.5) and not meeting the expected level at the time of program completion (ratings of ≤3.0). We calculated the number of residents who did not achieve target ratings for each of the subcompetencies. Results: In Spring 2022, of the 2,637 residents in the spring of their last year of training, 1,613 (61.2%) achieved a rating of ≥3.5 on every subcompetency and 1,024 (38.8%) failed to achieve that rating on at least one subcompetency. There were 250 residents (9.5%) who failed to achieve half of their expected subcompetency ratings and 105 (4.0%) who failed to achieve the expected rating (ie, rating was ≤3.0) on every subcompetency. Conclusion: When using an EM milestone rating threshold of 3.5, only 61.2% of physicians achieved the target ratings for program graduation; 4.0% of physicians failed to achieve target ratings for any milestone subcompetency; and 9.5% of physicians failed to achieve the target ratings for graduating residents in half of the subcompetencies.

Suicidal ideation, perception of personal safety, and career regret among emergency medicine residents during the COVID-19 pandemic.

Objectives: The COVID-19 pandemic was disruptive for trainees and may have affected career decisions for some learners. This study examined the impact of the pandemic on emergency medicine (EM) resident perceptions of their mental health, perceptions of personal safety, and career choice regret. Methods: This was a cross-sectional survey study administered following the 2021 American Board of Emergency Medicine In-Training Examination (ITE). Survey measures included suicidal ideation (SI), COVID concerns in terms of infection prevention and control (IPC) training, COVID risk to self and/or COVID risk to family, and COVID-related career regret. COVID concerns were compared by gender and race/ethnicity using Pearson's chi-square tests. Multivariable logistic regression models were used to test the association between SI and COVID concerns, resident characteristics, and program characteristics. Results: A total of 6980 out of 8491 EM residents (82.2%) from 244 programs completed the survey. Only 1.1% of participants reported insufficient training in COVID IPC practices. Participants were concerned about COVID risk to themselves (40.3%) and to their families (63.3%) due to their job roles. These concerns were more common among women or nonbinary (vs. men); all other races/ethnicities (vs. non-Hispanic Whites); senior residents (vs. PGY-1, PGY-2 residents); and residents who were married or in relationships (vs. single or divorced). A total of 6.1% of participants reported that COVID made them reconsider choosing EM as their career. Career regret in this cohort was higher than that in the proportion (3.2%) expressing career regret in the 2020 ITE (p < 0.001). Career regret was more common among women or nonbinary (vs. men); all other races/ethnicities (vs. non-Hispanic Whites); and senior residents (vs. PGY-1, PGY-2 residents). The overall SI rate was 2.6%, which did not differ from that of the 2020 sample of EM residents (2.5%, p = 0.88). Conclusions: Many EM residents reported concerns about COVID risks to themselves and their families. Although the rate of SI remained unchanged, more EM residents reported career regret during the COVID pandemic.

Prospective validity evidence for the abbreviated emergency medicine Copenhagen Burnout Inventory.

BACKGROUND: Prior research has provided retrospective validity evidence for an abbreviated Copenhagen Burnout Inventory (CBI) to measure burnout among emergency medicine (EM) residents. We sought to provide additional validity and reliability evidence for the two-factor, six-item abbreviated CBI. METHODS: This cross-sectional study used data from the abbreviated CBI that was administered following the 2022 American Board of Emergency Medicine In-training Examination. Confirmatory factor analysis (CFA) was performed and the prevalence of burnout among EM residents was determined. RESULTS: Of the 8918 eligible residents, 7465 (83.7%) completed the abbreviated CBI. CFA confirmed the previously developed model of two factors using six items answered with a 1- to 5-point Likert scale. The internal factor was derived from personal and work-related burnout and the external factor was related to caring for patients. The reliability was determined using Cronbach's alpha (0.87). The overall prevalence of burnout was 49.4%; the lowest prevalence was at the EM1 level (43.1%) and the highest was at the EM2 level (53.8%). CONCLUSIONS: CFA of the abbreviated CBI demonstrated good reliability and model fit. The two-factor, six-item survey instrument identified an increase in the prevalence of burnout among EM residents that coincided with working in the COVID-19 environment. The abbreviated CBI has sufficient reliability and validity evidence to encourage its broader use.

Emergency physicians perspectives of state continuing medical education requirements for medical licensure.

Objectives: This study aims to better understand the perspectives of emergency medicine physicians' on the role that state-mandated, topic-specific continuing medical education (CME) plays in addressing knowledge gaps, its relevance to current emergency practice, its reported burden and costs of CME activities to emergency physicians, and its perceived improvement in patient care. Methods: A cross-sectional survey was designed by the Coalition of Board-Certified Emergency Physicians (COBCEP) and distributed in February 2023 to all American Board of Emergency Medicine (ABEM)-certified physicians. Statistical tests of significance (Pearson's chi-square and Fisher's exact test) assessed the cost and time spent on CME as well as the perceived value placed on CME by ABEM-certified physicians to improve patient care. Data were summarized using descriptive statistics. Results: There were 5562 (13.0%) responses from the 43656 physicians who received the survey-5506 responses were included for analysis. Over half of the physicians (53.0%) had more than 15 years of post-residency practice experience. Most physicians (57.3%) spent less than $5,000 per year on obtaining CME. Most physicians practicing in states with state-mandated, topic-specific CME requirements believed that participation in ABEM continuing certification could be used to reduce the need for state-mandated, topic-specific CME requirements (83.6%) and state-mandated, topic-specific requirements were believed to be unlikely to improve patient care (70.8%). Conclusions: Although well-intended, state CME requirements may lack relevancy and can, at times, place an undue burden on emergency physicians. Tailoring CME requirements to increase relevance to their patient populations and reduce barriers to completing CME could enhance knowledge translation and improve patient outcomes.

Comparison of performance data between emergency medicine 1-3 and 1-4 program formats.

Objective: This study compares performance data from physicians completing 3-year versus 4-year emergency medicine residency training programs. Currently, there are 2 training formats and little is known about objective performance differences. Methods: This was a retrospective cross-sectional analysis of emergency residents and physicians. Multiple analyses were conducted comparing physicians' performances, including Accreditation Council of Graduate Medical Education Milestones and American Board of Emergency Medicine In-training Examination (ITE), Qualifying Examination (QE), Oral Certification Examination (OCE), and program extensions from 3-year and 4-year residency programs. Some confounding variables were not or could not be considered, such as rationale for medical students to choose one format over another, as well as application and final match rates. Results: Milestone scores are higher for emergency medicine 3 residents in 1-3 programs (3.51) versus emergency medicine 3 residents in 1-4 programs (3.07; P < 0.001, d = 1.47) and highest for emergency medicine 4 residents (3.67). There was no significant difference in program extension rates (emergency medicine 1-3, 8.1%; emergency medicine 1-4, 9.6%; P = 0.05, ω = 0.02). ITE scores were higher for emergency medicine 1, 2, and 3 residents from 1-3 programs and emergency medicine 4 residents from 1-4 programs scored highest. Mean QE score was slightly higher for emergency 1-3 physicians (83.55 vs 83.00; P < 0.01, d = 0.10). QE pass rate was higher for emergency 1-3 physicians (93.1% vs 90.8%; P < 0.001, ω = 0.08). Mean OCE score was slightly higher for emergency 1-4 physicians (5.67 vs 5.65; P = 0.03, d = -0.07) but did not reach a priori statistical significance (α < 0.01). OCE pass rate was also slightly higher for emergency 1-4 physicians (96.9% vs 95.5%; P = 0.06, ω = -0.07) but also non-significant. Conclusions: These results suggest that although performance measures demonstrate small differences between physicians from emergency medicine 1-3 and 1-4 programs, these differences are limited in their ability to make causal claims about performance on the basis of program format alone.

Reliability and validity support for an abbreviated Copenhagen burnout inventory using exploratory and confirmatory factor analysis.

Objective: The Copenhagen Burnout Inventory (CBI) is an open-access, valid, and reliable instrument measuring burnout that includes 19 items distributed across the following 3 domains (factors): personal burnout, work burnout, and patient burnout. The primary objective of this study was to determine the validity and reliability of an abbreviated CBI to assess burnout in emergency medicine residents. Methods: This cross-sectional study used data from the CBI that followed the 2021 American Board of Emergency Medicine In-training Examination. Exploratory factor analysis (EFA) was followed by confirmatory factor analysis (CFA). Results: Of the 8491 eligible residents, 7225 (85.1%) completed the survey; the EFA cohort included 3613 residents and the CFA cohort included 3612 residents. EFA showed 2 eigenvalues ≥1, an internal factor and an external factor. There were 6 CBI items that contributed to the 2 factors. The first factor was related to personal burnout and work-related burnout and the second factor was related to working with patients. There were 4 CBI items that contributed to the internal factor and 2 CBI items that contributed to the external factor. Using the abbreviated CBI, the incidence of a resident having 1 or both types of burnout was 34.1%. Conclusions: This study provides validity evidence and reliability support for the use of a 6-item, 2-factor abbreviated CBI. A shorter, reliable, valid, and publicly accessible burnout inventory provides numerous advantages for burnout research in emergency medicine.

Prevalence of Discrimination, Abuse, and Harassment in Emergency Medicine Residency Training in the US.

Importance: The prevalence of workplace mistreatment and its association with the well-being of emergency medicine (EM) residents is unclear. More information about the sources of mistreatment might encourage residency leadership to develop and implement more effective strategies to improve professional well-being not only during residency but also throughout the physician's career. Objective: To examine the prevalence, types, and sources of perceived workplace mistreatment during training among EM residents in the US and the association between mistreatment and suicidal ideation. Design, Setting, and Participants: In this survey study conducted from February 25 to 29, 2020, all residents enrolled in EM residencies accredited by the Accreditation Council for Graduate Medical Education (ACGME) who participated in the 2020 American Board of Emergency Medicine computer-based In-training Examination were invited to participate. A multiple-choice, 35-item survey was administered after the examination asking residents to self-report the frequency, sources, and types of mistreatment experienced during residency training and whether they had suicidal thoughts. Main Outcomes and Measures: The types and frequency of workplace mistreatment and the sources of the mistreatment were identified, and rates of self-reported suicidality were obtained. Multivariable logistic regression models were used to examine resident and program characteristics associated with suicidal thoughts. Results: Of 8162 eligible EM residents, 7680 (94.1%) responded to at least 1 question on the survey; 6503 (79.7%) completed the survey in its entirety. A total of 243 ACGME-accredited residency programs participated, and 1 did not. The study cohort included 4768 male residents (62.1%), 2698 female residents (35.1%), 4919 non-Hispanic White residents (64.0%), 2620 residents from other racial/ethnic groups (Alaska Native, American Indian, Asian or Pacific Islander, African American, Mexican American, Native Hawaiian, Puerto Rican, other Hispanic, or mixed or other race) (34.1%), 483 residents who identified as lesbian, gay, bisexual, transgender, queer, or other (LGBTQ+) (6.3%), and 5951 residents who were married or in a relationship (77.5%). Of the total participants, 3463 (45.1%) reported exposure to some type of workplace mistreatment (eg, discrimination, abuse, or harassment) during the most recent academic year. A frequent source of mistreatment was identified as patients and/or patients' families; 1234 respondents (58.7%) reported gender discrimination, 867 (67.5%) racial discrimination, 282 (85.2%) physical abuse, and 723 (69.1%) sexual harassment from patients and/or family members. Suicidal thoughts occurring during the past year were reported by 178 residents (2.5%), with similar prevalence by gender (108 men [2.4%]; 59 women [2.4%]) and race/ethnicity (113 non-Hispanic White residents [2.4%]; 65 residents from other racial/ethnic groups [2.7%]). Conclusions and Relevance: In this survey study, EM residents reported that workplace mistreatment occurred frequently. The findings suggest common sources of mistreatment for which educational interventions may be developed to help ensure resident wellness and career satisfaction.

Attitudes of food consumers at universities towards recycling human urine as crop fertiliser: A multinational survey dataset.

We present here a data set generated from a multinational survey on opinions of university community members on the prospect of consuming food grown with human urine as fertiliser and about their urine recycling perceptions in general. The data set comprises answers from 3,763 university community members (students, faculty/researchers, and staff) from 20 universities in 16 countries and includes demographic variables (age bracket, gender, type of settlement of origin, academic discipline, and role in the university). Questions were designed based on Ajzen's theory of planned behaviour to elicit information about three components of behavioural intention-attitudes, subjective norms, and perceived behavioural control. Survey questions covered perceived risks and benefits (attitudes), perceptions of colleagues (injunctive social norm) and willingness to consume food grown with cow urine/faeces (descriptive social norm), and willingness to pay a price premium for food grown with human urine as fertiliser (perceived behavioural control). We also included a question about acceptable urine recycling and disposal options and assessed general environmental outlook via the 15-item revised New Ecological Paradigm (NEP) scale. Data were collected through a standardised survey instrument translated into the relevant languages and then administered via an online form. Invitations to the survey were sent by email to university mailing lists or to a systematic sample of the university directory. Only a few studies on attitudes towards using human urine as fertiliser have been conducted previously. The data described here, which we analysed in "Willingness among food consumers at universities to recycle human urine as crop fertiliser: Evidence from a multinational survey" [1], may be used to further understand potential barriers to acceptance of new sanitation systems based on wastewater source separation and urine recycling and can help inform the design of future sociological studies.

Willingness among food consumers to recycle human urine as crop fertiliser: Evidence from a multinational survey.

Source-separating sanitation systems offer the possibility of recycling nutrients present in wastewater as crop fertilisers. Thereby, they can reduce agriculture's impacts on global sources, sinks, and cycles for nitrogen and phosphorous, as well as their associated environmental costs. However, it has been broadly assumed that people would be reluctant to perform the new sanitation behaviours that are necessary for implementing such systems in practice. Yet, few studies have tried to systematically gather evidence in support of this assumption. To address this gap, we surveyed 3763 people at 20 universities in 16 countries using a standardised questionnaire. We identified and systematically assessed cross-cultural and country-level explanatory factors that were strongly associated with people's willingness to consume food grown using human urine as fertiliser. Overall, 68% of the respondents favoured recycling human urine, 59% stated a willingness to eat urine-fertilised food, and only 11% believed that urine posed health risks that could not be mitigated by treatment. Most people did not expect to pay less for urine-fertilised food, but only 15% were willing to pay a price premium. Consumer perceptions were found to differ greatly by country and the strongest predictive factors for acceptance overall were cognitive factors (perceptions of risks and benefits) and social norms. Increasing awareness and building trust among consumers about the effectiveness of new sanitation systems via cognitive and normative messaging can help increase acceptance. Based on our findings, we believe that in many countries, acceptance by food consumers will not be the major social barrier to closing the loop on human urine. That a potential market exists for urine-fertilised food, however, needs to be communicated to other stakeholders in the sanitation service chain.

Actions, indicators, and outputs in urban biodiversity plans: A multinational analysis of city practice.

Urban biodiversity offers important benefits to residents and may be crucial to reaching global biodiversity conservation targets, but little research has been conducted on how cities actually plan for biodiversity. In this study, we conducted a mixed methods content analysis of biodiversity plans by 39 cities around the world to determine whether they measured their actions, how they did so (via quantitative indicators and qualitative outputs), and what topics these actions and measures covered. We based our analytical framework on the Singapore Index on Cities' Biodiversity (also known as the City Biodiversity Index), a widely applied 23-indicator index that helps cities track their progress in biodiversity planning. The Singapore Index groups its indicators into the following three core components: native biodiversity, ecosystem services, and governance and management. For actions and measures not classifiable by the Singapore Index, we inductively derived additional categories. Across all plans, we identified 2,231 actions, 346 indicators, and 444 outputs. We found that all of the plans included actions, while 82% included measures (67% included indicators and 72% included outputs). Only 29% of actions were associated with a measure. Overall, the plans covered all of the categories in the Singapore Index, particularly within the core components of native biodiversity and governance and management, though some plans had a narrower focus. The 20 additional urban biodiversity topics that were not covered by the Singapore Index framework included socioeconomic considerations, data collection, genetic diversity, urban agriculture and forestry, green infrastructure, human-wildlife conflicts, indigenous concerns, and citizen science. Indicators were the most common measures for native biodiversity and ecosystem service topics, while outputs were the most common measures for governance and management. Our results may inform the revision and development of urban biodiversity indicators in the post-2020 framework and of other initiatives that guide cities in contributing to local and global biodiversity goals.

A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence.

PURPOSE: To investigate whether clinical competency committees (CCCs) were consistent in applying milestone ratings for first-year residents over time or whether ratings increased or decreased. METHOD: Beginning in December 2013, the Accreditation Council for Graduate Medical Education (ACGME) initiated a phased-in requirement for reporting milestones; emergency medicine (EM), diagnostic radiology (DR), and urology (UR) were among the earliest reporting specialties. The authors analyzed CCC milestone ratings of first-year residents from 2013 to 2016 from all ACGME-accredited EM, DR, and UR programs for which they had data. The number of first-year residents in these programs ranged from 2,838 to 2,928 over this time period. The program-level average milestone rating for each subcompetency was regressed onto the time of observation using a random coefficient multilevel regression model. RESULTS: National average program-level milestone ratings of first-year residents decreased significantly over the observed time period for 32 of the 56 subcompetencies examined. None of the other subcompetencies showed a significant change. National average in-training examination scores for each of the specialties remained essentially unchanged over the time period, suggesting that differences between the cohorts were not likely an explanatory factor. CONCLUSIONS: The findings indicate that CCCs tend to become more stringent or maintain consistency in their ratings of beginning residents over time. One explanation for these results is that CCCs may become increasingly comfortable in assigning lower ratings when appropriate. This finding is consistent with an increase in confidence with the milestone rating process and the quality of feedback it provides.

Straight Line Scoring by Clinical Competency Committees Using Emergency Medicine Milestones.

BACKGROUND: In 2013, milestone ratings became a reporting requirement for emergency medicine (EM) residency programs. Programs rate each resident in the fall and spring on 23 milestone subcompetencies. OBJECTIVE: This study examined the incidence of straight line scoring (SLS) for EM Milestone ratings, defined as a resident being assessed the same score across the milestone subcompetencies. METHODS: This descriptive analysis measured the frequencies of SLS for all Accreditation Council for Graduate Medical Education (ACGME)-accredited EM programs during the 2015-2016 academic year. Outcomes were the frequency of SLS in the fall and spring milestone assessments, changes in the number of SLS reports, and reporting trends. Chi-square analysis compared nominal variables. RESULTS: There were 6257 residents in the fall and 6588 in the spring. Milestone scores were reported for 6173 EM residents in the fall (99% of 6257) and spring (94% of 6588). In the fall, 93% (5753 residents) did not receive SLS ratings and 420 (7%) did, with no significant difference compared with the spring (5776 [94%] versus 397 [6%]). Subgroup analysis showed higher SLS results for residents' first ratings (183 of 2136 versus 237 of 4220, P < .0001) and for their final ratings (200 of 2019 versus 197 of 4354, P < .0001). Twenty percent of programs submitted 10% or more SLS ratings, and a small percentage submitted more than 50% of ratings as SLS. CONCLUSIONS: Most programs did not submit SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.