Measuring Emergency Department Workload Perception Using Electronic Medical Record Measures of Patient Volume and Acuity.

BACKGROUND: Workload in the emergency department (ED) fluctuates and there is no established model for measurement of clinician-level ED workload. OBJECTIVE: The aim of this study was to measure perceived ED workload and assess the relationship between perceived workload and objective measures of workload from the electronic medical record (EMR). METHODS: This study was conducted at a tertiary care, academic ED from July 1, 2020 through April 13, 2021. Attending workload perceptions were collected using a 5-point scale in three care areas with variable acuity. We collected eight EMR measures thought to correlate with perceived workload. EMR values were compared across areas of the department using ANOVA and correlated with attending workload ratings using linear regression. RESULTS: We collected 315 unique workload ratings, which were normally distributed. For the entire department, there was a weak positive correlation between reported workload perception and mean percentage of inpatient admissions (r = 0.23; p < 0.001), intensive care unit admissions (r = 0.2; p < 0.001), patient arrivals per shift (r = 0.14; p = 0.017), critical care billed visits (r = 0.22; p < 0.001), cardiopulmonary resuscitation code activations (r = 0.2; p < 0.001), and level 5 visits (r = 0.13; p = 0.02). There was weak negative correlation for ED discharges (r = -0.23; p < 0.001). Several correlations were stronger in individual care areas, including percent admissions in the lowest-acuity area (r = 0.43; p = 0.033) and patient arrivals in the highest-acuity area (r = 0.44; p < .01). No significant correlation was found in any area for observation admissions or trauma activations. CONCLUSIONS: In this study, EMR measures of workload were not closely correlated with ED attending physician workload perception. Future study should examine additional factors contributing to physician workload outside of the EMR.

Variability in Emergency Department Procedure Rates and Distributions in a Regional Health System: A Cross-Sectional Observational Study.

STUDY OBJECTIVE: Procedural competency is essential to the practice of emergency medicine. However, there are limited data quantifying emergency department procedural volumes to inform the work of educators and credentialing bodies. In this study, we characterize procedural scope and volume in a regional health care system and compare rates between practice settings and over time. METHODS: Cross-sectional data were acquired from electronic medical records of a regional health care system from March 2017 through February 2022. Nonspecific entries, esoteric procedures, and nonprocedural clinical skills were excluded. Procedural rates were compared: (1) between academic and community hospitals, (2) across study years, and (3) across seasons. Analyses were repeated for pediatric encounters, and with study year 4 removed to assess the influence of the first year of the coronavirus disease 2019 pandemic on results. RESULTS: There were 131,976 instances of 40 qualifying procedures in 1,979,935 unique visits across 9 EDs. Several high-acuity procedures had similar rates in academic and community settings, including cardiac pacing, cricothyrotomy, and lateral canthotomy. Year-over-year procedural rates were stable or increasing for most procedures, with a notable exception of lumbar puncture. Most procedures did not have significant seasonal variation, and most findings were stable when study year 4 was removed from the analysis. CONCLUSION: All procedures were performed in all settings and rates of several emergent procedures were similar in both settings, underscoring the importance of broad procedural competence for all emergency physicians. Educators and credentialing organizations can use these data to inform decisions regarding curriculum design and certification requirements.

The test developer's dilemma: Evaluating the balance of feasibility and empiric performance of test development techniques for repeated written assessments.

PURPOSE: Written assessments face challenges when administered repeatedly, including resource-intensive item development and the potential for performance improvement secondary to item recall as opposed to understanding. This study examines the efficacy of three-item development techniques in addressing these challenges. METHODS: Learners at five training programs completed two 60-item repeated assessments. Items from the first test were randomized to one of three treatments for the second assessment: (1) Verbatim repetition, (2) Isomorphic changes, or (3) Total revisions. Primary outcomes were the stability of item psychometrics across test versions and evidence of item recall influencing performance as measured by the rate of items answered correctly and then incorrectly (correct-to-incorrect rate), which suggests guessing. RESULTS: Forty-six learners completed both tests. Item psychometrics were comparable across test versions. Correct-to-incorrect rates differed significantly between groups with the highest guessing rate (lowest recall effect) in the Total Revision group (0.15) and the lowest guessing rate (highest recall effect) in the Verbatim group (0.05), p = 0.01. CONCLUSIONS: Isomorphic and total revisions demonstrated superior performance in mitigating the effect of recall on repeated assessments. Given the high costs of total item revisions, there is promise in exploring isomorphic items as an efficient and effective approach to repeated written assessments.[Box: see text].

Hawks and Doves in Standardized Letters of Evaluation: 6 Years of Rating Distributions and Trends in Emergency Medicine.

Background Standardized Letters of Evaluation (SLOEs) are an important part of resident selection in many specialties. Often written by a group, such letters may ask writers to rate applicants in different domains. Prior studies have noted inflated ratings; however, the degree to which individual institutions are "doves" (higher rating) or "hawks" (lower rating) is unclear. Objective To characterize institutional SLOE rating distributions to inform readers and developers regarding potential threats to validity from disparate rating practices. Methods Data from emergency medicine (EM) SLOEs between 2016 and 2021 were obtained from a national database. SLOEs from institutions with at least 10 letters per year in all years were included. Ratings on one element of the SLOE-the "global assessment of performance" item (Top 10%, Top Third, Middle Third, and Lower Third)-were analyzed numerically and stratified by predefined criteria for grading patterns (Extreme Dove, Dove, Neutral, Hawk, Extreme Hawk) and adherence to established guidelines (Very High, High, Neutral, Low, Very Low). Results Of 40 286 SLOEs, 20 407 met inclusion criteria. Thirty-five to 50% of institutions displayed Neutral grading patterns across study years, with most other institutional patterns rated as Dove or Extreme Dove. Adherence to guidelines was mixed and fewer than half of institutions had Very High or High adherence each year. Most institutions underutilize the Lower Third rating. Conclusions Despite explicit guidelines for the distribution of global assessment ratings in the EM SLOE, there is high variability in institutional rating practices.

Differences in Emergency Medicine Resident Procedural Reporting by Gender in the United States.

Background Studies across specialties have demonstrated gender disparities in feedback, learner assessments, and operative cases. However, data are limited on differences in numbers of procedures among residents. Objective To quantify the association between gender and the number of procedures reported among emergency medicine (EM) residents. Methods We conducted a retrospective review of procedural differences by self-identified gender among graduating EM residents at 8 separate programs over a 10-year period (2013 to 2022). Sites were selected to ensure diversity of program length, program type, and geography. Residents from combined training programs, those who did not complete their full training at that institution, and those who did not have data available were excluded. We calculated the mean, SD, median, and IQR for each procedure by gender. We compared reported procedures by gender using linear regression, controlling for institution, and performed a sensitivity analysis excluding outlier residents with procedure totals >3 SD from the mean. Results We collected data from 914 residents, with 880 (96.3%) meeting inclusion criteria. There were 358 (40.7%) women and 522 (59.3%) men. The most common procedures were point-of-care ultrasound, adult medical resuscitation, adult trauma resuscitation, and intubations. After adjusting for institutions, the number of dislocation reductions, chest tube insertions, and sedations were higher for men. The sensitivity analysis findings were stable except for central lines, which were also more common in men. Conclusions In a national sample of EM programs, there were increased numbers of dislocation reductions, chest tube insertions, and sedations reported by men compared with women.

Medical Malpractice Cases Involving Medical Students.

PURPOSE: This study uses a large national legal database to analyze characteristics of malpractice claims involving U.S. medical students. METHOD: The Westlaw database was searched in September 2023 for malpractice cases involving medical students from January 1, 1900, through September 1, 2023. Each case was independently reviewed by 2 authors, abstracting each variable. Categorical data were summarized as frequency of occurrence (i.e., number and percentage), and continuous data were summarized with means, medians, and ranges. All authors independently reviewed the dataset to identify potential themes and codes. RESULTS: There were 65 cases that met the inclusion criteria. Reported patient outcomes were death (19, 29%), pain (25, 38%), and disability (36, 55%). The most common specialties involved were emergency medicine (16, 25%), general surgery (14, 22%), and obstetrics and gynecology (13, 20%). The most common primary alleged errors attributed to students related to medical decision-making (30, 46%), procedural complication (24, 37%), and poor communication (11, 17%). Among 23 (35%) cases reporting year of training, 1 (4%) included a second-year student, 13 (57%) included third-year students, and 9 (39%) included fourth-year students. Of the 65 lawsuits, 28 (43%) resulted in a settlement or verdict against the medical student. The total amount paid in these cases was $78,192,612, with a mean (median) of $3,007,408 ($1,050,000) per case. Of these 28 cases, 14 (50%) cited minimal or no physician supervision. CONCLUSIONS: Medical malpractice claims involving medical students are rare but commonly relate to medical decision-making, procedural complication, and poor communication, with a lack of supervision being frequently cited. These results can be used to guide students and supervising physicians on how to avoid scenarios that may increase vulnerability to medical malpractice lawsuits.

Faculty consensus on competitiveness for the new competency-based emergency medicine standardized letter of evaluation.

Background: Emergency medicine (EM) has introduced a new, competency-based standardized letter of evaluation (SLOE) template. While a previous version of the SLOE has been shown to promote a high degree of faculty consensus regarding competitiveness, this has not been shown for the new SLOE template. Objective: The objective was to evaluate faculty consensus on competitiveness for the new EM SLOE 2.0. Methods: Fifty mock SLOE 2.0 letters using the new template were drafted and sent to a group of experienced EM educators. The 50 letters were ranked by the experienced faculty as well as a point-based prediction model and a regression model and the results were compared. Results: Faculty consensus on competitiveness remained strong when using the new SLOE 2.0 format. The points-based prediction model and regression model both demonstrated a high level of agreement with faculty consensus rankings for the SLOE 2.0. Conclusions: Introduction of the new, competency-based SLOE 2.0 format did not have a deleterious effect on faculty consensus rankings of competitiveness.

Decoding competitiveness: Exploring how emergency medicine faculty interpret standardized letters of evaluation.

Background: Given the importance of the standardized letter of evaluation (SLOE) for application to emergency medicine (EM) residency, it is important that SLOE developers and authors understand how reviewers determine SLOE competitiveness. To inform SLOE design and authorship, the authors set out to build a novel theory to explain how faculty holistically interpret SLOE competitiveness. Methods: The authors used constructivist grounded theory to explore how EM faculty determine SLOE competitiveness. They used purposive sampling to recruit EM faculty participants with at least 1 year of experience in scoring SLOEs. One author conducted hour-long, semistructured interviews over Zoom between August 2023 and March 2024. Two authors iteratively coded the data to develop the initial codebook, organize codes into categories, and build connections to construct the resulting theory. Results: The authors interviewed 11 EM faculty from throughout the United States. Participants described a complex process to determine SLOE competitiveness. They began by contextualizing the SLOE to determine its trustworthiness and value before using various components of the SLOE to stratify and refine their understanding of competitiveness. Finally, when participants noted the inconsistency between different aspects of the SLOE, they used various methods to reconcile discordances and determine competitiveness. Conclusions: This study illuminates the framework used by EM faculty to determine applicant competitiveness based on the SLOE and highlights several factors that SLOE authors should consider to ensure the accurate and efficient transfer of information.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation.

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Differences in emergency medicine resident procedural reporting by race and ethnicity.

Background: The recruitment, retention, and training of physicians from groups underrepresented in medicine (UiM) is critically important to the practice of emergency medicine (EM). Studies across specialties have demonstrated disparities in operative experiences among UiM resident learners who are UiM; however, there are limited data on procedural disparities in EM. Objective: We sought to quantify the association between racial and ethnic identities that are UiM and the number of procedures reported among EM residents. Methods: We conducted a retrospective review of procedural differences by UiM status (using self-identified race and ethnicity) among graduating EM residents at nine training programs over a 10-year period. Sites were selected to ensure diversity of program length, program type, and geography. Data from residents in combined training programs, those who did not complete their full training at that institution, and those with missing data or electing not to report race/ethnicity were excluded. We calculated median and interquartile ranges for each procedure by UiM status. We conducted multivariable regression analyses accounting for UiM status, gender, and site as well as a sensitivity analysis excluding values >3 standard deviations from the mean for each procedure. Results: We collected data from 988 total residents, with 718 (73%) being non-UiM, 204 (21%) being UiM, 48 (5%) electing not to specific race/ethnicity, and 18 (2%) missing race/ethnicity data. While unadjusted data demonstrated a difference between UiM and non-UiM resident numbers across several procedures, there were no significant differences in procedures reported after accounting for gender and site in the primary or sensitivity analyses. Conclusions: We did not identify a statistically significant difference in reported procedures between UiM and non-UiM residents in EM. Future work should include qualitative investigations of UiM resident experience surrounding procedures as well as mixed-methods studies to examine how these data interact.

Flexible endoscopic intubation in emergency medicine: A mixed-methods needs assessment.

Objectives: This needs assessment aimed to improve understanding of flexible endoscopic intubation training and practice in emergency medicine (EM), providing insights to educators and practice leaders seeking to improve education and practices. Methods: We conducted a multicenter, mixed-methods needs assessment of emergency physicians (EPs) incorporating focus groups and a survey. Focus groups comprised community EPs, academic EPs, and resident EPs. We analyzed focus group transcripts using grounded theory, qualitatively describing EM endoscopic intubation. The qualitative analysis shaped our survey instrument, which we deployed in cross-sectional fashion. We report survey data with descriptive statistics. Results: Focus groups with 13 EPs identified three themes: indications for use of endoscopic intubation, factors impacting a physician's decision to endoscopically intubate, and attaining and maintaining endoscopic intubation competency. Of 257 surveyed EPs (33% response rate), 79% had received endoscopic intubation training during residency, though 82% had performed this procedure 10 or fewer times in their career. Despite 97% acknowledging the necessity of competency, only 23% felt highly confident in their ability to perform endoscopic intubation. Participants (93%) reported scarce opportunities to perform the procedure and identified factors believed to facilitate competency acquisition and maintenance, including opportunities to perform endoscopic intubation in practice (98%), local champions (93%), and performing nasopharyngoscopy (87%). Conclusions: While most EPs acknowledged the importance of competency in endoscopic intubation, they reported scarce procedural opportunities and commonly expressed low confidence. Further research is needed on this topic, and we propose avenues to enhance education and practices related to endoscopic intubation. These include development of robust procedural curricula, support of local champions, and incorporating nasopharyngoscopy into EM practice.

Educator's blueprint: Holistic applicant file review in undergraduate and postgraduate medical education.

Medical schools and graduate medical education programs are tasked each year with selecting the next class of trainees, often from large applicant pools with enormous quantities of data to be processed. Review of applicant files must therefore be efficient, equitable, and effective in maximizing the likelihood of trainee success and alignment with institutional missions and values. In this article, we discuss 10 strategies to optimize the file review process for trainee selection. Using these strategies, educators can ensure rigorous and accountable file review processes for their training programs.

Educator's blueprint: A how-to guide for developing high-quality multiple-choice questions.

Multiple-choice questions are commonly used for assessing learners' knowledge, as part of educational programs and scholarly endeavors. To ensure that questions accurately assess the learners and provide meaningful data, it is important to understand best practices in multiple-choice question design. This Educator's Blueprint paper provides 10 strategies for developing high-quality multiple-choice questions. These strategies include determining the purpose, objectives, and scope of the question; assembling a writing team; writing succinctly; asking questions that assess knowledge and comprehension rather than test-taking ability; ensuring consistent and independent answer choices; using plausible foils; avoiding grouped options; selecting the ideal response number and order; writing high-quality explanations; and gathering validity evidence before and evaluating the questions after use.

Trends in point-of-care ultrasound use among emergency medicine residency programs over a 10-year period.

Background: Point-of-care ultrasound (POCUS) is increasingly utilized in emergency medicine (EM). While residents are required by the Accreditation Council for General Medical Education to complete a minimum of 150 POCUS examinations before graduation, the distribution of examination types is not well-described. This study sought to assess the number and distribution of POCUS examinations completed during EM residency training and evaluate trends over time. Methods: This was a 10-year retrospective review of POCUS examinations across five EM residency programs. The study sites were deliberately selected to represent diversity in program type, program length, and geography. Data from EM residents graduating from 2013 to 2022 were eligible for inclusion. Exclusion criteria were residents in combined training programs, residents who did not complete all training at one institution, and residents who did not have data available. Examination types were identified from the American College of Emergency Physicians guidelines for POCUS. Each site obtained POCUS examination totals for every resident upon graduation. We calculated the mean and 95% confidence interval for each procedure across study years. Results: A total of 535 residents were eligible for inclusion, with 524 (97.9%) meeting all inclusion criteria. The mean number of POCUS examinations per resident increased by 46.9% from 277 in 2013 to 407 in 2022. All examination types had stable or increasing frequency. Focused assessment with sonography in trauma (FAST), cardiac, obstetric/gynecologic, and renal/bladder were performed most frequently. Ocular, deep venous thrombosis, musculoskeletal, skin/soft tissue, thoracic, and cardiac examinations had the largest percentage increase in numbers over the 10-year period, while bowel and testicular POCUS remained rare. Conclusions: There was an overall increase in the number of POCUS examinations performed by EM residents over the past 10 years, with FAST, cardiac, obstetric/gynecologic, and renal/bladder being the most common examination types. Among less common procedures, increased frequency may be needed to ensure competence and avoid skill decay for those examination types. This information can help inform POCUS training in residency and accreditation requirements.

Trends in emergency medicine resident procedural reporting over a 10-year period.

Background: Procedural competency is expected of all emergency medicine (EM) residents upon graduation. The ACGME requires a minimum number of essential procedures to successfully complete training. However, data are limited on the actual number of procedures residents perform and prior studies are limited to single institutions over short time periods. This study sought to assess the number of Key Index Procedures completed during EM residency training and evaluate trends over time. Methods: We conducted a retrospective review of graduating EM resident procedure logs across eight ACGME accredited residency programs over the last 10 years (2013-2022). Sites were selected to ensure diversity of program length, program type, and geography. All data from EM residents graduating in 2013-2022 were eligible for inclusion. Data from residents from combined training programs, those who did not complete their full training at that institution (i.e., transferred in/out), or those who did not have data available were excluded. We determined the list of procedures based upon the ACGME Key Index Procedures list. Sites obtained totals for each of the identified procedures for each resident upon graduation. We calculated the mean and 95% CI for each procedure. Results: We collected data from a total of 914 residents, with 881 (96.4%) meeting inclusion criteria. The most common procedures were point-of-care ultrasound, adult medical resuscitation, adult trauma resuscitation, and intubation. The least frequent procedures included pericardiocentesis, cricothyroidotomy, cardiac pacing, vaginal delivery, and chest tubes. Most procedures were stable over time with the exception of lumbar punctures (decreased) and point-of-care ultrasound (increased). Conclusions: In a national sample of EM programs, procedural numbers remained stable except for lumbar puncture and ultrasound. This information can inform residency training curricula and accreditation requirements.

Virtual student experiences: A case study of objectives, outcomes, and anticipated consequences.

Due to the COVID-19 pandemic, most in-person visiting clerkship opportunities have been canceled. Many institutions have developed virtual experiences to fill this void; however, the format and objectives of these experiences are variable. This article describes an education intervention for visiting students where both learner-oriented and program-oriented outcomes of a nonclerkship virtual student experience are explored. This intervention consisted of five 1-hour case-based teaching sessions in addition to attendance at weekly residency conference over a 1-week period. The primary outcomes were student ratings of how the experience impacted their 1) perception of the program's educational environment and culture, 2) medical knowledge, and 3) program ranking before and after the experience. Of 24 participants, 15 students (63%) completed the final evaluation, all with positive summative rankings in every domain measured. Of the 15 respondents, 12 (80%) reported an increase in where they would rank the program after the experience compared to before, and nine of 15 (60%) stated that they would rank the program #1 if making a list at the time they completed the final evaluation. Future study should examine long-term outcomes of these interventions including decisions regarding where students apply, who programs interview, and ranking decisions of students and programs.