Measuring Emergency Department Workload Perception Using Electronic Medical Record Measures of Patient Volume and Acuity.

BACKGROUND: Workload in the emergency department (ED) fluctuates and there is no established model for measurement of clinician-level ED workload. OBJECTIVE: The aim of this study was to measure perceived ED workload and assess the relationship between perceived workload and objective measures of workload from the electronic medical record (EMR). METHODS: This study was conducted at a tertiary care, academic ED from July 1, 2020 through April 13, 2021. Attending workload perceptions were collected using a 5-point scale in three care areas with variable acuity. We collected eight EMR measures thought to correlate with perceived workload. EMR values were compared across areas of the department using ANOVA and correlated with attending workload ratings using linear regression. RESULTS: We collected 315 unique workload ratings, which were normally distributed. For the entire department, there was a weak positive correlation between reported workload perception and mean percentage of inpatient admissions (r = 0.23; p < 0.001), intensive care unit admissions (r = 0.2; p < 0.001), patient arrivals per shift (r = 0.14; p = 0.017), critical care billed visits (r = 0.22; p < 0.001), cardiopulmonary resuscitation code activations (r = 0.2; p < 0.001), and level 5 visits (r = 0.13; p = 0.02). There was weak negative correlation for ED discharges (r = -0.23; p < 0.001). Several correlations were stronger in individual care areas, including percent admissions in the lowest-acuity area (r = 0.43; p = 0.033) and patient arrivals in the highest-acuity area (r = 0.44; p < .01). No significant correlation was found in any area for observation admissions or trauma activations. CONCLUSIONS: In this study, EMR measures of workload were not closely correlated with ED attending physician workload perception. Future study should examine additional factors contributing to physician workload outside of the EMR.

Variability in Emergency Department Procedure Rates and Distributions in a Regional Health System: A Cross-Sectional Observational Study.

STUDY OBJECTIVE: Procedural competency is essential to the practice of emergency medicine. However, there are limited data quantifying emergency department procedural volumes to inform the work of educators and credentialing bodies. In this study, we characterize procedural scope and volume in a regional health care system and compare rates between practice settings and over time. METHODS: Cross-sectional data were acquired from electronic medical records of a regional health care system from March 2017 through February 2022. Nonspecific entries, esoteric procedures, and nonprocedural clinical skills were excluded. Procedural rates were compared: (1) between academic and community hospitals, (2) across study years, and (3) across seasons. Analyses were repeated for pediatric encounters, and with study year 4 removed to assess the influence of the first year of the coronavirus disease 2019 pandemic on results. RESULTS: There were 131,976 instances of 40 qualifying procedures in 1,979,935 unique visits across 9 EDs. Several high-acuity procedures had similar rates in academic and community settings, including cardiac pacing, cricothyrotomy, and lateral canthotomy. Year-over-year procedural rates were stable or increasing for most procedures, with a notable exception of lumbar puncture. Most procedures did not have significant seasonal variation, and most findings were stable when study year 4 was removed from the analysis. CONCLUSION: All procedures were performed in all settings and rates of several emergent procedures were similar in both settings, underscoring the importance of broad procedural competence for all emergency physicians. Educators and credentialing organizations can use these data to inform decisions regarding curriculum design and certification requirements.

The test developer's dilemma: Evaluating the balance of feasibility and empiric performance of test development techniques for repeated written assessments.

PURPOSE: Written assessments face challenges when administered repeatedly, including resource-intensive item development and the potential for performance improvement secondary to item recall as opposed to understanding. This study examines the efficacy of three-item development techniques in addressing these challenges. METHODS: Learners at five training programs completed two 60-item repeated assessments. Items from the first test were randomized to one of three treatments for the second assessment: (1) Verbatim repetition, (2) Isomorphic changes, or (3) Total revisions. Primary outcomes were the stability of item psychometrics across test versions and evidence of item recall influencing performance as measured by the rate of items answered correctly and then incorrectly (correct-to-incorrect rate), which suggests guessing. RESULTS: Forty-six learners completed both tests. Item psychometrics were comparable across test versions. Correct-to-incorrect rates differed significantly between groups with the highest guessing rate (lowest recall effect) in the Total Revision group (0.15) and the lowest guessing rate (highest recall effect) in the Verbatim group (0.05), p = 0.01. CONCLUSIONS: Isomorphic and total revisions demonstrated superior performance in mitigating the effect of recall on repeated assessments. Given the high costs of total item revisions, there is promise in exploring isomorphic items as an efficient and effective approach to repeated written assessments.[Box: see text].

Hawks and Doves in Standardized Letters of Evaluation: 6 Years of Rating Distributions and Trends in Emergency Medicine.

Background Standardized Letters of Evaluation (SLOEs) are an important part of resident selection in many specialties. Often written by a group, such letters may ask writers to rate applicants in different domains. Prior studies have noted inflated ratings; however, the degree to which individual institutions are "doves" (higher rating) or "hawks" (lower rating) is unclear. Objective To characterize institutional SLOE rating distributions to inform readers and developers regarding potential threats to validity from disparate rating practices. Methods Data from emergency medicine (EM) SLOEs between 2016 and 2021 were obtained from a national database. SLOEs from institutions with at least 10 letters per year in all years were included. Ratings on one element of the SLOE-the "global assessment of performance" item (Top 10%, Top Third, Middle Third, and Lower Third)-were analyzed numerically and stratified by predefined criteria for grading patterns (Extreme Dove, Dove, Neutral, Hawk, Extreme Hawk) and adherence to established guidelines (Very High, High, Neutral, Low, Very Low). Results Of 40 286 SLOEs, 20 407 met inclusion criteria. Thirty-five to 50% of institutions displayed Neutral grading patterns across study years, with most other institutional patterns rated as Dove or Extreme Dove. Adherence to guidelines was mixed and fewer than half of institutions had Very High or High adherence each year. Most institutions underutilize the Lower Third rating. Conclusions Despite explicit guidelines for the distribution of global assessment ratings in the EM SLOE, there is high variability in institutional rating practices.

Medical Malpractice Cases Involving Medical Students.

PURPOSE: This study uses a large national legal database to analyze characteristics of malpractice claims involving U.S. medical students. METHOD: The Westlaw database was searched in September 2023 for malpractice cases involving medical students from January 1, 1900, through September 1, 2023. Each case was independently reviewed by 2 authors, abstracting each variable. Categorical data were summarized as frequency of occurrence (i.e., number and percentage), and continuous data were summarized with means, medians, and ranges. All authors independently reviewed the dataset to identify potential themes and codes. RESULTS: There were 65 cases that met the inclusion criteria. Reported patient outcomes were death (19, 29%), pain (25, 38%), and disability (36, 55%). The most common specialties involved were emergency medicine (16, 25%), general surgery (14, 22%), and obstetrics and gynecology (13, 20%). The most common primary alleged errors attributed to students related to medical decision-making (30, 46%), procedural complication (24, 37%), and poor communication (11, 17%). Among 23 (35%) cases reporting year of training, 1 (4%) included a second-year student, 13 (57%) included third-year students, and 9 (39%) included fourth-year students. Of the 65 lawsuits, 28 (43%) resulted in a settlement or verdict against the medical student. The total amount paid in these cases was $78,192,612, with a mean (median) of $3,007,408 ($1,050,000) per case. Of these 28 cases, 14 (50%) cited minimal or no physician supervision. CONCLUSIONS: Medical malpractice claims involving medical students are rare but commonly relate to medical decision-making, procedural complication, and poor communication, with a lack of supervision being frequently cited. These results can be used to guide students and supervising physicians on how to avoid scenarios that may increase vulnerability to medical malpractice lawsuits.

Differences in Emergency Medicine Resident Procedural Reporting by Gender in the United States.

Background Studies across specialties have demonstrated gender disparities in feedback, learner assessments, and operative cases. However, data are limited on differences in numbers of procedures among residents. Objective To quantify the association between gender and the number of procedures reported among emergency medicine (EM) residents. Methods We conducted a retrospective review of procedural differences by self-identified gender among graduating EM residents at 8 separate programs over a 10-year period (2013 to 2022). Sites were selected to ensure diversity of program length, program type, and geography. Residents from combined training programs, those who did not complete their full training at that institution, and those who did not have data available were excluded. We calculated the mean, SD, median, and IQR for each procedure by gender. We compared reported procedures by gender using linear regression, controlling for institution, and performed a sensitivity analysis excluding outlier residents with procedure totals >3 SD from the mean. Results We collected data from 914 residents, with 880 (96.3%) meeting inclusion criteria. There were 358 (40.7%) women and 522 (59.3%) men. The most common procedures were point-of-care ultrasound, adult medical resuscitation, adult trauma resuscitation, and intubations. After adjusting for institutions, the number of dislocation reductions, chest tube insertions, and sedations were higher for men. The sensitivity analysis findings were stable except for central lines, which were also more common in men. Conclusions In a national sample of EM programs, there were increased numbers of dislocation reductions, chest tube insertions, and sedations reported by men compared with women.

Differences in emergency medicine resident procedural reporting by race and ethnicity.

Background: The recruitment, retention, and training of physicians from groups underrepresented in medicine (UiM) is critically important to the practice of emergency medicine (EM). Studies across specialties have demonstrated disparities in operative experiences among UiM resident learners who are UiM; however, there are limited data on procedural disparities in EM. Objective: We sought to quantify the association between racial and ethnic identities that are UiM and the number of procedures reported among EM residents. Methods: We conducted a retrospective review of procedural differences by UiM status (using self-identified race and ethnicity) among graduating EM residents at nine training programs over a 10-year period. Sites were selected to ensure diversity of program length, program type, and geography. Data from residents in combined training programs, those who did not complete their full training at that institution, and those with missing data or electing not to report race/ethnicity were excluded. We calculated median and interquartile ranges for each procedure by UiM status. We conducted multivariable regression analyses accounting for UiM status, gender, and site as well as a sensitivity analysis excluding values >3 standard deviations from the mean for each procedure. Results: We collected data from 988 total residents, with 718 (73%) being non-UiM, 204 (21%) being UiM, 48 (5%) electing not to specific race/ethnicity, and 18 (2%) missing race/ethnicity data. While unadjusted data demonstrated a difference between UiM and non-UiM resident numbers across several procedures, there were no significant differences in procedures reported after accounting for gender and site in the primary or sensitivity analyses. Conclusions: We did not identify a statistically significant difference in reported procedures between UiM and non-UiM residents in EM. Future work should include qualitative investigations of UiM resident experience surrounding procedures as well as mixed-methods studies to examine how these data interact.

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation.

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Flexible endoscopic intubation in emergency medicine: A mixed-methods needs assessment.

Objectives: This needs assessment aimed to improve understanding of flexible endoscopic intubation training and practice in emergency medicine (EM), providing insights to educators and practice leaders seeking to improve education and practices. Methods: We conducted a multicenter, mixed-methods needs assessment of emergency physicians (EPs) incorporating focus groups and a survey. Focus groups comprised community EPs, academic EPs, and resident EPs. We analyzed focus group transcripts using grounded theory, qualitatively describing EM endoscopic intubation. The qualitative analysis shaped our survey instrument, which we deployed in cross-sectional fashion. We report survey data with descriptive statistics. Results: Focus groups with 13 EPs identified three themes: indications for use of endoscopic intubation, factors impacting a physician's decision to endoscopically intubate, and attaining and maintaining endoscopic intubation competency. Of 257 surveyed EPs (33% response rate), 79% had received endoscopic intubation training during residency, though 82% had performed this procedure 10 or fewer times in their career. Despite 97% acknowledging the necessity of competency, only 23% felt highly confident in their ability to perform endoscopic intubation. Participants (93%) reported scarce opportunities to perform the procedure and identified factors believed to facilitate competency acquisition and maintenance, including opportunities to perform endoscopic intubation in practice (98%), local champions (93%), and performing nasopharyngoscopy (87%). Conclusions: While most EPs acknowledged the importance of competency in endoscopic intubation, they reported scarce procedural opportunities and commonly expressed low confidence. Further research is needed on this topic, and we propose avenues to enhance education and practices related to endoscopic intubation. These include development of robust procedural curricula, support of local champions, and incorporating nasopharyngoscopy into EM practice.

Educator's blueprint: Holistic applicant file review in undergraduate and postgraduate medical education.

Medical schools and graduate medical education programs are tasked each year with selecting the next class of trainees, often from large applicant pools with enormous quantities of data to be processed. Review of applicant files must therefore be efficient, equitable, and effective in maximizing the likelihood of trainee success and alignment with institutional missions and values. In this article, we discuss 10 strategies to optimize the file review process for trainee selection. Using these strategies, educators can ensure rigorous and accountable file review processes for their training programs.

Educator's blueprint: A how-to guide for developing high-quality multiple-choice questions.

Multiple-choice questions are commonly used for assessing learners' knowledge, as part of educational programs and scholarly endeavors. To ensure that questions accurately assess the learners and provide meaningful data, it is important to understand best practices in multiple-choice question design. This Educator's Blueprint paper provides 10 strategies for developing high-quality multiple-choice questions. These strategies include determining the purpose, objectives, and scope of the question; assembling a writing team; writing succinctly; asking questions that assess knowledge and comprehension rather than test-taking ability; ensuring consistent and independent answer choices; using plausible foils; avoiding grouped options; selecting the ideal response number and order; writing high-quality explanations; and gathering validity evidence before and evaluating the questions after use.

Trends in emergency medicine resident procedural reporting over a 10-year period.

Background: Procedural competency is expected of all emergency medicine (EM) residents upon graduation. The ACGME requires a minimum number of essential procedures to successfully complete training. However, data are limited on the actual number of procedures residents perform and prior studies are limited to single institutions over short time periods. This study sought to assess the number of Key Index Procedures completed during EM residency training and evaluate trends over time. Methods: We conducted a retrospective review of graduating EM resident procedure logs across eight ACGME accredited residency programs over the last 10 years (2013-2022). Sites were selected to ensure diversity of program length, program type, and geography. All data from EM residents graduating in 2013-2022 were eligible for inclusion. Data from residents from combined training programs, those who did not complete their full training at that institution (i.e., transferred in/out), or those who did not have data available were excluded. We determined the list of procedures based upon the ACGME Key Index Procedures list. Sites obtained totals for each of the identified procedures for each resident upon graduation. We calculated the mean and 95% CI for each procedure. Results: We collected data from a total of 914 residents, with 881 (96.4%) meeting inclusion criteria. The most common procedures were point-of-care ultrasound, adult medical resuscitation, adult trauma resuscitation, and intubation. The least frequent procedures included pericardiocentesis, cricothyroidotomy, cardiac pacing, vaginal delivery, and chest tubes. Most procedures were stable over time with the exception of lumbar punctures (decreased) and point-of-care ultrasound (increased). Conclusions: In a national sample of EM programs, procedural numbers remained stable except for lumbar puncture and ultrasound. This information can inform residency training curricula and accreditation requirements.

Trends in point-of-care ultrasound use among emergency medicine residency programs over a 10-year period.

Background: Point-of-care ultrasound (POCUS) is increasingly utilized in emergency medicine (EM). While residents are required by the Accreditation Council for General Medical Education to complete a minimum of 150 POCUS examinations before graduation, the distribution of examination types is not well-described. This study sought to assess the number and distribution of POCUS examinations completed during EM residency training and evaluate trends over time. Methods: This was a 10-year retrospective review of POCUS examinations across five EM residency programs. The study sites were deliberately selected to represent diversity in program type, program length, and geography. Data from EM residents graduating from 2013 to 2022 were eligible for inclusion. Exclusion criteria were residents in combined training programs, residents who did not complete all training at one institution, and residents who did not have data available. Examination types were identified from the American College of Emergency Physicians guidelines for POCUS. Each site obtained POCUS examination totals for every resident upon graduation. We calculated the mean and 95% confidence interval for each procedure across study years. Results: A total of 535 residents were eligible for inclusion, with 524 (97.9%) meeting all inclusion criteria. The mean number of POCUS examinations per resident increased by 46.9% from 277 in 2013 to 407 in 2022. All examination types had stable or increasing frequency. Focused assessment with sonography in trauma (FAST), cardiac, obstetric/gynecologic, and renal/bladder were performed most frequently. Ocular, deep venous thrombosis, musculoskeletal, skin/soft tissue, thoracic, and cardiac examinations had the largest percentage increase in numbers over the 10-year period, while bowel and testicular POCUS remained rare. Conclusions: There was an overall increase in the number of POCUS examinations performed by EM residents over the past 10 years, with FAST, cardiac, obstetric/gynecologic, and renal/bladder being the most common examination types. Among less common procedures, increased frequency may be needed to ensure competence and avoid skill decay for those examination types. This information can help inform POCUS training in residency and accreditation requirements.

Notes From the Field: Automatic Item Generation, Standard Setting, and Learner Performance in Mastery Multiple-Choice Tests.

Mastery learning assessments have been described in simulation-based educational interventions; however, studies applying mastery learning to multiple-choice tests (MCTs) are lacking. This study investigates an approach to item generation and standard setting for mastery learning MCTs and evaluates the consistency of learner performance across sequential tests. Item models, variables for question stems, and mastery standards were established using a consensus process. Two test forms were created using item models. Tests were administered at two training programs. The primary outcome, the test-retest consistency of pass-fail decisions across versions of the test, was 94% (κ = .54). Decision-consistency classification was .85. Item-level consistency was 90% (κ = .77, SE = .03). These findings support the use of automatic item generation to create mastery MCTs which produce consistent pass-fail decisions. This technique broadens the range of assessment methods available to educators that require serial MCT testing, including mastery learning curricula.

Virtual student experiences: A case study of objectives, outcomes, and anticipated consequences.

Due to the COVID-19 pandemic, most in-person visiting clerkship opportunities have been canceled. Many institutions have developed virtual experiences to fill this void; however, the format and objectives of these experiences are variable. This article describes an education intervention for visiting students where both learner-oriented and program-oriented outcomes of a nonclerkship virtual student experience are explored. This intervention consisted of five 1-hour case-based teaching sessions in addition to attendance at weekly residency conference over a 1-week period. The primary outcomes were student ratings of how the experience impacted their 1) perception of the program's educational environment and culture, 2) medical knowledge, and 3) program ranking before and after the experience. Of 24 participants, 15 students (63%) completed the final evaluation, all with positive summative rankings in every domain measured. Of the 15 respondents, 12 (80%) reported an increase in where they would rank the program after the experience compared to before, and nine of 15 (60%) stated that they would rank the program #1 if making a list at the time they completed the final evaluation. Future study should examine long-term outcomes of these interventions including decisions regarding where students apply, who programs interview, and ranking decisions of students and programs.

Affective, cognitive, and behavioral outcomes from a resident personal finance curriculum pilot project.

BACKGROUND: The transition to residency marks a significant shift in the financial circumstances of medical trainees. Despite existing resources, residents still cite uncertainty in this domain. A personal finance curriculum is needed to close this educational gap and improve the financial well-being of trainees. METHODS: The curriculum was developed using Kern's framework. Two needs assessments informed the consensus development of goals and objectives, educational strategies, and assessments. Course material was hosted online for asynchronous review and complemented by two 1-hour webinars. The curriculum was piloted at one institution. Participants completed (1) knowledge assessments before and after the intervention, (2) a survey of reactions to the curriculum, and (3) an assessment of financial behavioral changes after the intervention. RESULTS: Thirty-seven residents (37/49, 76%) enrolled in the curriculum. Among participants, 20 (20/37, 54%) completed the curriculum. Most participants agreed or strongly agreed that the content was relevant (20/20, 100%) and clearly presented (19/20, 95%) and that they would recommend the curriculum to other residents (20/20, 100%). Performance on the knowledge assessment improved 21% after the intervention (mean ± SD = pretest 57% ± 17%, posttest = 78% ± 12%; p < 0.001). Most residents (17/20, 85%) also reported behavioral changes including setting new financial goals (12/20, 60%), taking new action toward financial planning (11/20, 55%), and changing financial habits (6/20, 30%). There were no direct financial costs incurred in the implementation of this pilot. CONCLUSIONS: This is a successful pilot of a virtual personal finance curriculum with positive outcomes data. Addressing this problem at scale will require buy-in from educators around the country to deliver this information to residents that may not otherwise seek it out. Future study should assess curricular outcomes in other settings and the durability of acquired knowledge and behavioral changes over time.