Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Development of a lecture evaluation tool rooted in cognitive load theory: A modified Delphi study.

Background: Didactics play a key role in medical education. There is no standardized didactic evaluation tool to assess quality and provide feedback to instructors. Cognitive load theory provides a framework for lecture evaluations. We sought to develop an evaluation tool, rooted in cognitive load theory, to assess quality of didactic lectures. Methods: We used a modified Delphi method to achieve expert consensus for items in a lecture evaluation tool. Nine emergency medicine educators with expertise in cognitive load participated in three modified Delphi rounds. In the first two rounds, experts rated the importance of including each item in the evaluation rubric on a 1 to 9 Likert scale with 1 labeled as "not at all important" and 9 labeled as "extremely important." In the third round, experts were asked to make a binary choice of whether the item should be included in the final evaluation tool. In each round, the experts were invited to provide written comments, edits, and suggested additional items. Modifications were made between rounds based on item scores and expert feedback. We calculated descriptive statistics for item scores. Results: We completed three Delphi rounds, each with 100% response rate. After Round 1, we removed one item, made major changes to two items, made minor wording changes to nine items, and modified the scale of one item. Following Round 2, we eliminated three items, made major wording changes to one item, and made minor wording changes to one item. After the third round, we made minor wording changes to two items. We also reordered and categorized items for ease of use. The final evaluation tool consisted of nine items. Conclusions: We developed a lecture assessment tool rooted in cognitive load theory specific to medical education. This tool can be applied to assess quality of instruction and provide important feedback to speakers.

Clinical reasoning education in the clerkship years: A cross-disciplinary national needs assessment.

BACKGROUND: Improving clinical reasoning education has been identified as an important strategy to reduce diagnostic error-an important cause of adverse patient outcomes. Clinical reasoning is fundamental to each specialty, yet the extent to which explicit instruction in clinical reasoning occurs across specialties in the clerkship years remains unclear. METHOD: The Alliance for Clinical Education (ACE) Clinical Reasoning Workgroup and the Directors of Clinical Skills Courses (DOCS) Clinical Reasoning Workgroup collaborated to develop a clinical reasoning needs assessment survey. The survey questionnaire covered seven common clinical reasoning topics including illness scripts, semantic qualifiers, cognitive biases and dual process theory. Questionnaires were delivered electronically through ACE member organizations, which are primarily composed of clerkship leaders across multiple specialties. Data was collected between March of 2019 and May of 2020. RESULTS: Questionnaires were completed by 305 respondents across the six organizations. For each of the seven clinical reasoning topics, the majority of clerkship leaders (range 77.4% to 96.8%) rated them as either moderately important or extremely important to cover during the clerkship curriculum. Despite this perceived importance, these topics were not consistently covered in respondents' clerkships (range 29.4% to 76.4%) and sometimes not covered anywhere in the clinical curriculum (range 5.1% to 22.9%). CONCLUSIONS: Clerkship educators across a range of clinical specialties view clinical reasoning instruction as important, however little curricular time is allocated to formally teach the various strategies. Faculty development and restructuring of curricular time may help address this potential gap.

Effect of Perceived Level of Interaction on Faculty Evaluations of 3rd Year Medical Students.

INTRODUCTION: Several factors are known to affect the way clinical performance evaluations (CPEs) of medical students are completed by supervising physicians. We sought to explore the effect of faculty perceived "level of interaction" (LOI) on these evaluations. METHODS: Our third-year CPE requires evaluators to identify perceived LOI with each student as low, moderate, or high. We examined CPEs completed during the academic year 2018-2019 for differences in (1) clinical and professionalism ratings, (2) quality of narrative comments, (3) quantity of narrative comments, and (4) percentage of evaluation questions left unrated. RESULTS: A total of 3682 CPEs were included in the analysis. ANOVA revealed statistically significant differences between LOI and clinical ratings (p ≤ .001), with mean ratings from faculty with a high LOI significantly higher than from faculty with a moderate or low LOI (p ≤ .001). Chi-squared analysis demonstrated differences based on faculty LOI and whether questions were left unrated (p ≤ .001), quantity of narrative comments (p ≤ .001), and specificity of narrative comments (p ≤ .001). CONCLUSIONS: Faculty who perceive higher LOI were more likely to assign that student higher ratings, complete more of the clinical evaluation and were more likely to provide narrative feedback with more specific, higher-quality comments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01307-w.

The GI Simulated Clinic: A Clinical Reasoning Exercise Supporting Medical Students' Basic and Clinical Science Integration.

Introduction: Cognitive integration is required to perform clinical decision-making tasks, even in the preclinical curriculum of medical school. Simulation supports students' cognitive integration by providing practical application of basic science knowledge in a relevant clinical context. To address the need for integrative activities in our curriculum, we implemented a simulated clinic exercise with cases representing gastrointestinal diseases for first-year medical students. Methods: Basic science and clinical skills course directors collaborated to design this simulated clinic event, during which student small groups rotated through a series of standardized patient encounters. During each encounter, one student performed the history and physical exam, following which the small group collaboratively developed a prioritized differential diagnosis. Afterwards, the gastroenterology course director debriefed students to highlight key learning points. We collected learner evaluation data following the event. Results: Two hundred eighty first-year medical students participated in the simulated clinic in 2018 and 2019. Students rated these events as effective for learning about clinical features of the diseases presented and for reinforcing skills learned in the clinical skills course. Students agreed that the small-group format, pace, and duration were appropriate and that the problem-solving aspect was intellectually stimulating. The most effective aspects were opportunities to solidify illness scripts, apply knowledge to solve a problem, and encounter diseases in a realistic clinical context. Discussion: This simulated clinic model effectively supported preclinical students' basic and clinical science integration to complete diagnostic reasoning tasks for gastrointestinal gastrointestinal conditions and was evaluated favorably by learners.

Optimizing Lectures From a Cognitive Load Perspective.

Lectures are a common instructional method in medical education. Understanding the cognitive processes and theories involved in learning is essential for lecturers to be effective. Cognitive load theory is one theory that is becoming increasingly recognized in medical education and addresses the appropriate use of one's working memory. Memory is essential to knowledge acquisition. Two types of memory can be considered, working memory (processing of information) and long-term memory (storage of information). Working memory has a limited capacity. Cognitive load refers to the amount of information processing activity imposed on working memory and can be divided into three domains: intrinsic, extraneous, and germane. By attending to cognitive load, educators can promote learning. This paper highlights various ways of improving cognitive load for learners during lecture-based instruction by minimizing extraneous load, optimizing intrinsic load, and promoting germane load.

Practical Application of Assessment Principles in the Development of an Innovative Clinical Performance Evaluation in the Entrustable Professional Activity Era.

Educators have been challenged to create assessments that are competency-based and grounded in accepted standards such as the entrustable professional activities (EPAs). The clinical performance evaluation (CPE) is a commonly utilized assessment modality, which allows multiple evaluators to provide feedback on a learner's performance in the clinical workplace. In this paper, we describe the relevant principles that served as a guide as we developed a new CPE for medical students that fully incorporate EPAs. This may help ease the transition for other institutions looking to introduce a new student CPE.

A Narrative Review of the Evidence Supporting Factors Used by Residency Program Directors to Select Applicants for Interviews.

BACKGROUND: Residency applicants feel increasing pressure to maximize their chances of successfully matching into the program of their choice, and are applying to more programs than ever before. OBJECTIVE: In this narrative review, we examined the most common and highly rated factors used to select applicants for interviews. We also examined the literature surrounding those factors to illuminate the advantages and disadvantages of using them as differentiating elements in interviewee selection. METHODS: Using the 2018 NRMP Program Director Survey as a framework, we examined the last 10 years of literature to ascertain how residency directors are using these common factors to grant residency interviews, and whether these factors are predictive of success in residency. RESULTS: Residency program directors identified 12 factors that contribute substantially to the decision to invite applicants for interviews. Although United States Medical Licensing Examination (USMLE) Step 1 is often used as a comparative factor, most studies do not demonstrate its predictive value for resident performance, except in the case of test failure. We also found that structured letters of recommendation from within a specialty carry increased benefit when compared with generic letters. Failing USMLE Step 1 or 2 and unprofessional behavior predicted lower performance in residency. CONCLUSIONS: We found that the evidence basis for the factors most commonly used by residency directors is decidedly mixed in terms of predicting success in residency and beyond. Given these limitations, program directors should be skeptical of making summative decisions based on any one factor.

Scurvy: Dietary Discretion in a Developed Country.

Although the causes have changed, scurvy (vitamin C deficiency) is still diagnosed in developed countries. We report a case of an 18-year-old female who presented to our emergency department with thrombocytopenia, sinus tachycardia, hypotension, fatigue, gingival hyperplasia, knee effusion, petechiae and ecchymosis in lower extremities. The differential diagnosis included hematologic abnormalities, infectious etiologies, vasculitis and vitamin deficiency. A brief dietary history was performed revealing poor fruit and vegetable intake, thus increasing our suspicion for vitamin C deficiency. This experience illustrates the importance of a dietary history and reminds us to keep scurvy in the differential diagnosis.

A Multidisciplinary Self-Directed Learning Module Improves Knowledge of a Quality Improvement Instrument: The HEART Pathway.

We created and tested an educational intervention to support implementation of an institution wide QI project (the HEART Pathway) designed to improve care for patients with acute chest pain. Although online learning modules have been shown effective in imparting knowledge regarding QI projects, it is unknown whether these modules are effective across specialties and healthcare professions. Participants, including nurses, advanced practice clinicians, house staff and attending physicians (N = 486), were enrolled into an online, self-directed learning course exploring the key concepts of the HEART Pathway. The module was completed by 97% of enrollees (469/486) and 90% passed on the first attempt (422/469). Out of 469 learners, 323 completed the pretest, learning module and posttest in the correct order. Mean test scores across learners improved significantly from 74% to 89% from the pretest to the posttest. Following the intervention, the HEART Pathway was used for 88% of patients presenting to our institution with acute chest pain. Our data demonstrate that this online, self-directed learning module can improve knowledge of the HEART Pathway across specialties-paving the way for more efficient and informed care for acute chest pain patients.

Erratum: This Article Corrects: "Trends in NRMP Data from 2007-2014 for U.S. Seniors Matching into Emergency Medicine".

[This corrects the article on p. 105 in vol. 18, PMID: 28116018.].

Trends in NRMP Data from 2007-2014 for U.S. Seniors Matching into Emergency Medicine.

INTRODUCTION: Since 1978, the National Residency Matching Program (NRMP) has published data demonstrating characteristics of applicants who have matched into their preferred specialty in the NRMP main residency match. These data have been published approximately every two years. There is limited information about trends within these published data for students matching into emergency medicine (EM). Our objective was to investigate and describe trends in NRMP data to include the following: the ratio of applicants to available EM positions; United State Medical Licensing Examination (USMLE) Step 1 and Step 2 scores (compared to the national means); number of programs ranked; and Alpha Omega Alpha Honor Medical Society (AOA) membership among U.S. seniors matching into EM. METHODS: This was a retrospective observational review of NRMP data published between 2007 and 2016. We analyzed the data using analysis of variance (ANOVA) or Kruskal-Wallis testing, and Fischer's exact or chi-squared testing, as appropriate to determine statistical significance. RESULTS: The ratio of applicants to available EM positions remained essentially stable from 2007 to 2014 but did increase slightly in 2016. We observed a net upward trend in overall Step 1 and Step 2 scores for EM applicants. However, this did not outpace the national trend increase in Step 1 and 2 scores overall. There was an increase in the mean number of programs ranked by EM applicants over the years studied from 7.8 (SD4.2) to 9.2 (SD5.0, p<0.001), driven predominantly by the cohort of U.S. students successful in the match. Among time intervals, there was a difference in the number of EM applicants with AOA membership (p=0.043) due to a drop in the number of AOA students in 2011. No sustained statistical trend in AOA membership was identified over the seven-year period studied. CONCLUSION: NRMP data demonstrate trends among EM applicants that are similar to national trends in other specialties for USMLE board scores, and a modest increase in number of programs ranked. AOA membership was largely stable. EM does not appear to have become more competitive relative to other specialties or previous years in these categories.

Large Posterior Communicating Artery Aneurysm: Initial Presentation with Reproducible Facial Pain Without Cranial Nerve Deficit.

Unruptured posterior communicating artery (PCOM) aneurysms can be difficult to diagnose and, when large (≥ 7mm), represent a substantial risk to the patient. While most unruptured PCOM aneurysms are asymptomatic, when symptoms do occur, clinical manifestations typically include severe headache (HA), visual acuity loss, and cranial nerve deficit. This case report describes an atypical initial presentation of a large unruptured PCOM aneurysm with symptoms mimicking trigeminal neuralgia, without other associated cranial nerve palsies or neurologic deficits. The patient returned to the emergency department four days later with a HA, trigeminal neuralgia, and a new cranial nerve III palsy. After appropriate imaging, she was found to have a large PCOM aneurysm, which was treated with surgical clipping with significant improvement in patient's symptoms.

Combined Versus Detailed Evaluation Components in Medical Student Global Rating Indexes.

INTRODUCTION: To determine if there is any correlation between any of the 10 individual components of a global rating index on an emergency medicine (EM) student clerkship evaluation form. If there is correlation, to determine if a weighted average of highly correlated components loses predictive value for the final clerkship grade. METHODS: This study reviewed medical student evaluations collected over two years of a required fourth-year rotation in EM. Evaluation cards, comprised of a detailed 10-part evaluation, were completed after each shift. We used a correlation matrix between evaluation category average scores, using Spearman's rho, to determine if there was any correlation of the grades between any of the 10 items on the evaluation form. RESULTS: A total of 233 students completed the rotation over the two-year period of the study. There were strong correlations (>0.80) between assessment components of medical knowledge, history taking, physical exam, and differential diagnosis. There were also strong correlations between assessment components of team rapport, patient rapport, and motivation. When these highly correlated were combined to produce a four-component model, linear regression demonstrated similar predictive power in terms of final clerkship grade (R(2)=0.71, CI95=0.65-0.77 and R(2)=0.69, CI95=0.63-0.76 for the full and reduced models respectively). CONCLUSION: This study revealed that several components of the evaluation card had a high degree of correlation. Combining the correlated items, a reduced model containing four items (clinical skills, interpersonal skills, procedural skills, and documentation) was as predictive of the student's clinical grade as the full 10-item evaluation. Clerkship directors should be aware of the performance of their individual global rating scales when assessing medical student performance, especially if attempting to measure greater than four components.

Medical student milestones in emergency medicine.

OBJECTIVES: Medical education is a continuum from medical school through residency to unsupervised clinical practice. There has been a movement toward competency-based medical education prompted by the Accreditation Council for Graduate Medical Education (ACGME) using milestones to assess competence. While implementation of milestones for residents sets specific standards for transition to internship, there exists a need for the development of competency-based instruments to assess medical students as they progress toward internship. The objective of this study was to develop competency-based milestones for fourth-year medical students completing their emergency medicine (EM) clerkships (regardless of whether the students were planning on entering EM) using a rigorous method to attain validity evidence. METHODS: A literature review was performed to develop a list of potential milestones. An expert panel, which included a medical student and 23 faculty members (four program directors, 16 clerkship directors, and five assistant deans) from 19 different institutions, came to consensus on these milestones through two rounds of a modified Delphi protocol. The Delphi technique builds content validity and is an accepted method to develop consensus by eliciting expert opinions through multiple rounds of questionnaires. RESULTS: Of the initial 39 milestones, 12 were removed at the end of round 1 due to low agreement on importance of the milestone or because of redundancy with other milestones. An additional 12 milestones were revised to improve clarity or eliminate redundancy, and one was added based on expert panelists' suggestions. Of the 28 milestones moving to round 2, consensus with a high level of agreement was achieved for 24. These were mapped to the ACGME EM residency milestone competency domains, as well as the Association of American Medical Colleges (AAMC) core entrustable professional activities for entering residency to improve content validity. CONCLUSIONS: This study found consensus support by experts for a list of 24 milestones relevant to the assessment of fourth-year medical student performance by the completion of their EM clerkships. The findings are useful for development of a valid method for assessing medical student performance as students approach residency.

Emergency medicine clerkship directors: current workforce.

INTRODUCTION: The emergency medicine clerkship director serves an important role in the education of medical students. The authors sought to update the demographic and academic profile of the emergency medicine clerkship director. METHODS: We developed and implemented a comprehensive questionnaire, and used it to survey all emergency medicine clerkship directors at United States allopathic medical schools accredited by the Liaison Committee on Medical Education. We analyzed and interpreted data using descriptive statistics. RESULTS: One hundred seven of 133 (80.4%) emergency medicine clerkship directors completed the survey. Clerkship Director's mean age was 39.7 years (SD-7.2), they were more commonly male 68.2%, of Caucasian racial backgrounds and at the instructor or assistant professor (71.3%) level. The mean number of years of experience as clerkship director was 5.5 (SD-4.5). The mean amount of protected time for clerkship administration reported by respondents was 7.3 hours weekly (SD-5.1), with the majority (53.8%) reporting 6 or more hours of protected time per week. However, 32.7% of emergency medicine clerkship directors reported not having any protected time for clerkship administration. Most clerkship directors (91.6%) held additional teaching responsibilities beyond their clerkship and many were involved in educational research (49.5%). The majority (79.8%), reported being somewhat or very satisfied with their job as clerkship director. CONCLUSION: Most clerkship directors were junior faculty at the instructor or assistant professor rank and were involved with a variety of educational endeavors beyond the clerkship.

State of undergraduate education in emergency medicine: a national survey of clerkship directors.

BACKGROUND: The discipline of emergency medicine (EM) has rapidly changed over the past 10 years, resulting in greater involvement of the specialty in undergraduate medical education. OBJECTIVES: The authors sought to present a review of how, when, and where EM is currently taught in U.S. medical schools and to include general program characteristics, such as number of required clerkships, clinical expectations and experiences, use of the national curriculum guide, didactic content, and methods of assessment and grading. METHODS: The authors surveyed representatives of the 128 U.S. allopathic medical schools on medical education in 2010. Contacts were drawn from established databases, direct inquiries, and medical school websites. Up to five attempts were made to contact representatives through e-mail and telephone. Descriptive statistics were used to summarize the data. RESULTS: The survey response rate was 83.6%. Fifty-two percent of medical schools now require students to complete EM clerkships. Required EM clerkships usually last 4 weeks and take place during the fourth year of medical school. They require students to complete a mean (±SD) of 14.3 (±2.8) shifts, which average 8.9 (±1.4) hours in length. Programs include a mean (±SD) of 18 (±10.4) hours of didactics. Approximately 60% of respondents report that both residents and attending physicians precept students. Assessments of students primarily include written clinical performance assessments and end-of-rotation written tests. These assessments contribute 66.8 and 24.5%, respectively, to the clerkship grade. CONCLUSIONS: Currently more than half of all U.S. medical schools require EM clerkships in their undergraduate medical curricula. This article reports an overview of EM programs at the undergraduate level.

Increasing Faculty Attendance at Emergency Medicine Resident Conferences: Does CME Credit Make a Difference?

BACKGROUND: Faculty involvement in resident teaching events is beneficial to resident education, yet evidence about the factors that promote faculty attendance at resident didactic conferences is limited. OBJECTIVE: To determine whether offering continuing medical education (CME) credits would result in an increase in faculty attendance at weekly emergency medicine conferences and whether faculty would report the availability of CME credit as a motivating factor. METHODS: Our prospective, multi-site, observational study of 5 emergency medicine residency programs collected information on the number of faculty members present at CME and non-CME lectures for 9 months and collected information from faculty on factors influencing decisions to attend resident educational events and from residents on factors influencing their learning experience. RESULTS: Lectures offering CME credit on average were attended by 5 additional faculty members per hour, compared with conferences that did not offer CME credit (95% confidence interval [CI], 3.9-6.1; P < .001). Faculty reported their desire to "participate in resident education" was the most influential factor prompting them to attend lectures, followed by "explore current trends in emergency medicine" and the lecture's "specific topic." Faculty also reported that "clinical/administrative duties" and "family responsibilities" negatively affected their ability to attend. Residents reported that the most important positive factor influencing their conference experience was "lectures given by faculty." CONCLUSIONS: Although faculty reported that CME credit was not an important factor in their decision to attend resident conferences, offering CME credit resulted in significant increases in faculty attendance. Residents reported that "lectures given by faculty" and "faculty attendance" positively affected their learning experience.

Developing and assessing initiatives designed to improve clinical teaching performance.

To improve the teaching performance of emergency physicians, it is necessary to understand the attributes of expert teachers and the optimal methods to deliver faculty development. A working group of medical educators was formed to review the literature, summarize what is known on the topic, and provide recommendations for future research. This occurred as a track of the 2012 Academic Emergency Medicine (AEM) consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success." The group concluded that the current state of research on these topics is limited. Improvement in understanding will come through research focusing on Kirkpatrick's higher levels of evaluation (behavior and results).