Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Correction: Clinical reasoning education in the clerkship years: A cross-disciplinary national needs assessment.

[This corrects the article DOI: 10.1371/journal.pone.0273250.].

Development of a lecture evaluation tool rooted in cognitive load theory: A modified Delphi study.

Background: Didactics play a key role in medical education. There is no standardized didactic evaluation tool to assess quality and provide feedback to instructors. Cognitive load theory provides a framework for lecture evaluations. We sought to develop an evaluation tool, rooted in cognitive load theory, to assess quality of didactic lectures. Methods: We used a modified Delphi method to achieve expert consensus for items in a lecture evaluation tool. Nine emergency medicine educators with expertise in cognitive load participated in three modified Delphi rounds. In the first two rounds, experts rated the importance of including each item in the evaluation rubric on a 1 to 9 Likert scale with 1 labeled as "not at all important" and 9 labeled as "extremely important." In the third round, experts were asked to make a binary choice of whether the item should be included in the final evaluation tool. In each round, the experts were invited to provide written comments, edits, and suggested additional items. Modifications were made between rounds based on item scores and expert feedback. We calculated descriptive statistics for item scores. Results: We completed three Delphi rounds, each with 100% response rate. After Round 1, we removed one item, made major changes to two items, made minor wording changes to nine items, and modified the scale of one item. Following Round 2, we eliminated three items, made major wording changes to one item, and made minor wording changes to one item. After the third round, we made minor wording changes to two items. We also reordered and categorized items for ease of use. The final evaluation tool consisted of nine items. Conclusions: We developed a lecture assessment tool rooted in cognitive load theory specific to medical education. This tool can be applied to assess quality of instruction and provide important feedback to speakers.

Clinical reasoning education in the clerkship years: A cross-disciplinary national needs assessment.

BACKGROUND: Improving clinical reasoning education has been identified as an important strategy to reduce diagnostic error-an important cause of adverse patient outcomes. Clinical reasoning is fundamental to each specialty, yet the extent to which explicit instruction in clinical reasoning occurs across specialties in the clerkship years remains unclear. METHOD: The Alliance for Clinical Education (ACE) Clinical Reasoning Workgroup and the Directors of Clinical Skills Courses (DOCS) Clinical Reasoning Workgroup collaborated to develop a clinical reasoning needs assessment survey. The survey questionnaire covered seven common clinical reasoning topics including illness scripts, semantic qualifiers, cognitive biases and dual process theory. Questionnaires were delivered electronically through ACE member organizations, which are primarily composed of clerkship leaders across multiple specialties. Data was collected between March of 2019 and May of 2020. RESULTS: Questionnaires were completed by 305 respondents across the six organizations. For each of the seven clinical reasoning topics, the majority of clerkship leaders (range 77.4% to 96.8%) rated them as either moderately important or extremely important to cover during the clerkship curriculum. Despite this perceived importance, these topics were not consistently covered in respondents' clerkships (range 29.4% to 76.4%) and sometimes not covered anywhere in the clinical curriculum (range 5.1% to 22.9%). CONCLUSIONS: Clerkship educators across a range of clinical specialties view clinical reasoning instruction as important, however little curricular time is allocated to formally teach the various strategies. Faculty development and restructuring of curricular time may help address this potential gap.

Effect of Perceived Level of Interaction on Faculty Evaluations of 3rd Year Medical Students.

INTRODUCTION: Several factors are known to affect the way clinical performance evaluations (CPEs) of medical students are completed by supervising physicians. We sought to explore the effect of faculty perceived "level of interaction" (LOI) on these evaluations. METHODS: Our third-year CPE requires evaluators to identify perceived LOI with each student as low, moderate, or high. We examined CPEs completed during the academic year 2018-2019 for differences in (1) clinical and professionalism ratings, (2) quality of narrative comments, (3) quantity of narrative comments, and (4) percentage of evaluation questions left unrated. RESULTS: A total of 3682 CPEs were included in the analysis. ANOVA revealed statistically significant differences between LOI and clinical ratings (p ≤ .001), with mean ratings from faculty with a high LOI significantly higher than from faculty with a moderate or low LOI (p ≤ .001). Chi-squared analysis demonstrated differences based on faculty LOI and whether questions were left unrated (p ≤ .001), quantity of narrative comments (p ≤ .001), and specificity of narrative comments (p ≤ .001). CONCLUSIONS: Faculty who perceive higher LOI were more likely to assign that student higher ratings, complete more of the clinical evaluation and were more likely to provide narrative feedback with more specific, higher-quality comments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01307-w.

Optimizing Lectures From a Cognitive Load Perspective.

Lectures are a common instructional method in medical education. Understanding the cognitive processes and theories involved in learning is essential for lecturers to be effective. Cognitive load theory is one theory that is becoming increasingly recognized in medical education and addresses the appropriate use of one's working memory. Memory is essential to knowledge acquisition. Two types of memory can be considered, working memory (processing of information) and long-term memory (storage of information). Working memory has a limited capacity. Cognitive load refers to the amount of information processing activity imposed on working memory and can be divided into three domains: intrinsic, extraneous, and germane. By attending to cognitive load, educators can promote learning. This paper highlights various ways of improving cognitive load for learners during lecture-based instruction by minimizing extraneous load, optimizing intrinsic load, and promoting germane load.

Practical Application of Assessment Principles in the Development of an Innovative Clinical Performance Evaluation in the Entrustable Professional Activity Era.

Educators have been challenged to create assessments that are competency-based and grounded in accepted standards such as the entrustable professional activities (EPAs). The clinical performance evaluation (CPE) is a commonly utilized assessment modality, which allows multiple evaluators to provide feedback on a learner's performance in the clinical workplace. In this paper, we describe the relevant principles that served as a guide as we developed a new CPE for medical students that fully incorporate EPAs. This may help ease the transition for other institutions looking to introduce a new student CPE.

The GI Simulated Clinic: A Clinical Reasoning Exercise Supporting Medical Students' Basic and Clinical Science Integration.

Introduction: Cognitive integration is required to perform clinical decision-making tasks, even in the preclinical curriculum of medical school. Simulation supports students' cognitive integration by providing practical application of basic science knowledge in a relevant clinical context. To address the need for integrative activities in our curriculum, we implemented a simulated clinic exercise with cases representing gastrointestinal diseases for first-year medical students. Methods: Basic science and clinical skills course directors collaborated to design this simulated clinic event, during which student small groups rotated through a series of standardized patient encounters. During each encounter, one student performed the history and physical exam, following which the small group collaboratively developed a prioritized differential diagnosis. Afterwards, the gastroenterology course director debriefed students to highlight key learning points. We collected learner evaluation data following the event. Results: Two hundred eighty first-year medical students participated in the simulated clinic in 2018 and 2019. Students rated these events as effective for learning about clinical features of the diseases presented and for reinforcing skills learned in the clinical skills course. Students agreed that the small-group format, pace, and duration were appropriate and that the problem-solving aspect was intellectually stimulating. The most effective aspects were opportunities to solidify illness scripts, apply knowledge to solve a problem, and encounter diseases in a realistic clinical context. Discussion: This simulated clinic model effectively supported preclinical students' basic and clinical science integration to complete diagnostic reasoning tasks for gastrointestinal gastrointestinal conditions and was evaluated favorably by learners.

Faculty physicians and new physicians disagree about which procedures are essential to learn in medical school.

BACKGROUND: Clinical procedures taught in the undergraduate medical curriculum are important for subsequent residency training and clinical practice. Published reports suggest that medical schools may not be adequately teaching procedures. This study identifies procedures considered essential by residents completing internship and by medical school faculty, and determines agreement on their importance for medical student education. METHODS: Two hundred and thirty-five physicians (184 new physicians who recently completed internship and 51 medical school teaching faculty) categorized 31 clinical procedures based on the importance for internship. New physicians who had completed internship reported the level of training received in medical school for each procedure. RESULTS: Survey responses were 76% (faculty) and 70% (new physicians who had completed internship). The faculty majority identified 14 procedures as 'Must Know.' New physicians disagreed on 8 of these and categorized an additional 5 as essential. There was 32% concordance for the 19 procedures identified by either group. New physicians reported 'Limited Hands-On Training' for all 19 procedures but 'Comprehensive Hands-On Training' for only two. CONCLUSIONS: New physicians who have completed internship and medical school faculty do not agree on procedures essential for internship. A core educational list of 19 procedures was identified using the responses from these two groups.

A case of intrauterine molar pregnancy with coexistent ectopic pregnancy.

We describe a woman who presented to the Emergency Department (ED) with vaginal bleeding and abdominal pain. She was initially diagnosed by the emergency physician with a molar pregnancy by transvaginal ultrasound, which was confirmed and treated by the consulting obstetrical service with a dilatation and curettage the following day. The patient was discharged home later that same day and subsequently returned to the ED after several hours complaining of an acute worsening of her abdominal pain with associated fatigue and lightheadedness. Transabdominal ultrasound performed by the emergency physician demonstrated intra-abdominal free fluid, and the obstetrical service was immediately contacted. Subsequent operative management identified a separate ruptured ectopic pregnancy in the fallopian tube that was confirmed by pathologic analysis after laparoscopic removal.

A Narrative Review of the Evidence Supporting Factors Used by Residency Program Directors to Select Applicants for Interviews.

BACKGROUND: Residency applicants feel increasing pressure to maximize their chances of successfully matching into the program of their choice, and are applying to more programs than ever before. OBJECTIVE: In this narrative review, we examined the most common and highly rated factors used to select applicants for interviews. We also examined the literature surrounding those factors to illuminate the advantages and disadvantages of using them as differentiating elements in interviewee selection. METHODS: Using the 2018 NRMP Program Director Survey as a framework, we examined the last 10 years of literature to ascertain how residency directors are using these common factors to grant residency interviews, and whether these factors are predictive of success in residency. RESULTS: Residency program directors identified 12 factors that contribute substantially to the decision to invite applicants for interviews. Although United States Medical Licensing Examination (USMLE) Step 1 is often used as a comparative factor, most studies do not demonstrate its predictive value for resident performance, except in the case of test failure. We also found that structured letters of recommendation from within a specialty carry increased benefit when compared with generic letters. Failing USMLE Step 1 or 2 and unprofessional behavior predicted lower performance in residency. CONCLUSIONS: We found that the evidence basis for the factors most commonly used by residency directors is decidedly mixed in terms of predicting success in residency and beyond. Given these limitations, program directors should be skeptical of making summative decisions based on any one factor.

Clerkship directors in emergency medicine: statement of purpose.

The Academy of Clerkship Directors in Emergency Medicine (CDEM) provides a forum for the collaborative exchange of ideas among emergency medicine (EM) medical student educators, a platform for the advancement of education, research, and faculty development, and establishes for the first time a national voice for undergraduate medical education within our specialty. CDEM plans to take a leading role in providing medical student educators with additional educational resources and opportunities for faculty development and networking. CDEM will work to foster the professional growth and development of undergraduate medical educators within our specialty. The advancement of undergraduate education within our specialty and beyond will come primarily from the support, hard work, and dedication of the educators. To accomplish our goals, at the departmental, medical school, and national level, we must come together to further promote our specialty across the spectrum of undergraduate medical education. The first step has already been taken with the formation of the Academy of CDEM.

A skin abscess model for teaching incision and drainage procedures.

BACKGROUND: Skin and soft tissue infections are increasingly prevalent clinical problems, and it is important for health care practitioners to be well trained in how to treat skin abscesses. A realistic model of abscess incision and drainage will allow trainees to learn and practice this basic physician procedure. METHODS: We developed a realistic model of skin abscess formation to demonstrate the technique of incision and drainage for educational purposes. The creation of this model is described in detail in this report. RESULTS: This model has been successfully used to develop and disseminate a multimedia video production for teaching this medical procedure. Clinical faculty and resident physicians find this model to be a realistic method for demonstrating abscess incision and drainage. CONCLUSION: This manuscript provides a detailed description of our model of abscess incision and drainage for medical education. Clinical educators can incorporate this model into skills labs or demonstrations for teaching this basic procedure.

Diagnosis of a preputial cavity abscess with bedside ultrasound in the emergency department.

Bedside ultrasound has become an important diagnostic tool for emergency physicians. Clinical investigators have demonstrated that evaluating soft tissue infections with ultrasound is useful for the detection of subcutaneous abscesses. Bedside ultrasound of a preputial cavity abscess in the Emergency Department has not been previously described in the English medical literature. A preputial cavity abscess, a rare complication of balanoposthitis, is a collection of pus between the foreskin and the distal penis. This case report describes the use of ultrasound to diagnose a penile abscess and reviews the related literature.

Penetrating trauma to the extremity.

Penetrating trauma to the extremities is a complex disease that foremost requires the evaluation for vascular injury. This monograph will address an algorithm to assess for associated vascular injury that includes current evaluation techniques. Approaches to wound management and use of antibiotics in the ED are also addressed.

Scurvy: Dietary Discretion in a Developed Country.

Although the causes have changed, scurvy (vitamin C deficiency) is still diagnosed in developed countries. We report a case of an 18-year-old female who presented to our emergency department with thrombocytopenia, sinus tachycardia, hypotension, fatigue, gingival hyperplasia, knee effusion, petechiae and ecchymosis in lower extremities. The differential diagnosis included hematologic abnormalities, infectious etiologies, vasculitis and vitamin deficiency. A brief dietary history was performed revealing poor fruit and vegetable intake, thus increasing our suspicion for vitamin C deficiency. This experience illustrates the importance of a dietary history and reminds us to keep scurvy in the differential diagnosis.

A Multidisciplinary Self-Directed Learning Module Improves Knowledge of a Quality Improvement Instrument: The HEART Pathway.

We created and tested an educational intervention to support implementation of an institution wide QI project (the HEART Pathway) designed to improve care for patients with acute chest pain. Although online learning modules have been shown effective in imparting knowledge regarding QI projects, it is unknown whether these modules are effective across specialties and healthcare professions. Participants, including nurses, advanced practice clinicians, house staff and attending physicians (N = 486), were enrolled into an online, self-directed learning course exploring the key concepts of the HEART Pathway. The module was completed by 97% of enrollees (469/486) and 90% passed on the first attempt (422/469). Out of 469 learners, 323 completed the pretest, learning module and posttest in the correct order. Mean test scores across learners improved significantly from 74% to 89% from the pretest to the posttest. Following the intervention, the HEART Pathway was used for 88% of patients presenting to our institution with acute chest pain. Our data demonstrate that this online, self-directed learning module can improve knowledge of the HEART Pathway across specialties-paving the way for more efficient and informed care for acute chest pain patients.

Erratum: This Article Corrects: "Trends in NRMP Data from 2007-2014 for U.S. Seniors Matching into Emergency Medicine".

[This corrects the article on p. 105 in vol. 18, PMID: 28116018.].