Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study.

PURPOSE: In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD: For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS: The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS: TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

Revisiting the Core Entrustable Professional Activities for Entering Residency.

The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.

Developing an Entrustment Process: Insights from the AAMC CoreEPA Pilot.

One of the main goals of the CoreEPA pilot has been to determine the feasibility of developing a process to make summative entrustment decisions regarding entrustable professional activities (EPAs). Five years into the pilot, we report results of a research study we conducted to explore approaches to the entrustment process undertaken by our ten participating schools. We sought to identify the choices that participating schools made regarding the entrustment process and why these decisions were made. We are sharing these results, highlighting ongoing challenges that were identified with the intent of helping other medical schools that are moving toward EPA-based assessment. We conducted semi-structured interviews with representatives of all 10 medical schools in the CoreEPA pilot to understand their choices in designing the entrustment process. Additional information was obtained through follow-up communication to ensure completeness and accuracy of the findings. Several common themes are described. Our results indicate that, while approaches to the entrustment process vary considerably, all schools demonstrated consistent adherence to the guiding principles of the pilot. Several common barriers to the entrustment process emerged, and there was a consensus that more experience is needed with the process before consequential entrustment decisions can be made. The CoreEPA pilot schools continue to address challenges identified in implementing entrustment processes and making entrustment decisions for our students graduating in the Class of 2020.

Progress on a New Kind of Progress Test: Assessing Medical Students' Clinical Skills.

PROBLEM: Progress testing of medical knowledge has advantages over traditional medical school examination strategies. However, little is known about its use in assessing medical students' clinical skills or their integration of clinical skills with necessary science knowledge. The authors previously reported on the feasibility of the Progress Clinical Skills Examination (PCSE), piloted with a group of early learners. This subsequent pilot test studied the exam's validity to determine whether the PCSE is sensitive to the growth in students' clinical skills across the four years of medical school. APPROACH: In 2014, 38 medical student volunteers (years 1-4) in the traditional 2 + 2 curriculum at Michigan State University College of Human Medicine participated in the eight-station PCSE. Faculty and standardized patients assessed students' clinical skills, and faculty assessed students' responses to postencounter necessary science questions. Students performed pretest self-assessment across multiple measures and completed a posttest evaluation of their PCSE experience. OUTCOMES: Student performance generally increased by year in medical school for communication, history-taking, and physical examination skills. Necessary science knowledge increased substantially from first-year to second-year students, with less change thereafter. Students felt the PCSE was a fair test of their clinical skills and provided an opportunity to demonstrate their understanding of the related necessary science. NEXT STEPS: The authors have been piloting a wider pool of cases. In 2016, they adopted the PCSE as part of the summative assessment strategy for the medical school's new integrated four-year curriculum. Continued assessment of student performance trajectories is planned.

Finding a Path to Entrustment in Undergraduate Medical Education: A Progress Report From the AAMC Core Entrustable Professional Activities for Entering Residency Entrustment Concept Group.

PROBLEM: To better prepare graduating medical students to transition to the professional responsibilities of residency, 10 medical schools are participating in an Association of American Medical Colleges pilot to evaluate the feasibility of explicitly teaching and assessing 13 Core Entrustable Professional Activities for Entering Residency. The authors focused on operationalizing the concept of entrustment as part of this process. APPROACH: Starting in 2014, the Entrustment Concept Group, with representatives from each of the pilot schools, guided the development of the structures and processes necessary for formal entrustment decisions associated with students' increased responsibilities at the start of residency. OUTCOMES: Guiding principles developed by the group recommend that formal, summative entrustment decisions in undergraduate medical education be made by a trained group, be based on longitudinal performance assessments from multiple assessors, and incorporate day-to-day entrustment judgments by workplace supervisors. Key to entrustment decisions is evidence that students know their limits (discernment), can be relied on to follow through (conscientiousness), and are forthcoming despite potential personal costs (truthfulness), in addition to having the requisite knowledge and skills. The group constructed a developmental framework for discernment, conscientiousness, and truthfulness to pilot a model for transparent entrustment decision making. NEXT STEPS: The pilot schools are studying a number of questions regarding the pathways to and decisions about entrustment. This work seeks to inform meaningful culture change in undergraduate medical education through a shared understanding of the assessment of trust and a shared trust in that assessment.

What are the implications of implementation science for medical education?

BACKGROUND: Derived from multiple disciplines and established in industries outside of medicine, Implementation Science (IS) seeks to move evidence-based approaches into widespread use to enable improved outcomes to be realized as quickly as possible by as many as possible. METHODS: This review highlights selected IS theories and models, chosen based on the experience of the authors, that could be used to plan and deliver medical education activities to help learners better implement and sustain new knowledge and skills in their work settings. RESULTS: IS models, theories and approaches can help medical educators promote and determine their success in achieving desired learner outcomes. We discuss the importance of incorporating IS into the training of individuals, teams, and organizations, and employing IS across the medical education continuum. Challenges and specific strategies for the application of IS in educational settings are also discussed. CONCLUSIONS: Utilizing IS in medical education can help us better achieve changes in competence, performance, and patient outcomes. IS should be incorporated into curricula across disciplines and across the continuum of medical education to facilitate implementation of learning. Educators should start by selecting, applying, and evaluating the teaching and patient care impact one or two IS strategies in their work.

Documenting clinical performance problems among medical students: feedback for learner remediation and curriculum enhancement.

INTRODUCTION: We operationalized the taxonomy developed by Hauer and colleagues describing common clinical performance problems. Faculty raters pilot tested the resulting worksheet by observing recordings of problematic simulated clinical encounters involving third-year medical students. This approach provided a framework for structured feedback to guide learner improvement and curricular enhancement. METHODS: Eighty-two problematic clinical encounters from M3 students who failed their clinical competency examination were independently rated by paired clinical faculty members to identify common problems related to the medical interview, physical examination, and professionalism. RESULTS: Eleven out of 26 target performance problems were present in 25% or more encounters. Overall, 37% had unsatisfactory medical interviews, with 'inadequate history to rule out other diagnoses' most prevalent (60%). Seventy percent failed because of physical examination deficiencies, with missing elements (69%) and inadequate data gathering (69%) most common. One-third of the students did not introduce themselves to their patients. Among students failing based on standardized patient (SP) ratings, 93% also failed to demonstrate competency based on the faculty ratings. CONCLUSIONS: Our review form allowed clinical faculty to validate pass/fail decisions based on standardized patient ratings. Detailed information about performance problems contributes to learner feedback and curricular enhancement to guide remediation planning and faculty development.

Better data for teachers, better data for learners, better patient care: college-wide assessment at Michigan State University's College of Human Medicine.

When our school organized the curriculum around a core set of medical student competencies in 2004, it was clear that more numerous and more varied student assessments were needed. To oversee a systematic approach to the assessment of medical student competencies, the Office of College-wide Assessment was established, led by the Associate Dean of College-wide Assessment. The mission of the Office is to 'facilitate the development of a seamless assessment system that drives a nimble, competency-based curriculum across the spectrum of our educational enterprise.' The Associate Dean coordinates educational initiatives, developing partnerships to solve common problems, and enhancing synergy within the College. The Office also works to establish data collection and feedback loops to guide rational intervention and continuous curricular improvement. Aside from feedback, implementing a systems approach to assessment provides a means for identifying performance gaps, promotes continuity from undergraduate medical education to practice, and offers a rationale for some assessments to be located outside of courses and clerkships. Assessment system design, data analysis, and feedback require leadership, a cooperative faculty team with medical education expertise, and institutional support. The guiding principle is 'Better Data for Teachers, Better Data for Learners, Better Patient Care.' Better data empowers faculty to become change agents, learners to create evidence-based improvement plans and increases accountability to our most important stakeholders, our patients.

Development and evaluation of a Health Record Online Submission Tool (HOST).

INTRODUCTION: Health records (HRs) are crucial to quality patient care. The Michigan State University College of Human Medicine begins teaching health record (HR) writing during the second-year clinical skills courses. Prior to this project, we used a cumbersome paper system to allow graduate assistants to grade and give feedback on students' HRs. This study discusses the development and evaluates the effectiveness of the new Health Record Online Submission Tool (HOST). METHODS: We developed an electronic submission system with the goals of decreasing the logistical demands of the paper-based system; improving the effectiveness, consistency, and oversight of HR instruction and evaluation; expanding the number of students who could serve as written record graduate assistants (WRGAs); and to begin preparing students for the use of electronic health records (EHRs). We developed the initial web-based system in 2003 and upgraded it to its present form, HOST, in 2007. We evaluated the system using course evaluations, surveys of WRGAs and clinical students, and queries of course faculty and staff. RESULTS: Course evaluation by 1,106 students during years 2001 through 2008 revealed that the students' self-assessment of ability to write HRs improved briefly with the introduction of HOST but then returned to baseline. The initial change to electronic submission was well received, though with continued use its rating dropped. A survey of 65 (response rate 61.3%) clinical students indicated that HOST did not completely prepare them for EHRs. The WRGAs (n = 14; response rate 58%) found the system easy to use to give feedback to students. Faculty (n = 3) and staff (n = 2) found that it saved time and made the review of students' HRs and WRGAs grading simpler. Student perception of grading consistency did not improve. CONCLUSIONS: HOST is the first published online method of in-depth HR training for preclinical students using information gathered in clinical encounters. With it we were able to maintain effective instruction, streamline course management, and significantly decrease staff time. HOST did not improve student perception of grading consistency and did not prepare students for specific EHR use. Within the context of our class size expansion and our community-based educational program, HOST bridges geography and can support future improvements in HR instruction and faculty development. Medical educators at other institutions could use a similar system to accomplish these goals.

A repurposed tool: the Programme Evaluation SOAP Note.

CONTEXT: Doctors have used the subjective-objective-assessment-plan (SOAP) note format to organise data about patient problems and create plans to address each of them. We retooled this into the 'Programme Evaluation SOAP Note', which serves to broaden the clinician faculty member's perspective on programme evaluation to include the curriculum and the system, as well as students. METHODS: The SOAP Note was chosen as the method for data recording because of its familiarity to clinician-educators and its strengths as a representation of a clinical problem-solving process with elements analogous to educational programme evaluation. We pilot-tested the Programme Evaluation SOAP Note to organise faculty members' interpretations of integrated student performances during the Year 3 patient care skills objective structured clinical examination (OSCE). RESULTS: Eight community clerkship directors and lead clerkship faculty members participated as observers in the 2007 gateway examination and completed the Programme Evaluation SOAP Note. Problems with the curriculum and system far outnumbered problems identified with students. CONCLUSIONS: Using the Programme Evaluation SOAP Note, clerkship leaders developed expanded lists of 'differential diagnoses' that could explain possible learner performance inadequacies in terms of system, curriculum and learner problems. This has informed programme improvement efforts currently underway. We plan to continue using the Programme Evaluation SOAP Note for ongoing programme improvement.

The patient safety OSCE for PGY-1 residents: a centralized response to the challenge of culture change.

BACKGROUND: Accreditation and Institute of Medicine mandates require retooling of graduate medical education curriculum and assessment processes. This Objective Structured Clinical Exam (OSCE) focused on patient safety-specific skills important to stakeholders from multiple institutions. PURPOSES: A 10-station OSCE was designed to assess patient safety-related competencies in new Postgraduate Year 1 (PGY-1) residents. The OSCE emphasized performance of essential skills and teamwork, and it provided early formative feedback to trainees and leadership. METHODS: Group nominal process selected 10 final OSCE stations. Two stations were designed to assess team competencies and response to feedback. Two hundred thirty-five trainees enrolled in 64 programs participated during summer 2006. Skill-set aggregation was employed to improve the validity of individual feedback. RESULTS: Significant performance deficits were noted. Trainee and administrator evaluation of the experience was positive. CONCLUSIONS: Multi-institutional test development and centralized testing was well received and produced worrisome results. Early assessment can guide the development of task-specific personalized learning plans and systemwide curricular improvement. Further research is needed to determine whether such an effort directed at PGY-1 trainees can improve trainee performance and patient safety.