The Minnesota Method: A Learner-Driven, Entrustable Professional Activity-Based Comprehensive Program of Assessment for Medical Students.

PROBLEM: Assessment has been the Achilles heel of competency-based medical education. It requires a program of assessment in which outcomes are clearly defined, students know where they are in the development of the competencies, and what the next steps are to attaining them. Achieving this goal in a feasible manner has been elusive with traditional assessment methods alone. The Education in Pediatrics Across the Continuum (EPAC) program at the University of Minnesota developed a robust program of assessment that has utility and recognizes when students are ready for the undergraduate to graduate medical education transition. APPROACH: The authors developed a learner-driven program of assessment in the foundational clinical training of medical students in the EPAC program based on the Core Entrustable Professional Activities for Entering Residency (Core EPAs). Frequent workplace-based assessments, coupled with summative assessments, informed a quarterly clinical competency committee and individualized learning plans. The data were displayed on real time dashboards for the students to review. OUTCOMES: Over 4 cohorts from 2015 to 2019, students (n = 13) averaged approximately 200 discrete Core EPA workplace-based assessments during their foundational clinical training year. Assessments were completed by an average of 9 different preceptors each month across 8 different specialties. The data were displayed in a way students and faculty could monitor development and inform a clinical competency committee's ability to determine readiness to transition to advanced clinical rotations and residency. NEXT STEPS: The next steps include continuing to scale the program of assessment to a larger cohort of students.

Validity Evidence for Assessing Entrustable Professional Activities During Undergraduate Medical Education.

PURPOSE: To explore validity evidence for the use of entrustable professional activities (EPAs) as an assessment framework in medical education. METHOD: Formative assessments on the 13 Core EPAs for entering residency were collected for 4 cohorts of students over a 9- to 12-month longitudinal integrated clerkship as part of the Education in Pediatrics Across the Continuum pilot at the University of Minnesota Medical School. The students requested assessments from clinical supervisors based on direct observation while engaging in patient care together. Based on each observation, the faculty member rated the student on a 9-point scale corresponding to levels of supervision required. Six EPAs were included in the present analyses. Student ratings were depicted as curves describing their performance over time; regression models were employed to fit the curves. The unit of analyses for the learning curves was observations rather than individual students. RESULTS: (1) Frequent assessments on EPAs provided a developmental picture of competence consistent with the negative exponential learning curve theory; (2) This finding was true across a variety of EPAs and across students; and (3) The time to attain the threshold level of performance on the EPA for entrustment varied by student and EPA. CONCLUSIONS: The results provide validity evidence for an EPA-based program of assessment. Students assessed using multiple observations performing the Core EPAs for entering residency demonstrate classic developmental progression toward the desired level of competence resulting in entrustment decisions. Future work with larger data samples will allow further psychometric analyses of assessment of EPAs.

Twelve tips for learners to succeed in a CBME program.

The international movement to competency-based medical education (CBME) marks a major transition in medical education that requires a shift in educators' and learners' approach to clinical experiences, the way assessment data are collected and integrated, and in learners' mindsets. Learners entering a CBME curriculum must actively drive their learning experiences and education goals. For some, this expectation may be a significant change from their previous approach to learning in medicine. This paper highlights 12 tips to help learners succeed within a CBME model.

Shared Mental Models Among Clinical Competency Committees in the Context of Time-Variable, Competency-Based Advancement to Residency.

PURPOSE: To evaluate response process validity evidence for clinical competency committee (CCC) assessments of first-year residents on a subset of General Pediatrics Entrustable Professional Activities (EPAs) and milestones in the context of a national pilot of competency-based, time-variable (CBTV) advancement from undergraduate to graduate medical education. METHOD: Assessments of 2 EPAs and 8 milestones made by the trainees' actual CCCs and 2 different blinded "virtual" CCCs for 48 first-year pediatrics residents at 4 residency programs between 2016 and 2018 were compared. Residents had 3 different training paths from medical school to residency: time-variable graduation at the same institution as their residency, time-fixed graduation at the same institution, or time-fixed graduation from a different institution. Assessments were compared using ordinal mixed-effects models. RESULTS: Actual CCCs assigned residents higher scores than virtual CCCs on milestones and one EPA's supervision levels. Residents who graduated from a different institution than their residency received lower milestone ratings than either group from the same institution; CBTV residents received higher ratings on one milestone (ICS4) and similar ratings on all others compared with non-CBTV residents who completed medical school at the same institution. CONCLUSIONS: First-year residents who graduated from CBTV medical school programs were assessed as having the same level of competence as residents who graduated from traditional medical school programs, but response process evidence suggests that members of CCCs may also draw on undocumented personal knowledge of the learner to draw conclusions about resident competence.

Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test.

Construct: We investigated whether a situational judgment test (SJT) designed to measure professionalism in physicians predicts residents' performance on (a) Accreditation Council for Graduate Medical Education (ACGME) competencies and (b) a multisource professionalism assessment (MPA). Background: There is a consensus regarding the importance of assessing professionalism and interpersonal and communication skills in medical students, residents, and practicing physicians. Nonetheless, these noncognitive competencies are not well measured during medical education selection processes. One promising method for measuring these noncognitive competencies is the SJT. In a typical SJT, respondents are presented with written or video-based scenarios and asked to make choices from a set of alternative courses of action. Interpersonally oriented SJTs are commonly used for selection to medical schools in the United Kingdom and Belgium and for postgraduate selection of trainees to medical practice in Belgium, Singapore, Canada, and Australia. However, despite international evidence suggesting that SJTs are useful predictors of in-training performance, end-of-training performance, supervisory ratings of performance, and clinical skills licensing objective structured clinical examinations, the use of interpersonally oriented SJTs in residency settings in the United States has been infrequently investigated. The purpose of this study was to investigate whether residents' performance on an SJT designed to measure professionalism-related competencies-conscientiousness, integrity, accountability, aspiring to excellence, teamwork, stress tolerance, and patient-centered care-predicts both their current and future performance as residents on two important but conceptually distinct criteria: ACGME competencies and the MPA. Approach: We developed an SJT to measure seven dimensions of professionalism. During calendar year 2017, 21 residency programs from 2 institutions administered the SJT. We conducted analyses to determine the validity of SJT and USMLE scores in predicting milestone performance in ACGME core competency domains and the MPA in June 2017 and 3 months later in September 2017 for the MPA and 1 year later, in June 2018, for ACGME domains. Results: At both periods, the SJT score predicted overall ACGME milestone performance (r = .13 and .17, respectively; p < .05) and MPA performance (r = .19 and .21, respectively; p < .05). In addition, the SJT predicted ACGME patient care, systems-based practice, practice-based learning and improvement, interpersonal and communication skills, and professionalism competencies (r = .16, .15, .15, .17, and .16, respectively; p < .05) 1 year later. The SJT score contributed incremental validity over USMLE scores in predicting overall ACGME milestone performance (ΔR = .07) 1 year later and MPA performance (ΔR = .05) 3 months later. Conclusions: SJTs show promise as a method for assessing noncognitive attributes in residency program applicants. The SJT's incremental validity to the USMLE series in this study underscores the importance of moving beyond these standardized tests to a more holistic review of candidates that includes both cognitive and noncognitive measures.

Narrative Performance Level Assignments at Initial Entrustment and Graduation: Integrating EPAs and Milestones to Improve Learner Assessment.

PURPOSE: To determine which narrative performance level for each general pediatrics entrustable professional activity (EPA) reflects the minimum level clinical competency committees (CCCs) felt should be associated with graduation as well as initial entrustment and compare expected narrative performance levels (ENPLs) for each EPA with actual narrative performance levels (ANPLs) assigned to residents at initial entrustment. METHOD: A series of 5 narratives, corresponding to the 5 milestone performance levels, were developed for each of the 17 general pediatrics EPAs. In academic year (AY) 2015-2016, the CCCs at 22 Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network member sites reported ENPLs for initial entrustment and at time of graduation. From AYs 2015-2016 to 2017-2018, programs reported ANPLs for initial entrustment decisions. ENPLs and ANPLs were compared using a logistic mixed effects model. RESULTS: ENPLs for graduation and entrustment were most often level 3 (competent) followed by level 4 (proficient). For 8 EPAs, the ENPLs for graduation and entrustment were the same. For the remaining 9, some programs would entrust residents before graduation or graduate them before entrusting them. There were 4,266 supervision level reports for initial entrustment for which an ANPL was provided. ANPLs that were lower than the ENPLs were significantly more likely to be assigned to the medical home-well child (OR = 0.39; 95% CI: 0.26-0.57), transition to adult care (OR = 0.43; 95% CI: 0.19-0.95), behavioral or mental health (OR = 0.36; 95% CI: 0.18-0.71), make referrals (OR = 0.31; 95% CI: 0.17-0.55), lead a team (OR = 0.34; 95% CI: 0.22-0.52), and handovers (OR = 0.18; 95% CI: 0.09-0.36) EPAs. CONCLUSIONS: CCCs reported lower ENPLs for graduation than for entrustment for 5 EPAs, possibly indicating curricular gaps that milestones and EPAs could help identify.

Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities.

Importance: Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. Objective: To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Design, Setting, and Participants: Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Interventions: Longitudinal, prospective assessment using EPAs. Main Outcomes and Measures: Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Results: Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). Conclusions and Relevance: This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Patient Satisfaction With Medical Student Participation in a Longitudinal Integrated Clerkship: A Controlled Trial.

PURPOSE: To determine whether longitudinal student involvement improves patient satisfaction with care. METHOD: The authors conducted a satisfaction survey of patients followed by 10 University of Minnesota Medical School students enrolled in 2016-2017 in the Veterans Affairs Longitudinal Undergraduate Medical Education (VALUE) program, a longitudinal integrated clerkship at the Minneapolis Veterans Health Care System. Students were embedded in an ambulatory practice with primary preceptors who assigned students a panel of 14 to 32 patients to follow longitudinally in inpatient and outpatient settings. Control patients, matched on disease severity, were chosen from the preceptor's panel. Two to five months after the students completed the VALUE program, the authors conducted a phone survey of the VALUE and control patients using a validated, customized questionnaire. RESULTS: Results are reported from 97 VALUE patients (63% response rate) and 72 controls (47% response rate) who had similar baseline characteristics. Compared with control patients, VALUE patients reported greater satisfaction with explanations provided by their health care provider, their provider's knowledge of their personal history, and their provider's looking out for their best interests (P < .05). Patients in the VALUE panel selected the top category more often than control patients for overall satisfaction with their health care (65% vs 43%, P < .05). CONCLUSIONS: The results of this controlled trial demonstrate that VALUE student longitudinal participation in patient care improves patient satisfaction and patient-perceived quality of health care for VALUE patients compared with controls matched by primary care provider and disease severity. These findings may have implications outside the Veterans Administration population.

Finding Greater Value in the Fourth Year of Medical School: Accelerating the Transition to Residency.

The recent focus on competency-based medical education has heralded a true change in U.S. medical education. Accelerating the transition from medical school to residency may reduce student debt, encourage competency-based educational advancement, and produce residency graduates better prepared for the independent and unsupervised practice of medicine. With some purposeful design considerations, innovative time-variable programs or fixed-time accelerated tracks can be implemented within current regulatory parameters and without major alteration of existing institutional regulatory guidelines, state licensing requirements, or specialty certification requirements. Conferring an MD degree in less than 4 full academic years provides opportunities to customize and find greater value in the fourth year of medical school as well as to redeploy time from undergraduate medical education to graduate medical education; this could shorten the overall time to completion of training and/or provide for customization of training in the final years of residency. In this article, the authors discuss the regulatory requirements for successful implementation; consider issues related to "off-cycle" graduates advancing to residency training outside of the Match; and share examples of 3 innovative accelerated programs in pediatrics, family medicine, and orthopaedics that have yielded advantages to individual learners, including reduced educational debt, as well as to the health care system.

Coproducing Health Professions Education: A Prerequisite to Coproducing Health Care Services?

In 2016, Batalden et al proposed a coproduction model for health care services. Starting from the argument that health care services should demonstrate service-dominant rather than goods-dominant logic, they argued that health care outcomes are the result of the intricate interaction of the provider and patient in concert with the system, community, and, ultimately, society. The key notion is that the patient is as much an expert in determining outcomes as the provider, but with different expertise. Patients come to the table with expertise in their lived experiences and the context of their lives.The authors posit that education, like health care services, should follow a service-dominant logic. Like the relationship between patients and providers, the relationship between learner and teacher requires the integrated expertise of each nested in the context of their system, community, and society to optimize outcomes. The authors then argue that health professions learners cannot be educated in a traditional, paternalistic model of education and then expected to practice in a manner that prioritizes coproductive partnerships with colleagues, patients, and families. They stress the necessity of adapting the health care services coproduction model to health professions education. Instead of asking whether the coproduction model is possible in the current system, they argue that the current system is not sustainable and not producing the desired kind of clinicians.A current example from a longitudinal integrated clerkship highlights some possibilities with coproduced education. Finally, the authors offer some practical ways to begin changing from the traditional model. They thus provide a conceptual framework and ideas for practical implementation to move the educational model closer to the coproduction health care services model that many strive for and, through that alignment, to set the stage for improved health outcomes for all.

Entrustable Professional Activities in General Surgery: Development and Implementation.

Entrustable Professional Activities are a holistic assessment framework developed in the Netherlands in 2005, which have recently been adopted in undergraduate medical education in the United States. As part of an increased focus on competency-based assessment, the specialty of pediatrics has led the way in incorporating them into graduate medical education. We describe the development and initial pilot process of implementation of EPAs into the assessment of General Surgery trainees.

Providing Compassionate Off-Ramps for Medical Students Is a Moral Imperative.

It is highly unusual for learners to leave medical training in the United States even though some individuals' goals may change and others may not achieve expected competence. There are a number of possible reasons for this: (1) Students may feel that they have progressed too far into their careers and amassed too much debt to leave medical training; (2) students may be allowed to graduate despite marginal performance; and (3) students may have entered medical training with risk factors for poor performance that were not addressed. As stewards of the educational process, medical educators have an ethical obligation to students and the public to create off-ramps, or points along the educational continuum at which learners can reassess their goals and educators can assess competence, that allow students to leave medicine.Given the nationwide focus on physician health and wellness, the authors believe the creation of options to leave medical training without compromising one's self-esteem or incurring unmanageable debt (i.e., compassionate off-ramps) is a moral imperative. The practice of medicine should not be an exercise in survival; it should allow people to develop and thrive over the course of their careers. Offering students options to make use of the medical competencies they have accumulated in other attractive careers would enable medical educators to behave compassionately toward individual students and fulfill their societal obligation to graduate competent and committed physicians. To this end, the authors present six recommendations for consideration.

Crossing the Gap: Using Competency-Based Assessment to Determine Whether Learners Are Ready for the Undergraduate-to-Graduate Transition.

In 2011, the Education in Pediatrics Across the Continuum (EPAC) Study Group recruited four medical schools (University of California, San Francisco; University of Colorado; University of Minnesota; and University of Utah) and their associated pediatrics clerkship and residency program directors to be part of a consortium to pilot a model designed to advance learners from undergraduate medical education (UME) to graduate medical education (GME) and then to fellowship or practice based on competence rather than time spent in training. The central design features of this pilot included predetermined expectations of performance and transition criteria to ensure readiness to progress from UME to GME, using the Core Entrustable Professional Activities for Entering Residency (Core EPAs) as a common assessment framework. Using this framework, each site team (which included, but was not limited to, the EPAC course, pediatric clerkship, and pediatric residency program directors) monitored learners' progress, with the site's clinical competency committee marking the point of readiness to transition from UME to GME (i.e., the attainment of supervision level 3a). Two of the sites implemented time-variable transition from UME to GME, based on when a learner met the performance expectations and transition criteria. In this Article, the authors describe each of the four sites' implementation of Core EPA assessment and their approach to gathering the data necessary to determine readiness for transition. They conclude by offering recommendations and lessons learned from the pilot's first seven years of development, adaptation, and implementation of assessment strategies across the sites, and discussing next steps.

Entrustable professional activities for quality and patient safety.

BACKGROUND: Further efforts are warranted to identify innovative approaches to best implement competencies in nursing education. To bridge the gap between competency-based education, practice, and implementation of knowledge, skills, and attitudes, one emerging approach is entrustable professional activities (EPAs). PURPOSE: The objective of this study was to introduce the concept of EPAs as a framework for curriculum and assessment in graduate nursing education and training. METHODS: Seven steps are provided to develop EPAs for nurses through the example of a quality and safety EPA. The example incorporates the Quality and Safety Education for Nurses (QSEN) patient safety competencies and evidence-based literature. FINDINGS: EPAs provide a practical approach to integrating competencies in nursing as quality and safety are the cornerstones of nursing practice, education, and research. DISCUSSION: Introducing the EPA concept in nursing is timely as we look to identify opportunities to enhance nurse practitioner (NP) training models and implement nurse residency programs.

A Lack of Continuity in Education, Training, and Practice Violates the "Do No Harm" Principle.

The paradigm shift to competency-based medical education (CBME) is under way, but incomplete implementation is blunting the potential impact on learning and patient outcomes. The fundamental principles of CBME call for standardizing outcomes addressing population health needs, then allowing time-variable progression to achieving them. Operationalizing CBME principles requires continuity within and across phases of the education, training, and practice continuum. However, the piecemeal origin of the phases of the "continuum" has resulted in a sequence of undergraduate to graduate medical education to practice that may be continuous temporally but bears none of the integration of a true continuum.With these timed interruptions during phase transitions, learning is not reinforced because of a failure to integrate experiences. Brief block rotations for learners and ever-shorter supervisory assignments for faculty preclude the development of relationships. Without these relationships, feedback falls on deaf ears. Block rotations also disrupt learners' relationships with patients. The harms resulting from such a system include decreases in patient satisfaction with their care and learner satisfaction with their work. Learners in this block system also demonstrate an erosion of empathy compared with those in innovative longitudinal training models. In addition, higher patient mortality during intern transitions has been demonstrated.The current medical education system is violating the first principle of medicine: "Do no harm." Full implementation of competency-based, time-variable education and training, with fixed outcomes aligned with population health needs, continuity in learning and relationships, and support from a developmental program of assessment, holds great potential to stop this harm.

Entrustable professional activities: A useful concept for neurology education.

Medical education is currently undergoing a paradigm shift from process-based to competency-based education, focused on measuring the desired competence of a physician. In an attempt to improve the assessment framework used for medical education, the concept of entrustable professional activities (EPAs) has gained traction. EPAs are defined as professional activities that can be entrusted to an individual in a clinical context. The Association of American Medical Colleges (AAMC) defined a set of 13 such EPAs to define the core of what all students should be able to do on day 1 of residency, regardless of specialty choice. The AAMC is currently piloting these EPAs with 10 medical schools to determine if EPAs can be used as a way to observe, measure, and entrust medical students with core clinical activities by the end of the clinical immersion experiences of the third year. The specialty of pediatrics is piloting the use of specialty-specific EPAs at 5 medical schools to assess readiness for transitions from medical school into pediatric residency training and practice. To date, no neurology-specific EPAs have been published for use in neurology clerkships or neurology residencies. This article introduces the concept of EPAs in the context of competency-based medical education and describes how EPAs might be relevant and applicable in neurologic education across the continuum. The Undergraduate Education Subcommittee of the American Academy of Neurology advocates for a proactive approach to incorporating core EPAs in undergraduate medical education and to considering an EPA-based specialty-specific assessment framework for neurology.

Criterion-Based Assessment in a Norm-Based World: How Can We Move Past Grades?

In the United States, the medical education community has begun a shift from the Flexnerian time-based model to a competency-based medical education model. The graduate medical education (GME) community is substantially farther along in this transition than is the undergraduate medical education (UME) community.GME has largely adopted the use of competencies and their attendant milestones and increasingly is employing the framework of entrustable professional activities (EPAs) to assess trainee competence. The UME community faces several challenges to successfully navigating a similar transition. First is the reliance on norm-based reference standards in the UME-GME transition, comparing students' performance versus their peers' with grades, United States Medical Licensing Examination Step 1 and Step 2 score interpretation, and the structured Medical School Performance Evaluation, or dean's letter. Second is the reliance on proxy assessments rather than direct observation of learners. Third is the emphasis on summative rather than formative assessments.Educators have overcome a major barrier to change by establishing UME outcomes assessment criteria with the advent and general acceptance of the physician competency reference set and the Core EPAs for Entering Residency in UME. Now is the time for the hard work of developing assessments steeped in direct observation that can be accepted by learners and faculty across the educational continuum and can be shown to predict clinical performance in a much more meaningful way than the current measures of grades and examinations. The acceptance of such assessments will facilitate the UME transition toward competency-based medical education.

Education in Pediatrics Across the Continuum (EPAC): First Steps Toward Realizing the Dream of Competency-Based Education.

The Education in Pediatrics Across the Continuum (EPAC) Study Group is developing the first competency-based, time-variable progression from undergraduate medical education (UME) to graduate medical education (GME) in the history of medical education in the United States. EPAC, an innovation project sponsored by the Association of American Medical Colleges and supported by the Josiah Macy Jr. Foundation, was developed through a collaboration between five medical schools and multiple professional organizations with an interest in undergraduate and graduate medical education. The planning and implementation process demanded cooperatively addressing practical barriers such as education requirements for licensure and developing approaches to learner assessment that provided meaningful information about competency. Each participating school now has at least three cohorts of learners participating, and the program is transitioning its first cohort of students from UME to GME based on achievement of predetermined competencies that allow this transition. Members of the first cohort of learners in this program have begun their pediatric residency training at different times beginning in late 2016, confirming the feasibility of competency-based advancement from UME to GME in pediatrics. Although there is still much to learn about the outcomes of EPAC learners' professional development in residency training and beyond, EPAC has defined an operational approach to a different path through medical school and into residency training, based on the attainment of competence.