How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study.

BACKGROUND: Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS: The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS: Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION: The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Medical Students' Perspectives on Opportunities to Engage in the Clinical Care of COVID-19 Patients during the Pandemic.

OBJECTIVES: Recent disease modeling suggests that pandemics are likely to increase in frequency and severity. As such, medical educators must learn from their experiences with coronavirus disease 2019 (COVID-19) to develop systematic strategies for ensuring that medical students receive hands-on training in the management of emerging diseases. Here, we outline the process by which the Florida International University Herbert Wertheim College of Medicine developed and updated guidelines for student participation in the care of patients with COVID-19 and report on students' experiences. METHODS: During the 2020-2021 academic year, Florida International University Herbert Wertheim College of Medicine students were not permitted to care for patients with COVID-19; however, academic year 2021-2022 guidelines did permit fourth-year students on subinternships or Emergency Medicine rotations to voluntarily care for patients with COVID-19. At the end of the 2021-2022 academic year, students completed an anonymous survey about their experience caring for patients with COVID-19. Likert-type and multiple-choice questions were analyzed using descriptive statistics and the short-answer responses were analyzed qualitatively. RESULTS: One hundred two students (84%) responded to the survey. Sixty-four percent of respondents opted to provide care for patients with COVID-19. Most students (63%) cared for patients with COVID-19 during their required Emergency Medicine Selective. Twenty-eight percent of students wished they had more COVID-19 patient care opportunities, and 29% did not feel prepared to care for patients with COVID-19 on their first day of residency. CONCLUSIONS: Many graduating students felt unprepared to care for patients with COVID-19 during residency and many wished they had had more opportunities to care for patients with COVID-19 during medical school. Curricular policies must evolve to allow students to gain competency in the care of patients with COVID-19 so that they are prepared for day one of residency.

Entrustable professional activities as a training and assessment framework in undergraduate medical education: A case study of a multi-institutional pilot.

In 2014, the Association of American Medical Colleges (AAMC) published 13 Core Entrustable Professional Activities (EPAs) that graduating students should be able to perform with indirect supervision when entering residency. A ten-school multi-year pilot was commissioned to test feasibility of implementing training and assessment of the AAMC's 13 Core EPAs. In 2020-21, a case study was employed to describe pilot schools' implementation experiences. Teams from nine of ten schools were interviewed to identify means and contexts of implementing EPAs and lessons learned. Audiotapes were transcribed then coded by investigators using conventional content analysis and a constant comparative method. Coded passages were organized in a database and analyzed for themes. Consensus among school teams regarding facilitators of EPA implementation included team commitment to piloting EPAs; agreement that: proximal EPA adoption with curriculum reform facilitates EPA implementation; EPAs 'naturally fit' in clerkships and provided opportunity for schools to reflect on and adjust curricula and assessments; and inter-school collaboration bolstered individual school progress. Schools did not make high-stakes decisions about student progress (e.g., promotion, graduation), yet EPA assessment results complemented other forms of assessment in providing students with robust formative feedback about their progress. Teams had varied perceptions of school capability to implement an EPA framework, influenced by various levels of dean involvement, willingness, and capability of schools to invest in data systems and provide other resources, strategic deployment of EPAs and assessments, and faculty buy-in. These factors affected varied pace of implementation. Teams agreed on the worthiness of piloting the Core EPAs, but substantial work is still needed to fully employ an EPA framework at the scale of entire classes of students with enough assessments per EPA and with required data validity/reliability. Recommendations stemming from findings may help inform further implementation efforts across other schools adopting or considering an EPA framework.

Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students' Self-Assessed Skills by Specialty.

BACKGROUND: The Association of American Medical Colleges described 13 Core Entrustable Professional Activities (EPAs) that graduating students should be prepared to perform under indirect supervision on day one of residency. Surgery program directors recently recommended entrustability in these Core EPAs for incoming surgery interns. We sought to determine if graduating students intending to enter surgery agreed they had the skills to perform these Core EPAs. STUDY DESIGN: Using de-identified, individual-level data collected from and about 2019 Association of American Medical Colleges Graduation Questionnaire respondents, latent profile analysis was used to group respondents based on their self-assessed Core EPAs skills' response patterns. Associations between intended specialty, among other variables, and latent profile analysis group were assessed using independent sample t -tests and chi-square tests and multivariable logistic regression methods. RESULTS: Among 12,308 Graduation Questionnaire respondents, latent profile analysis identified 2 respondent groups: 7,863 (63.9%) in a high skill acquisition agreement (SAA) group and 4,445 (36.1%) in a moderate SAA group. Specialty was associated with SAA group membership (p < 0.001), with general surgery, orthopaedic surgery, and emergency medicine respondents (among others) overrepresented in the high SAA group. In the multivariable logistic regression models, each of anesthesiology, ophthalmology, pediatrics, psychiatry, and radiology (vs general surgery) specialty intention was associated with a lower odds of high SAA group membership. CONCLUSION: Graduating students' self-assessed Core EPAs skills were higher for those intending general surgery than for those intending some other specialties. Our findings can inform collaborative efforts to ensure graduates' acquisition of the skills expected of them at the start of residency.

Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020.

Importance: Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency. Objective: To assess progress in developing an entrustment process in the Core EPAs framework. Design, Setting, and Participants: In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database. Interventions: Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations. Main Outcomes and Measures: On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded. Results: Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs. Conclusions and Relevance: These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.

Providing Care with Cultural, Racial, and Ethnic Humility: the Framework for an Additional Entrustable Professional Activity.

Healthcare inequities are rampant in the USA. There is a lack of standardization for training medical students in providing care with cultural, racial, and ethnic humility. This innovation is the framework of an entrustable professional activity that can be utilized to address this vital issue.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters.

PURPOSE: In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. METHOD: The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. RESULTS: Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. CONCLUSIONS: In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study.

PURPOSE: In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD: For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS: The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS: TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

Core Entrustable Professional Activities (EPAs) and the Transition from Medical School to Residency: the Postgraduate Year One Resident Perspective.

INTRODUCTION: The Association of American Medical Colleges (AAMC) proposed thirteen core Entrustable Professional Activities (EPAs) that all graduates should be able to perform under indirect supervision upon entering residency. As an underlying premise is that graduates ready to do so will be better prepared to transition to the responsibilities of residency, we explored the relationship between postgraduate year (PGY)-1 residents' self-assessed preparedness to perform core EPAs under indirect supervision at the start of residency with their ease of transition to residency. METHODS: Using response data to a questionnaire administered in September 2019 to PGY-1 residents who graduated from AAMC core EPA pilot schools, we examined between-group differences and independent associations for each of PGY-1 position type, specialty, and "EPA-preparedness" score (proportion of EPAs the resident reported as prepared to perform under indirect supervision at the start of residency) and ease of transition to residency (from 1 = much harder to 5 = much easier than expected). RESULTS: Of 274 questionnaire respondents (19% of 1438 graduates), 241 (88% of 274) had entered PGY-1 training and completed all questionnaire items of interest. EPA-preparedness score (mean 0.71 [standard deviation 0.26]) correlated with ease of transition (3.1 [0.9]; correlation = .291, p < .001). In linear regression controlling for specialty (among other variables), EPA-preparedness score (ß-coefficient 1.08; 95% confidence interval .64-1.52; p < .001) predicted ease of transition to residency. CONCLUSION: Graduates who felt prepared to perform many of the core EPAs under indirect supervision at the start of PGY-1 training reported an easier-than-expected transition to residency. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01370-3.

Taking CBL to the Lecture Hall: a Comparison of Outcomes Between Traditional Small Group CBL and a Novel Large Group Team-Based CBL Teaching Method.

PURPOSE: Case-based learning (CBL), an important component of medical school curricula, is an effective inquiry-based teaching method associated with high levels of student and teacher satisfaction. However, because traditional CBL requires small groups, its feasibility is limited by faculty and resources. We developed and tested a novel team-based CBL (TB-CBL) method to be implemented in the lecture hall. METHODS: All second-year students at our institution (n = 121) were randomized to either traditional small group CBL or TB-CBL during the Endocrine block and to the other modality during the Renal block. All students were exposed to both methods. Case content was identical, and sessions were run concurrently. This cross-over, non-inferiority study tested the hypothesis that no difference in knowledge acquisition, clinical reasoning, or student satisfaction would be detected between groups. RESULTS: Based on student performance on case-relevant exam questions, no difference in knowledge acquisition was seen between groups for either block (p = 0.62 Endocrine, p = 0.38 Renal). There was also no difference in overall final exam performance between groups (p = 0.56 Endocrine, p = 0.26 Renal). Case-relevant script concordance testing revealed no difference in clinical reasoning skills between groups (p = 0.87 Endocrine, p = 0.17 Renal). Satisfaction was higher for the TB-CBL format (p = 0.005). Cost analysis revealed that each small group CBL session costs $2654, while each TB-CBL session costs approximately $221. CONCLUSIONS: TB-CBL, a novel case-based teaching method, appears to produce similar learner outcomes and higher student satisfaction when compared with small group CBL. TB-CBL may be used to supplement case-based curricula while optimizing resource allocation.

Developing an Entrustment Process: Insights from the AAMC CoreEPA Pilot.

One of the main goals of the CoreEPA pilot has been to determine the feasibility of developing a process to make summative entrustment decisions regarding entrustable professional activities (EPAs). Five years into the pilot, we report results of a research study we conducted to explore approaches to the entrustment process undertaken by our ten participating schools. We sought to identify the choices that participating schools made regarding the entrustment process and why these decisions were made. We are sharing these results, highlighting ongoing challenges that were identified with the intent of helping other medical schools that are moving toward EPA-based assessment. We conducted semi-structured interviews with representatives of all 10 medical schools in the CoreEPA pilot to understand their choices in designing the entrustment process. Additional information was obtained through follow-up communication to ensure completeness and accuracy of the findings. Several common themes are described. Our results indicate that, while approaches to the entrustment process vary considerably, all schools demonstrated consistent adherence to the guiding principles of the pilot. Several common barriers to the entrustment process emerged, and there was a consensus that more experience is needed with the process before consequential entrustment decisions can be made. The CoreEPA pilot schools continue to address challenges identified in implementing entrustment processes and making entrustment decisions for our students graduating in the Class of 2020.

From the Classroom to Entrustment - The Development of Motivational Interviewing Skills as an Entrustable Professional Activity.

This article was migrated. The article was marked as recommended. Introduction The move towards value-based care and population health has highlighted the prominent role of social and behavioral factors in determining health outcomes. Patient-centered behavioral guidance to improve patient self-management is recognized as an evidence-based intervention for a variety of chronic conditions but has yet to be adopted as a core competency or core entrustable professional activity (EPA). Motivational Interviewing (MI) is an evidence-based behavioral intervention involving an integrated set of competencies, featuring reflective listening, affirmation, evocation, and collaborative planning. An MI encounter is an observable, discrete task that can be framed as an EPA. Successful implementation of EPAs in the workplace requires institutional engagement, a thoughtful curricular approach, faculty development, and feasible, valid workplace-based assessment (WBA). Methods We implemented competency-based MI training and assessed competency outcomes for students and faculty. After joining the Association of American Medical Colleges Core EPA Pilot, we applied an iterative group process to develop an EPA and workplace-based assessment based on established MI competencies. Results Drawing upon nine years of developing MI curriculum, we present competency data for a student training study and a faculty coaching study, describe how we transitioned training from the classroom to the clinical setting employing an EPA framework, and present a one-page schematic and related WBA for an EPA based on MI. Conclusion We propose that MI is a core EPA for future physicians practicing value-based care, and offer a roadmap for curriculum implementation.

Furthering the Validity of a Tool to Assess Simulated Pregnancy Options Counseling Skills.

OBJECTIVE: To further the validity of a tool to assess nondirective pregnancy options counseling skills. METHODS: Using a cross-sectional design, we explored four sources of construct validity evidence for an objective structured clinical examination for training and assessment of nondirective pregnancy options counseling: content, response process, internal structure, and relations to other variables. Content of the previously developed tool was enhanced through input from five family medicine educators. The objective structured clinical examination was implemented in a family medicine clerkship with third-year medical students from 2014 to 2015 using trained raters. Response process was addressed after a pilot round. Three new raters evaluated videotapes of 46 performances. Cronbach's alpha, intraclass correlation coefficients, and Spearman's rho were estimated with 95% confidence intervals. RESULTS: The content validity was affirmed. Cronbach's alpha was 0.71. According to Landis and Koch's criteria, all but two items unique to the clinical situation of pregnancy options counseling generated substantial to perfect agreement (0.62-1.00). Relations to other variables within the checklist were strong, ranging from 0.66 to 0.87. DISCUSSION: This tool for assessing pregnancy options counseling skills has excellent content and strong internal structure. Further work to improve the Global Rating Scale may be necessary for summative use.

Advanced Patient-Centered Communication for Health Behavior Change: Motivational Interviewing Workshops for Medical Learners.

INTRODUCTION: Medical settings are critical access points for behavior change counseling, and lifestyle behavior change is considered a key component of chronic disease management. The Association of American Medical Colleges recommends that future physicians be competent in shared decision making and patient-centered behavioral guidance to prevent illness and improve patient self-management of chronic disease. Motivational interviewing (MI) is a patient-centered, directive method of communication to enhance behavior change. Specific teachable strategies underlie the collaborative MI communication style that aims to reduce discord and build motivation for change. METHODS: We present our three-session 12-hour MI curriculum as an advanced form of patient-centered communication. Each session includes presession assignment, large-group interactive lecture, and small-group activities for practice. An interdisciplinary team consisting of medical educators and health behavior change research-educators who are also members of the Motivational Interviewing Network of Trainers created the submission. The purpose of this resource is to provide medical educators with a short curriculum that incorporates materials and learning activities to promote skill in MI. RESULTS: In addition to positive feedback from student evaluations including the areas of relevance to training and self-rated skills improvement, preliminary pre- and posttraining scores from the medical students show significant improvement in expression of empathy and the ratio of reflections to questions. DISCUSSION: Implementation of the curriculum allows learners the opportunity to practice evidence-based communication that promotes intrinsic motivation for health behavior change in patients, a key treatment focus in chronic disease management.

Simulation technology for skills training and competency assessment in medical education.

Medical education during the past decade has witnessed a significant increase in the use of simulation technology for teaching and assessment. Contributing factors include: changes in health care delivery and academic environments that limit patient availability as educational opportunities; worldwide attention focused on the problem of medical errors and the need to improve patient safety; and the paradigm shift to outcomes-based education with its requirements for assessment and demonstration of competence. The use of simulators addresses many of these issues: they can be readily available at any time and can reproduce a wide variety of clinical conditions on demand. In lieu of the customary (and arguably unethical) system, whereby novices carry out the practice required to master various techniques--including invasive procedures--on real patients, simulation-based education allows trainees to hone their skills in a risk-free environment. Evaluators can also use simulators for reliable assessments of competence in multiple domains. For those readers less familiar with medical simulators, this article aims to provide a brief overview of these educational innovations and their uses; for decision makers in medical education, we hope to broaden awareness of the significant potential of these new technologies for improving physician training and assessment, with a resultant positive impact on patient safety and health care outcomes.

A multicenter study to provide evidence of construct validity in a computer-based outcome measure of neurology clinical skills.

BACKGROUND: Using computer-based simulation to assess clinical skill-a key competence for medical trainees-enables standardization and exposure to a broad sample of physical findings. The purpose of this study is to provide evidence of construct validity for a computer-based outcome measure of neurology clinical skills. METHOD: A total of 128 medical students and neurology residents at four institutions volunteered to take a 34-question computer-based test designed to measure neurology clinical skills. Subjects were classified into three groups based on level of training: novice, intermediate, and experienced. RESULTS: Overall performance increased with level of training. Question difficulty discriminated between groups as predicted. Twenty-six of 34 individual items discriminated between novices and more advanced learners. This test separated learners at different levels of training with a consistency of .92. CONCLUSION: This study provided evidence of construct validity for a computer-based outcome measure of neurology clinical skills.