Learner Assessment and Program Evaluation: Supporting Precision Education.

ABSTRACT: Precision education (PE) systematically leverages data and advanced analytics to inform educational interventions that, in turn, promote meaningful learner outcomes. PE does this by incorporating analytic results back into the education continuum through continuous feedback cycles. These data-informed sequences of planning, learning, assessing, and adjusting foster competence and adaptive expertise. PE cycles occur at individual (micro), program (meso), or system (macro) levels. This article focuses on program- and system-level PE.Data for PE come from a multitude of sources, including learner assessment and program evaluation. The authors describe the link between these data and the vital role evaluation plays in providing evidence of educational effectiveness. By including prior program evaluation research supporting this claim, the authors illustrate the link between training programs and patient outcomes. They also describe existing national reports providing feedback to programs and institutions, as well as 2 emerging, multiorganization program- and system-level PE efforts. The challenges encountered by those implementing PE and the continuing need to advance this work illuminate the necessity for increased cross-disciplinary collaborations and a national cross-organizational data-sharing effort.Finally, the authors propose practical approaches for funding a national initiative in PE as well as potential models for advancing the field of PE. Lessons learned from successes by others illustrate the promise of these recommendations.

Social network analysis of publication collaboration of accelerating change in MedEd consortium.

INTRODUCTION: The American Medical Association formed the Accelerating Change in Medical Education Consortium through grants to effect change in medical education. The dissemination of educational innovations through scholarship was a priority. The objective of this study was to explore the patterns of collaboration of educational innovation through the consortium's publications. METHOD: Publications were identified from grantee schools' semi-annual reports. Each publication was coded for the number of citations, Altmetric score, domain of scholarship, and collaboration with other institutions. Social network analysis explored relationships at the midpoint and end of the grant. RESULTS: Over five years, the 32 Consortium institutions produced 168 publications, ranging from 38 papers from one institution to no manuscripts from another. The two most common domains focused on health system science (92 papers) and competency-based medical education (30 papers). Articles were published in 54 different journals. Forty percent of publications involved more than one institution. Social network analysis demonstrated rich publishing relationships within the Consortium members as well as beyond the Consortium schools. In addition, there was growth of the network connections and density over time. CONCLUSION: The Consortium fostered a scholarship network disseminating a broad range of educational innovations through publications of individual school projects and collaborations.

The critical role of infrastructure and organizational culture in implementing competency-based education and individualized pathways in undergraduate medical education.

In 2010, several key works in medical education predicted the changes necessary to train modern physicians to meet current and future challenges in health care, including the standardization of learning outcomes paired with individualized learning processes. The reframing of a medical expert as a flexible, adaptive team member and change agent, effective within a larger system and responsive to the community's needs, requires a new approach to education: competency-based medical education (CBME). CBME is an outcomes-based developmental approach to ensuring each trainee's readiness to advance through stages of training and continue to grow in unsupervised practice. Implementation of CBME with fidelity is a complex and challenging endeavor, demanding a fundamental shift in organizational culture and investment in appropriate infrastructure. This paper outlines how member schools of the American Medical Association Accelerating Change in Medical Education Consortium developed and implemented CBME, including common challenges and successes. Critical supporting factors include adoption of the master adaptive learner construct, longitudinal views of learner development, coaching, and a supportive learning environment.

Realizing the vision of the Lancet Commission on Education of Health Professionals for the 21st Century: Transforming medical education through the Accelerating Change in Medical Education Consortium.

In the last two decades, prompted by the anticipated arrival of the 21st Century and on the centenary of the publication of the Flexner Report, many in medical education called for change to address the expanding chasm between the requirements of the health care system and the educational systems producing the health care workforce. Calls were uniform. Curricular content was missing. There was a mismatch in where people trained and where they were needed to practice, legacy approaches to pedagogical methods that needed to be challenged, an imbalance in diversity of trainees, and a lack of research on educational outcomes, resulting in a workforce that was described as ill-equipped to provide health care in the current and future environment. The Lancet Commission on Education of Health Professionals for the 21st Century published a widely acclaimed report in 2010 that called for a complete and authoritative re-examination of health professional education. This paper describes the innovations of the American Medical Association Accelerating Change in Medical Education Consortium schools as they relate to the recommendations of the Lancet Commission. We outline the successes, challenges, and lessons learned in working to deeply reform medical education.

Revisiting the Core Entrustable Professional Activities for Entering Residency.

The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.

A Responsible Educational Handover: Improving Communication to Improve Learning.

An important tenet of competency-based medical education is that the educational continuum should be seamless. The transition from undergraduate medical education (UME) to graduate medical education (GME) is far from seamless, however. Current practices around this transition drive students to focus on appearing to be competitively prepared for residency. A communication at the completion of UME-an educational handover-would encourage students to focus on actually preparing for the care of patients. In April 2018, the American Medical Association's Accelerating Change in Medical Education consortium meeting included a debate and discussion on providing learner performance measures as part of a responsible educational handover from UME to GME. In this Perspective, the authors describe the resulting 5 recommendations for developing such a handover: (1) The purpose of the educational handover should be to provide medical school performance data to guide continued improvement in learner ability and performance, (2) the process used to create an educational handover should be philosophically and practically aligned with the learner's continuous quality improvement, (3) the educational handover should be learner driven with a focus on individualized learning plans that are coproduced by the learner and a coach or advisor, (4) the transfer of information within an educational handover should be done in a standardized format, and (5) together, medical schools and residency programs must invest in adequate infrastructure to support learner improvement. These recommendations are shared to encourage implementation of the educational handover and to generate a potential research agenda that can inform policy and best practices.

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency.

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Assessing learning in the adaptive curriculum.

BACKGROUND: Medical education is a dynamic process that will continuously evolve to respond to changes in the foundations of medicine, the clinical practice of medicine and in health systems science. PURPOSE: In this paper, we review how assessing learning in such a dynamic environment requires comprehensive flexible and adaptable methodological approaches designed to assess knowledge attainment and transfer, clinical skills/competency development, and ethical/professional behavior. Adaptive assessments should measure the learner's ability to observe where changes in health care delivery are needed and how to implement them. Balancing formative and summative assessments will promote reflective learning so that each student will reach her/his highest potential. From the programmatic perspective, measuring the design and delivery of instruction in relation to students? efforts to achieve competency will improve learning and foster continuous professional development of faculty and advance the science of learning. APPROACH: We describe how two medical schools are approaching adaptive assessment, including using portfolio systems that encompass teaching and learning experiences while offering real-time longitudinal tracking of digital data toward improving learning and provide curricula continuous improvement cycles. Using latest technologies, portfolios produce actionable data displays with precise guidance for learning and program development.

Describing the Journey and Lessons Learned Implementing a Competency-Based, Time-Variable Undergraduate Medical Education Curriculum.

Oregon Health & Science University School of Medicine launched a completely new undergraduate medical education curriculum in 2014. This initiative dramatically transformed the MD degree program, changing the instructional content taught, the pedagogical methods used by the faculty, and the methods of assessment, and it added new elements such as academic coaching and programmatic entrustment to the program. One of the most exciting and impactful aspects to date of this curricular transformation has been the deliberate implementation of a competency-based framework that incorporates frequent assessment, tracking of student progression using an electronic portfolio, and academic coaching to optimize learning and customize curricular elements for each student. The next major step in this process-the implementation of time-variable progression-is currently ongoing as a planning group at the school works through the conceptual, logistical, legal, and regulatory issues related to implementing such a system. When implementation is complete, MD students will graduate only once they have earned entrustment for all 13 Core Entrustable Professional Activities for Entering Residency. This article describes the school's progress to date in its curricular transformation and articulates lessons learned thus far in driving substantive and dramatic institutional changes that profoundly impact students, faculty, and administrators in one academic health center.

Implementing an Entrustable Professional Activities Framework in Undergraduate Medical Education: Early Lessons From the AAMC Core Entrustable Professional Activities for Entering Residency Pilot.

In 2014, the Association of American Medical Colleges (AAMC) published a list of 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs) that medical school graduates might be expected to perform, without direct supervision, on the first day of residency. Soon after, the AAMC commissioned a five-year pilot with 10 medical schools across the United States, seeking to implement the Core EPA framework to improve the transition from undergraduate to graduate medical education.In this article, the pilot team presents the organizational structure and early results of collaborative efforts to provide guidance to other institutions planning to implement the Core EPA framework. They describe the aims, timeline, and organization of the pilot as well as findings to date regarding the concepts of entrustment, assessment, curriculum development, and faculty development. On the basis of their experiences over the first two years of the pilot, the authors offer a set of guiding principles for institutions intending to implement the Core EPA framework. They also discuss the impact of the pilot, its limitations, and next steps, as well as how the pilot team is engaging the broader medical education community. They encourage ongoing communication across institutions to capitalize on the expertise of educators to tackle challenges related to the implementation of this novel approach and to generate common national standards for entrustment. The Core EPA pilot aims to better prepare medical school graduates for their professional duties at the beginning of residency with the ultimate goal of improving patient care.

Fostering the Development of Master Adaptive Learners: A Conceptual Model to Guide Skill Acquisition in Medical Education.

Change is ubiquitous in health care, making continuous adaptation necessary for clinicians to provide the best possible care to their patients. The authors propose that developing the capabilities of a Master Adaptive Learner will provide future physicians with strategies for learning in the health care environment and for managing change more effectively. The concept of a Master Adaptive Learner describes a metacognitive approach to learning based on self-regulation that can foster the development and use of adaptive expertise in practice. The authors describe a conceptual literature-based model for a Master Adaptive Learner that provides a shared language to facilitate exploration and conversation about both successes and struggles during the learning process.

Applying the institutional review board data repository approach to manage ethical considerations in evaluating and studying medical education.

ISSUE: Medical educators and educational researchers continue to improve their processes for managing medical student and program evaluation data using sound ethical principles. This is becoming even more important as curricular innovations are occurring across undergraduate and graduate medical education. Dissemination of findings from this work is critical, and peer-reviewed journals often require an institutional review board (IRB) determination. APPROACH: IRB data repositories, originally designed for the longitudinal study of biological specimens, can be applied to medical education research. The benefits of such an approach include obtaining expedited review for multiple related studies within a single IRB application and allowing for more flexibility when conducting complex longitudinal studies involving large datasets from multiple data sources and/or institutions. In this paper, we inform educators and educational researchers on our analysis of the use of the IRB data repository approach to manage ethical considerations as part of best practices for amassing, pooling, and sharing data for educational research, evaluation, and improvement purposes. IMPLICATIONS: Fostering multi-institutional studies while following sound ethical principles in the study of medical education is needed, and the IRB data repository approach has many benefits, especially for longitudinal assessment of complex multi-site data.