Validity of the Health Systems Science Examination: Relationship Between Examinee Performance and Time of Training.

The health systems science (HSS) framework articulates systems-relevant topics that medical trainees must learn to be prepared for physician practice. As new HSS-related curricula are developed, measures demonstrating appropriate levels of reliability and validity are needed. The authors describe a collaborative effort between a consortium of medical schools and the National Board of Medical Examiners to create a multiple-choice HSS examination in the areas of evidence-based medicine/population health, patient safety, quality improvement, and teamwork. Fifteen schools administered the 100-question examination through 2 academic years a total of 1887 times to 1837 first-time takers. Total test score mean was 67% (SD 11%). Total test reliability as measured by coefficient α was .83. This examination differentiated between medical students who completed the examination before, during, and after relevant training/instruction. This new HSS examination can support and inform the efforts of institutions as they integrate HSS-related content into their curricula.

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency.

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Longitudinal assessments in continuing specialty certification and lifelong learning.

Longitudinal assessment (LA) involves the regular, spaced delivery of a limited number of questions on practice relevant content on a computer or mobile internet platform. Depending on the platform, participants may indicate relevance of the content to their practice and confidence in their answer prior to receiving immediate feedback (including critiques) on each question. Individual dashboards may be included to assist participants in tracking progress and identifying areas of strength and weaknesss across a content blueprint. This paper provides an overview of the theoretical underpinnings underlying LA programs, briefly describes current uses of LA in medicine and suggests areas for evaluating the role of LA in continuing medical specialty certification and continuing professional development.

The Time Is Now: Using Graduates' Practice Data to Drive Medical Education Reform.

Medical educators are not yet taking full advantage of the publicly available clinical practice data published by federal, state, and local governments, which can be attributed to individual physicians and evaluated in the context of where they attended medical school and residency training. Understanding how graduates fare in actual practice, both in terms of the quality of the care they provide and the clinical challenges they face, can aid educators in taking an evidence-based approach to medical education. Although in their infancy, efforts to link clinical outcomes data to educational process data hold the potential to accelerate medical education research and innovation. This approach will enable unprecedented insight into the long-term impact of each stage of medical education on graduates' future practice. More work is needed to determine best practices, but the barrier to using these public data is low, and the potential for early results is immediate. Using practice data to evaluate medical education programs can transform how the future physician workforce is trained and better align continuously learning medical education and health care systems.

Concerns and Responses for Integrating Health Systems Science Into Medical Education.

With the aim of improving the health of individuals and populations, medical schools are transforming curricula to ensure physician competence encompasses health systems science (HSS), which includes population health, health policy, high-value care, interprofessional teamwork, leadership, quality improvement, and patient safety. Large-scale, meaningful integration remains limited, however, and a major challenge in HSS curricular transformation efforts relates to the receptivity and engagement of students, educators, clinicians, scientists, and health system leaders. The authors identify several widely perceived challenges to integrating HSS into medical school curricula, respond to each concern, and provide potential strategies to address these concerns, based on their experiences designing and integrating HSS curricula. They identify two broad categories of concerns: the (1) relevance and importance of learning HSS-including the perception that there is inadequate urgency for change; HSS education is too complex and should occur in later years; early students would not be able to contribute, and the roles already exist; and the science is too nascent-and (2) logistics and practicality of teaching HSS-including limited curricular time, scarcity of faculty educators with expertise, lack of support from accreditation agencies and licensing boards, and unpreparedness of evolving health care systems to partner with schools with HSS curricula. The authors recommend the initiation and continuation of discussions between educators, clinicians, basic science faculty, health system leaders, and accrediting and regulatory bodies about the goals and priorities of medical education, as well as about the need to collaborate on new methods of education to reach these goals.

Barriers to Medical Students' Electronic Health Record Access Can Impede Their Preparedness for Practice.

Medical students need hands-on experience documenting clinical encounters as well as entering orders to prepare for residency and become competent physicians. In the era of paper medical records, students consistently acquired experience writing notes and entering orders as part of their clinical experience. Over the past decade, however, patient records have transitioned from paper to electronic form. This change has had the unintended consequence of limiting medical students' access to patient records. This restriction has meant that many students leave medical school without the appropriate medical record skills for transitioning to residency.In this article, the authors explore medical students' current access to electronic health records (EHRs) as well as policy proposals from medical societies, innovative models implemented at some U.S. medical schools, and other possible solutions to ensure that students have sufficient experiential learning opportunities with EHRs in clinical settings. They also contend that competence in the use of EHRs is necessary for students to become physicians who can harness the full potential of these tools rather than physicians for whom EHRs hinder excellent patient care. Finally, the authors argue that meaningful experiences using EHRs should be consistently incorporated into medical school curricula and that EHR-related skills should be rigorously assessed with other clinical skills.

How Can Medical Students Add Value? Identifying Roles, Barriers, and Strategies to Advance the Value of Undergraduate Medical Education to Patient Care and the Health System.

PURPOSE: As health systems evolve, the education community is seeking to reimagine student roles that combine learning with meaningful contributions to patient care. The authors sought to identify potential stakeholders regarding the value of student work, and roles and tasks students could perform to add value to the health system, including key barriers and associated strategies to promote value-added roles in undergraduate medical education. METHOD: In 2016, 32 U.S. medical schools in the American Medical Association's (AMA's) Accelerating Change in Education Consortium met for a two-day national meeting to explore value-added medical education; 121 educators, systems leaders, clinical mentors, AMA staff leadership and advisory board members, and medical students were included. A thematic qualitative analysis of workshop discussions and written responses was performed, which extracted key themes. RESULTS: In current clinical roles, students can enhance value by performing detailed patient histories to identify social determinants of health and care barriers, providing evidence-based medicine contributions at the point-of-care, and undertaking health system research projects. Novel value-added roles include students serving as patient navigators/health coaches, care transition facilitators, population health managers, and quality improvement team extenders. Six priority areas for advancing value-added roles are student engagement, skills, and assessments; balance of service versus learning; resources, logistics, and supervision; productivity/billing pressures; current health systems design and culture; and faculty factors. CONCLUSIONS: These findings provide a starting point for collaborative work to positively impact clinical care and medical education through the enhanced integration of value-added medical student roles into care delivery systems.

Fostering the Development of Master Adaptive Learners: A Conceptual Model to Guide Skill Acquisition in Medical Education.

Change is ubiquitous in health care, making continuous adaptation necessary for clinicians to provide the best possible care to their patients. The authors propose that developing the capabilities of a Master Adaptive Learner will provide future physicians with strategies for learning in the health care environment and for managing change more effectively. The concept of a Master Adaptive Learner describes a metacognitive approach to learning based on self-regulation that can foster the development and use of adaptive expertise in practice. The authors describe a conceptual literature-based model for a Master Adaptive Learner that provides a shared language to facilitate exploration and conversation about both successes and struggles during the learning process.

Ensuring Competent Care by Senior Physicians.

The increasing number of senior physicians and calls for increased accountability of the medical profession by the public have led regulators and policymakers to consider implementing age-based competency screening. Some hospitals and health systems have initiated age-based screening, but there is no agreed upon assessment process. Licensing and certifying organizations generally do not require that senior physicians pass additional assessments of health, competency, or quality performance. Studies suggest that physician performance, on average, declines with increasing years in medical practice, but the effect of age on an individual physician's competence is highly variable. Many senior physicians practice effectively and should be allowed to remain in practice as long as quality and safety are not endangered. Stakeholders in the medical profession should consider the need to develop guidelines and methods for monitoring and/or screening to ensure that senior physicians provide safe and effective care for patients. Any screening process needs to achieve a balance between protecting patients from harm due to substandard practice, while at the same time ensuring fairness to physicians and avoiding unnecessary reductions in workforce.

The ABMS MOC Part III Examination: Value, Concerns, and Alternative Formats.

This article describes the presentations and discussions at a conference co-convened by the Council on Medical Education of the American Medical Association (AMA) and by the American Board of Medical Specialties (ABMS). The conference focused on the ABMS Maintenance of Certification (MOC) Part III Examination. This article, reflecting the conference agenda, covers the value of and evidence supporting the examination, as well as concerns about the cost of the examination, and-given the current format-its relevance. In addition, the article outlines alternative formats for the examination that four ABMS member boards are currently developing or implementing. Lastly, the article presents contrasting views on the approach to professional self-regulation. One view operationalizes MOC as a high-stakes, pass-fail process while the other perspective holds MOC as an organized approach to support continuing professional development and improvement. The authors hope to begin a conversation among the AMA, the ABMS, and other professional stakeholders about how knowledge assessment in MOC might align with the MOC program's educational and quality improvement elements and best meet the future needs of both the public and the physician community.

Implementation of competency-based medical education: are we addressing the concerns and challenges?

CONTEXT: Competency-based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time-independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation. OBJECTIVES: Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community. METHODS: The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised. CONCLUSIONS: The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities-based frameworks. Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task-based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work-based assessment.

American Board of Medical Specialties Maintenance of Certification: theory and evidence regarding the current framework.

The American Board of Medical Specialties Maintenance of Certification Program (ABMS MOC) is designed to provide a comprehensive approach to physician lifelong learning, self-assessment, and quality improvement (QI) through its 4-part framework and coverage of the 6 competencies previously adopted by the ABMS and the Accreditation Council for Graduate Medical Education (ACGME). In this article, the theoretical rationale and exemplary empiric data regarding the MOC program and its individual parts are reviewed. The value of each part is considered in relation to 4 criteria about the relationship of the competencies addressed within that part to (1) patient outcomes, (2) physician performance, (3) validity of the assessment or educational methods utilized, and (4) learning or improvement potential. Overall, a sound theoretical rationale and a respectable evidence base exists to support the current structure and elements of the MOC program. However, it is incumbent on the ABMS and ABMS member boards to continue to examine their programs moving forward to assure the public and the profession that they are meeting expectations, are clinically relevant, and provide value to patients and participating physicians, and to refine and improve them as ongoing research indicates.

Validity considerations in the assessment of professionalism.

During the last decade, interest in assessing professionalism in medical education has increased exponentially and has led to the development of many new assessment tools. Efforts to validate the scores produced by tools designed to assess professionalism have lagged well behind the development of these tools. This paper provides a structured framework for collecting evidence to support the validity of assessments of professionalism. The paper begins with a short history of the concept of validity in the context of psychological assessment. It then describes Michael Kane's approach to validity as a structured argument. The majority of the paper then focuses on how Kane's framework can be applied to assessments of professionalism. Examples are provided from the literature, and recommendations for future investigation are made in areas where the literature is deficient.

Commentary: Building the evidence base in support of the American Board of Medical Specialties maintenance of certification program.

In this issue, Lipner and colleagues describe research supporting the value of the examinations used in the maintenance of certification (MOC) programs of the American Board of Internal Medicine and the American Board of Surgery. The authors of this commentary review the contribution of this research and previous investigations that underscore the value of this component of the American Board of Medical Specialties (ABMS) MOC program. In addition, they point out that the MOC examination is one element of a comprehensive approach to physician lifelong learning, assessment, and quality improvement. The ABMS MOC program requires diplomates of the ABMS member boards to engage in continuous professional development in the six domains of competence and performance previously defined by the ABMS and the Accreditation Council for Graduate Medical Education. Although evidence and a sound rationale exist to support educational and assessment methods that target all six domains, it will be important to continue to build the body of evidence demonstrating the value of MOC to the public and to the profession.

Constructing a validity argument for the mini-Clinical Evaluation Exercise: a review of the research.

PURPOSE: The mini-Clinical Evaluation Exercise (mCEX) is increasingly being used to assess the clinical skills of medical trainees. Existing mCEX research has typically focused on isolated aspects of the instrument's reliability and validity. A more thorough validity analysis is necessary to inform use of the mCEX, particularly in light of increased interest in high-stakes applications of the methodology. METHOD: Kane's (2006) validity framework, in which a structured argument is developed to support the intended interpretation(s) of assessment results, was used to evaluate mCEX research published from 1995 to 2009. In this framework, evidence to support the argument is divided into four components (scoring, generalization, extrapolation, and interpretation/decision), each of which relates to different features of the assessment or resulting scores. The strength and limitations of the reviewed research were identified in relation to these components, and the findings were synthesized to highlight overall strengths and weaknesses of existing mCEX research. RESULTS: The scoring component yielded the most concerns relating to the validity of mCEX score interpretations. More research is needed to determine whether scoring-related issues, such as leniency error and high interitem correlations, limit the utility of the mCEX for providing feedback to trainees. Evidence within the generalization and extrapolation components is generally supportive of the validity of mCEX score interpretations. CONCLUSIONS: Careful evaluation of the circumstances of mCEX assessment will help to improve the quality of the resulting information. Future research should address issues of rater selection, training, and monitoring which can impact rating accuracy.

Toward meeting the challenge of physician competence assessment: the University of California, San Diego Physician Assessment and Clinical Education (PACE) Program.

Physician competence and performance problems contribute to medical errors and substandard health care quality. Assessment of the clinical competence of practicing physicians, however, is challenging. Although physician competence assessment undoubtedly does take place at the local level (e.g., hospital, medical group), it is difficult to objectively assess a partner, colleague, or friend. Moreover, the methodologies used and the outcomes are necessarily veiled by peer review statutes. Consequently, there is a need for regional or national assessment centers with the knowledge, skill, and experience to perform clinical competence assessments on individual physicians and provide or direct remediation, when appropriate. The University of California, San Diego (UCSD) Physician Assessment and Clinical Education (PACE) Program was founded at the UCSD School of Medicine in 1996 for this purpose. From inception in 1996 through the first quarter of 2009, 867 physicians have participated in the UCSD PACE Program. The PACE Program is divided into two components. Phase I includes two days of multilevel, multimodal testing, and Phase II is a five-day, preceptor-based formative assessment program taking place in the residency program of the physician's specialty. From July 2002 through December 2005, a study of 298 physician participants of the UCSD PACE Program was conducted.The future of the comprehensive assessment of practicing physicians depends on (1) development and standardization of instruments, techniques, and procedures for measuring competence and performance, including in-practice measures, (2) collaborative networking of assessment programs, (3) cost control, and (4) continued development of remedial measures that correspond to assessment findings.

Assessment of medical professionalism: who, what, when, where, how, and ... why?

Medical professionalism is increasingly recognized as a core competence of medical trainees and practitioners. Although the general and specific domains of professionalism are thoroughly characterized, procedures for assessing them are not well-developed. This article outlines an approach to designing and implementing an assessment program for medical professionalism that begins and ends with asking and answering a series of critical questions about the purpose and nature of the program. The process of exposing an assessment program to a series of interrogatives that comprise an integrated and iterative framework for thinking about the assessment process should lead to continued improvement in the quality and defensibility of that program.