Longitudinal assessments in continuing specialty certification and lifelong learning.

Longitudinal assessment (LA) involves the regular, spaced delivery of a limited number of questions on practice relevant content on a computer or mobile internet platform. Depending on the platform, participants may indicate relevance of the content to their practice and confidence in their answer prior to receiving immediate feedback (including critiques) on each question. Individual dashboards may be included to assist participants in tracking progress and identifying areas of strength and weaknesss across a content blueprint. This paper provides an overview of the theoretical underpinnings underlying LA programs, briefly describes current uses of LA in medicine and suggests areas for evaluating the role of LA in continuing medical specialty certification and continuing professional development.

Implementation of competency-based medical education: are we addressing the concerns and challenges?

CONTEXT: Competency-based medical education (CBME) has emerged as a core strategy to educate and assess the next generation of physicians. Advantages of CBME include: a focus on outcomes and learner achievement; requirements for multifaceted assessment that embraces formative and summative approaches; support of a flexible, time-independent trajectory through the curriculum; and increased accountability to stakeholders with a shared set of expectations and a common language for education, assessment and regulation. OBJECTIVES: Despite the advantages of CBME, numerous concerns and challenges to the implementation of CBME frameworks have been described, including: increased administrative requirements; the need for faculty development; the lack of models for flexible curricula, and inconsistencies in terms and definitions. Additionally, there are concerns about reductionist approaches to assessment in CBME, lack of good assessments for some competencies, and whether CBME frameworks include domains of current importance. This study will outline these issues and discuss the responses of the medical education community. METHODS: The concerns and challenges expressed are primarily categorised as: (i) those related to practical, administrative and logistical challenges in implementing CBME frameworks, and (ii) those with more conceptual or theoretical bases. The responses of the education community to these issues are then summarised. CONCLUSIONS: The education community has begun to address the challenges involved in implementing CBME. Models and guidance exist to inform implementation strategies across the continuum of education, and focus on the more efficient use of resources and technology, and the use of milestones and entrustable professional activities-based frameworks. Inconsistencies in CBME definitions and frameworks remain a significant obstacle. Evolution in assessment approaches from in vitro task-based methods to in vivo integrated approaches is responsive to many of the theoretical and conceptual concerns about CBME, but much work remains to be done to bring rigour and quality to work-based assessment.

Validity considerations in the assessment of professionalism.

During the last decade, interest in assessing professionalism in medical education has increased exponentially and has led to the development of many new assessment tools. Efforts to validate the scores produced by tools designed to assess professionalism have lagged well behind the development of these tools. This paper provides a structured framework for collecting evidence to support the validity of assessments of professionalism. The paper begins with a short history of the concept of validity in the context of psychological assessment. It then describes Michael Kane's approach to validity as a structured argument. The majority of the paper then focuses on how Kane's framework can be applied to assessments of professionalism. Examples are provided from the literature, and recommendations for future investigation are made in areas where the literature is deficient.

Validity of the Health Systems Science Examination: Relationship Between Examinee Performance and Time of Training.

The health systems science (HSS) framework articulates systems-relevant topics that medical trainees must learn to be prepared for physician practice. As new HSS-related curricula are developed, measures demonstrating appropriate levels of reliability and validity are needed. The authors describe a collaborative effort between a consortium of medical schools and the National Board of Medical Examiners to create a multiple-choice HSS examination in the areas of evidence-based medicine/population health, patient safety, quality improvement, and teamwork. Fifteen schools administered the 100-question examination through 2 academic years a total of 1887 times to 1837 first-time takers. Total test score mean was 67% (SD 11%). Total test reliability as measured by coefficient α was .83. This examination differentiated between medical students who completed the examination before, during, and after relevant training/instruction. This new HSS examination can support and inform the efforts of institutions as they integrate HSS-related content into their curricula.

Assessment of medical professionalism: who, what, when, where, how, and ... why?

Medical professionalism is increasingly recognized as a core competence of medical trainees and practitioners. Although the general and specific domains of professionalism are thoroughly characterized, procedures for assessing them are not well-developed. This article outlines an approach to designing and implementing an assessment program for medical professionalism that begins and ends with asking and answering a series of critical questions about the purpose and nature of the program. The process of exposing an assessment program to a series of interrogatives that comprise an integrated and iterative framework for thinking about the assessment process should lead to continued improvement in the quality and defensibility of that program.

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency.

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Making self-assessment more effective.

Self-assessment has been held out as an important mechanism for lifelong learning and self-improvement for health care professionals. However, there is growing concern that individual learners often interpret the results inaccurately. This idea has led to skepticism that self-assessment in its current form can ever be truly useful for lifelong professional development. We examine the proposal that self-assessment can and should be made more effective. First, relevance should be improved. The process should be tied more explicitly to the individual's actual practice profile, rather than being loosely relevant to broader constructs around the permitted scope of practice (eg, certification or licensure). In addition, self-assessment should include not only knowledge and reasoning but also what is done every day in practice, thereby broadening from competence in simulated settings to performance in real settings. Second, the impact of self-assessment should be substantially strengthened by periodic external validation of self-assessment results, together with goals set as a result and plans for further improvement. This offers to the individual the very tangible benefit of satisfying external mandates (eg, licensure and certification). In addition, impact should be reinforced by linking the results of self-assessment to subsequent learning activities including Continuing Medical Education (CME). Although these enhancements individually may not cure all of what ails self-assessment, they might ensure greater effectiveness for the purposes of lifelong learning.

The Time Is Now: Using Graduates' Practice Data to Drive Medical Education Reform.

Medical educators are not yet taking full advantage of the publicly available clinical practice data published by federal, state, and local governments, which can be attributed to individual physicians and evaluated in the context of where they attended medical school and residency training. Understanding how graduates fare in actual practice, both in terms of the quality of the care they provide and the clinical challenges they face, can aid educators in taking an evidence-based approach to medical education. Although in their infancy, efforts to link clinical outcomes data to educational process data hold the potential to accelerate medical education research and innovation. This approach will enable unprecedented insight into the long-term impact of each stage of medical education on graduates' future practice. More work is needed to determine best practices, but the barrier to using these public data is low, and the potential for early results is immediate. Using practice data to evaluate medical education programs can transform how the future physician workforce is trained and better align continuously learning medical education and health care systems.

Barriers to Medical Students' Electronic Health Record Access Can Impede Their Preparedness for Practice.

Medical students need hands-on experience documenting clinical encounters as well as entering orders to prepare for residency and become competent physicians. In the era of paper medical records, students consistently acquired experience writing notes and entering orders as part of their clinical experience. Over the past decade, however, patient records have transitioned from paper to electronic form. This change has had the unintended consequence of limiting medical students' access to patient records. This restriction has meant that many students leave medical school without the appropriate medical record skills for transitioning to residency.In this article, the authors explore medical students' current access to electronic health records (EHRs) as well as policy proposals from medical societies, innovative models implemented at some U.S. medical schools, and other possible solutions to ensure that students have sufficient experiential learning opportunities with EHRs in clinical settings. They also contend that competence in the use of EHRs is necessary for students to become physicians who can harness the full potential of these tools rather than physicians for whom EHRs hinder excellent patient care. Finally, the authors argue that meaningful experiences using EHRs should be consistently incorporated into medical school curricula and that EHR-related skills should be rigorously assessed with other clinical skills.

Concerns and Responses for Integrating Health Systems Science Into Medical Education.

With the aim of improving the health of individuals and populations, medical schools are transforming curricula to ensure physician competence encompasses health systems science (HSS), which includes population health, health policy, high-value care, interprofessional teamwork, leadership, quality improvement, and patient safety. Large-scale, meaningful integration remains limited, however, and a major challenge in HSS curricular transformation efforts relates to the receptivity and engagement of students, educators, clinicians, scientists, and health system leaders. The authors identify several widely perceived challenges to integrating HSS into medical school curricula, respond to each concern, and provide potential strategies to address these concerns, based on their experiences designing and integrating HSS curricula. They identify two broad categories of concerns: the (1) relevance and importance of learning HSS-including the perception that there is inadequate urgency for change; HSS education is too complex and should occur in later years; early students would not be able to contribute, and the roles already exist; and the science is too nascent-and (2) logistics and practicality of teaching HSS-including limited curricular time, scarcity of faculty educators with expertise, lack of support from accreditation agencies and licensing boards, and unpreparedness of evolving health care systems to partner with schools with HSS curricula. The authors recommend the initiation and continuation of discussions between educators, clinicians, basic science faculty, health system leaders, and accrediting and regulatory bodies about the goals and priorities of medical education, as well as about the need to collaborate on new methods of education to reach these goals.

How Can Medical Students Add Value? Identifying Roles, Barriers, and Strategies to Advance the Value of Undergraduate Medical Education to Patient Care and the Health System.

PURPOSE: As health systems evolve, the education community is seeking to reimagine student roles that combine learning with meaningful contributions to patient care. The authors sought to identify potential stakeholders regarding the value of student work, and roles and tasks students could perform to add value to the health system, including key barriers and associated strategies to promote value-added roles in undergraduate medical education. METHOD: In 2016, 32 U.S. medical schools in the American Medical Association's (AMA's) Accelerating Change in Education Consortium met for a two-day national meeting to explore value-added medical education; 121 educators, systems leaders, clinical mentors, AMA staff leadership and advisory board members, and medical students were included. A thematic qualitative analysis of workshop discussions and written responses was performed, which extracted key themes. RESULTS: In current clinical roles, students can enhance value by performing detailed patient histories to identify social determinants of health and care barriers, providing evidence-based medicine contributions at the point-of-care, and undertaking health system research projects. Novel value-added roles include students serving as patient navigators/health coaches, care transition facilitators, population health managers, and quality improvement team extenders. Six priority areas for advancing value-added roles are student engagement, skills, and assessments; balance of service versus learning; resources, logistics, and supervision; productivity/billing pressures; current health systems design and culture; and faculty factors. CONCLUSIONS: These findings provide a starting point for collaborative work to positively impact clinical care and medical education through the enhanced integration of value-added medical student roles into care delivery systems.

Fostering the Development of Master Adaptive Learners: A Conceptual Model to Guide Skill Acquisition in Medical Education.

Change is ubiquitous in health care, making continuous adaptation necessary for clinicians to provide the best possible care to their patients. The authors propose that developing the capabilities of a Master Adaptive Learner will provide future physicians with strategies for learning in the health care environment and for managing change more effectively. The concept of a Master Adaptive Learner describes a metacognitive approach to learning based on self-regulation that can foster the development and use of adaptive expertise in practice. The authors describe a conceptual literature-based model for a Master Adaptive Learner that provides a shared language to facilitate exploration and conversation about both successes and struggles during the learning process.

Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study.

BACKGROUND: Multivariate generalizability analysis was used to investigate the performance of a commonly used clinical evaluation tool. METHOD: Practicing physicians were trained to use the mini-Clinical Skills Examination (CEX) rating form to rate performances from the United States Medical Licensing Examination Step 2 Clinical Skills examination. RESULTS: Differences in rater stringency made the greatest contribution to measurement error; more raters rating each examinee, even on fewer occasions, could enhance score stability. Substantial correlated error across the competencies suggests that decisions about one scale unduly influence those on others. CONCLUSIONS: Given the appearance of a halo effect across competencies, score interpretations that assume assessment of distinct dimensions of clinical performance should be made with caution. If the intention is to produce a single composite score by combining results across competencies, the presence of these effects may be less critical.

Ensuring Competent Care by Senior Physicians.

The increasing number of senior physicians and calls for increased accountability of the medical profession by the public have led regulators and policymakers to consider implementing age-based competency screening. Some hospitals and health systems have initiated age-based screening, but there is no agreed upon assessment process. Licensing and certifying organizations generally do not require that senior physicians pass additional assessments of health, competency, or quality performance. Studies suggest that physician performance, on average, declines with increasing years in medical practice, but the effect of age on an individual physician's competence is highly variable. Many senior physicians practice effectively and should be allowed to remain in practice as long as quality and safety are not endangered. Stakeholders in the medical profession should consider the need to develop guidelines and methods for monitoring and/or screening to ensure that senior physicians provide safe and effective care for patients. Any screening process needs to achieve a balance between protecting patients from harm due to substandard practice, while at the same time ensuring fairness to physicians and avoiding unnecessary reductions in workforce.

The ABMS MOC Part III Examination: Value, Concerns, and Alternative Formats.

This article describes the presentations and discussions at a conference co-convened by the Council on Medical Education of the American Medical Association (AMA) and by the American Board of Medical Specialties (ABMS). The conference focused on the ABMS Maintenance of Certification (MOC) Part III Examination. This article, reflecting the conference agenda, covers the value of and evidence supporting the examination, as well as concerns about the cost of the examination, and-given the current format-its relevance. In addition, the article outlines alternative formats for the examination that four ABMS member boards are currently developing or implementing. Lastly, the article presents contrasting views on the approach to professional self-regulation. One view operationalizes MOC as a high-stakes, pass-fail process while the other perspective holds MOC as an organized approach to support continuing professional development and improvement. The authors hope to begin a conversation among the AMA, the ABMS, and other professional stakeholders about how knowledge assessment in MOC might align with the MOC program's educational and quality improvement elements and best meet the future needs of both the public and the physician community.

Effects of training in direct observation of medical residents' clinical competence: a randomized trial.

BACKGROUND: Faculty observation of residents and students performing clinical skills is essential for reliable and valid evaluation of trainees. OBJECTIVE: To evaluate the efficacy of a new multifaceted method of faculty development called direct observation of competence training. DESIGN: Controlled trial of faculty from 16 internal medicine residency programs using a cluster randomization design. SETTING: Academic medical centers. PARTICIPANTS: 40 internal medicine teaching faculty members: 17 in the intervention group and 23 in the control group. MEASUREMENTS: Changes in faculty comfort performing direct observation, faculty satisfaction with workshop, and changes in faculty rating behaviors 8 months after completing the training. INTERVENTION: The direct observation of competence workshop combines didactic mini-lectures, interactive small group and videotape evaluation exercises, and evaluation skill practice with standardized residents and patients. RESULTS: 37 faculty members (16 in the intervention group and 21 in the control group) completed the study. Most of the faculty in the intervention group (14 [88%]) reported that they felt significantly more comfortable performing direct observation compared with control group faculty (4 [19%]) (P = 0.04), and all intervention faculty rated the training as outstanding. For 9 videotaped clinical encounters, intervention group faculty were more stringent than controls in their evaluations of medical interviewing, physical examination, and counseling; differences in ratings for medical interviewing and physical examination remained statistically significant even after adjustment for baseline rating behavior. LIMITATIONS: The study involved a limited number of residency programs, and faculty did not rate the performance of actual residents. CONCLUSION: Direct observation of competence training, a new multifaceted approach to faculty development, leads to meaningful changes in rating behaviors and in faculty comfort with evaluation of clinical skills.

The impact of postgraduate training and timing on USMLE Step 3 performance.

PURPOSE: This study examined the extent to which differences in clinical experience, gained in postgraduate training programs, affect performance on Step 3 of the United States Medical Licensing Examination (USMLE). METHOD: Subjects in the study were 36,805 U.S. and Canadian medical school graduates who took USMLE Step 3 for the first time between November 1999 and December 2002. Regression analyses examined the relation between length and type of postgraduate training and Step 3 score after controlling for prior performance on previous USMLE examinations. RESULTS: Results indicate that postgraduate training in programs that provide exposure to a broad range of patient problems, and continued training in such areas, improves performance on Step 3. CONCLUSIONS: Study data reaffirm the validity of the USMLE Step 3 examination, and the information found in the pattern of results across specialties may be useful to residents and program directors.