The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Advancing anti-oppression and social justice in healthcare through competency-based medical education (CBME).

Competency-based medical education (CBME) focuses on preparing physicians to improve the health of patients and populations. In the context of ongoing health disparities worldwide, medical educators must implement CBME in ways that advance social justice and anti-oppression. In this article, authors describe how CBME can be implemented to promote equity pedagogy, an approach to education in which curricular design, teaching, assessment strategies, and learning environments support learners from diverse groups to be successful. The five core components of CBME programs - outcomes competency framework, progressive sequencing of competencies, learning experiences tailored to learners' needs, teaching focused on competencies, and programmatic assessment - enable individualization of learning experiences and teaching and encourage learners to partner with their teachers in driving their learning. These educational approaches appreciate each learner's background, experiences, and strengths. Using an exemplar case study, the authors illustrate how CBME can afford opportunities to enhance anti-oppression and social justice in medical education and promote each learner's success in meeting the expected outcomes of training. The authors provide recommendations for individuals and institutions implementing CBME to enact equity pedagogy.

The Associations Between United States Medical Licensing Examination Performance and Outcomes of Patient Care.

PURPOSE: The United States Medical Licensing Examination (USMLE) comprises a series of assessments required for the licensure of U.S. MD-trained graduates as well as those who are trained internationally. Demonstration of a relationship between these examinations and outcomes of care is desirable for a process seeking to provide patients with safe and effective health care. METHOD: This was a retrospective cohort study of 196,881 hospitalizations in Pennsylvania over a 3-year period (January 1, 2017 to December 31, 2019) for 5 primary diagnoses: heart failure, acute myocardial infarction, stroke, pneumonia, or chronic obstructive pulmonary disease. The 1,765 attending physicians for these hospitalizations self-identified as family physicians or general internists. A converted score based on USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores was available, and the outcome measures were in-hospital mortality and log length of stay (LOS). The research team controlled for characteristics of patients, hospitals, and physicians. RESULTS: For in-hospital mortality, the adjusted odds ratio was 0.94 (95% confidence interval [CI] = 0.90, 0.99; P < .02). Each standard deviation increase in the converted score was associated with a 5.51% reduction in the odds of in-hospital mortality. For log LOS, the adjusted estimate was 0.99 (95% CI = 0.98, 0.99; P < .001). Each standard deviation increase in the converted score was associated with a 1.34% reduction in log LOS. CONCLUSIONS: Better provider USMLE performance was associated with lower in-hospital mortality and shorter log LOS for patients, although the magnitude of the latter is unlikely to be of practical significance. These findings add to the body of evidence that examines the validity of the USMLE licensure program.

The Undergraduate to Graduate Medical Education Transition as a Systems Problem: A Root Cause Analysis.

The transition from undergraduate medical education (UME) to graduate medical education (GME) constitutes a complex system with important implications for learner progression and patient safety. The transition is currently dysfunctional, requiring students and residency programs to spend significant time, money, and energy on the process. Applications and interviews continue to increase despite stable match rates. Although many in the medical community acknowledge the problems with the UME-GME transition and learners have called for prompt action to address these concerns, the underlying causes are complex and have defied easy fixes. This article describes the work of the Coalition for Physician Accountability's Undergraduate Medical Education to Graduate Medical Education Review Committee (UGRC) to apply a quality improvement approach and systems thinking to explore the underlying causes of dysfunction in the UME-GME transition. The UGRC performed a root cause analysis using the 5 whys and an Ishikawa (or fishbone) diagram to deeply explore problems in the UME-GME transition. The root causes of problems identified include culture, costs and limited resources, bias, systems, lack of standards, and lack of alignment. Using the principles of systems thinking (components, connections, and purpose), the UGRC considered interactions among the root causes and developed recommendations to improve the UME-GME transition. Several of the UGRC's recommendations stemming from this work are explained. Sustained monitoring will be necessary to ensure interventions move the process forward to better serve applicants, programs, and the public good.

"Cephalgia" or "migraine"? Solving the headache of assessing clinical reasoning using natural language processing.

In this op-ed, we discuss the advantages of leveraging natural language processing (NLP) in the assessment of clinical reasoning. Clinical reasoning is a complex competency that cannot be easily assessed using multiple-choice questions. Constructed-response assessments can more directly measure important aspects of a learner's clinical reasoning ability, but substantial resources are necessary for their use. We provide an overview of INCITE, the Intelligent Clinical Text Evaluator, a scalable NLP-based computer-assisted scoring system that was developed to measure clinical reasoning ability as assessed in the written documentation portion of the now-discontinued USMLE Step 2 Clinical Skills examination. We provide the rationale for building a computer-assisted scoring system that is aligned with the intended use of an assessment. We show how INCITE's NLP pipeline was designed with transparency and interpretability in mind, so that every score produced by the computer-assisted system could be traced back to the text segment it evaluated. We next suggest that, as a consequence of INCITE's transparency and interpretability features, the system may easily be repurposed for formative assessment of clinical reasoning. Finally, we provide the reader with the resources to consider in building their own NLP-based assessment tools.

Blue Skies With Clouds: Envisioning the Future Ideal State and Identifying Ongoing Tensions in the UME-GME Transition.

The transition from medical school to residency in the United States consumes large amounts of time for students and educators in undergraduate and graduate medical education (UME, GME), and it is costly for both students and institutions. Attempts to improve the residency application and Match processes have been insufficient to counteract the very large number of applications to programs. To address these challenges, the Coalition for Physician Accountability charged the Undergraduate Medical Education to Graduate Medical Education Review Committee (UGRC) with crafting recommendations to improve the system for the UME-GME transition. To guide this work, the UGRC defined and sought stakeholder input on a "blue-skies" ideal state of this transition. The ideal state views the transition as a system to support a continuum of professional development and learning, thus serving learners, educators, and the public, and engendering trust among them. It also supports the well-being of learners and educators, promotes diversity, and minimizes bias. This manuscript uses polarity thinking to analyze 3 persistent key tensions in the system that require ongoing management. First, the formative purpose of assessment for learning and growth is at odds with the use of assessment data for ranking and sorting candidates. Second, the function of residents as learners can conflict with their role as workers contributing service to health care systems. Third, the current residency Match process can position the desire for individual choice-among students and their programs-against the workforce needs of the profession and the public. This Scholarly Perspective presents strategies to balance the upsides and downsides inherent to these tensions. By articulating the ideal state of the UME-GME transition and anticipating tensions, educators and educational organizations can be better positioned to implement UGRC recommendations to improve the transition system.

An Examination of the Associations Among USMLE Step 3 Scores and the Likelihood of Disciplinary Action in Practice.

PURPOSE: As the last examination in the United States Medical Licensing Examination (USMLE) sequence, Step 3 provides a safeguard before physicians enter into unsupervised practice. There is, however, little validity research focusing on Step 3 scores beyond examining its associations with other educational and professional assessments thought to cover similar content. This study examines the associations between Step 3 scores and subsequent receipt of disciplinary action taken by state medical boards for problematic behavior in practice. It analyzes Step 3 total, Step 3 computer-based case simulation (CCS), and Step 3 multiple-choice question (MCQ) scores. METHOD: The final sample included 275,392 board-certified physicians who graduated from MD-granting medical schools and who passed Step 3 between 2000 and 2017. Cross-classified multilevel logistic regression models were used to examine the effects of Step 3 scores on the likelihood of receiving a disciplinary action, controlling for other USMLE scores and accounting for jurisdiction and specialty. RESULTS: Results showed that physicians with higher Step 3 total, CCS, and MCQ scores tended to have lower chances of receiving a disciplinary action, after accounting for other USMLE scores. Specifically, a 1-standard-deviation increase in Step 3 total, CCS, and MCQ score was associated with a 23%, 11%, and 17% decrease in the odds of receiving a disciplinary action, respectively. The effect of Step 2 CK score on the likelihood of receiving a disciplinary action was statistically significant, while the effect of Step 1 score became statistically nonsignificant when other Step scores were included in the analysis. CONCLUSIONS: Physicians who perform better on Step 3 are less likely to receive a disciplinary action from a state medical board for problematic behavior in practice. These findings provide some validity evidence for the use of Step 3 scores when making medical licensure decisions in the United States.

Do programme coordinators contribute to the professional development of residents? an exploratory study.

BACKGROUND: With the development of training programmes for health professions, the role of programme coordinators has become increasingly important. However, their role in providing educational support for the professional development of resident trainees has not been investigated well. This study aimed to qualitatively analyse the involvement of programme coordinators in educational support for residents. METHODS: Semi-structured reflective writing on 'support for residents' was collected from programme coordinators in teaching hospitals in Japan in 2017-18 using a web-based questionnaire. Descriptions were qualitatively analysed thematically, using the professional identity formation (PIF) framework. RESULTS: A total of 39 cases of "support for residents" by 31 coordinators were analysed. We found that residents most commonly faced prior personal problems, including mental health issues and insufficient social skills/unprofessional behaviour. A thematic analysis revealed that coordinators played a variety of educational roles: 1) requesting supervisors to reconsider their teaching; 2) protecting residents from the negative influence of clinical experiences; 3) facilitating residents' self-assessment and confidence; 4) creating a safer learning environment; 5) providing support for prior personal problems through 5-1) fostering a better atmosphere for the mental health of residents, and 5-2) intervening for residents with insufficient social skills/unprofessional behaviour; 6) providing support for isolated residents; and 7) preventing problems with peers. CONCLUSIONS: This study identified seven educational roles of programme coordinators for residents from a standpoint of PIF of residents. Based on these findings, four valuable attributes for coordinators were established: non-hierarchical relationships with residents, parenting attitudes, sensitivity to residents' changes, and the perspective of the citizen and a member of the public. These attributes would underpin coordinators' educational roles and facilitate the professional development of residents. This study provides a basis for defining and revising the role profiles of programme coordinators, and for improving staff development.

Revisiting Retake Policy: Analyzing the Success Rates of Examinees With Multiple Attempts on the United States Medical Licensing Examination.

PURPOSE: A testing program's decision making on retake policy considers a number of factors, including fairness to examinees, examination security, examination purpose, and classification accuracy. For high-stakes licensure and certification examinations charged with protection of the public, this includes balancing fairness issues inherent in the potential for false negatives with the public protection need to minimize false positives. Since 2012, the United States Medical Licensing Examination (USMLE) has maintained a policy of ≤ 6 attempts on any examination component. The purpose of this study was to empirically examine the appropriateness of existing USMLE retake policy. METHOD: A sample of over 300,000 examinees who were administered each Step examination between 2008 and 2018 was examined to address how pass rates (per attempt and cumulatively) differed for examinees and examinee subgroups via graphic depictions of pass rates, the impact of restricting the number of attempts on examinees and examinee subgroups via logistic regression analyses, and the impact of restricting the number of attempts on classification error via classification error analyses. RESULTS: Repeaters displayed consistently lower pass rates on subsequent attempts. Regardless of Step, most passing examinees did so by their fourth attempt. Models predicting the log odds of examinees taking ≥ 5 attempts showed a significant effect of gender, race, and medical school status, although the associated sample sizes are small. Misclassification analyses showed that, as attempts increased, the percentage of false passers increased and percentage of false failers decreased. CONCLUSIONS: These considerations led to the USMLE's decision to reduce its attempt limit from 6 to 4 on any Step or Step component effective July 2021. This change balances the competing interests of fairness and examination access to the examinee population, while simultaneously maintaining a minimum standard consistent with the mission of the USMLE program to protect the public.

Differential Item Functioning Analysis of United States Medical Licensing Examination Step 1 Items.

PURPOSE: Previous studies have examined and identified demographic group score differences on United States Medical Licensing Examination (USMLE) Step examinations. It is necessary to explore potential etiologies of such differences to ensure fairness of examination use. Although score differences are largely explained by preceding academic variables, one potential concern is that item-level bias may be associated with remaining group score differences. The purpose of this 2019-2020 study was to statistically identify and qualitatively review USMLE Step 1 exam questions (items) using differential item functioning (DIF) methodology. METHOD: Logistic regression DIF was used to identify and classify the effect size of DIF on Step 1 items meeting minimum sample size criteria. After using DIF to flag items statistically, subject matter expert (SME) review was used to identify potential reasons why items may have performed differently between racial and gender groups, including characteristics such as content, format, wording, context, or stimulus materials. USMLE SMEs reviewed items to identify the group difference they believed was present, if any; articulate a rationale behind the group difference; and determine whether that rationale would be considered construct relevant or construct irrelevant. RESULTS: All identified DIF rationales were relevant to the constructs being assessed and therefore did not reflect item bias. Where SME-generated rationales aligned with statistical differences (flags), they favored self-identified women on items tagged to women's health content categories and were judged to be construct relevant. CONCLUSIONS: This study did not find evidence to support the hypothesis that group-level performance differences beyond those explained by prior academic performance variables are driven by item-level bias. Health professions examination programs have an obligation to assess for group differences, and when present, investigate to what extent, if any, measurement bias plays a role.

Exploring the Association Between USMLE Scores and ACGME Milestone Ratings: A Validity Study Using National Data From Emergency Medicine.

PURPOSE: The United States Medical Licensing Examination (USMLE) sequence and the Accreditation Council for Graduate Medical Education (ACGME) milestones represent 2 major components along the continuum of assessment from undergraduate through graduate medical education. This study examines associations between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and ACGME emergency medicine (EM) milestone ratings. METHOD: In February 2019, subject matter experts (SMEs) provided judgments of expected associations for each combination of Step examination and EM subcompetency. The resulting sets of subcompetencies with expected strong and weak associations were selected for convergent and discriminant validity analysis, respectively. National-level data for 2013-2018 were provided; the final sample included 6,618 EM residents from 158 training programs. Empirical bivariate correlations between milestone ratings and Step scores were calculated, then those correlations were compared with the SMEs' judgments. Multilevel regression analyses were conducted on the selected subcompetencies, in which milestone ratings were the dependent variable, and Step 1 score, Step 2 CK score, and cohort year were independent variables. RESULTS: Regression results showed small but statistically significant positive relationships between Step 2 CK score and the subcompetencies (regression coefficients ranged from 0.02 [95% confidence interval (CI), 0.01-0.03] to 0.12 [95% CI, 0.11-0.13]; all P < .05), with the degree of association matching the SMEs' judgments for 7 of the 9 selected subcompetencies. For example, a 1 standard deviation increase in Step 2 CK score predicted a 0.12 increase in MK-01 milestone rating, when controlling for Step 1. Step 1 score showed a small statistically significant effect with only the MK-01 subcompetency (regression coefficient = 0.06 [95% CI, 0.05-0.07], P < .05). CONCLUSIONS: These results provide incremental validity evidence in support of Step 1 and Step 2 CK score and EM milestone rating uses.

Multiple United States Medical Licensing Examination Attempts and the Estimated Risk of Disciplinary Actions Among Graduates of U.S. and Canadian Medical Schools.

PURPOSE: The United States Medical Licensing Examination (USMLE) recently announced 2 policy changes: shifting from numeric score reporting on the Step 1 examination to pass/fail reporting and limiting examinees to 4 attempts for each Step component. In light of these policies, exam measures other than scores, such as the number of examination attempts, are of interest. Attempt limit policies are intended to ensure minimum standards of physician competency, yet little research has explored how Step attempts relate to physician practice outcomes. This study examined the relationship between USMLE attempts and the likelihood of receiving disciplinary actions from state medical boards. METHOD: The sample population was 219,018 graduates from U.S. and Canadian MD-granting medical schools who passed all USMLE Step examinations by 2011 and obtained a medical license in the United States, using data from the NBME and the Federation of State Medical Boards. Logistic regressions estimated how attempts on Steps 1, 2 Clinical Knowledge (CK), and 3 examinations influenced the likelihood of receiving disciplinary actions by 2018, while accounting for physician characteristics. RESULTS: A total of 3,399 physicians (2%) received at least 1 disciplinary action. Additional attempts needed to pass Steps 1, 2 CK, and 3 were associated with an increased likelihood of receiving disciplinary actions (odds ratio [OR]: 1.07, 95% confidence interval [CI]: 1.01, 1.13; OR: 1.09, 95% CI: 1.03, 1.16; OR: 1.11, 95% CI: 1.04, 1.17, respectively), after accounting for other factors. CONCLUSIONS: Physicians who took multiple attempts to pass Steps 1, 2 CK, and 3 were associated with higher estimated likelihood of receiving disciplinary actions. This study offers support for licensure and practice standards to account for physicians' USMLE attempts. The relatively small effect sizes, however, caution policy makers from placing sole emphasis on this relationship.

Entrustable Professional Activity-Based Assessments in Undergraduate Medical Education: A Survey of Pediatric Educators.

OBJECTIVE: To explore the use of entrustable professional activities (EPAs) in undergraduate medical education (UME), characterization of EPAs by pediatric educators, and opportunities and challenges with an EPA framework. METHODS: In 2020, 9 survey questions were administered to members of the Council on Medical Student Education in Pediatrics, a national pediatric UME group. Clark's Commitment and Necessary Effort model on motivation served as the theoretical framework for our study. Quantitative and qualitative data were analyzed using descriptive statistics and conventional content analysis, respectively. RESULTS: One hundred and sixty-seven (31%) of 479 recipients, representing 75% of accredited schools responded. Eigty-three percent agreed that they understood what EPAs were, yet a minority reported using EPAs. Eighty-five percent of EPA users expressed satisfaction with EPAs in providing a shared framework and an opportunity to track student competence; dissatisfaction was expressed toward faculty resource needs. Among nonusers, barriers hindering implementation included faculty development challenges and faculty time. Qualitative analyses revealed a rich understanding of EPAs: that they offer a framework to measure learner competence by assessing performance in workplace-based tasks that can be used for entrustment decisions and for program evaluation. CONCLUSIONS: Although most pediatric UME educators report understanding EPAs (contributing to self-efficacy) and recognize their benefits (value), a minority report actually using EPAs. EPAs, while providing a valuable framework, pose challenges from contextual factors affecting personal agency, which could affect educator commitment in implementation. For more widespread adoption of the EPAs, efforts should focus on minimizing these perceived barriers.

Otitis Media Module in the Pediatric Preclerkship Educational Exercises (PRECEDE) Curriculum.

Introduction: The Johns Hopkins Pediatrics Clerkship developed the PRECEDE (preclerkship educational exercises) curriculum with the primary goal of offering students formative instruction in essential pediatric clinical skills to prepare them for their clerkship. PRECEDE sessions occur at the beginning of each basic clerkship for new clinical clerkship students. The otitis media module is one in a series of modules presented in the curriculum and consists of a lecture and four short skills-development stations, each with a faculty facilitator. Methods: This 2-hour module began with a 1-hour didactic overview of otitis media. Medical students were divided into three groups. One group learned about writing prescriptions via two otitis media clinical vignettes. Another group explored visualization and diagnosis of otitis media via video. The last student group was subdivided and learned proper techniques for positioning and restraining pediatric patients during otoscopic exams and the psychomotor skills for performing otoscopic examinations, including pneumatic otoscopy. Student groups rotated through all four activity stations. Students were guided through discussion to develop interpretation, diagnostic, and treatment skills for acute otitis media. Results: Between 2010 and 2012, 254 third- and fourth-year medical students participated in this module. When asked to evaluate overall quality, 86% of learners rated the module as excellent, and 14% rated it as good. Discussion: By establishing these important skills, students may be better equipped to develop appropriate otitis media assessments, diagnoses, and care plans for patients and to use otitis media as a platform for broad education in other essential pediatric skills.

Effects of Moving the United States Medical Licensing Examination Step 1 After Core Clerkships on Step 2 Clinical Knowledge Performance.

PURPOSE: To investigate the effect of a change in the United States Medical Licensing Examination Step 1 timing on Step 2 Clinical Knowledge (CK) scores, the effect of lag time on Step 2 CK performance, and the relationship of incoming Medical College Admission Test (MCAT) score to Step 2 CK performance pre and post change. METHOD: Four schools that moved Step 1 after core clerkships between academic years 2008-2009 and 2017-2018 were analyzed. Standard t tests were used to examine the change in Step 2 CK scores pre and post change. Tests of differences in proportions were used to evaluate whether Step 2 CK failure rates differed between curricular change groups. Linear regressions were used to examine the relationships between Step 2 CK performance, lag time and incoming MCAT score, and curricular change group. RESULTS: Step 2 CK performance did not change significantly (P = .20). Failure rates remained highly consistent (pre change: 1.83%; post change: 1.79%). The regression indicated that lag time had a significant effect on Step 2 CK performance, with scores declining with increasing lag time, with small but significant interaction effects between MCAT and Step 2 CK scores. Students with lower incoming MCAT scores tended to perform better on Step 2 CK when Step 1 was after clerkships. CONCLUSIONS: Moving Step 1 after core clerkships appears to have had no significant impact on Step 2 CK scores or failure rates, supporting the argument that such a change is noninferior to the traditional model. Students with lower MCAT scores benefit most from the change.