Recommendations to address and research systemic bias in assessment: perspectives from directors of research in medical education.

INTRODUCTION: Addressing systemic bias in medical school assessment is an urgent task for medical education. This paper outlines recommendations on topic areas for further research on systemic bias, developed from a workshop discussion at the 2023 annual meeting of the Society of Directors of Research in Medical Education. MATERIALS AND METHODS: During the workshop, directors engaged in small-group discussions on guidelines to address bias in assessment practices following a proposed categorization of 'Do's,' 'Don'ts,' and 'Don't knows' and listed their insights using anonymous sticky notes, which were shared and discussed with the larger group of participants. The authors performed a content analysis of the notes through deductive and inductive coding. We reviewed and discussed our analysis to reach consensus. RESULTS: The workshop included 31 participants from 28 institutions across the US and Canada, generating 51 unique notes. Participants identified 23 research areas in need of further study. The inductive analysis of proposed research areas revealed four main topics: 1) The role of interventions, including pre-medical academic interventions, medical-education interventions, assessment approaches, and wellness interventions; 2) Professional development, including the definition and assessment of professionalism and professional identity formation; 3) Context, including patient care and systemic influences; and 4) Research approaches. DISCUSSION: While limited to data from a single workshop, the results offered perspectives about areas for further research shared by a group of directors of medical education research units from diverse backgrounds. The workshop produced valuable insights into the need for more evidence-based interventions that promote more equitable assessment practices grounded in real-world situations and that attenuate the effects of bias.

Deliberate Practice as an Effective Method for Reducing Diagnostic Error in Identifying Burn and Bruise Injuries Suspicious for an Abusive Injury.

OBJECTIVE: To examine the effectiveness of an education intervention for reducing physician diagnostic error in identifying pediatric burn and bruise injuries suspicious for abuse, and to determine case-specific variables associated with an increased risk of diagnostic error. STUDY DESIGN: This was a multicenter, prospective, cross-sectional study. A convenience sample of pediatricians and other front-line physicians who treat acutely injured children in the United States and Canada were eligible for participation. Using a web-based education and assessment platform, physicians deliberately practiced with a spectrum of 300 pediatric burn and bruise injury image-based cases. Participants were asked if there was a suspicion for abuse present or absent, were given corrective feedback after every case, and received summative diagnostic performance overall (accuracy), suspicion for abuse present (sensitivity), and absent (specificity). RESULTS: Of the 93/137 (67.9%) physicians who completed all 300 cases, there was a significant reduction in diagnostic error (initial 16.7%, final 1.6%; delta -15.1%; 95% CI -13.5, -16.7), sensitivity error (initial 11.9%, final 0.7%; delta -11.2%; 95% CI -9.8, -12.5), and specificity error (initial 23.3%, final 6.6%; delta -16.7%; 95% CI -14.8, -18.6). Based on 35 627 case interpretations, variables associated with diagnostic error included patient age, sex, skin color, mechanism of injury, and size and pattern of injury. CONCLUSIONS: The education intervention substantially reduced diagnostic error in differentiating the presence vs absence of a suspicion for abuse in children with burn and bruise injuries. Several case-based variables were associated with diagnostic error, and these data can be used to close specific skill gaps in this clinical domain.

Precision medicine within health professions education: Defining a research agenda for emergency medicine using a foresight and strategy technique (FaST) review.

Background: Precision medicine, sometimes referred to as personalized medicine, is rapidly changing the possibilities for how people will engage health care in the near future. As technology to support precision medicine exponentially develops, there is an urgent need to proactively improve our understanding of precision medicine and pose important research questions (RQs) related to its inclusion in the education and training of future emergency physicians. Methods: A seven-step process was employed to develop a research agenda exploring the intersection of precision and emergency medicine education/training. A literature search of articles about precision medicine was conducted first, which informed the creation of future four scenarios in which trainees and practicing physicians regularly discuss and incorporate precision medicine tools into their discussions and work. Based on these futurist narratives, potential education RQs were generated by an expert panel. A total of 59 initial questions were subsequently categorized and refined to a priority list through a nominal group voting method. The top/priority questions were presented at the 2023 SAEM Consensus Conference on Precision Medicine, Austin, Texas, for further input. Results: Eight high-value education RQs were developed, reflecting a holistic view of the challenges and opportunities for precision medicine education in the knowledge, skills, and attitudes relevant to emergency medicine. These questions contend with topics such as most effective pedagogical methods; intended resulting outcomes and behaviors; the generational differences between practicing emergency physicians, educators, and future trainees; and the desires and expectations of patients. Conclusions: Emergency medicine and emergency physicians must be prepared to understand precision medicine and incorporate this information into their "toolbox" of thinking, problem solving, and communication with patients and colleagues. This research agenda on how best to educate future emergency physicians in the use of personalized data to provide optimal health care is the focus of this article.

Determination, categorization, and hierarchy of content for a pediatric emergency medicine curriculum designed for emergency medicine residents.

Background: Currently, the Accreditation Council of Graduate Medical Education requires time-based pediatric experiences for emergency medicine (EM) residents in both pediatric emergency medicine (PEM) and critical care settings. The American Board of Emergency Medicine has published the Model of the Clinical Practice of Emergency Medicine, which is a list of content an EM resident should learn. However, this list is large and without prioritization and therefore can be difficult to incorporate into time-limited curricula. Objectives: The primary objective of this study was to develop comprehensive categorization of PEM content using an EM lens. The second objective was to suggest a prioritization for the EM learner of the enumerated PEM elements. Methods: We first assembled a comprehensive list of PEM concepts, diagnoses, and procedures that might be taught to EM residents. We then convened focus groups composed of key stakeholders to help formulate content and concept themes important for EM resident training. Once the themes were identified, we divided the list of PEM topics into appropriate themes and then carried out a second round of focus groups expanded to include more diverse expert input for prioritizing the elements of the comprehensive list within each theme. Results: We prioritized 168 important PEM concepts from previous standards and emerging PEM literature among 10 identified themes: the pediatric normal, the bottom-line boil-it-down approach, common presentations, high-acuity pediatric cases and procedures, differences between children and adults, same between children and adults, red flags, infrequency of caring for a child compared with an adult, keep breadth but promote self-directed depth, and triage and disposition. Conclusions: Based on input from stakeholders in EM resident education, we identified key themes within PEM education and created a framework for the hierarchical categorization of PEM content for within an EM residency.

How to Use and Report on p-values.

The use of the p-value in quantitative research, particularly its threshold of "P < 0.05" for determining "statistical significance," has long been a cornerstone of statistical analysis in research. However, this standard has been increasingly scrutinized for its potential to mislead findings, especially when the practical significance, the number of comparisons, or the suitability of statistical tests are not properly considered. In response to controversy around use of p-values, the American Statistical Association published a statement in 2016 that challenged the research community to abandon the term "statistically significant". This stance has been echoed by leading scientific journals to urge a significant reduction or complete elimination in the reliance on p-values when reporting results. To provide guidance to researchers in health professions education, this paper provides a succinct overview of the ongoing debate regarding the use of p-values and the definition of p-values. It reflects on the controversy by highlighting the common pitfalls associated with p-value interpretation and usage, such as misinterpretation, overemphasis, and false dichotomization between "significant" and "non-significant" results. This paper also outlines specific recommendations for the effective use of p-values in statistical reporting including the importance of reporting effect sizes, confidence intervals, the null hypothesis, and conducting sensitivity analyses for appropriate interpretation. These considerations aim to guide researchers toward a more nuanced and informative use of p-values.

The Association of Master Adaptive Learning With Less Burnout and More Resilience in Obstetrics and Gynecology Residents.

PURPOSE: Medical education should prepare learners for complex and evolving work, and should ideally include the Master Adaptive Learner (MAL) model-meta-learning skills for continuous self-regulated learning. This study aimed to measure obstetrics and gynecology (OB/GYN) residents' MAL attributes, assess associations with burnout and resilience, and explore learning task associations with MAL. METHOD: OB/GYN residents were surveyed electronically at an in-training examination in January 2022. The survey included demographic information, the 2-item Maslach Burnout Inventory, the 2-item Connor-Davidson Resilience Scale, 4 MAL items (e.g., "I take every opportunity to learn new things"), and questions about training and learning experiences. RESULTS: Of 5,761 residents, 3,741 respondents (65%) were included. A total of 1,478 of 3,386 (39%) demonstrated burnout (responded positive for burnout on emotional exhaustion or depersonalization items). The mean (SD) Connor-Davidson Resilience Scale score was 6.4 (1.2) of a total possible score of 8. The mean (SD) MAL score was 16.3 (2.8) of a total possible score of 20. The MAL score was inversely associated with burnout, with lower MAL scores for residents with (mean [SD] MAL score, 16.5 [2.4]) vs without (mean [SD], 16.0 [2.3]) burnout (P < .001). Higher MAL scores were associated with higher resilience (R = 0.29, P < .001). Higher MAL scores were associated with the statement, "I feel that I was well prepared for my first year of residency" (R = 0.19, P < .001) and a plan to complete subspecialty training after residency (mean [SD] of 16.6 [2.4] for "yes" and 16.2 [2.4] for "no," P < .001). CONCLUSIONS: Residents who scored higher on MAL showed more resilience and less burnout. Whether less resilient, burned-out residents did not have the agency to achieve MAL status or whether MAL behaviors filled the resiliency reservoir and protected against burnout is not clear.

Data sharing and big data in health professions education: Ottawa consensus statement and recommendations for scholarship.

Changes in digital technology, increasing volume of data collection, and advances in methods have the potential to unleash the value of big data generated through the education of health professionals. Coupled with this potential are legitimate concerns about how data can be used or misused in ways that limit autonomy, equity, or harm stakeholders. This consensus statement is intended to address these issues by foregrounding the ethical imperatives for engaging with big data as well as the potential risks and challenges. Recognizing the wide and ever evolving scope of big data scholarship, we focus on foundational issues for framing and engaging in research. We ground our recommendations in the context of big data created through data sharing across and within the stages of the continuum of the education and training of health professionals. Ultimately, the goal of this statement is to support a culture of trust and quality for big data research to deliver on its promises for health professions education (HPE) and the health of society. Based on expert consensus and review of the literature, we report 19 recommendations in (1) framing scholarship and research through research, (2) considering unique ethical practices, (3) governance of data sharing collaborations that engage stakeholders, (4) data sharing processes best practices, (5) the importance of knowledge translation, and (6) advancing the quality of scholarship through multidisciplinary collaboration. The recommendations were modified and refined based on feedback from the 2022 Ottawa Conference attendees and subsequent public engagement. Adoption of these recommendations can help HPE scholars share data ethically and engage in high impact big data scholarship, which in turn can help the field meet the ultimate goal: high-quality education that leads to high-quality healthcare.

Promotion in Place: A Model for Competency-Based, Time-Variable Graduate Medical Education.

PROBLEM: Competency-based medical education is increasingly regarded as a preferred framework for physician training, but implementation is limited. U.S. residency programs remain largely time based, with variable assessments and limited opportunities for individualization. Gaps in graduates' readiness for unsupervised care have been noted across specialties. Logistical barriers and regulatory requirements constrain movement toward competency-based, time-variable (CBTV) graduate medical education (GME), despite its theoretical benefits. APPROACH: The authors describe a vision for CBTV-GME and an implementation model that can be applied across specialties. Termed "Promotion in Place" (PIP), the model relies on enhanced assessment, clear criteria for advancement, and flexibility to adjust individuals' responsibilities and time in training based on demonstrated competence. PIP allows a resident's graduation to be advanced or delayed accordingly. Residents deemed competent for early graduation can transition to attending physician status within their training institution and benefit from a period of "sheltered independence" until the standard graduation date. Residents who need extended time to achieve competency have graduation delayed to incorporate additional targeted education. OUTCOMES: A proposal to pilot the PIP model of CBTV-GME received funding through the American Medical Association's "Reimagining Residency" initiative in 2019. Ten of 46 residency programs in a multihospital system expressed interest and pursued initial planning. Seven programs withdrew for reasons including program director transitions, uncertainty about resident reactions, and the COVID-19 pandemic. Three programs petitioned their specialty boards for exemptions from time-based training. One program was granted the needed exemption and launched a PIP pilot, now in year 4, demonstrating the feasibility of implementing this model. Implementation tools and templates are described. NEXT STEPS: Larger-scale implementation with longer-term assessment is needed to evaluate the impact and generalizability of this CBTV-GME model.

Researching models of innovation and adoption in health professions education.

BACKGROUND: Despite the constant presence of change and innovation in health professions education (HPE), there has been relatively little theoretical modelling of such change, the experiences of change, the ideology associated with change or the unexpected consequences of change. In this paper, the authors explore theoretical approaches to the adoption of innovations in HPE as a way of mapping a broader theoretical landscape of change. METHOD: The authors, HPE researchers with an interest in technology adoption and systemic change, present a narrative review of the literature based on a series of thought experiments regarding how communities and individuals respond to the introduction of new ideas or methods. This research investigates the stages of innovation adoption, from the emergence and hype around new ideas to the concrete experiences of early adopters. RESULTS: When an innovation first emerges, there is often little concrete information available to inform potential adopters, leaving it susceptible to hype, both positive and negative. This can be described using the Gartner Hype Cycle model, albeit with important caveats. Once the adoption of an innovation gets underway, early adopter user experiences can inform those that follow. This can be described using Rogers' diffusion of innovation model, again with caveats. Notably, neither model goes beyond the point of single point-in-time, yes/no, individual adoption. Other approaches, such as learning curve theory, are needed to track uptake and maintenance by individuals over time. SIGNIFICANCE: This expanded theoretical base, while still somewhat instrumentalist, combined with complementary theoretical perspectives can afford opportunities to better explore reasons for variance, volunteerism and resistance to change. In summary, change is complicated and nuanced, and better models and theories are needed to understand and work meaningfully with change in HPE. To that end, the authors seek to encourage richer and more thoughtful research and scholarly thinking about change and a more nuanced approach to the pursuit of change in HPE as a whole.

Considerations on the Use of Neonatal and Pediatric Resuscitation Guidelines for Hospitalized Neonates and Infants: On Behalf of the American Heart Association Emergency Cardiovascular Care Committee and the American Academy of Pediatrics.

Between 0.25% and 3% of admissions to the NICU, PICU, and PCICU receive cardiopulmonary resuscitation (CPR). Most CPR events occur in patients <1 year old. The incidence of CPR is 10 times higher in the NICU than at birth. Therefore, optimizing the approach to CPR in hospitalized neonates and infants is important. At birth, the resuscitation of newborns is performed according to neonatal resuscitation guidelines. In older infants and children, resuscitation is performed according to pediatric resuscitation guidelines. Neonatal and pediatric guidelines differ in several important ways. There are no published recommendations to guide the transition from neonatal to pediatric guidelines. Therefore, hospitalized neonates and infants can be resuscitated using neonatal guidelines, pediatric guidelines, or a hybrid approach. This report summarizes the current neonatal and pediatric resuscitation guidelines, considers how to apply them to hospitalized neonates and infants, and identifies knowledge gaps and future priorities. The lack of strong scientific data makes it impossible to provide definitive recommendations on when to transition from neonatal to pediatric resuscitation guidelines. Therefore, it is up to health care teams and institutions to decide if neonatal or pediatric guidelines are the best choice in a given location or situation, considering local circumstances, health care team preferences, and resource limitations.

Twelve Tips for using Learning Curves in Health Professions Education Research.

Learning curves can be used to design, implement, and evaluate educational interventions. Attention to key aspects of the method can improve the fidelity of this representation of learning as well as its suitability for education and research purposes. This paper addresses when to use a learning curve, which graphical properties to consider, how to use learning curves quantitatively, and how to use observed thresholds to communicate meaning. We also address the associated ethics and policy considerations. We conclude with a best practices checklist for both educators and researchers seeking to use learning curves in their work.

It Takes a Village: Optimal Graduate Medical Education Requires a Deliberately Developmental Organization.

Coaching is proposed as a means of improving the learning culture of medicine. By fostering trusting teacher-learner relationships, learners are encouraged to embrace feedback and make the most of failure. This paper posits that a cultural shift is necessary to fully harness the potential of coaching in graduate medical education. We introduce the deliberately developmental organization framework, a conceptual model focusing on three core dimensions: developmental communities, developmental aspirations, and developmental practices. These dimensions broaden the scope of coaching interactions. Implementing this organizational change within graduate medical education might be challenging, yet we argue that embracing deliberately developmental principles can embed coaching into everyday interactions and foster a culture in which discussing failure to maximize learning becomes acceptable. By applying the dimensions of developmental communities, aspirations, and practices, we present a six-principle roadmap towards transforming graduate medical education training programs into deliberately developmental organizations.

Estimating the Irreducible Uncertainty in Visual Diagnosis: Statistical Modeling of Skill Using Response Models.

BACKGROUND: For the representative problem of prostate cancer grading, we sought to simultaneously model both the continuous nature of the case spectrum and the decision thresholds of individual pathologists, allowing quantitative comparison of how they handle cases at the borderline between diagnostic categories. METHODS: Experts and pathology residents each rated a standardized set of prostate cancer histopathological images on the International Society of Urological Pathologists (ISUP) scale used in clinical practice. They diagnosed 50 histologic cases with a range of malignancy, including intermediate cases in which clear distinction was difficult. We report a statistical model showing the degree to which each individual participant can separate the cases along the latent decision spectrum. RESULTS: The slides were rated by 36 physicians in total: 23 ISUP pathologists and 13 residents. As anticipated, the cases showed a full continuous range of diagnostic severity. Cases ranged along a logit scale consistent with the consensus rating (Consensus ISUP 1: mean -0.93 [95% confidence interval {CI} -1.10 to -0.78], ISUP 2: -0.19 logits [-0.27 to -0.12]; ISUP 3: 0.56 logits [0.06-1.06]; ISUP 4 1.24 logits [1.10-1.38]; ISUP 5: 1.92 [1.80-2.04]). The best raters were able to meaningfully discriminate between all 5 ISUP categories, showing intercategory thresholds that were quantifiably precise and meaningful. CONCLUSIONS: We present a method that allows simultaneous quantification of both the confusability of a particular case and the skill with which raters can distinguish the cases. IMPLICATIONS: The technique generalizes beyond the current example to other clinical situations in which a diagnostician must impose an ordinal rating on a biological spectrum. HIGHLIGHTS: Question: How can we quantify skill in visual diagnosis for cases that sit at the border between 2 ordinal categories-cases that are inherently difficult to diagnose?Findings: In this analysis of pathologists and residents rating prostate biopsy specimens, decision-aligned response models are calculated that show how pathologists would be likely to classify any given case on the diagnostic spectrum. Decision thresholds are shown to vary in their location and precision.Significance: Improving on traditional measures such as kappa and receiver-operating characteristic curves, this specialization of item response models allows better individual feedback to both trainees and pathologists, including better quantification of acceptable decision variation.

Interactive computer-assisted learning as an educational method for learning pediatric interproximal dental caries identification.

OBJECTIVE: We developed a web-based tool to measure the amount and rate of skill acquisition in pediatric interproximal caries diagnosis among pre- and postdoctoral dental students and identified variables predictive for greater image interpretation difficulty. STUDY DESIGN: In this multicenter prospective cohort study, a convenience sample of pre- and postdoctoral dental students participated in computer-assisted learning in the interpretation of bitewing radiographs of 193 children. Participants were asked to identify the presence or absence of interproximal caries and, where applicable, locate the lesions. After every case, participants received specific visual and text feedback on their diagnostic performance. They were requested to complete the 193-case set but could complete enough cases to achieve a competency performance standard of 75% accuracy, sensitivity, and specificity. RESULTS: Of 130 participants, 62 (47.7%) completed all cases. The mean change from initial to maximal diagnostic accuracy was +15.3% (95% CI, 13.0-17.7), sensitivity was +10.8% (95% CI, 9.0-12.7), and specificity was +15.5% (95% CI, 12.9-18.1). The median number of cases completed to achieve competency was 173 (interquartile range, 82-363). Of these 62 participants, 45 (72.6%) showed overall improvement in diagnostic accuracy. Greater numbers of interproximal lesions (P < .001) and the presence of noninterproximal caries (P < .001) predicted greater interpretation difficulty. CONCLUSIONS: Computer-assisted learning led to improved diagnosis of interproximal caries on bitewing radiographs among pre- and postdoctoral dental students.

Evaluation of Polytomous Item Locations in Multicomponent Measuring Instruments: A Note on a Latent Variable Modeling Procedure.

This note is concerned with evaluation of location parameters for polytomous items in multiple-component measuring instruments. A point and interval estimation procedure for these parameters is outlined that is developed within the framework of latent variable modeling. The method permits educational, behavioral, biomedical, and marketing researchers to quantify important aspects of the functioning of items with ordered multiple response options, which follow the popular graded response model. The procedure is routinely and readily applicable in empirical studies using widely circulated software and is illustrated with empirical data.

The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156.

The use of Artificial Intelligence (AI) in medical education has the potential to facilitate complicated tasks and improve efficiency. For example, AI could help automate assessment of written responses, or provide feedback on medical image interpretations with excellent reliability. While applications of AI in learning, instruction, and assessment are growing, further exploration is still required. There exist few conceptual or methodological guides for medical educators wishing to evaluate or engage in AI research. In this guide, we aim to: 1) describe practical considerations involved in reading and conducting studies in medical education using AI, 2) define basic terminology and 3) identify which medical education problems and data are ideally-suited for using AI.

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School.

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education.

Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.