The Inconspicuous Learner Handover: An Exploratory Study of U.S. Emergency Medicine Program Directors' Perceptions of Learner Handovers from Medical School to Residency.

Phenomenon: Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach: Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings: We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights: PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

Supportive and collaborative interdependence: Distinguishing residents' contributions within health care teams.

INTRODUCTION: Individual assessments disregard team contributions, while team assessments disregard an individual's contributions. Interdependence has been put forth as a conceptual bridge between our educational traditions of assessing individual performance and our imminent challenge of assessing team-based performance without losing sight of the individual. The purpose of this study was to develop a more refined conceptualisation of interdependence to inform the creation of measures that can assess the interdependence of residents within health care teams. METHODS: Following a constructivist grounded theory approach, we conducted 49 semi-structured interviews with various members of health care teams (e.g. physicians, nurses, pharmacists, social workers and patients) across two different clinical specialties-Emergency Medicine and Paediatrics-at two separate sites. Data collection and analysis occurred iteratively. Constant comparative inductive analysis was used, and coding consisted of three stages: initial, focused and theoretical. RESULTS: We asked participants to reflect upon interdependence and describe how it exists in their clinical setting. All participants acknowledged the existence of interdependence, but they did not view it as part of a linear spectrum where interdependence becomes independence. Our analysis refined the conceptualisation of interdependence to include two types: supportive and collaborative. Supportive interdependence occurs within health care teams when one member demonstrates insufficient expertise to perform within their scope of practice. Collaborative interdependence, on the other hand, was not triggered by lack of experience/expertise within an individual's scope of practice, but rather recognition that patient care requires contributions from other team members. CONCLUSION: In order to assess a team's collective performance without losing sight of the individual, we need to capture interdependent performances and characterise the nature of such interdependence. Moving away from a linear trajectory where independence is seen as the end goal can also help support efforts to measure an individual's competence as an interdependent member of a health care team.

The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156.

The use of Artificial Intelligence (AI) in medical education has the potential to facilitate complicated tasks and improve efficiency. For example, AI could help automate assessment of written responses, or provide feedback on medical image interpretations with excellent reliability. While applications of AI in learning, instruction, and assessment are growing, further exploration is still required. There exist few conceptual or methodological guides for medical educators wishing to evaluate or engage in AI research. In this guide, we aim to: 1) describe practical considerations involved in reading and conducting studies in medical education using AI, 2) define basic terminology and 3) identify which medical education problems and data are ideally-suited for using AI.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs.

BACKGROUND: Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions. METHODS: Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses. RESULTS: 687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame. CONCLUSIONS: Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

The shift from disbelieving underperformance to recognising failure: A tipping point model.

CONTEXT: Coming face to face with a trainee who needs to be failed is a stern test for many supervisors. In response, supervisors have been encouraged to report evidence of failure through numerous assessment redesigns. And yet, there are lingering signs that some remain reluctant to engage in assessment processes that could alter a trainee's progression in the programme. Failure is highly consequential for all involved and, although rare, requires explicit study. Recent work identified a phase of disbelief that preceded identification of underperformance. What remains unknown is how supervisors come to recognise that a trainee needs to be failed. METHODS: Following constructivist grounded theory methodology, 42 physicians and surgeons in British Columbia, Canada shared their experiences supervising trainees who profoundly underperformed, required extensive remediation or were dismissed from the programme. We identified recurring themes using an iterative, constant comparative process. RESULTS: The shift from disbelieving underperformance to recognising failure involves three patterns: accumulation of significant incidents, discovery of an egregious error after negligible deficits or illumination of an overlooked deficit when pointed out by someone else. Recognising failure was accompanied by anger, certainty and a sense of duty to prevent harm. CONCLUSION: Coming to the point of recognising that a trainee needs to fail is akin to the psychological process of a tipping point where people first realise that noise is signal and cross a threshold where the pattern is no longer an anomaly. The co-occurrence of anger raises the possibility for emotions to be a driver of, and not only a barrier to, recognising failure. This warrants caution because tipping points, and anger, can impede detection of improvement. Our findings point towards possibilities for supporting earlier identification of underperformance and overcoming reluctance to report failure along with countermeasures to compensate for difficulties in detecting improvement once failure has been verified.

Who's on your team? Specialty identity and inter-physician conflict during admissions.

PURPOSE: Despite the implementation of professionalism curricula and standardised communication tools, inter-physician conflict persists. In particular, the interface between emergency medicine (EM) and internal medicine (IM) has long been recognised as a source of conflict. The social nuances of this conflict remain underexplored, limiting educators' ability to comprehensively address these issues in the clinical learning environment. Thus, the authors explored EM and IM physicians' experiences with negotiating hospital admissions to better understand the social dynamics that contribute to inter-physician conflict and provide foundational guidance for communication best practices. METHODS: Using a constructivist grounded theory (CGT) approach, the authors conducted 18 semi-structured interviews between June and October 2020 with EM and IM physicians involved in conversations regarding admissions (CRAs). They asked participants to describe the social exchanges that influenced these conversations and to reflect on their experiences with inter-physician conflict. Data collection and analysis occurred iteratively. The relationships between the codes were discussed by the research team with the goal of developing conceptual connections between the emergent themes. RESULTS: Participants described how their approaches to CRAs were shaped by their specialty identity, and how allegiance to members of their group contributed to interpersonal conflict. This conflict was further promoted by a mutual sense of disempowerment within the organisation, misaligned expectations, and a desire to promote their group's prerogatives. Conflict was mitigated when patient care experiences fostered cross-specialty team formation and collaboration that dissolved traditional group boundaries. CONCLUSIONS: Conflict between EM and IM physicians during CRAs was primed by participants' specialty identities, their power struggles within the broader organisation, and their sense of duty to their own specialty. However, formation of collaborative inter-specialty physician teams and expansion of identity to include colleagues from other specialties can mitigate inter-physician conflict.

Is Lactating in the Emergency Department a Letdown? Exploring Barriers and Supports to Workplace Lactation in Emergency Medicine.

STUDY OBJECTIVE: To explore the social and environmental conditions in emergency departments that contribute to perceived barriers and supports for workplace lactation among individuals working in emergency medicine. METHODS: Constructivist grounded theory was used by our team to understand the social processes and behaviors associated with workplace lactation for health care professionals working in EDs. A total of 24 interviews of individuals in EDs with recent return-to-work experience after childbirth were performed. The interviews yielded 36 unique experiences (from 21 faculty, 12 trainees, and 3 nurses) because some participants had more than 1 child, in which case all lactation experiences were discussed. Interview transcriptions were coded and analyzed iteratively for the development of themes, per constructivist grounded theory. RESULTS: Using constant comparative inductive methods, we describe 3 pervasive themes as they relate to workplace lactation that emerged from the analysis of interview data: (1) emergency medicine culture, (2) workplace lactation policies, and (3) supports for workplace lactation. CONCLUSION: Although formalized workplace lactation policies and other identifiable supports for workplace lactation aid individuals desiring to lactate after returning to work in EDs, many individuals still experience cultural barriers to their desired lactation habits. Policies and individual support systems may continue to fall short of meeting the needs of lactating individuals in emergency medicine unless broader cultural change occurs. Our work offers initial recommendations for shifting the landscape of lactation practices in emergency medicine.

A scoping review of approaches for measuring 'interdependent' collaborative performances.

INTRODUCTION: Individual assessment disregards the team aspect of clinical work. Team assessment collapses the individual into the group. Neither is sufficient for medical education, where measures need to attend to the individual while also accounting for interactions with others. Valid and reliable measures of interdependence are critical within medical education given the collaborative manner in which patient care is provided. Medical education currently lacks a consistent approach to measuring the performance between individuals working together as part of larger healthcare team. This review's objective was to identify existing approaches to measuring this interdependence. METHODS: Following Arksey & O'Malley's methodology, we conducted a scoping review in 2018 and updated it to 2020. A search strategy involving five databases located >12 000 citations. At least two reviewers independently screened titles and abstracts, screened full texts (n = 161) and performed data extraction on twenty-seven included articles. Interviews were also conducted with key informants to check if any literature was missing and assess that our interpretations made sense. RESULTS: Eighteen of the twenty-seven articles were empirical; nine conceptual with an empirical illustration. Eighteen were quantitative; nine used mixed methods. The articles spanned five disciplines and various application contexts, from online learning to sports performance. Only two of the included articles were from the field of Medical Education. The articles conceptualised interdependence of a group, using theoretical constructs such as collaboration synergy; of a network, using constructs such as degree centrality; and of a dyad, using constructs such as synchrony. Both descriptive (eg social network analysis) and inferential (eg multi-level modelling) approaches were described. CONCLUSION: Efforts to measure interdependence are scarce and scattered across disciplines. Multiple theoretical concepts and inconsistent terminology may be limiting programmatic work. This review motivates the need for further study of measurement techniques, particularly those combining multiple approaches, to capture interdependence in medical education.

Examination of physician characteristics in opioid prescribing in the emergency department.

AIM: We aimed to better understand variation in opioid prescribing practices by investigating physician factors at one academic suburban Emergency Department (ED). METHODS: We retrospectively reviewed the electronic medical records of all patients given opioid prescriptions in the Stanford Health Care ED from 2009 to 2018. We described the variation in opioid prescriptions over time from 2009 to 2018, then dove deeper into a single year (July 1, 2017 to July 1, 2018). We described the number and type of opioid prescriptions at discharge and variation in attending physician opioid prescribing patterns using independent t-tests and a Fischer's exact test. RESULTS: From 2009 to 2018, 657,037 patient visits occurred; 92,612 (14.1%) opioid prescriptions were written. Opioid prescriptions increased from 2009, peaked in 2015, then decreased. Individual providers wrote opioid prescriptions for 1 to 17% of their discharged patients. There was no significant difference in opioid prescribing based on provider gender (p = 0.456), fellow or attending status (p = 0.390), residency completed at Stanford Hospital (p = 0.593), residency completed within California (p = 0.493), or residency completed after 2010 (p = 0.589). Of the 371 providers who wrote opioid prescriptions from 2009 through 2018, 120 wrote prescriptions for patients who had already received at least three opioid prescriptions in the same year from the same department. CONCLUSION: This study could inform policymakers by describing patterns of variation in opioid prescribing over time and between providers. Although we did see significant differences in prescribing patterns from one provider to the next, those were not explained by the factors we examined. Further studies could investigate factors such as provider experience with pain and addiction, bias regarding particular pathologies, and concern around patient satisfaction scores.

Impostor syndrome among physicians and physicians in training: A scoping review.

CONTEXT: Impostor syndrome (IS) is increasingly recognised as a condition among physicians and physicians in training. Impostor syndrome is especially problematic because of its association with increased rates of burnout and suicide. In order to address this issue, we need to fully understand its prevalence, scope, and factors associated with IS. The purpose of this scoping review is to analyse the existing literature on IS among practising physicians and physicians in training in order to identify current trends and directions for future research. METHODS: The authors conducted a literature search of nine databases for any articles on IS among practising physicians or physicians in training published prior to January 2019. Two reviewers independently screened articles and identified 18 papers meeting the study inclusion criteria. Two authors independently extracted data and performed quantitative and qualitative syntheses consistent with best practice recommendations for scoping reviews. RESULTS: Most studies utilised the Clance Impostor Phenomenon Scale and cited rates of IS ranging from 22% to 60%. Studies found that gender, low self-esteem and institutional culture were associated with higher rates of IS, whereas social support, validation of success, positive affirmation, and both personal and shared reflections were protective. Overall, IS was also associated with higher rates of burnout. CONCLUSIONS: This review summarises the existing literature on IS among practising physicians and physicians in training, providing valuable insights and areas for future research.

Seeing but not believing: Insights into the intractability of failure to fail.

CONTEXT: Inadequate documentation of observed trainee incompetence persists despite research-informed solutions targeting this failure to fail phenomenon. Documentation could be impeded if assessment language is misaligned with how supervisors conceptualise incompetence. Because frameworks tend to itemise competence as well as being vague about incompetence, assessment design may be improved by better understanding and describing of how supervisors experience being confronted with a potentially incompetent trainee. METHODS: Following constructivist grounded theory methodology, analysis using a constant comparison approach was iterative and informed data collection. We interviewed 22 physicians about their experiences supervising trainees who demonstrate incompetence; we quickly found that they bristled at the term 'incompetence,' so we began to use 'underperformance' in its place. RESULTS: Physicians began with a belief and an expectation: all trainees should be capable of learning and progressing by applying what they learn to subsequent clinical experiences. Underperformance was therefore unexpected and evoked disbelief in supervisors, who sought alternate explanations for the surprising evidence. Supervisors conceptualised underperformance as: an inability to engage with learning due to illness, a life event or learning disorders, so that progression was stalled, or an unwillingness to engage with learning due to lack of interest, insight or humility. CONCLUSION: Physicians conceptualise underperformance as problematic progression due to insufficient engagement with learning that is unresponsive to intensified supervision. Although failure to fail tends to be framed as a reluctance to document underperformance, the prior phase of disbelief prevents confident documentation of performance and delays identification of underperformance. The findings offer further insight and possible new solutions to address under-documentation of underperformance.

Elucidating system-level interdependence in electronic health record data: What are the ramifications for trainee assessment?

CONTEXT: The electronic health record (EHR) has been identified as a potential site for gathering data about trainees' clinical performance, but these data are not collected or organised for this purpose. Therefore, a careful and rigorous approach is required to explore how EHR data could be meaningfully used for assessment purposes. The purpose of this study was to identify EHR performance metrics that represent both the independent and interdependent clinical performance of emergency medicine (EM) trainees and explore how they might be meaningfully used for assessment and feedback. METHODS: Using constructivist grounded theory, we conducted 21 semi-structured interviews with EM faculty members and residents. Participants were asked to identify the clinical actions of trainees that would be valuable for assessment and feedback and describe how those activities are represented in the EHR. Data collection and analysis, which consisted of three stages of coding, occurred iteratively. RESULTS: When faculty members and trainees in EM were asked to reflect on the usefulness of using EHR performance metrics for resident assessment and feedback they expressed both widespread support for the idea in principle and hesitation that aspects of clinical performance captured in the data would not be representative of residents' individual performance, but would rather reflect their interdependence with other team members and the systems in which they work. We highlight three categorisations of system-level interdependence - medical directives, technological systems and organisational systems - identified by our participants, and discuss strategies participants employed to navigate these forms of interdependence within the health care system. CONCLUSIONS: System-level interdependence shapes physicians' performances, and yet, this impact is rarely corrected for or noted within clinical performance data. Educators have a responsibility to recognise system-level interdependence when teaching and consider system-level interdependence when assessing the performance of trainees in order to most effectively and fairly utilise the EHR as a source of assessment data.

The role of data science and machine learning in Health Professions Education: practical applications, theoretical contributions, and epistemic beliefs.

Data science is an inter-disciplinary field that uses computer-based algorithms and methods to gain insights from large and often complex datasets. Data science, which includes Artificial Intelligence techniques such as Machine Learning (ML), has been credited with the promise to transform Health Professions Education (HPE) by offering approaches to handle big (and often messy) data. To examine this promise, we conducted a critical review to explore: (1) published applications of data science and ML in HPE literature and (2) the potential role of data science and ML in shifting theoretical and epistemological perspectives in HPE research and practice. Existing data science studies in HPE are often not informed by theory, but rather oriented towards developing applications for specific problems, uses, and contexts. The most common areas currently being studied are procedural (e.g., computer-based tutoring or adaptive systems and assessment of technical skills). We found that epistemic beliefs informing the use of data science and ML in HPE poses a challenge for existing views on what constitutes objective knowledge and the role of human subjectivity for instruction and assessment. As a result, criticisms have emerged that the integration of data science in the field of HPE is in danger of becoming technically driven and narrowly focused in its approach to teaching, learning and assessment. Our findings suggest that researchers tend to formalize around the epistemological stance driven largely by traditions of a research paradigm. Future data science studies in HPE need to involve both education scientists and data scientists to ensure mutual advancements in the development of educational theory and practical applications. This may be one of the most important tasks in the integration of data science and ML in HPE research in the years to come.

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance.

Construct: This study seeks to determine validity evidence for the Quality of Assessment for Learning score (QuAL score), which was created to evaluate short qualitative comments that are related to specific scores entered into a workplace-based assessment, common within the competency-based medical education (CBME) context. Background: In the age of CBME, qualitative comments play an important role in clarifying the quantitative scores rendered by observers at the bedside. Currently there are few practical tools that evaluate mixed data (e.g. associated score-and-comment data), other than the comprehensive Completed Clinical Evaluation Report Rating tool (CCERR) that was originally derived to rate end-of-rotation reports. Approach: A multi-center, randomized cohort-based rating exercise was conducted to evaluate the rating properties of the QuAL score as compared to the CCERR. One group rated comments using the QuAL score, and the other group rated comments using the CCERR. A generalizability study (G-Study) and a decision study (D-study) were conducted to determine the number of meta-raters for a reliable rating (phi-coefficient target of >0.80). Both scores were correlated against rater's gestalt perceptions of utility for both faculty and residents reading the scores. Results: Twenty-five meta-raters from 20 sites participated in this rating exercise. The G-study revealed that the CCERR group (n = 13) rated the comments with a very high reliability (Phi = 0.97). Meanwhile, the QuAL group (n = 12) rated the comments with a similarly high reliability (Phi = 0.97). The QuAL score required only two raters to reach an acceptable target reliability of >0.80, while the CCERR required three. The QuAL score correlated with perceptions of utility (Meta-rater usefulness, Pearson's r = 0.69, p < 0.001; Perceived usefulness for trainee, r = 0.74, p < 0.001). The CCERR performed similarly, correlating with perceived faculty (r = 0.67, <0.001) and resident utility (0.79, <0.001). Conclusions: The QuAL score is reliable rating score that correlates well with perceptions of utility. The QuAL score may be useful for rating shorter comments generated by workplace-based assessments.

Considering the interdependence of clinical performance: implications for assessment and entrustment.

INTRODUCTION: Our ability to assess independent trainee performance is a key element of competency-based medical education (CBME). In workplace-based clinical settings, however, the performance of a trainee can be deeply entangled with others on the team. This presents a fundamental challenge, given the need to assess and entrust trainees based on the evolution of their independent clinical performance. The purpose of this study, therefore, was to understand what faculty members and senior postgraduate trainees believe constitutes independent performance in a variety of clinical specialty contexts. METHODS: Following constructivist grounded theory, and using both purposive and theoretical sampling, we conducted individual interviews with 11 clinical teaching faculty members and 10 senior trainees (postgraduate year 4/5) across 12 postgraduate specialties. Constant comparative inductive analysis was conducted. Return of findings was also carried out using one-to-one sessions with key informants and public presentations. RESULTS: Although some independent performances were described, participants spoke mostly about the exceptions to and disclaimers about these, elaborating their sense of the interdependence of trainee performances. Our analysis of these interdependence patterns identified multiple configurations of coupling, with the dominant being coupling of trainee and supervisor performance. We consider how the concept of coupling could advance workplace-based assessment efforts by supporting models that account for the collective dimensions of clinical performance. CONCLUSION: These findings call into question the assumption of independent performance, and offer an important step toward measuring coupled performance. An understanding of coupling can help both to better distinguish independent and interdependent performances, and to consider revising workplace-based assessment approaches for CBME.

Quality Evaluation Scores are no more Reliable than Gestalt in Evaluating the Quality of Emergency Medicine Blogs: A METRIQ Study.

Construct: We investigated the quality of emergency medicine (EM) blogs as educational resources. PURPOSE: Online medical education resources such as blogs are increasingly used by EM trainees and clinicians. However, quality evaluations of these resources using gestalt are unreliable. We investigated the reliability of two previously derived quality evaluation instruments for blogs. APPROACH: Sixty English-language EM websites that published clinically oriented blog posts between January 1 and February 24, 2016, were identified. A random number generator selected 10 websites, and the 2 most recent clinically oriented blog posts from each site were evaluated using gestalt, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) score, and the Medical Education Translational Resources: Impact and Quality (METRIQ-8) score, by a sample of medical students, EM residents, and EM attendings. Each rater evaluated all 20 blog posts with gestalt and 15 of the 20 blog posts with the ALiEM AIR and METRIQ-8 scores. Pearson's correlations were calculated between the average scores for each metric. Single-measure intraclass correlation coefficients (ICCs) evaluated the reliability of each instrument. RESULTS: Our study included 121 medical students, 88 EM residents, and 100 EM attendings who completed ratings. The average gestalt rating of each blog post correlated strongly with the average scores for ALiEM AIR (r = .94) and METRIQ-8 (r = .91). Single-measure ICCs were fair for gestalt (0.37, IQR 0.25-0.56), ALiEM AIR (0.41, IQR 0.29-0.60) and METRIQ-8 (0.40, IQR 0.28-0.59). CONCLUSION: The average scores of each blog post correlated strongly with gestalt ratings. However, neither ALiEM AIR nor METRIQ-8 showed higher reliability than gestalt. Improved reliability may be possible through rater training and instrument refinement.

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study.

STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students' scores using Many Facet Rasch Modeling.

INTRODUCTION: OSCEs are commonly conducted in multiple cycles (different circuits, times, and locations), yet the potential for students' allocation to different OSCE cycles is rarely considered as a source of variance-perhaps in part because conventional psychometrics provide limited insight. METHODS: We used Many Facet Rasch Modeling (MFRM) to estimate the influence of "examiner cohorts" (the combined influence of the examiners in the cycle to which each student was allocated) on students' scores within a fully nested multi-cycle OSCE. RESULTS: Observed average scores for examiners cycles varied by 8.6%, but model-adjusted estimates showed a smaller range of 4.4%. Most students' scores were only slightly altered by the model; the greatest score increase was 5.3%, and greatest score decrease was -3.6%, with 2 students passing who would have failed. DISCUSSION: Despite using 16 examiners per cycle, examiner variability did not completely counter-balance, resulting in an influence of OSCE cycles on students' scores. Assumptions were required for the MFRM analysis; innovative procedures to overcome these limitations and strengthen OSCEs are discussed. CONCLUSIONS: OSCE cycle allocation has the potential to exert a small but unfair influence on students' OSCE scores; these little-considered influences should challenge our assumptions and design of OSCEs.

Exploring How House Staff Unions Impact the Program Director-Resident Educational Alliance.

ABSTRACT: In 1999, the National Labor Relations Board determined that residents function as employees, thereby allowing them to freely unionize. From 2020 to 2023, house staff (i.e., resident physicians and fellows) unions have significantly increased, and 8 physician training centers, representing nearly 4,000 house staff, have unionized since March 2021. While unions provide residents with an important tool in effecting change in their workplace, their introduction into the educational milieu has the potential to alter the program director (PD)-resident relationship. In this article, the authors use the educational alliance framework to detail 3 factors required to support a quality educational relationship between a resident and their PD. They also elaborate on how the introduction of unions may impact the PD-resident relationship and explore the potential unintended consequences of unionization as it pertains to this relationship. The authors then use 2 social psychology theories, naïve realism and motivated reasoning, to describe common framing dynamics that lead to conflict during collective bargaining processes. They conclude by offering strategies that PDs may use to mitigate tensions that arise in contract negotiations, even without a direct seat at the table. Ultimately, PDs should anticipate continued growth of resident unions and prepare themselves and their programs for the tensions that may arise from this action. The PD role as a neutral third party ought to be preserved, which is possible if all parties set reasonable expectations for the changes in the PD's role and responsibilities under a union. PDs should understand the 3 core aspects of the educational alliance and the importance of establishing credibility with their residents early on to build a strong foundation.

Considering the Secondary Use of Clinical and Educational Data to Facilitate the Development of Artificial Intelligence Models.

ABSTRACT: Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives. More recently, however, these clinical and educational data are also increasingly being used to train artificial intelligence (AI) models. The implications of this relatively unique secondary use of data have not been well explored. These models can support the development of sophisticated AI products that can be commercialized. While these products have the potential to support and improve the educational system, there are challenges related to validity, patient and learner consent, and biased or discriminatory outputs. The authors consider the implications of developing AI models and products using educational and clinical data from learners, discuss the uses of these products within medical education, and outline considerations that should guide the appropriate use of data for this purpose. These issues are further explored by examining how they have been navigated in an educational collaborative.