Teaching foundational language equity concepts in the pre-clinical curriculum.

BACKGROUND: Despite the prevalence of non-English languages in the US population, existing medical training to teach communication with linguistically diverse communities is limited to electives or solely focuses on medical interpreting. Language-appropriate communication skills are seldom comprehensively integrated in medical education. This study describes the development and evaluation of an intervention to teach foundational language equity concepts. METHODS: The authors implemented a pre-clinical language equity course at three medical school campuses between August 2020 and March 2022. Sessions focused on the impact of language in health, physician language proficiency standards, and working with medical interpreters. The study sought to (1) understand students' language skills and prior clinical experiences with patients with non-English language preference and (2) evaluate the curriculum's impact. Students self-reported their language skills and experiences as part of a voluntary pre-questionnaire. Pre and post-questionnaires evaluated knowledge, attitudes, and intent to apply language equity concepts. Descriptive statistics and chi-squared tests were used to examine trends; themes were identified from free-text responses. RESULTS: Overall, 301 students completed the course, 252 (83%) completed at least one questionnaire; for each session, between 35% and 46% of learners completed both pre and post-questionnaires. Three quarters (189/252) reported non-English languages. Over half (138/252) reported previous non-English language patient care, and 28% (62/224) had served as ad hoc (untrained) interpreters. Only two students (< 1%) had ever been assessed for medical language abilities. Students demonstrated improved post-course language equity knowledge, strategies for interpreter-mediated encounters, and likelihood to report a plan for language skills assessment (all p < .001). Most plans were multifaceted (61%, 38/62), involving goals like completing a language course, taking a proficiency exam, openly discussing skills and uncertainties with team members, and increasing professional interpreter utilization. CONCLUSIONS: A longitudinal language equity curriculum can be feasibly integrated in pre-clinical education, highlight the linguistic diversity of the student body, and serve as a first step in ensuring that all students have a strong language equity foundation prior to clinical rotations. Future steps include evaluating the intervention's potential long-term effects on professional interpreter utilization, student clinical performance, and institutional culture that promotes multilingualism.

Flexible endoscopic intubation in emergency medicine: A mixed-methods needs assessment.

Objectives: This needs assessment aimed to improve understanding of flexible endoscopic intubation training and practice in emergency medicine (EM), providing insights to educators and practice leaders seeking to improve education and practices. Methods: We conducted a multicenter, mixed-methods needs assessment of emergency physicians (EPs) incorporating focus groups and a survey. Focus groups comprised community EPs, academic EPs, and resident EPs. We analyzed focus group transcripts using grounded theory, qualitatively describing EM endoscopic intubation. The qualitative analysis shaped our survey instrument, which we deployed in cross-sectional fashion. We report survey data with descriptive statistics. Results: Focus groups with 13 EPs identified three themes: indications for use of endoscopic intubation, factors impacting a physician's decision to endoscopically intubate, and attaining and maintaining endoscopic intubation competency. Of 257 surveyed EPs (33% response rate), 79% had received endoscopic intubation training during residency, though 82% had performed this procedure 10 or fewer times in their career. Despite 97% acknowledging the necessity of competency, only 23% felt highly confident in their ability to perform endoscopic intubation. Participants (93%) reported scarce opportunities to perform the procedure and identified factors believed to facilitate competency acquisition and maintenance, including opportunities to perform endoscopic intubation in practice (98%), local champions (93%), and performing nasopharyngoscopy (87%). Conclusions: While most EPs acknowledged the importance of competency in endoscopic intubation, they reported scarce procedural opportunities and commonly expressed low confidence. Further research is needed on this topic, and we propose avenues to enhance education and practices related to endoscopic intubation. These include development of robust procedural curricula, support of local champions, and incorporating nasopharyngoscopy into EM practice.

Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study.

PURPOSE: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into six elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. METHODS: Messick's unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from three pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018-2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). RESULTS: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone, r = 0.34, P = .019. CONCLUSION: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

BACKGROUND: While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. OBJECTIVE: To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). DESIGN: Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). SUBJECTS: Participating faculty in longitudinal rater training. MAIN MEASURES: Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). KEY RESULTS: Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (ß 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. CONCLUSIONS: Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Principles of Justice, Equity, Diversity, and Inclusion in Health Care Distance Simulation Education: Consensus Building via the Nominal Group Technique.

PURPOSE: Health care distance simulation (HCDS) replicates professional encounters through an immersive experience overseen by experts and in which technological infrastructure enriches the learning activity. As HCDS has gained traction, so has the movement to provide inclusive and accessible simulation experiences for all participants. However, established guidelines for best practices in HCDS regarding justice, equity, diversity, and inclusion (JEDI) are lacking. This study aimed to generate consensus statements on JEDI principles in synchronous HCDS education using the nominal group technique (NGT). METHOD: Professionals with experience in HCDS education were invited to generate, record, discuss, and vote on ideas that they considered best practices for JEDI. This process was followed by a thematic analysis of the NGT discussion to provide a deeper understanding of the final consensus statements. An independent group of HCDS educators individually reviewed and recorded their agreement or disagreement with the consensus statements created by the NGT process. RESULTS: Eleven independent experts agreed on 6 key practices for JEDI in HCDS. Educators need to (1) be aware of JEDI principles, (2) be able to define and differentiate JEDI, (3) model JEDI in their environment, (4) have expertise and comfort facilitating conversations and debriefing around JEDI issues, (5) be advocates within their organizations to ensure equitable educational experiences, and (6) achieve JEDI without compromising educational objectives. Experts were divided on the approach to technology to ensure equitable learning experiences: some believed that the most basic technology accessible to all learners should be used, and some believed that the technology used should be determined by the competency of the students or faculty. CONCLUSIONS: Structural and institutional barriers in HCDS education persist despite agreement on key JEDI practices. Conclusive research is needed to guide the optimal policy in HCDS toward creating equitable learning experiences while bridging the digital divide.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education.

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

Development and validity evidence for the intraprofessional conflict exercise: An assessment tool to support collaboration.

BACKGROUND: Effective collaboration is the foundation for delivering safe, high quality patient care. Health sciences curricula often include interprofessional collaboration training but may neglect conflicts that occur within a profession (intraprofessional). We describe the development of and validity evidence for an assessment of intraprofessional conflict management. METHODS AND FINDINGS: We designed a 22-item assessment, the Intraprofessional Conflict Exercise, to evaluate skills in managing intraprofessional conflicts based on a literature review of conflict management. Using Messick's validity framework, we collected evidence for content, response process, and internal structure during a simulated intraprofessional conflict from 2018 to 2019. We performed descriptive statistics, inter-rater reliability, Cronbach's alpha, generalizability theory, and factor analysis to gather validity evidence. Two trained faculty examiners rated 82 trainees resulting in 164 observations. Inter-rater reliability was fair, weighted kappa of 0.33 (SE = 0.03). Cronbach's alpha was 0.87. The generalizability study showed differentiation among trainees (19.7% person variance) and was highly reliable, G-coefficient 0.88, Phi-coefficient 0.88. The decision study predicted that using one rater would have high reliability, G-coefficient 0.80. Exploratory factor analysis demonstrated three factors: communication skills, recognition of limits, and demonstration of respect for others. Based on qualitative observations, we found all items to be applicable, highly relevant, and helpful in identifying how trainees managed intraprofessional conflict. CONCLUSIONS: The Intraprofessional Conflict Exercise provides a useful and reliable way to evaluate intraprofessional conflict management skills. It provides meaningful and actionable feedback to trainees and may help health educators in preparing trainees to manage intraprofessional conflict.

The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study.

OBJECTIVE: To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program. SUMMARY AND BACKGROUND DATA: ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors. METHODS: Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences. RESULTS: Over the 3 administrations, 1975 surgery residents participated from 125 distinct residency programs. Overall scores [Mean = 64% (SD = 7%)] remained consistent across the 3 years ( P = 0.670). There were no significant differences among resident characteristics (gender, age, international medical graduate status). The mean case discrimination index was 0.54 [SD = 0.15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Ф-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% versus 63%, Cohen's d = 0.23, P < 0.001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%). CONCLUSIONS: ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.

The Use of Learning Analytics to Enable Detection of Underperforming Trainees: An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data.

OBJECTIVE: This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets. SUMMARY BACKGROUND DATA: ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation. METHODS: National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings. RESULTS: A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies ( P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs. CONCLUSION: The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

Needs assessment in head and neck surgical oncology training: A qualitative study of expert opinions.

BACKGROUND: Few studies have investigated the needs of head and neck surgery trainees and areas for improvement of fellowship programs. METHODS: We conducted a qualitative study by interviewing a nationally representative sample of program directors and national leaders in head and neck surgery. We asked about the current state and strengths of training; and areas for further improvement. All interviews were independently coded and analyzed by two reviewers. RESULTS: All experts (100%) believed that the current training provides a strong foundation and furthermore, a standardized curriculum is beneficial. Multidisciplinary training (80%), participation in tumor boards (75%), and a syllabus (60%) were the most frequently mentioned components. Most believed that a formal certification process would be beneficial (73%), though there was no consensus on the format. CONCLUSION: Experts in head and neck surgery are generally in favor of a standardized curriculum. Further discussions of a formal certification process might be warranted.

Optimizing Clinical Reasoning Assessments With Analytic and Holistic Ratings: Examining the Validity, Reliability, and Cost of a Simplified Patient Note Scoring Procedure.

PURPOSE: Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback. METHOD: Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time). RESULTS: Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores. CONCLUSIONS: Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills.

OBJECTIVE: The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. DESIGN: The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. SETTING: The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. RESULTS: The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. CONCLUSIONS: ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Resident Readiness for Senior Level Decision Making: Identifying the Domains for Formative Assessment and Feedback.

OBJECTIVE: To establish expert consensus regarding the domains and topics for senior surgery residents (PGY-4) to make critical decisions and assume senior-level responsibilities, and to develop the formative American College of Surgeons Senior Resident Readiness Assessment (ACS SRRA) Program. DESIGN: The American College of Surgeons (ACS) education leadership team conducted a focus group with surgical experts to identify the content for an assessment tool to evaluate senior residents' readiness for their increased levels of responsibility. After the focus group, national experts were recruited to develop consensus on the topics through three rounds of surveys using Delphi methodology. The Delphi participants rated topics using Likert-type scales and their comments were incorporated into subsequent rounds. Consensus was defined as ≥ 80% agreement with internal-consistency reliability (Cronbach's alpha) ≥ 0.8. In a stepwise fashion, topics that did not achieve consensus for inclusion were removed from subsequent survey rounds. SETTING: The surveys were administered via an online questionnaire. PARTICIPANTS: Twelve program directors and assistant program directors made up the focus group. The 39 Delphi participants represented seven different surgical subspecialties and were from diverse practice settings. The median length of experience in general surgery resident education was 20 years (IQR 14.3-30.0) with 64% of the experts being either current or past general surgery residency program directors. RESULTS: The response rate was 100% and Cronbach's alpha was ≥ 0.9 for each round. The Delphi participants contributed a large number of comments. Of the 201 topics that were evaluated initially, 120 topics in 25 core clinical areas were included to create the final domains of ACS SRRA. CONCLUSIONS: National consensus on the domain of the ACS SRRA has been achieved via the modified Delphi method among expert surgeon educators. ACS SRRA will identify clinical topics and areas in which each senior resident needs improvement and provide data to residents and residency programs to develop individualized learning plans. This would help in preparing the senior residents to assume their responsibilities and support their readiness for future fellowship training or surgical practice.

Medical Spanish Endocrinology Educational Module.

Introduction: Hispanic/Latinx patients experience health disparities in endocrine disease, such as higher rates of mortality due to diabetes mellitus, worse outcomes in the surgical treatment of thyroid cancer, and lack of knowledge about bone health and osteoporotic fracture prevention. Educational strategies to teach cultural and linguistic medical Spanish knowledge to medical students have the potential to improve Hispanic/Latinx endocrine health. Methods: We created an 8-hour medical Spanish endocrine module targeting language and cultural skills acquisition. Specifically, students practiced obtaining a past medical history, obtaining a medications history, providing and explaining a diagnosis, explaining discharge instructions, and discussing sociocultural aspects of endocrine health. We divided the module into four 2-hour sessions: (1) preclass self-study, (2) class period, (3) role-play/interviewing practice session, and (4) case/cultural activity assignment. Participants completed a pre- and postassessment to measure student confidence level and knowledge. Results: We implemented the module four times at one medical school, with 47 fourth-year medical students with intermediate or higher general Spanish skills. Confidence increased for all learners in the endocrine-focused interview and exam in Spanish. Heritage Spanish learners, who were exposed to Spanish at home as children, reported higher postmodule confidence in eliciting the past medical history of endocrine problems. Discussion: The medical Spanish endocrine module improved the communication skills of medical students with intermediate through advanced Spanish proficiency. Future study should evaluate learner clinical performance metrics and applications to other groups of learners, such as resident physicians or health professions involved in the care of patients with endocrine disease.

Patient involvement in assessment of postgraduate medical learners: A scoping review.

CONTEXT: Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent. METHODS: We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment. RESULTS: We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise. CONCLUSION: Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

Mastery skill assessment in hepato-pancreato-biliary surgical ultrasound: It's a Matter of Entrustment.

BACKGROUND: A formative hepato-pancreato-biliary (HPB) ultrasound (US) skills practicum is offered annually to graduating HPB fellows, using entrustment assessments for open (IOUS) and laparoscopic (LAPUS) US. It is hypothesized that validity evidence will support the use of these assessments to determine if graduating fellows are well prepared to perform HPB US independently. METHODS: Expert faculty were surveyed to set Mastery Entrustment standards for fellow performance. Standards were applied to fellow performances during two annual US skills practicums. RESULTS: 11 faculty questionnaires were included. Mean Entrustment cut scores across all items were 4.9/5.0 and 4.8/5.0 and Global Entrustment cut scores were 5.0/5.0 and 4.8/5.0 for IOUS and LAPUS, respectively. 78.5% (29/37) fellows agreed to have their de-identified data evaluated. Mean fellow Entrustments (across all skills) were 4.1 (SD 0.6; 2.6-4.9) and 3.9 (SD 0.7; 2.7-5), while the Global Entrustments were 3.6 (SD 0.8; 2-5) and 3.5 (SD 1.0; 2-5) for IOUS and LAPUS, respectively. CONCLUSIONS: Two cohorts of graduating HPB fellows are not meeting Mastery Standards for HPB US performance determined by a panel of expert faculty.

Effect of a hospital-wide campaign on COVID-19 vaccination uptake among healthcare workers in the context of raised concerns for life-threatening side effects.

BACKGROUND: All healthcare workers (HCWs) in Yongin Severance Hospital were allocated to receive the ChAdOx1 nCov-19 vaccine according to national policy. A report of thrombosis and thrombocytopenia syndrome (TTS) associated with ChAdOx1 nCoV-19 led to hesitancy about receiving the second dose among HCWs who had received the first dose. METHODS: From 7 to 14 May, 2021, we performed a survey to identify the factors associated with hesitancy about receiving the second vaccine dose among HCWs at the hospital who had received the first dose of the vaccine. Based on survey results, a hospital-wide campaign was implemented on 18 May 2021 to improve vaccine coverage. HCWs who completed the second dose completed a self-administered questionnaire to evaluate the effect of the campaign. FINDINGS: Of 1,171 HCWs who had received the first dose of the vaccine, 71.5% completed the online survey, of whom 3.7% refused to take the second dose and 22.3% showed hesitancy. Hesitancy to receive a second dose was significantly associated with age under 30 years and concerns about TTS, and was less common among those who trusted effectiveness and safety of the vaccine. Among HCWs who received the first dose, 96.2% completed vaccination with the second dose between 27 May and 4 June, 2021. Of those who answered the questionnaire asked about the timing of their decision to receive the second dose, 57.1% reported that they were motivated by the hospital-wide campaign. CONCLUSION: A tailored intervention strategy based on a survey can improve COVID-19 vaccination uptake among HCWs.

A Meaningful and Actionable Professionalism Assessment: Validity Evidence for the Professionalism Mini-Evaluation Exercise (P-MEX) Across 8 Years.

PURPOSE: With the growing importance of professionalism in medical education, it is imperative to develop professionalism assessments that demonstrate robust validity evidence. The Professionalism Mini-Evaluation Exercise (P-MEX) is an assessment that has demonstrated validity evidence in the authentic clinical setting. Identifying the factorial structure of professionalism assessments determines professionalism constructs that can be used to provide diagnostic and actionable feedback. This study examines validity evidence for the P-MEX, a focused and standardized assessment of professionalism, in a simulated patient setting. METHOD: The P-MEX was administered to 275 pediatric residency applicants as part of a 3-station standardized patient encounter, pooling data over an 8-year period (2012 to 2019 residency admission years). Reliability and construct validity for the P-MEX were evaluated using Cronbach's alpha, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA). RESULTS: Cronbach's alpha for the P-MEX was 0.91. The EFA yielded 4 factors: doctor-patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. The CFA demonstrated good model fit with a root-mean-square error of approximation of .058 and a comparative fit index of .92, confirming the reproducibility of the 4-factor structure of professionalism. CONCLUSIONS: The P-MEX demonstrates construct validity as an assessment of professionalism, with 4 underlying subdomains in doctor-patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. These results yield new confidence in providing diagnostic and actionable subscores within the P-MEX assessment. Educators may wish to integrate the P-MEX assessment into their professionalism curricula.

Validity Evidence Supporting the Comunicación y Habilidades Interpersonales (CAI) Scale for Medical Spanish Communication and Interpersonal Skills Assessment.

PURPOSE: While many schools teach medical Spanish to improve communication with the growing Spanish-speaking population, a standardized assessment methodology is lacking. This study evaluates validity evidence for a newly developed Spanish Communication and Interpersonal Skills (CIS) scale. METHOD: The authors developed the Comunicación y Habilidades Interpersonales (CAI, pronounced /ki/) scale by culturally and linguistically adapting a 14-item validated English CIS scale. Sources of validity evidence were gathered targeting content, response process, internal structure, relations to other variables, and consequences. The CAI was administered as part of an objective structured clinical examination at 2 medical schools from April to June 2020. RESULTS: The authors mapped CAI items onto USMLE Step 2 CIS behavioral functions and medical Spanish learner competencies to examine content validity. The mean item difficulty and item discrimination estimates are 2.86 (SD = 0.34) and 0.81 (SD = 0.08), respectively, demonstrating good psychometric properties at the item level. Internal-consistency reliability for a single case was 0.96. Learner variance accounted for 12% of total variance, demonstrating ability to differentiate higher and lower performing learners; the person-case interaction accounted for 44% of total variance, indicating strong case specificity. CONCLUSIONS: The CAI has higher case specificity than previously reported in English, suggesting that the nuanced contextual elements of individual cases may matter even more when learners are using limited language skills. Implementing the CAI more widely may promote inclusion of patient-centered communication skills as part of curricular content and activities. This study's validity evidence may inform the development of a future medical Spanish certification examination.

Affective, cognitive, and behavioral outcomes from a resident personal finance curriculum pilot project.

BACKGROUND: The transition to residency marks a significant shift in the financial circumstances of medical trainees. Despite existing resources, residents still cite uncertainty in this domain. A personal finance curriculum is needed to close this educational gap and improve the financial well-being of trainees. METHODS: The curriculum was developed using Kern's framework. Two needs assessments informed the consensus development of goals and objectives, educational strategies, and assessments. Course material was hosted online for asynchronous review and complemented by two 1-hour webinars. The curriculum was piloted at one institution. Participants completed (1) knowledge assessments before and after the intervention, (2) a survey of reactions to the curriculum, and (3) an assessment of financial behavioral changes after the intervention. RESULTS: Thirty-seven residents (37/49, 76%) enrolled in the curriculum. Among participants, 20 (20/37, 54%) completed the curriculum. Most participants agreed or strongly agreed that the content was relevant (20/20, 100%) and clearly presented (19/20, 95%) and that they would recommend the curriculum to other residents (20/20, 100%). Performance on the knowledge assessment improved 21% after the intervention (mean ± SD = pretest 57% ± 17%, posttest = 78% ± 12%; p < 0.001). Most residents (17/20, 85%) also reported behavioral changes including setting new financial goals (12/20, 60%), taking new action toward financial planning (11/20, 55%), and changing financial habits (6/20, 30%). There were no direct financial costs incurred in the implementation of this pilot. CONCLUSIONS: This is a successful pilot of a virtual personal finance curriculum with positive outcomes data. Addressing this problem at scale will require buy-in from educators around the country to deliver this information to residents that may not otherwise seek it out. Future study should assess curricular outcomes in other settings and the durability of acquired knowledge and behavioral changes over time.