Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study.

PURPOSE: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into six elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. METHODS: Messick's unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from three pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018-2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). RESULTS: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone, r = 0.34, P = .019. CONCLUSION: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Giving Literal Thanks: An ACGME-Sponsored Initiative to Bring Residents Back to the Bedside.

Background Resident burnout is at an all-time high. In response, the Accreditation Council for Graduate Medical Education (ACGME) developed the Back to Bedside grant for resident-led burnout interventions that increase the time residents spend with patients. Objective We designed a resident-patient reading intervention, Giving Literal Thanks (GLT), intended to increase meaningful time residents spend with patients and thereby decrease burnout. Methods All 65 pediatric residents rotating through our academic hospital's inpatient units from Fall 2019 through Fall 2021 were invited to read and gift books to their patients. We studied our intervention's relationship to resident burnout using a convergent mixed-methods design, including anonymous, unlinked pre-, peri-, and post-intervention surveys and focus groups. Qualitative and quantitative data were analyzed separately, then integrated to describe burnout pre- and post-intervention. Results Forty-one of 65 residents (63.1%) completed pre-intervention surveys, and 8 of 65 (12.3%) completed post-intervention surveys. Twenty-seven resident-patient reading interactions were recorded, and 2 focus groups were held (1 pre- and 1 post-intervention). Five themes were identified: (1) limited opportunities exist to spend time at the bedside; (2) spending time at the bedside is valuable; (3) other responsibilities may preclude time at the bedside; (4) GLT could promote positive outcomes; and (5) GLT might not be the right tool to reduce burnout. Further quantitative data analysis was prevented by low survey response rates. While GLT was positively received and feasible, we were unable to show an improvement in burnout. Conclusions GLT was well-regarded but may not improve resident burnout.

Confidence of Faculty in Writing Letters of Recommendation for Pediatric Fellowship Applicants.

BACKGROUND: The assessment of pediatric residents applying to subspecialty fellowship programs relies on faculty letters of recommendation (LOR). However, it is unclear if pediatric faculty are confident that their LOR are effective. OBJECTIVE: This study aims to assess the confidence of pediatric faculty in writing an effective LOR for pediatric residents applying to subspecialty fellowship programs. METHODS: Survey development was conducted using evidence-based best practices. Surveys were distributed via email in 2021 to all full-time pediatric faculty members who taught pediatric residents in a large academic medical center. Categorical values were compared by chi-square test. RESULTS: Eighty-five out of 150 (57%) faculty members completed the survey. Forty-one percent of participants were very confident that their LOR provided adequate content to assess residents during the application process. Confidence was associated with higher academic rank (p=0.02), frequent contact with residents (p=0.01), and writing >2 LOR in the last five years (p=0.0002). Confident LOR writers were more likely to describe their own background, details about the resident's scholarly activity, and the resident's ability to work as part of a team. Thirty-five percent of respondents reported never considering gender bias when writing LOR, whereas 28% reported always considering gender bias. Eighty-seven percent of respondents reported an interest in receiving LOR writing guidelines. CONCLUSION: Half of the faculty respondents were not very confident in their ability to write an effective LOR for pediatric residents applying for a fellowship. Faculty development and standardized instructions on writing effective LOR may be helpful both at the institutional and national levels, including the importance of considering gender bias when writing LOR.

Development and Implementation of a Novel Case-Based Gun Violence Prevention Training Program for First-Year Residents.

PROBLEM: Gun violence results in approximately 40,000 deaths in the United States each year, yet physicians rarely discuss gun access and firearm safety with patients. Lack of education about how to have these conversations is an important barrier, particularly among trainees. APPROACH: A 2-part training curriculum was developed for first-year residents. It included (1) a didactic presentation outlining a framework to understand types of firearm-related violence, describing institutional resources, and reviewing strategies for approaching discussions about firearms with patients, and (2) interactive case scenarios, adjusted for clinical disciplines, with standardized patients. Before and after the training, participants completed surveys on the training's relevance, efficacy, and benefit. Standardized patients provided real-time feedback to participants and completed assessments based on prespecified learning objectives. OUTCOMES: In June-August 2019, 148 first-year residents in internal medicine (n = 74), general surgery (n = 12), emergency medicine (n = 15), pediatrics (n = 22), psychiatry (n = 16), and OB/GYN (n = 9) completed the training. Most (70%, n = 104) reported having no prior exposure to gun violence prevention education. Knowledge about available resources increased among participants from 3% (n = 5) pretraining to 97% (n = 143) post-training. Awareness about relevant laws, such as Extreme Risk Protection Orders, and their appropriate use increased from 3% (n = 4) pretraining to 98% (n = 145) post-training. Comfort discussing access to guns and gun safety with patients increased from a median of 5 pretraining to 8 post-training (on a scale of 1-10, with higher scores indicating more comfort). NEXT STEPS: Delivery of a case-based gun violence prevention training program was effective and feasible in a single institution. Next steps include expanding the training to other learners (across undergraduate and graduate medical education) and institutions and assessing how the program changes practice over time.

Creation of the Association of Pediatric Program Directors Faculty Development for Educators "Nuts and Bolts" Teaching Aids.

Faculty development (FD) continues to be a great need and challenge for faculty engaged in graduate medical education (GME) and such educator development should ultimately benefit learners in GME programs. As a Task Force within the Association of Pediatric Program Directors (APPD) FD Learning Community, our Educator subcommittee utilized multiple needs assessments to develop a novel educator development program called the APPD FD for Educators Nuts and Bolts (Nuts & Bolts) consisting of 1 to 2 page FD teaching aids. This paper outlines the development of these teaching aids and can be used as a reference for developing future Nuts & Bolts teaching aids. The development was based on 1) a defining framework (Glassick's criteria), 2) a clear model for evaluation of the program (Logic model), and 3) development of an assessment tool to demonstrate effectiveness of the teaching aids in FD at the local level. These 3 components help establish the scholarly nature of the FD Nuts & Bolts program. Multiple lessons learned from development of this program are presented to inform others engaged in educator program development. Detailed proposals for use of FD Nuts & Bolts for faculty improvement and plans for generating additional teaching aids are presented.

Establishing Psychological Safety to Obtain Feedback for Training Programs: A Novel Cross-Specialty Focus Group Exchange.

BACKGROUND: Formative feedback from residents is essential to improve residency programs, and focus groups may provide rich information. However, residents may withhold information due to fear of retaliation or speak less candidly to please focus group moderators. OBJECTIVE: We assessed participant perceptions and utility of feedback obtained from a confidential focus group exchange between 2 residency programs. METHODS: Anesthesiology and pediatric programs at the same institution participated in 2017. Residents voluntarily provided program feedback during 1 of 2 confidential focus groups for each program. Each focus group was moderated by the program director (PD) of the other specialty. The PDs used thematic analysis to identify themes for use by the respective programs in improvement efforts. An anonymous survey was distributed after the focus groups to collect participant perceptions (quantitative and narrative) on this approach. RESULTS: Thirteen residents of 140 (9.3%) participated (7 anesthesiology, 6 pediatrics). Thematic feedback from focus groups was largely consistent with known issues, although novel information was also obtained (eg, pediatric interns wanted earlier one-on-one meetings with their PD). Survey data suggest that residents were able to share more meaningful feedback than they would otherwise, and they did not feel that having an external moderator (a PD who may have been unfamiliar with the specialty) was a barrier to discussion. The approach required 6 hours of time for each PD and approximately $200 for dinners. CONCLUSIONS: The focus group exchange required modest resources, was perceived as safe by residents, and generated robust, actionable feedback for the programs.

Psyche Meets the Gatekeepers: Creating a More Humane Culture for Women in Medicine.

In the Greek myth of Psyche and Eros, Psyche must fulfill four seemingly impossible tasks to achieve full consciousness before she can be reunited with her bridegroom, Eros. From early childhood, girls and women can encounter gender stereotyping, sexual harassment and assault, and other gender-related challenges. Although both men and women can face mistreatment in medical school, female students experience sexual harassment and sexual assault at higher rates than male students. In medical training and career advancement, women often face additional obstacles unrelated to the formal education of physicians, such as salary and promotional disparities, and lack of mentorship, sponsorship, or role models. The suicide rate for male physicians is nearly 1.5 times higher than the general population; for female physicians, it is more than twice as high. Parenthood can pose additional challenges for mothers and fathers early in their academic careers, and women are vastly underrepresented as they try to move past the "gatekeepers" into leadership roles. Using the framework of the Psyche and Eros myth to examine the trajectory of a female physician's career, this article provides insights into these challenges, as well as strategies to address some of these inequities, such as programs to support female promotion and leadership, expanded mentorship and mentor models, sponsorship opportunities, leadership accountability, implicit bias training, and others. The authors call for the participation of both men and women as essential in ventures to create a more humane environment for the training and practice of medicine.

Promoting Emotional Well-Being Through an Innovative Personal and Professional Development Curriculum for Pediatric Residents.

Burnout and attenuation of empathy during training are significant problems facing pediatric residency programs. To proactively address these issues, a curriculum of Personal and Professional Development was created to build skills of reflection and exploring emotions. Data on 3 years of this program suggests that it prevents erosion of empathy.

Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns.

BACKGROUND: Clinical competency committee (CCC) identification of residents with performance concerns is critical for early intervention. METHODS: Program directors and 94 CCC members at 14 pediatric residency programs responded to a written survey prompt asking them to describe how they identify residents with performance concerns. Data was analyzed using thematic analysis. RESULTS: Six themes emerged from analysis and were grouped into two domains. The first domain included four themes, each describing a path through which residents could meet or exceed a concern threshold:1) written comments from rotation assessments are foundational in identifying residents with performance concerns, 2) concerning performance extremes stand out, 3) isolated data points may accumulate to raise concern, and 4) developmental trajectory matters. The second domain focused on how CCC members and program directors interpret data to make decisions about residents with concerns and contained 2 themes: 1) using norm- and/or criterion-referenced interpretation, and 2) assessing the quality of the data that is reviewed. CONCLUSIONS: Identifying residents with performance concerns is important for their education and the care they provide. This study delineates strategies used by CCC members across several programs for identifying these residents, which may be helpful for other CCCs to consider in their efforts.

Simulation for Milestone Assessment: Use of a Longitudinal Curriculum for Pediatric Residents.

INTRODUCTION: Determining clinical competency on the milestones requires direct observation of residents, which is difficult for faculty members who are also providing patient care. Simulation can potentially represent an effective standardized tool for high-stakes assessment. Using a longitudinal simulation curriculum with formative and summative components, we conducted a pilot investigation to examine whether (1) performance on the formative cases predicted performance on the summative cases and (2) performance on the summative cases correlated with the clinical competency committee's (CCC) milestone placement. METHODS: We developed 6 formative and 6 matched summative cases for pediatric interns that covered core pediatric topics. The interns progressed through the formative cases in pairs during the course of the academic year and then through the summative cases back to back individually at the end of the year. The interns were evaluated using a competency-based simulation evaluation. We determined the relationship between the formative and summative scores and between the summative scores and those from the CCC. RESULTS: The relationship between formative and summative scores was not statistically significant. There was a statistically significant relationship between summative and CCC scores. CONCLUSIONS: Significant methodological limitations preclude definite conclusions about the predictive power of simulation cases for Pediatric Milestones-based assessment. However, our work is an example of how simulation is a potentially useful tool for assessing residents' skill development on the Pediatric Milestones. More rigorous research is needed to determine the extent to which simulation can be used for high-stakes, milestones-based assessment.

A multi-institutional study of the emotional intelligence of resident physicians.

BACKGROUND: Although emotional intelligence (EI) may have a role in the development of Accreditation Council for Graduate Medical Education core competencies, few studies have measured resident EI across specialties. This study aimed to describe the EI of resident physicians across multiple specialties. METHODS: Three hundred twenty five surgery, pediatric, and pathology residents at 3 large academic institutions were invited to complete the psychometrically validated Trait Emotional Intelligence Questionnaire. RESULTS: The response rate was 42.8% (n = 139). Global EI of all residents (101.0 ± 8.1) was comparable with, but less variable than, the general population sample and was not statistically different between specialties. Compared with the norm sample, residents in the 3 specialty groups demonstrated unique combinations of areas of relative high and low development. CONCLUSIONS: There exist distinct strengths and opportunities for the development for surgery, pediatrics, and pathology residents. Future investigations could use EI profiling to create educational interventions to develop specific areas of EI and assess correlation with resident performance.

Are there gender differences in the emotional intelligence of resident physicians?

BACKGROUND: Because academic literature indicates that emotional intelligence (EI) is tied to work performance, job satisfaction, burnout, and client satisfaction, there is great interest in understanding physician EI. OBJECTIVE: To determine whether gender differences in resident EI profiles mirror EI gender differences in the general population. STUDY DESIGN (INCLUDE PARTICIPANTS AND SETTING): A total of 325 residents in 3 types of residency programs (pathology, pediatrics, and general surgery) at 3 large academic institutions were invited electronically to complete the validated Trait Emotional Intelligence Questionnaire (TEIQue), a tool consisting of 153 items that cluster to 15 independent facets, 4 composite factors, and 1 global EI score. RESULTS: The response rate was 42.8% (n = 139, women = 84). Global EI was not significantly different between men and women resident physicians (p = 0.74). Women scored higher than men in the TEIQue facets impulse control (p = 0.004) and relationships (p = 0.004). Men scored higher than women in 2 facets, stress management (p = 0.008) and emotion management (p = 0.023). Within surgery (n = 85, women = 46), women scored higher than men in impulse control (p = 0.006), whereas men scored higher in stress management (p = 0.008). CONCLUSIONS: Men and women residents across 3 specialties demonstrated near-identical global EI scores. However, gender differences in specific TEIQue facets suggest that similar to the general population, men and women residents may benefit from specific training of different EI domains to enhance well-rounded development. The lack of significant gender differences within surgery may indicate that surgery attracts individuals with particular EI profiles regardless of gender. Future research should focus on the functional relationship between educational interventions that promote targeted EI development and enhanced clinical performance.