Analysis of narrative assessments of internal medicine resident performance: are there differences associated with gender or race and ethnicity?

BACKGROUND: Equitable assessment is critical in competency-based medical education. This study explores differences in key characteristics of qualitative assessments (i.e., narrative comments or assessment feedback) of internal medicine postgraduate resident performance associated with gender and race and ethnicity. METHODS: Analysis of narrative comments included in faculty assessments of resident performance from six internal medicine residency programs was conducted. Content analysis was used to assess two key characteristics of comments- valence (overall positive or negative orientation) and specificity (detailed nature and actionability of comment) - via a blinded, multi-analyst approach. Differences in comment valence and specificity with gender and race and ethnicity were assessed using multilevel regression, controlling for multiple covariates including quantitative competency ratings. RESULTS: Data included 3,383 evaluations with narrative comments by 597 faculty of 698 residents, including 45% of comments about women residents and 13.2% about residents who identified with race and ethnicities underrepresented in medicine. Most comments were moderately specific and positive. Comments about women residents were more positive (estimate 0.06, p 0.045) but less specific (estimate - 0.07, p 0.002) compared to men. Women residents were more likely to receive non-specific, weakly specific or no comments (adjusted OR 1.29, p 0.012) and less likely to receive highly specific comments (adjusted OR 0.71, p 0.003) or comments with specific examples of things done well or areas for growth (adjusted OR 0.74, p 0.003) than men. Gendered differences in comment specificity and valence were most notable early in training. Comment specificity and valence did not differ with resident race and ethnicity (specificity: estimate 0.03, p 0.32; valence: estimate - 0.05, p 0.26) or faculty gender (specificity: estimate 0.06, p 0.15; valence: estimate 0.02 p 0.54). CONCLUSION: There were significant differences in the specificity and valence of qualitative assessments associated with resident gender with women receiving more praising but less specific and actionable comments. This suggests a lost opportunity for well-rounded assessment feedback to the disadvantage of women.

Exploring gender and thematic differences in qualitative assessments of internal medicine resident performance.

INTRODUCTION: Evidence suggests gender disparities in medical education assessment, including differences in ratings of competency and narrative comments provided in resident performance assessments. This study explores how gender manifests within the content of qualitative assessments (i.e., narrative comments or performance feedback) of resident performance. METHODS: Qualitative content analysis was used to explore gender-based differences in narrative comments included in faculty assessments of resident performance during inpatient medicine rotations at six Internal Medicine residency programs, 2016-2017. A blinded, multi-analyst approach was employed to identify themes across comments. Patterns in themes with resident gender and post-graduate year (PGY) were explored, focusing on PGY2 and PGY3 when residents are serving in the team leader role. RESULTS: Data included 3,383 evaluations with narrative comments of 385 men (55.2%) and 313 women residents (44.8%). There were thematic differences in narrative comments received by men and women residents and how these themes manifested within comments changed with training time. Compared to men, comments about women had a persistent relationship-orientation and emphasized confidence over training including as interns and in PGY2 and PGY3, when serving as team leader. The relationship-orientation was characterized not only by the residents' communal attributes but also their interpersonal and communication skills, including efforts supporting others and establishing the tone for the team. Comments about women residents often highlighted confidence, including recommendations around behaviors that convey confidence in decision-making and team leadership. DISCUSSION: There were gender-based thematic differences in qualitative assessments. Comments about women resident team leaders highlight relationship building skills and urge confidence and actions that convey confidence as team leader. Persistent attention to communal skills suggests gendered expectations for women resident team leaders and a lost opportunity for well-rounded feedback to the disadvantage of women residents. These findings may inform interventions to promote equitable assessment, such as providing feedback across the competencies.

Implementation of a Problem-Based Presentation Format to Improve Residents' Ambulatory Patient Presentations.

Background: The format for residents to present hospitalized patients to teaching faculty is well defined; however, guidance for presenting in clinic is not uniform. Objective: We report the development, implementation, and evaluation of a new standardized format for presenting in clinic: the Problem-Based Presentation (PBP). Methods: After a needs assessment, we implemented the format at the teaching clinics of our internal medicine residency program. We surveyed participants on innovation outcomes, feasibility, and acceptability (pre-post design; 2019-2020; 5-point scale). Residents' primary outcomes were confidence in presentation content and presentation order, presentation efficiency, and presentation organization. Faculty were asked about the primary outcomes of resident presentation efficiency, presentation organization, and satisfaction with resident presentations. Results: Participants were 111 residents and 22 faculty (pre-intervention) and 110 residents and 20 faculty (post-intervention). Residents' confidence in knowing what the attending physician wants to hear in an outpatient presentation, confidence in what order to present the information, and how organized they felt when presenting in clinic improved (all P<.001; absolute increase of the top 2 ratings of 25%, 28%, and 31%, respectively). Residents' perceived education in their outpatient clinic also improved (P=.002; absolute increase of the top 2 ratings of 19%). Faculty were more satisfied with the structured presentations (P=.008; absolute increase of the top 2 ratings of 27%). Conclusions: Implementation of a new format for presenting in clinic was associated with increased resident confidence in presentation content, order of items, overall organization, and a perceived increase in the frequency of teaching points reviewed by attending physicians.

Continuity and Health Outcomes in Resident Clinics: A Scoping Review of the Literature.

Continuity of care is an essential component of primary care, resulting in improved satisfaction, management of chronic conditions, and adherence to screening recommendations. The impact of continuity of care in teaching practices remains unclear. We performed a scoping review of the literature to understand the impact of continuity on patients and trainees in teaching practices. A systematic search was performed through PubMed to identify articles published prior to January 2020 addressing continuity of care and health outcomes in resident primary care clinic settings. A total of 543 abstracts were evaluated by paired independent reviewers. In total, 24 articles met the inclusion criteria and were abstracted by four authors. These articles included a total of 6,973 residents (median = 96, range = 9-5,000) and over 1,000,000 patients (median = 428, range = 70-1,000,000). Most publications demonstrated that higher continuity was associated with better diabetic care (71%, n = five of seven), receipt of preventive care per guidelines (60%, n = three of five), and lower costs or administrative burden of care (100%, n = three of three). A smaller proportion of publications reported a positive association between continuity and hypertension control (28%, n = two of seven). The majority of publications evaluating patient/resident satisfaction demonstrated that better continuity was associated with higher patient (67%, n = four of six) and resident (67%, n = six of nine) satisfaction. A review of the existing literature revealed that higher continuity of care in resident primary care clinics was associated with better patient health outcomes and patient/resident satisfaction. Interventions to improve continuity in training settings are needed.

Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education.

BACKGROUND: Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern. OBJECTIVE: Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity. DESIGN: Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square. PARTICIPANTS: A total of 689 IM residents, including 330 women and URiM residents (48%). MAIN MEASURES: ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F(2, 1321) 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37). CONCLUSION: Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.

Association Between Resident Race and Ethnicity and Clinical Performance Assessment Scores in Graduate Medical Education.

PURPOSE: To assess the association between internal medicine (IM) residents' race/ethnicity and clinical performance assessments. METHOD: The authors conducted a cross-sectional analysis of clinical performance assessment scores at 6 U.S. IM residency programs from 2016 to 2017. Residents underrepresented in medicine (URiM) were identified using self-reported race/ethnicity. Standardized scores were calculated for Accreditation Council for Graduate Medical Education core competencies. Cross-classified mixed-effects regression assessed the association between race/ethnicity and competency scores, adjusting for rotation time of year and setting; resident gender, postgraduate year, and IM In-Training Examination percentile rank; and faculty gender, rank, and specialty. RESULTS: Data included 3,600 evaluations by 605 faculty of 703 residents, including 94 (13.4%) URiM residents. Resident race/ethnicity was associated with competency scores, with lower scores for URiM residents (difference in adjusted standardized scores between URiM and non-URiM residents, mean [standard error]) in medical knowledge (-0.123 [0.05], P = .021), systems-based practice (-0.179 [0.05], P = .005), practice-based learning and improvement (-0.112 [0.05], P = .032), professionalism (-0.116 [0.06], P = .036), and interpersonal and communication skills (-0.113 [0.06], P = .044). Translating this to a 1 to 5 scale in 0.5 increments, URiM resident ratings were 0.07 to 0.12 points lower than non-URiM resident ratings in these 5 competencies. The interaction with faculty gender was notable in professionalism (difference between URiM and non-URiM for men faculty -0.199 [0.06] vs women faculty -0.014 [0.07], P = .01) with men more than women faculty rating URiM residents lower than non-URiM residents. Using the 1 to 5 scale, men faculty rated URiM residents 0.13 points lower than non-URiM residents in professionalism. CONCLUSIONS: Resident race/ethnicity was associated with assessment scores to the disadvantage of URiM residents. This may reflect bias in faculty assessment, effects of a noninclusive learning environment, or structural inequities in assessment.

Association of Gender With Learner Assessment in Graduate Medical Education.

Importance: Gender bias may affect assessment in competency-based medical education. Objective: To evaluate the association of gender with assessment of internal medicine residents. Design, Setting, and Participants: This multisite, retrospective, cross-sectional study included 6 internal medicine residency programs in the United States. Data were collected from July 1, 2016, to June 30, 2017, and analyzed from June 7 to November 6, 2019. Exposures: Faculty assessments of resident performance during general medicine inpatient rotations. Main Outcomes and Measures: Standardized scores were calculated based on rating distributions for the Accreditation Council for Graduate Medical Education's core competencies and internal medicine Milestones at each site. Standardized scores are expressed as SDs from the mean. The interaction of gender and postgraduate year (PGY) with standardized scores was assessed, adjusting for site, time of year, resident In-Training Examination percentile rank, and faculty rank and specialty. Results: Data included 3600 evaluations for 703 residents (387 male [55.0%]) by 605 faculty (318 male [52.6%]). Interaction between resident gender and PGY was significant in 6 core competencies. In PGY2, female residents scored significantly higher than male residents in 4 of 6 competencies, including patient care (mean standardized score [SE], 0.10 [0.04] vs 0.22 [0.05]; P = .04), systems-based practice (mean standardized score [SE], -0.06 [0.05] vs 0.13 [0.05]; P = .003), professionalism (mean standardized score [SE], -0.04 [0.06] vs 0.21 [0.06]; P = .001), and interpersonal and communication skills (mean standardized score [SE], 0.06 [0.05] vs 0.32 [0.06]; P < .001). In PGY3, male residents scored significantly higher than female patients in 5 of 6 competencies, including patient care (mean standardized score [SE], 0.47 [0.05] vs 0.32 [0.05]; P = .03), medical knowledge (mean standardized score [SE], 0.47 [0.05] vs 0.24 [0.06]; P = .003), systems-based practice (mean standardized score [SE], 0.30 [0.05] vs 0.12 [0.06]; P = .02), practice-based learning (mean standardized score [SE], 0.39 [0.05] vs 0.16 [0.06]; P = .004), and professionalism (mean standardized score [SE], 0.35 [0.05] vs 0.18 [0.06]; P = .03). There was a significant increase in male residents' competency scores between PGY2 and PGY3 (range of difference in mean adjusted standardized scores between PGY2 and PGY3, 0.208-0.391; P ≤ .002) that was not seen in female residents' scores (range of difference in mean adjusted standardized scores between PGY2 and PGY3, -0.117 to 0.101; P ≥ .14). There was a significant increase in male residents' scores between PGY2 and PGY3 cohorts in 6 competencies with female faculty and in 4 competencies with male faculty. There was no significant change in female residents' competency scores between PGY2 to PGY3 cohorts with male or female faculty. Interaction between faculty-resident gender dyad and PGY was significant in the patient care competency (ß estimate [SE] for female vs male dyad in PGY1 vs PGY3, 0.184 [0.158]; ß estimate [SE] for female vs male dyad in PGY2 vs PGY3, 0.457 [0.181]; P = .04). Conclusions and Relevance: In this study, resident gender was associated with differences in faculty assessments of resident performance, and differences were linked to PGY. In contrast to male residents' scores, female residents' scores displayed a peak-and-plateau pattern whereby assessment scores peaked in PGY2. Notably, the peak-and-plateau pattern was seen in assessments by male and female faculty. Further study of factors that influence gender-based differences in assessment is needed.

NCCN Guidelines Insights: B-Cell Lymphomas, Version 3.2019.

Diffuse large B-cell lymphomas (DLBCLs) and follicular lymphoma (FL) are the most common subtypes of B-cell non-Hodgkin's lymphomas in adults. Histologic transformation of FL to DLBCL (TFL) occurs in approximately 15% of patients and is generally associated with a poor clinical outcome. Phosphatidylinositol 3-kinase (PI3K) inhibitors have shown promising results in the treatment of relapsed/refractory FL. CAR T-cell therapy (axicabtagene ciloleucel and tisagenlecleucel) has emerged as a novel treatment option for relapsed/refractory DLBCL and TFL. These NCCN Guidelines Insights highlight important updates to the NCCN Guidelines for B-Cell Lymphomas regarding the treatment of TFL and relapsed/refractory FL and DLBCL.

Gender Bias in Resident Assessment in Graduate Medical Education: Review of the Literature.

BACKGROUND: Competency-based medical education relies on meaningful resident assessment. Implicit gender bias represents a potential threat to the integrity of resident assessment. We sought to examine the available evidence of the potential for and impact of gender bias in resident assessment in graduate medical education. METHODS: A systematic literature review was performed to evaluate the presence and influence of gender bias on resident assessment. We searched Medline and Embase databases to capture relevant articles using a tiered strategy. Review was conducted by two independent, blinded reviewers. We included studies with primary objective of examining the impact of gender on resident assessment in graduate medical education in the USA or Canada published from 1998 to 2018. RESULTS: Nine studies examined the existence and influence of gender bias in resident assessment and data included rating scores and qualitative comments. Heterogeneity in tools, outcome measures, and methodologic approach precluded meta-analysis. Five of the nine studies reported a difference in outcomes attributed to gender including gender-based differences in traits ascribed to residents, consistency of feedback, and performance measures. CONCLUSION: Our review suggests that gender bias poses a potential threat to the integrity of resident assessment in graduate medical education. Future study is warranted to understand how gender bias manifests in resident assessment, impact on learners and approaches to mitigate this bias.

What Attracts Medical Students to Primary Care? A Nominal Group Evaluation.

OBJECTIVE: To examine the perceptions of first-year medical students on their experiences in primary care. METHODS: Nominal group technique sessions were conducted with first-year medical students for 5 years. Questions were designed to evaluate primary care experiences and the role of primary care physicians. The questions explored what would make them consider primary care, what would detract from it, and what primary care has to offer that no other specialty can. Responses were weighted and ranked. The main outcome was the top five responses to three questions that were obtained at each session. RESULTS: Thirty-four students generated 280 responses to 3 questions. The top 5 responses for each year resulted in 29 experiences that strengthen enthusiasm: patient interactions (weighted sum, 43%), physician interactions/role modeling (22%), community interactions (20%), healthcare system/finances (8%), and other (6%). The top 5 responses resulted in 26 experiences that weaken enthusiasm, including hidden curriculum (45%), poor role models (29%), uncertainties about the healthcare system such as finances and documentation (20%), and patient interactions (6%). The top 5 responses regarding the uniqueness of primary care resulted in 37 experiences, including patient interactions (38%), continuity of care (20%), knowledge base (13%), community impact (10%), lifestyle benefits (10%), and education/prevention (9%). CONCLUSIONS: Medical students highlighted unique relationships with patients and continuity of care as experiences that increase their enthusiasm for primary care. Negative experiences that weakened enthusiasm for primary care included hidden curriculum and poor role models. Programs that provide experiences in primary care can increase student interest in primary care careers.

Signed, Sealed, Delivered: Increasing Patient Notification of Test Results in an Internal Medicine Resident Continuity Clinic.

OBJECTIVES: Informing patients of their test results is an important patient safety issue, yet many physicians perform dismally in this regard. Residents often face additional barriers to communicating test results to patients. We wanted to determine whether streamlining the notification process, communicating expectations, and having residents audit their performance would increase result notification rates. METHODS: We used a quasi-experimental design, and a single-group before-and-after intervention. Our multifold intervention consisted of development and standardization of a notification process in the electronic medical record, an education component, and a self-audit component. During a 15-minute session, we educated residents on the use of the new process. We also restated expectations regarding notifying patients of their results. Residents audited their own charts for a period before the intervention and during a second, postintervention period. An independent review of notification rates took place simultaneously as well as during an additional period several months later. RESULTS: In total, 87 residents were eligible for participation. All 87 completed the project, giving a 100% participation rate. Resident-reported laboratory test notification rates increased from 16% to 91%; other test result rates increased from 33% to 84%. The three independent reviews showed laboratory test notification rates increased from 18.5% to 71.7% to 87.1%, and notification of other test results increased from 23.5% to 66.7% to 91.7%. CONCLUSIONS: Baseline rates of notification for diagnostic tests results were low, but streamlining the notification process, clearly stating expectations for using it, and using resident self-audit can improve notification rates.

NCCN Guidelines Insights: Chronic Lymphocytic Leukemia/Small Lymphocytic Lymphoma, Version 1.2017.

Chronic lymphocytic leukemia (CLL) and small lymphocytic lymphoma (SLL) are different manifestations of the same disease and managed in much the same way. The advent of novel CD20 monoclonal antibodies led to the development of effective chemoimmunotherapy regimens. More recently, small molecule inhibitors targeting kinases involved in a number of critical signaling pathways and a small molecule inhibitor of the BCL-2 family of proteins have demonstrated activity for the treatment of patients with CLL/SLL. These NCCN Guidelines Insights highlight important updates to the NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) for CLL/SLL for the treatment of patients with newly diagnosed or relapsed/refractory CLL/SLL.

Impact of an educational intervention on provider knowledge, attitudes, and comfort level regarding counseling women ages 40-49 about breast cancer screening.

BACKGROUND: Mammography screening for women under the age of 50 is controversial. Groups such as the US Preventive Services Task Force recommend counseling women 40-49 years of age about mammography risks and benefits in order to incorporate the individual patient's values in decisions regarding screening. We assessed the impact of a brief educational intervention on the knowledge and attitudes of clinicians regarding breast cancer screening. METHODS: The educational intervention included a review of the risks and benefits of screening, individual risk assessment, and counseling methods. Sessions were led by a physician expert in breast cancer screening. Participants were physicians and nurses in 13 US Department of Veterans Affairs primary care clinics in Alabama. Outcomes were as follows: 1) knowledge assessment of mammogram screening recommendations; 2) counseling practices on the risks and benefits of screening; and 3) comfort level with counseling about screening. Outcomes were assessed by survey before and after the intervention. RESULTS: After the intervention, significant changes in attitudes about breast cancer screening were seen. There was a decrease in the percentage of participants who reported that they would screen all women ages 40-49 years (82% before the intervention, 9% afterward). There was an increase in the percentage of participants who reported that they would wait until the patient was 50 years old before beginning to screen (12% before the intervention, 38% afterward). More participants (5% before, 53% after; P<0.001) said that they would discuss the patient's preferences. Attitudes favoring discussion of screening benefits increased, though not significantly, from 94% to 99% (P=0.076). Attitudes favoring discussion of screening risks increased from 34% to 90% (P<0.001). The comfort level with discussing benefits increased from a mean of 3.8 to a mean of 4.5 (P<0.001); the comfort level with discussing screening risks increased from 2.7 to 4.3 (P<0.001); and the comfort level with discussing cancer risks and screening preferences with patients increased from 3.2 to 4.3 (P<0.001). (The comfort levels measurements were assessed by using a Likert scale, for which 1= not comfortable and 5= very comfortable.). CONCLUSION: Most clinicians in the US Department of Veterans Affairs ambulatory practices in Alabama reported that they routinely discuss mammography benefits but not potential harms with patients. An educational intervention detailing recommendations and counseling methods affected the knowledge and attitudes about breast cancer screening. Participants expressed greater likelihood of discussing screening options in the future.

A qualitative assessment of internal medicine resident perceptions of graduate medical education following implementation of the 2011 ACGME duty hour standards.

BACKGROUND: In 2011, the Accreditation Council of Graduate Medical Education implemented updated guidelines for medical resident duty hours, further limiting continuous work hours for first-year residents. We sought to investigate the impact of these restrictions on graduate medical education among internal medicine residents. METHODS: We conducted eight focus groups with internal medicine residents at the University of Alabama at Birmingham in 06/2012-07/2012. Discussion questions included, "How do you feel the 2011 ACGME work hour restrictions have impacted your graduate medical education?" Transcripts of the focus groups were reviewed and themes identified using a deductive/inductive approach. Participants completed a survey to collect demographic information and future practice plans. RESULTS: Thirty-four residents participated in our focus groups. Five themes emerged: decreased teaching, decreased experiential learning, shift-work mentality, tension between residency classes, and benefits and opportunities. Residents reported that since implementation of the guidelines, teaching was often deferred to complete patient-care tasks. Residents voiced concern that PGY-1 s were not receiving adequate clinical experience and that procedural and clinical reasoning skills are being negatively impacted. PGY-1 s reported being well-rested and having increased time for independent study. CONCLUSIONS: Residents noted a decline in teaching and are concerned with the decrease in "hands-on" clinical education that is inevitably impacted by fewer hours in the hospital, though some benefits were also reported. Future studies are needed to further elucidate the impact of decreased resident work hours on graduate medical education.

Preparing for oral scientific and clinical vignette presentations.

BACKGROUND: Little is known about how faculty, residents, and fellows practice for oral presentations at academic meetings. We sought to categorize presenters' practice styles and the impact of feedback. METHODS: We surveyed oral presenters at 5 annual academic general internal medicine meetings between 2008 and 2010, using a cross-sectional design. Main measures were frequency and settings of practice, most helpful practice setting, changes made in response to feedback, impact of feedback, and perceived quality of presentation. RESULTS: The response rate was 63% (333/525 responders). Respondents represented 59 academic medical centers. Presenters reported practicing in a mean ± SD of 2.3 (±1.3) of 5 different settings. Of the 46% of presenters (152/333) who practiced in front of a group of more experienced colleagues, 80% of presenters (122/152) reported it was the most helpful setting. Eighty-one percent of presenters (268/333) practiced alone, and 25% of presenters (82/333) reported practicing alone was the most helpful setting. The mean numbers of change types reported by faculty were fewer than those reported by residents and fellows (mean 2.3 ± 1.8, and 3.1 ± 2.0, respectively; P < .001). Practicing alone was not associated with changes in content (P  =  .30), visual aids (P  =  .12), or delivery style (P  =  .53). CONCLUSIONS: Practicing in front of a group of experienced colleagues was the most helpful setting in which to prepare for an oral academic meeting presentation, but it was not universally utilized. Feedback given at these sessions was more likely to result in changes made to the presentation; however, broader implementation of such sessions 5 require institutional support.