Values-Based Resident Selection in an Internal Medicine-Pediatrics Residency Program.

BACKGROUND: There is no consensus regarding values important for medical resident success, and current methods for selecting residents correlate poorly with success in residency. OBJECTIVE: We developed and validated a set of values demonstrated by exemplary residents in the Internal Medicine-Pediatrics program at the University of Utah and used them to inform our resident selection process. DESIGN: We utilized a modified Delphi method to identify and internally validate values of successful residents. We implemented these values into the interview evaluation rubric. PARTICIPANTS: Four members of the Internal Medicine-Pediatrics residency program leadership and eleven current residents aided in value generation. Nine faculty from leadership positions in the residency programs of Internal Medicine-Pediatrics, Internal Medicine, and Pediatrics formed a local expert panel for validation. APPROACH: We performed a literature review and engaged local stakeholders in a semi-structured group interview to generate 107 values. After consolidation based on redundancy, two iterative cycles of expert review using a modified Delphi approach, and alignment with the Accreditation Council for Graduate Medical Education core competencies, eleven values achieved expert agreement and were integrated into an interview rubric to aid in resident selection. KEY RESULTS: We identified eleven values important for resident success: academic strength, intellectual curiosity, compassion, communication, work ethic, teamwork, leadership, self-awareness, DEI (diversity, equity, and inclusion), professionalism, and adaptability. The rank list from 2021 was found to correlate with a score based on values, but not Step 2 score, as it did in 2017. CONCLUSIONS: We applied a modified Delphi method to generate eleven observable values present in the ideal Internal Medicine-Pediatric resident at one academic health center in the Intermountain West. Higher Step 2 scores no longer correlated with higher ranking when we used these values to inform our rank list.

Association of Mobile Workstations and Rounding-in-Flow with Resident Efficiency: A Controlled Study at an Academic Internal Medicine Department.

BACKGROUND: Residents commonly use a batched workflow to round on hospitalized patients, creating redundancy and decreasing efficiency. OBJECTIVE: To improve resident efficiency through a novel workflow using mobile laptops and modified rounding-in-flow. DESIGN, SETTING, PARTICIPANTS: A controlled experimental study conducted at an academic medical center for 3 months. One internal medicine team served as the intervention group, and two other teams served as a control group; 34 interns and 20 senior residents participated. INTERVENTION: Residents in the intervention group were provided a novel workflow and a mobile laptop to allow them to round "in-flow." Control group residents rounded as usual (batched workflow without laptops). MAIN MEASURES: Fourteen interns were monitored for a time-motion study. Time-stamped electronic medical record (EMR) data were used to assess percentage of progress notes and orders placed during rounds (9 a.m.-12:30 p.m.) and percentage of discharge summaries signed within 24 h of discharge. A post-intervention survey measured perceived efficiency. RESULTS: A time-motion study showed non-significant differences between time in the intervention group and that in the control group: communication time with patients (128 min vs 105 min, p = 0.37) and computer time (289 min vs 306 min, p = 0.71). EMR data for 664 visits in the control group and 374 in the intervention group showed that rounding-in-flow was associated with an odds ratio (OR) of 1.5 for placing progress notes during rounds (95% CI: 1.2-1.7, p < 0.001), an OR of 1.1 for placing non-discharge orders during rounds (95% CI: 1.0-1.2, p = 0.01), and an OR of 3.9 for signing discharge summaries within 24 h of discharge (95% CI: 2.3-7.2, p < 0.001). Post-intervention survey, completed by 23 of 34 interns, showed that interns in the intervention group perceived that orders were completed during rounds more often than the control group (OR 7.8; 95% CI: 1.3-60.1, p = 0.03). CONCLUSIONS: Using mobile laptops with modified rounding-in-flow was associated with earlier completion of residents' work, suggesting improved efficiency.

Using Natural Language Processing and Machine Learning to Identify Internal Medicine-Pediatrics Residency Values in Applications.

PROBLEM: Although holistic review has been used successfully in some residency programs to decrease bias, such review is time-consuming and unsustainable for many programs without initial prescreening. The unstructured qualitative data in residency applications, including notable experiences, letters of recommendation, personal statement, and medical student performance evaluations, require extensive time, resources, and metrics to evaluate; therefore, previous applicant screening relied heavily on quantitative metrics, which can be socioeconomically and racially biased. APPROACH: Using residency applications to the University of Utah internal medicine-pediatrics program from 2015 to 2019, the authors extracted relevant snippets of text from the narrative sections of applications. Expert reviewers annotated these snippets into specific values (academic strength; intellectual curiosity; compassion; communication; work ethic; teamwork; leadership; self-awareness; diversity, equity, and inclusion; professionalism; and adaptability) previously identified as associated with resident success. The authors prospectively applied a machine learning model (MLM) to snippets from applications from 2023, and output was compared with a manual holistic review performed without knowledge of MLM results. OUTCOMES: Overall, the MLM had a sensitivity of 0.64, specificity of 0.97, positive predictive value of 0.62, negative predictive value of 0.97, and F1 score of 0.63. The mean (SD) total number of annotations per application was significantly correlated with invited for interview status (invited: 208.6 [59.1]; not invited: 145.2 [57.2]; P < .001). In addition, 8 of the 10 individual values were significantly predictive of an applicant's invited for interview status. NEXT STEPS: The authors created an MLM that can identify several values important for resident success in internal medicine-pediatrics programs with moderate sensitivity and high specificity. The authors will continue to refine the MLM by increasing the number of annotations, exploring parameter tuning and feature engineering options, and identifying which application sections have the highest correlation with invited for interview status.

Formation and Evaluation of an Academic Elective for Residents in a Combined Internal Medicine-Pediatrics Residency Program.

Background Recently, there has been increasing focus on skills that are crucial for success in residency that is not explicitly taught. Specifically, the four domains of teaching skills, evidence appraisal, wellness, and education on structural racism have been identified as topics that are important and underrepresented in current resident education curriculums, largely due to time constraints. Methods A task force consisting of one post-graduate year 2 (PGY-2) resident, one PGY-4 resident, the Associate Program Director, and the Program Director of the Internal Medicine-Pediatrics residency program was formed to explore current deficiencies in resident curriculum and to research possible solutions. As an intervention, we created and executed a four-week academic elective with dedicated time for upper-level residents to learn and explore the four domains of resident teaching, evidence-based clinical practice, wellness, and anti-racism work. The elective included several clinical sessions dedicated to implementing the skills taught in the elective. The month-long elective completed in January 2021. All residents evaluated each lecture or experience based on how valuable it was to their education on a Likert scale from 1 to 7, with 1 defined as "not valuable at all" and 7 defined as "extremely valuable." Results Residents rated the overall value of teaching in each domain highly. Education and activities in wellness lectures were found to have the highest value-added material (6.20 ± 0.41, n = 18), followed by residents-as-teachers lectures (5.93 ± 0.25, n = 48), anti-racism (5.57 ± 1.11, n = 9), and evidence-based clinical practice (5.18 ± 0.50, n = 43). In addition, each domain was found to have at least one high-yield topic. Conclusions We were able to create and execute an academic elective with dedicated time for upper-level residents to develop and utilize valuable skills in teaching, evidence appraisal, wellness, and anti-racism. Future work will focus on refining the curriculum based on resident evaluations and expanding this elective to the Internal Medicine and Pediatrics categorical programs at our institution.