Engaging the Entire Learner: Pathway Program Administrators' Experiences of Providing Students with Research Experiences in Academic Medicine.

Objective: Pathway programs designed to recruit and retain students from groups historically excluded from science and medicine have focused on providing academic and social support through programs that provide mentored experiences. However, for students in science, technology, engineering, math, and medicine (STEMM) majors, students from underrepresented groups tend to leave science-oriented programs at higher rates than students who are not underrepresented. As such, they are also underrepresented in medical fields, including academic medicine. Insight into how pathway programs contribute to addressing this issue is critical. Methods: This study took a qualitative approach to investigating the experiences of pathway program administrators in academic medicine. Interviews were conducted with 12 program administrators working on 8 different programs throughout the country. Interviews were analyzed using directed content analysis while also allowing for the development of new themes based on the data. Results: The codes were organized into 6 overarching themes: mentorship, student engagement, determining program success, administrative time and program logistics, diversity and inclusion, and transition to virtual learning (due to COVID-19). Within each of these themes, program administrators described challenges along with some strategies programs employed to overcome these challenges. Conclusions: The greatest overall challenge described was finding and sustaining relationships with faculty and nonfaculty mentors. To address this issue, many programs have worked within their institutions to incentivize this work. For student engagement, program administrators reported issues with tailoring to skill sets and interests of multiple students while still fostering community. Program administrators have also expanded definitions for determining program success. Program administration is a challenge, and more support staff or time to devote to these programs is often needed. Diversity challenges encompass recruiting faculty and students from groups underrepresented in STEMM and the logistics of getting all necessary accommodations for students. Finally, transition to virtual learning, due to the COVID-19 pandemic, brought about challenges and opportunities.

Enhancing the Value of Surgical Entrustable Professional Activities through Integrative Learning Analytics.

OBJECTIVE: To demonstrate the value of integrating surgical resident Entrustable Professional Activity (EPA) data into a learning analytics platform that provides meaningful feedback for formative and summative decision-making. DESIGN: Description of the Surgical Entrustable Professional Activities (SEPA) analytics dashboard, and examples of summary analytics and intuitive display features. SETTING: Department of Surgery, University of Wisconsin Hospital and Clinics. PARTICIPANTS: Surgery residents, faculty, and residency program administrators. RESULTS: We outline the major functionalities of the SEPA dashboard and offer concrete examples of how these features are utilized by various stakeholders to support progressive entrustment decisions for surgical residents. CONCLUSIONS: Our intuitive analytics platform allows for seamless integration of SEPA microassessment data to support Clinical Competency Committee (CCC) decisions for resident evaluation and provides point of training feedback to faculty and trainees in support of progressive autonomy.

Measuring the impact of simulation debriefing on the practices of interprofessional trauma teams using natural language processing.

BACKGROUND: Natural language processing (NLP) may be a tool for automating trauma teamwork assessment in simulated scenarios. METHODS: Using the Trauma Nontechnical Skills Assessment (T-NOTECHS), raters assessed video recordings of trauma teams in simulated pre-debrief (Sim1) and post-debrief (Sim2) trauma resuscitations. We developed codes through directed content analysis and created algorithms capturing teamwork-related discourse through NLP. Using a within subjects pre-post design (n = 150), we compared changes in teams' Sim1 versus Sim2 T-NOTECHS scores and automatically coded discourse to identify which NLP algorithms could identify skills assessed by the T-NOTECHS. RESULTS: Automatically coded behaviors revealed significant post-debrief increases in teams' simulation discourse: Verbalizing Findings, Acknowledging Communication, Directed Communication, Directing Assessment and Role Assignment, and Leader as Hub for Information. CONCLUSIONS: Our results suggest NLP can capture changes in trauma team discourse. These findings have implications for the expedition of team assessment and innovations in real-time feedback when paired with speech-to-text technology.

Gender Differences in Entrustable Professional Activity Evaluations of General Surgery Residents.

OBJECTIVE: To determine differences in entrustable professional activity (EPA) assessments between male and female general surgery residents. SUMMARY BACKGROUND DATA: Evaluations play a critical role in career advancement for physicians. However, female physicians in training receive lower evaluations and underrate their own performance. Competency-based assessment frameworks, such as EPAs, may help address gender bias in surgery by linking evaluations to specific, observable behaviors. METHODS: In this cohort study, EPA assessments were collected from July 2018 to May 2020. The effect of resident sex on EPA entrustment levels was analyzed using multiple linear and ordered logistic regressions. Narrative comments were analyzed using latent dirichlet allocation to identify topics correlated with resident sex. RESULTS: Of the 2480 EPAs, 1230 EPAs were submitted by faculty and 1250 were submitted by residents. After controlling for confounding factors, faculty evaluations of residents were not impacted by resident sex (estimate = 0.09, P = 0.08). However, female residents rated themselves lower by 0.29 (on a 0-4 scale) compared to their male counterparts (P < 0.001). Within narrative assessments, topics associated with resident sex demonstrated that female residents focus on the "guidance" and "supervision" they received while performing an EPA, while male residents were more likely to report "independent" action. CONCLUSIONS: Faculty assessments showed no difference in EPA levels between male and female residents. Female residents rate themselves lower by nearly an entire post graduate year (PGY) level compared to male residents. Latent dirichlet allocation -identified topics suggest this difference in self-assessment is related to differences in perception of autonomy.

Simulation-Based Medical Education: Development of an Assessment Tool for Novice Use.

BACKGROUND: Simulation-based medical education, an educational model in which students engage in simulated patient scenarios, improves performance. However, assessment tools including the Oxford Non-Technical Skills (NOTECHS) scale require expert assessors. We modified this tool for novice use. METHODS: Medical students participated in 5 nontechnical simulations. The NOTECHS scale was modified to allow for novice evaluation. Three novices and 2 experts assessed performance, with intraclass correlation used to assess validity. RESULTS: Twenty-two learners participated in the simulations. Novice reviewers had moderate to excellent correlation among evaluations (0.66 < intraclass correlation coefficients [ICC] < 0.95). Novice and expert reviewers had moderate to good correlation among evaluations (0.51 < ICC < 0.88). DISCUSSION: The modified NOTECHS scales can be utilized by novices to evaluate simulation performance. Novice assessment correlates with expert review. These tools may encourage the use of simulation-based medical education.

Instructional methods of attendings as exhibited during intraoperative takeovers: A pilot study.

BACKGROUND: This pilot study examined intraoperative instructional techniques during "takeovers," defined as the act of an attending taking control of a case from a resident. This work describes what happens during takeovers and identifies possible reasons for takeovers. METHODS: Intraoperative audio-video recordings during 25 laparoscopic inguinal hernia repair procedures were collected. Participants included 2 postgraduate year-5 residents and 5 attendings. Postoperative evaluation forms were completed by attendings. Coding schemes for takeovers during hernia reduction and mesh placement steps were developed using conventional and directed content analysis in an iterative process by study team members, including individuals with expertise in education, surgery, and surgical education. RESULTS: Takeovers occurred in 72% of cases. Frequency of takeovers was not related to case difficulty or differences in resident technical skill levels, nor did they decrease over the duration of the 2-month rotation. Takeovers most commonly occurred when a resident struggled to progress the case. They also occurred when anatomy was unclear or when the attending wanted to teach a specific skill. Differences were identified among attendings regarding frequency of takeovers. The majority of takeover behaviors were directed at instructing residents; however, attendings' teaching techniques did not vary by resident. CONCLUSION: Attending teaching habits appear to be independent of resident skills and depend on the attending's teaching style rather than residents' learning needs. Findings highlight the need for faculty development to help surgical educators learn how to tailor instruction to individual trainees. Additionally, future research is needed to establish the effectiveness of instruction through takeovers in the operating room.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Entrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment.

OBJECTIVE: Effective self-assessment is a cornerstone of lifelong professional development; however, evidence suggests physicians have a limited ability to self-assess. Novel strategies to improve the accuracy of learner self-assessment are needed. Our institution's surgical entrustable professional activity (EPA) implementation strategy incorporates resident self-assessment to address this issue. This study evaluates the accuracy of resident self-assessment versus faculty assessment across 5 EPAs in general surgery. DESIGN, SETTING, PARTICIPANTS: Within a single academic general surgery residency program, assessment data for 5 surgery EPAs was prospectively collected using a mobile application. Matched assessments (resident and faculty assessments for the same clinical encounter) were identified and the remainder excluded. Assessment scores were compared using Welch's t test. Agreement was analyzed using Cohen's kappa with squared weights. RESULTS: One thousand eight hundred and fifty-seven EPA assessments were collected in 17 months following implementation. One thousand one hundred and fifty-five (62.2%) were matched pairs. Residents under-rated their own performance relative to faculty assessments (2.36 vs 2.65, p < 0.01). This pattern held true for all subsets except for Postgraduate Year (PGY)2 residents and Inguinal Hernia EPAs. There was at least moderate agreement between matched resident and faculty EPA assessment scores (κ = 0.57). This was consistent for all EPAs except Trauma evaluations, which were completed by faculty from 2 different departments. Surgery resident self-assessments more strongly agreed with Surgery faculty assessments than Emergency Medicine faculty assessments (κ = 0.58 vs 0.36). CONCLUSIONS: Resident EPA self-assessments are equivalent or slightly lower than faculty assessments across a wide breadth of clinical scenarios. Resident and faculty matched assessments demonstrate moderate agreement.

Implementation of Entrustable Professional Activities into a General Surgery Residency.

OBJECTIVE: Concerns over resident ability to practice effectively after graduation have led to the competency-based medical education movement. Entrustable professional activities (EPAs) may facilitate competency-based medical education in surgery, but implementation is challenging. This manuscript describes 1 strategy used to implement EPAs into an academic general surgery residency. DESIGN, SETTING, PARTICIPANTS: A mobile application was developed incorporating 5 EPAs developed by the American Board of Surgery; residents and faculty from the Departments of Surgery, Emergency Medicine, and Hospital Medicine at a single tertiary care center were trained in its use. Entrustment levels and free text feedback were collected. Self-assessment was paired with supervisor assessment, and faculty assessments were used to inform clinical competency committee entrustment decisions. Feedback was regularly solicited from app users and results distributed on a monthly basis. RESULTS: One thousand seven hundred and twenty microassessments were collected over the first 16 months of implementation; 898 (47.8%) were performed by faculty with 569 (66.0%) matched pairs. Engagement was skewed with small numbers of high performers in both resident and faculty groups. Continued development of resident and faculty was required to sustain engagement with the program. Nonsurgical specialties contributed significantly to resident assessments (496, 28.8%). CONCLUSIONS: EPAs are being successfully integrated into the assessment framework at our institution. EPA implementation in surgery residency is a long-term process that requires investment, but may address limitations in the current assessment framework.

Teaching practicing surgeons what not to do: An analysis of instruction fluidity during a simulation-based continuing medical education course.

BACKGROUND: Interest is growing in simulation-based continuing medical education courses for practicing surgeons. However, little research has explored the instruction employed during these courses. This study examines instruction practices used during an annual simulation-based continuing medical education course. METHODS: Audio-video data were collected from surgeon instructors (n = 12) who taught a simulated laparoscopic hernia repair continuing medical education course across 2 years. Surgeon learners (n = 58) were grouped by their self-reported laparoscopic and hernia repair experience. Instructors' transcribed dialogue was automatically coded for 5 types of responses to the following questions: anecdotes, confirming, correcting, guidance, and what not to do. Differences in these responses were measured against the progress of the simulations and across learners with different experience levels. Postcourse interviews with instructors were conducted for additional qualitative validation. RESULTS: Performing t tests of instructor responses revealed that they were significantly more likely to answer in forms coded as anecdotes when responding to relative experts and in forms coded as what not to do when responding to novices. Linear regressions of each code against normalized progressions of each simulation revealed a significant relationship between progression through a simulation and frequency of the what not to do code for less-experienced learners. Postcourse interviews revealed that instructors continuously assess participants throughout a session and modify their teaching strategies. CONCLUSION: Instructors significantly modified the focus of their teaching as a function both of their learners' self-reported experience levels, their assessment of learner needs, and learner progression through the training sessions.

What do you want to know? Operative experience predicts the type of questions practicing surgeons ask during a CME laparoscopic hernia repair course.

BACKGROUND: Given their variegated backgrounds, surgeons taking continuing medical education (CME) courses possess different learning needs. This study examines the relationship between surgeons' levels of experience and the questions they asked in a simulation-based CME course. METHODS: We analyzed transcribed audio-video data collected from surgeons participating in a simulated laparoscopic hernia repair CME course and identified four types of questions learners posed to their instructors. Linear regressions compared how often these questions were asked versus self-reported operative experience. RESULTS: Both Requesting Guidance and Requesting Confirmation were inversely proportional to experience, whereas Asking About a Specific Case was directly proportional to experience. Requesting Instructor Preference exhibited no significant correlation with experience. CONCLUSION: Practicing surgeons with relatively less experience tend to ask for confirmation and guidance, whereas those with greater experience tend to focus on specific hypothetical scenarios. This data can be used to tailor instruction based on learners' self-reported experience level.

The hands and head of a surgeon: Modeling operative competency with multimodal epistemic network analysis.

BACKGROUND: This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate psychomotor, procedural, and cognitive skills to manage errors. METHODS: Audio-video data were collected from general surgery residents (N = 45) performing a simulated laparoscopic ventral hernia repair. Errors were identified using a standard checklist, and speech was coded for elements related to error recognition and management. Epistemic network analysis (ENA) was used to model the integration of error management skills. RESULTS: There was no correlation between number or type of errors committed and operative outcome. However, ENA models showed significant differences in the integration of error management skills between high-performing and low-performing residents. CONCLUSION: These results suggest that error checklists and surgeons' speech can be used to model the integration of psychomotor, procedural, and cognitive aspects of intraoperative performance. Moreover, ENA can identify and quantify this integration, providing insight on performance gaps in both individuals and populations.