Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

PURPOSE: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. METHOD: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. RESULTS: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. CONCLUSIONS: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Implementing and optimizing a communication curriculum in medical teaching: Stakeholders' perspectives.

OBJECTIVE: The relevance of communication in medical education is continuously increasing. At the Medical Faculty of Hamburg, the communication curriculum was further developed and optimized during this project. This article aims to describe the stakeholders' perceived challenges and supporting factors in the implementation and optimization processes. METHODS: The initial communication curriculum and its development after a one-year optimization process were assessed with a curricular mapping. A SWOT analysis and group discussions were carried out to provide information on the need for optimization and on challenges the different stakeholders faced. RESULTS: The curricular mapping showed that the communication curriculum is comprehensive, coherent, integrated and longitudinal. In both the implementation and the project-related optimization processes, support from the dean, cooperation among all stakeholders and structural prerequisites were deemed the most critical factors for successfully integrating communication content into the curriculum. CONCLUSION: The initiative and support of all stakeholders, including the dean, teachers and students, were crucial for the project's success. PRACTICE IMPLICATIONS: Although the implementation of a communication curriculum is recommended for all medical faculties, their actual implementation processes may differ. In a "top-down" and "bottom-up" approach, all stakeholders should be continuously involved in the process to ensure successful integration.

Outcomes of embryo transfers performed by Reproductive Endocrinology and Infertility fellows vs. faculty: an 11-year retrospective review.

OBJECTIVE: To compare the clinical pregnancy rate (CPR) and live birth rate (LBR) of embryo transfer episodes (ETEs) performed by Reproductive Endocrinology and Infertility fellows vs. those of ETEs performed by faculty physicians. DESIGN: Retrospective cohort analysis. SETTING: Academic reproductive endocrinology and infertility practice. PATIENT(S): In total, 3,073 ETEs for 1,488 unique patients were performed by fellows or faculty physicians between January 2009 and January 2020. INTERVENTION(S): None. MAIN OUTCOME MEASURE(S): Clinical pregnancy rate and LBR. RESULT(S): Fifteen fellows performed 1,225 (39.9%) of 3,073 ETEs after completing 30 mock transfers. On comparing outcomes among fellowship years (FY1, FY2, and FY3), CPR (44.1% vs. 43.2% vs. 45.7%, respectively, P = .83) and LBR (39.1% vs. 38.1% vs. 38.4%, respectively, P = .97) were not significantly different. Fellowship year 1 fellows' initial 30 ETEs vs. all the remaining FY1 ETEs had a significantly higher CPR (48.1% vs. 40.5%, respectively, P = .030) and LBR (45.4% vs. 34.3%, respectively, P = .001). There were no significant differences between faculty versus fellow ETEs in terms of CPR (43.0% vs. 45.0%, respectively, P = .30) or LBR (37.3% vs. 39.8%, respectively, P = .16), even after adjusting for patient age, body mass index, primary infertility diagnosis, autologous vs. donor oocyte, fresh vs. frozen embryo, number of embryos transferred, type of transfer catheter, and year of transfer (P = .32 for CPR, P = .22 for LBR). CONCLUSION(S): Appropriately trained FY1 fellows had success rates maintained throughout all FYs. There were no significant differences in clinical outcomes between fellow- and faculty-performed transfers. These data demonstrated that allowing fellows to perform live embryo transfers is not detrimental to clinical outcomes.

Downstream funding success of early career researchers for resubmitted versus new applications: A matched cohort.

BACKGROUND: Early career researchers face a hypercompetitive funding environment. To help identify effective intervention strategies for early career researchers, we examined whether first-time NIH R01 applicants who resubmitted their original, unfunded R01 application were more successful at obtaining any R01 funding within 3 and 5 years than original, unfunded applicants who submitted new NIH applications, and we examined whether underrepresented minority (URM) applicants differentially benefited from resubmission. Our observational study is consistent with an NIH working group's recommendations to develop interventions to encourage resubmission. METHODS AND FINDINGS: First-time applicants with US medical school academic faculty appointments who submitted an unfunded R01 application between 2000-2014 yielded 4,789 discussed and 7,019 not discussed applications. We then created comparable groups of first-time R01 applicants (resubmitted original R01 application or submitted new NIH applications) using optimal full matching that included applicant and application characteristics. Primary and subgroup analyses used generalized mixed models with obtaining any NIH R01 funding within 3 and 5 years as the two outcomes. A gamma sensitivity analysis was performed. URM applicants represented 11% and 12% of discussed and not discussed applications, respectively. First-time R01 applicants resubmitting their original, unfunded R01 application were more successful obtaining R01 funding within 3 and 5 years than applicants submitting new applications-for both discussed and not discussed applications: discussed within 3 years (OR 4.17 [95 CI 3.53, 4.93]) and 5 years (3.33 [2.82-3.92]); and not discussed within 3 years (2.81 [2.52, 3.13]) and 5 years (2.47 [2.22-2.74]). URM applicants additionally benefited within 5 years for not discussed applications. CONCLUSIONS: Encouraging early career researchers applying as faculty at a school of medicine to resubmit R01 applications is a promising potential modifiable factor and intervention strategy. First-time R01 applicants who resubmitted their original, unfunded R01 application had log-odds of obtaining downstream R01 funding within 3 and 5 years 2-4 times higher than applicants who did not resubmit their original application and submitted new NIH applications instead. Findings held for both discussed and not discussed applications.

NASA-TLX assessment of workload in resident physicians and faculty surgeons covering trauma, surgical intensive care unit, and emergency general surgery services.

BACKGROUND: Higher workload is associated with burnout and lower performance. Therefore, we aim to assess shift-related factors associated with higher workload on EGS, ICU, and trauma surgery services. METHODS: In this prospective cohort study, faculty surgeons and surgery residents completed a survey after each EGS, ICU, or trauma shift, including shift details and a modified NASA-TLX. RESULTS: Seventeen faculty and 12 residents completed 174 and 48 surveys after working scheduled 12-h and 24-h shifts, respectively (response rates: faculty - 62%, residents - 42%). NASA-TLX was significantly increased with a higher physician subjective fatigue level. Further, seeing more consults or performing more operations than average significantly increased workload. Finally, NASA-TLX was significantly higher for faculty when they felt their shift was more difficult than expected. CONCLUSIONS: Higher volume clinical responsibilities and higher subjective fatigue levels are independently associated with higher workload. Designing shift coverage to expand on busier days may decrease workload, impacting burnout and shift performance.

Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies.

PURPOSE: Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback. This study evaluated which NLP techniques best classify the quality of surgical trainee formative feedback recorded as part of a workplace assessment. METHOD: During the 2016-2017 academic year, the SIMPL (Society for Improving Medical Professional Learning) app was used to record operative performance narrative feedback for residents at 3 university-based general surgery residency training programs. Feedback comments were collected for a sample of residents representing all 5 postgraduate year levels and coded for quality. In May 2019, the coded comments were then used to train NLP models to automatically classify the quality of feedback across 4 categories (effective, mediocre, ineffective, or other). Models included support vector machines (SVM), logistic regression, gradient boosted trees, naive Bayes, and random forests. The primary outcome was mean classification accuracy. RESULTS: The authors manually coded the quality of 600 recorded feedback comments. Those data were used to train NLP models to automatically classify the quality of feedback across 4 categories. The NLP model using an SVM algorithm yielded a maximum mean accuracy of 0.64 (standard deviation, 0.01). When the classification task was modified to distinguish only high-quality vs low-quality feedback, maximum mean accuracy was 0.83, again with SVM. CONCLUSIONS: To the authors' knowledge, this is the first study to examine the use of NLP for classifying feedback quality. SVM NLP models demonstrated the ability to automatically classify the quality of surgical trainee evaluations. Larger training datasets would likely further increase accuracy.

Reassessing career pathways of surgical leaders: An examination of surgical leaders' early accomplishments.

BACKGROUND: The American College of Surgeon (ACS), American Surgical Association (ASA), Association of Women Surgeons (AWS), and Society of Black Academic Surgeons (SBAS) partnered to gain insight into whether inequities found in surgical society presidents may be present earlier. METHODS: ACS, ASA, AWS, and SBAS presidents' CVs were assessed for demographics and scholastic achievements at the time of first faculty appointment. Regression analyses controlling for age were performed to determine relative differences across societies. RESULTS: 66 of the 68 presidents' CVs were received and assessed (97% response rate). 50% of AWS future presidents were hired as Instructors rather than Assistant professors, compared to 29.4% of SBAS, 25% of ASA and 29.4% of ACS. The future ACS, ASA, and SBAS presidents had more total publications than the AWS presidents, but similar numbers of 1st and Sr. author publications. CONCLUSION: Gender inequities in academic surgeon hiring practices and perceived scholastic success may be present at first hire.

A Faculty Development Workshop for Planning and Implementing Interactive Virtual Case-Based Teaching.

Introduction: The virtual learning environment has become increasingly important due to physical distance requirements put in place during the COVID-19 pandemic. The transition to a virtual format has been challenging for case-based teaching sessions, which involve substantial audience participation. We developed a faculty development workshop aimed at teaching health professions educators how to use various interactive virtual tools within videoconferencing platforms to facilitate virtual case-based sessions. Methods: Two 90-minute workshops were piloted as a faculty development initiative. The facilitators demonstrated interactive teaching tools that could be used within virtual case-based sessions. Then, participants discussed how to incorporate these tools into case-based teaching sessions of different class sizes in small-group breakout sessions. Participants completed an online survey following each workshop to evaluate the sessions. Results: A total of 18 and 26 subjects participated in the first and second workshops, respectively. Survey response rates were 100% (n = 18) and 65% (n = 17) for the first and second workshops, respectively. Both groups provided overall high ratings and reported that the workshop was clear, organized, and relevant. Participants were more familiar and comfortable with the use of various interactive tools for online teaching. Discussion: Distance online teaching will be increasingly required for an undetermined time. Faculty development efforts are crucial to facilitate effective interactive teaching sessions that engage learners and maximize learning. This virtual teaching workshop is a simple and straightforward way to introduce a more interactive format to virtual case-based teaching in the health professions.

Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory.

BACKGROUND: The American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills. METHODS: Surgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis. RESULTS: Three-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks. CONCLUSIONS: Variability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.

Augmenting physician examiner scoring in objective structured clinical examinations: including the standardized patient perspective.

In Canada, high stakes objective structured clinical examinations (OSCEs) administered by the Medical Council of Canada have relied exclusively on physician examiners (PEs) for scoring. Prior research has looked at using SPs to replace PEs. This paper reports on two studies that implement and evaluate a standardized patient (SP) scoring tool to augment PE scoring. The unique aspect of this study is that it explores the benefits of combining SP and PE scores. SP focus groups developed rating scales for four dimensions they labelled: Listening, Communication, Empathy/Rapport, and Global Impression. In Study I, 43 SPs from one site of a national PE-scored OSCE rated 60 examinees with the initial SP rating scales. In Study II, 137 SPs used slightly revised rating scales with optional narrative comments to score 275 examinees at two sites. Examinees were blinded to SP scoring and SP ratings did not count. Separate PE and SP scoring was examined using descriptive statistics and correlations. Combinations of SP and PE scoring were assessed using pass-rates, reliability, and decision consistency and accuracy indices. In Study II, SP and PE comments were examined. SPs showed greater variability in their scoring, and rated examinees lower than PEs on common elements, resulting in slightly lower pass rates when combined. There was a moderate tendency for both SPs and PEs to make negative comments for the same examinee but for different reasons. We argue that SPs and PE assess performance from different perspectives, and that combining scores from both augments overall reliability of scores and pass/fail decisions. There is potential to provide examinees with feedback comments from each group.

Driving lesson or driving test? : A metaphor to help faculty separate feedback from assessment.

Although there is consensus in the medical education world that feedback is an important and effective tool to support experiential workplace-based learning, learners tend to avoid the feedback associated with direct observation because they perceive it as a high-stakes evaluation with significant consequences for their future. The perceived dominance of the summative assessment paradigm throughout medical education reduces learners' willingness to seek feedback, and encourages supervisors to mix up feedback with provision of 'objective' grades or pass/fail marks. This eye-opener article argues that the provision and reception of effective feedback by clinical supervisors and their learners is dependent on both parties' awareness of the important distinction between feedback used in coaching towards growth and development (assessment for learning) and reaching a high-stakes judgement on the learner's competence and fitness for practice (assessment of learning). Using driving lessons and the driving test as a metaphor for feedback and assessment helps supervisors and learners to understand this crucial difference and to act upon it. It is the supervisor's responsibility to ensure that supervisor and learner achieve a clear mutual understanding of the purpose of each interaction (i.e. feedback or assessment). To allow supervisors to use the driving lesson-driving test metaphor for this purpose in their interactions with learners, it should be included in faculty development initiatives, along with a discussion of the key importance of separating feedback from assessment, to promote a feedback culture of growth and support programmatic assessment of competence.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Impact of Trainee Involvement on Video-Assisted Thoracoscopic Lobectomy for Cancer.

BACKGROUND: Previous literature in other surgical disciplines regarding the impact of resident and fellow involvement on operative time and outcomes has yielded mixed results. The impact of trainee involvement on minimally invasive thoracic surgery is unknown. This study compared risk-adjusted differences in operative time and outcomes of video-assisted thoracoscopic lobectomy for cancer between cases performed with and without residents and fellows involved. METHODS: All patients undergoing elective video-assisted thoracoscopic lobectomy for cancer between 2008 and 2018 were identified in the Veterans Affairs Surgical Quality Improvement Program database. Patients were stratified into 2 cohorts: cases with residents and fellows involved, and cases performed only by attending surgeons. Primary outcomes included operative time, postoperative hospital length of stay, and composite 30-day morbidity and mortality. Secondary outcomes included factors associated with high and low trainee operative autonomy. RESULTS: A total of 3678 patients met study inclusion criteria. In all, 1780 cases were performed with residents and fellows involved (median postgraduate year, 5; interquartile range, 4-7). Multivariate analysis showed that operative time was significantly shorter in resident- and fellow-involved cases compared with attending-only cases (mean [SD], 3.6 [1.4] versus 3.8 [1.6] hours; P < .001). There were no significant differences in composite 30-day morbidity and mortality (16.0% versus 17.1%; adjusted odds ratio = 0.93; 95% confidence interval, 0.77-1.11; P = .40) or length of stay. Substratification of trainees by postgraduate year resulted in similar findings. Cases performed in July through October and those in the Northeastern United States were associated with low autonomy. CONCLUSIONS: Current training paradigms in thoracic surgery are safe, and the involvement of motivated and skilled trainees with appropriate supervision may benefit operative duration.

Energizing the Conversation: How to Identify and Overcome Gender Inequalities in Academic Medicine.

Gender inequality exists in advanced faculty and leadership positions at academic medical centers; however, despite growing awareness, how to best approach and rectify the issue is unknown. To energize the conversation on gender inequality at one academic medical center, chairs and women faculty were surveyed to identify barriers faced by women navigating their careers. A symposium with short talks to increase awareness, a panel with University leaders to discuss issues and successful strategies to overcome gaps, and focus groups to delve further into key areas that underlie inequity through an active café style format were planned and implemented. This multifaceted approach resulted in a wealth of knowledge. The symposium and panel highlighted important relevant issues and offered personal strategies for successful career advancement, while the focus group discussions further identified barriers and inspired ongoing efforts across departments and novel approaches to overcome three key issues (work-life integration, deliberate promotion of mentor/sponsor relationships, and overcoming unconscious bias) identified through the initial surveys. Compiled data were then disseminated to participants and University leaders to enhance awareness of available programs and prompt action in critical areas lacking support. Overall, the approach indicated that securing support from leaders and the academic community alike are pertinent to emphasize actions needed to overcome issues affecting women in academic medicine. Moreover, bringing leaders and faculty together for an informational session and brainstorming appears to energize the conversation. Such efforts can ultimately instill change and establish an inclusive environment where all members of the academic medicine community can thrive.

[Status Quo - The requirements for medical habilitation in Germany]. / Status Quo ­ Die Anforderungen an die medizinische Habilitation in Deutschland.

BACKGROUND: In Germany, the habilitation proves one's qualification for independent research and teaching. It is a prerequisite for obtaining the teaching qualification in the respective specialist area. The prerequisites are laid down in the habilitation regulations of universities and equivalent institutions. This review article aims to show the requirements for the habilitation of medical faculties. METHODS: The current regulations regarding habilitation and implementation of all 39 German medical faculties were analyzed according to the following criteria: total publications (n), first and last authorships (n), teaching achievements, considerations of third-party funding, patent rights, abstracts at congresses, participation in further didactic training, cumulative habilitation, prerequisite of doctorate or equivalent achievements, reviewer (n) and their affiliation, giving a university-wide and scientific lecture, as well as a teaching sample. RESULTS: A total of 37 habilitation requirements were included in the analysis. The requirements differ in several central points: above all in numbers of required first and last authorships, total numbers of publications and evaluation of the publication performance. A cumulative habilitation is possible at 97 % (n = 36) of the universities. There are also distinct differences regarding the requirements of the reviewers (only internal, internal and external, only external). CONCLUSIONS: Each requirement and their weighting are often inconsistent between orders. In order to increase the transparency and appreciation of the scientific achievement, a standardization of the requirements seems expedient.

Didactic qualification of teaching staff in primary care medicine - a position paper of the Primary Care Committee of the Society for Medical Education.

Having teaching staff with didactic qualifications in university teaching leads to a measurable improvement in academic skills among students. Previous recommendations on the type and scope of medical didactic qualification measures primarily apply to teaching staff at university and in-patient settings. The situation of primary care medicine, which often employs external lecturers and whose teaching takes place to a considerable extent in decentralized training facilities (teaching practices) is not adequately addressed. Taking into account a survey on the status quo at higher education institutions for General Practice in Germany, recommendations for minimum standards are made, based on national and international recommendations on the content and scope of medical didactic qualification measures. These recommendations include preliminary work by the Personnel and Organizational Development in Teaching (POiL) Committee of the Society for Medical Education (GMA), the MedicalTeachingNetwork (MDN), the Society of University Teaching Staff in General Medicine (GHA) as well as the experiences of the committee members, who hail from the field of general medicine, internal medicine and pediatrics amongst others.