Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education.

BACKGROUND: Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. OBJECTIVE: This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. METHODS: NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. RESULTS: The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). CONCLUSIONS: The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Equity and Game-Theory Strategies to Promote Gender Diversity and Inclusion in an Academic Health Science Centre.

BACKGROUND: Achieving diversity, inclusion, and gender equity remains an elusive challenge for many institutions worldwide and is understudied in Canadian academic health science centres. METHODS: McMaster University's Department of Medicine undertook surveys and analyses to determine whether there was inequity in leadership positions and salaries, or unprofessional behaviour within the department. Measures of academic productivity in relation to gender for both educators and researchers were analyzed. The department began shifting policies to foster greater gender diversity and inclusion. A revision of the leadership selection process, incorporating tenets of equity and a new game theory-based strategy called Diversitive Agreement Versus Nash Equilibrium (DAvNE) was evaluated. RESULTS: The department's survey revealed underrepresentation of women and people of colour in leadership positions, with perceived barriers to their promotion. Both women and people of colour reported experiencing unprofessional behaviour directed toward them. A gender gap in base salary was observed, with female full professors being paid less. No difference in academic productivity was seen between male and female educators or researchers. The leadership competitions conducted under new selection processes emphasizing diversity resulted in 66% of participating women securing a leadership position, in comparison to 25% of participating men. People of colour made up 27% of members participating in these leadership competitions, but none was successful in obtaining a position. CONCLUSIONS: Diversity and inclusion disparities in the Department of Medicine at McMaster University indicate a need for further efforts and innovation to bring about greater gender and racial equity.

CONTEXTE: La mise en place des conditions nécessaires à la diversité, à l'inclusion et à l'équité des genres demeure un défi difficile à relever pour de nombreux établissements dans le monde entier. C'est une démarche peu étudiée dans les centres universitaires canadiens des sciences de la santé. MÉTHODOLOGIE: Le département de médecine de l'Université McMaster a mené des sondages et des analyses en vue de déterminer s'il existait en son sein même des iniquités en matière d'attribution des postes de direction et de fixation des salaires, ou des comportements non professionnels. Des mesures de la productivité universitaire des éducateurs et des chercheurs ont été analysées en fonction des genres. Le département a entrepris une réforme de ses politiques afin de favoriser une plus grande diversité et inclusion des genres. Une révision du processus de sélection des candidats aux postes de direction, intégrant les principes d'équité et une nouvelle stratégie fondée sur la théorie des jeux appelée Diversitive Agreement Versus Nash Equilibrium (DAvNE [accord de diversité vs équilibre de Nash]), a été évaluée. RÉSULTATS: Le sondage du département a révélé qu'il existait une sous-représentation des femmes et des personnes de couleur parmi les titulaires des postes de direction, ainsi que des obstacles perçus à leur promotion. Tant les femmes que les personnes de couleur ont fait état de comportements non professionnels à leur endroit. Un écart entre les genres a été observé au chapitre de la rémunération de base, les professeurs titulaires de sexe féminin étant moins rémunérées. Aucune différence n'a été observée sur le plan de la productivité universitaire entre les femmes et les hommes au sein de l'effectif des éducateurs et des chercheurs. Les concours pour les postes de direction ont été organisés en fonction de nouveaux processus de sélection mettant l'accent sur la diversité. Au final, 66 % des candidats qui ont vu leur candidature retenue étaient des femmes et 25 %, des hommes. Dans le cadre de ces concours, 27 % des candidats étaient des personnes de couleur, mais aucun n'a réussi à obtenir un poste. CONCLUSIONS: Les disparités notées en matière de diversité et d'inclusion au sein du département de médecine de l'Université McMaster montrent que l'atteinte d'une plus grande équité des genres et des races passe par des efforts et une innovation soutenus.