Effect of station format on the psychometric properties of Multiple Mini Interviews.

BACKGROUND: Given the widespread use of Multiple Mini Interviews (MMIs), their impact on the selection of candidates and the considerable resources invested in preparing and administering them, it is essential to ensure their quality. Given the variety of station formats used and the degree to which that factor resides in the control of training programmes that we know so little about, format's effect on MMI quality is a considerable oversight. This study assessed the effect of two popular station formats (interview vs. role-play) on the psychometric properties of MMIs. METHODS: We analysed candidate data from the first 8 years of the Integrated French MMIs (IF-MMI) (2010-2017, n = 11 761 applicants), an MMI organised yearly by three francophone universities and administered at four testing sites located in two Canadian provinces. There were 84 role-play and 96 interview stations administered, totalling 180 stations. Mixed design analyses of variance (ANOVAs) were used to test the effect of station format on candidates' scores and stations' discrimination. Cronbach's alpha coefficients for interview and role-play stations were also compared. Predictive validity of both station formats was estimated with a mixed multiple linear regression model testing the relation between interview and role-play scores with average clerkship performance for those who gained entry to medical school (n = 462). RESULTS: Role-play stations (M = 20.67, standard deviation [SD] = 3.38) had a slightly lower mean score than interview stations (M = 21.36, SD = 3.08), p < 0.01, Cohen's d = 0.2. The correlation between role-play and interview stations scores was r = 0.5 (p < 0.01). Discrimination coefficients, Cronbach's alpha and predictive validity statistics did not vary by station format. CONCLUSION: Interview and role-play stations have comparable psychometric properties, suggesting format to be interchangeable. Programmes should select station format based on match to the personal qualities for which they are trying to select.

Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study.

When determining the score given to candidates in multiple mini-interview (MMI) stations, raters have to translate a narrative judgment to an ordinal rating scale. When adding individual scores to calculate final ranking, it is generally presumed that the values of possible scores on the evaluation grid are separated by constant intervals, following a linear function, although this assumption is seldom validated with raters themselves. Inaccurate interval values could lead to systemic bias that could potentially distort candidates' final cumulative scores. The aim of this study was to establish rating scale values based on rater's intent, to validate these with an independent quantitative method, to explore their impact on final score, and to appraise their meaning according to experienced MMI interviewers. A 4-round consensus-group exercise was independently conducted with 42 MMI interviewers who were asked to determine relative values for the 6-point rating scale (from A to F) used in the Canadian integrated French MMI (IFMMI). In parallel, relative values were also calculated for each option of the scale by comparing the average scores concurrently given to the same individual in other stations every time that option was selected during three consecutive IFMMI years. Data from the same three cohorts was used to simulate the impact of using new score values on final rankings. Comments from the consensus group exercise were reviewed independently by two authors to explore raters' rationale for choosing specific values. Relative to the maximum (A = 100%) and minimum (F = 0%), experienced raters concluded to values of 86.7% (95% CI 86.3-87.1), 69.5% (68.9-70.1), 51.2% (50.6-51.8), and 29.3% (28.1-30.5), for scores of B, C, D and E respectively. The concurrent score approach was based on 43,412 IFMMI stations performed by 4345 medical school applicants. It provided quasi-identical values of 87.1% (82.4-91.5), 70.4% (66.1-74.7), 51.2% (47.1-55.3) and 31.8% (27.9-35.7), respectively. Qualitative analysis explained that while high scores are usually based on minor details of relatively low importance, low scores are usually attributed for more serious offenses and were assumed by the raters to carry more weight in the final score. Individual drop or increase in final MMI ranking with the use of new scale values ranged from - 21 to + 5 percentiles, with the average candidate changing by ± 1.4 percentiles. Consulting with experienced interviewers is a simple and effective approach to establish rating scale values that truly reflects raters' intent in MMI, thus improving the accuracy of the instrument and contributing to the general fairness of the process.

Black students applying and admitted to medicine in the province of Quebec, Canada: what do we know so far?

To address the underrepresentation of Black students in medical schools in Canada and identify barriers in selection processes, we compare data from the latest Canadian census to that of an exit-survey conducted after a situational judgment test (Casper) among medical school applicants and from questionnaires done after selection interviews in Quebec, Canada. The proportion of Black people aged 15-34 years old in Quebec in 2016 was 5.3% province-wide and 8.2% in the Montreal metropolitan area. The proportion in the applicant pool for 2020 in Quebec was estimated to be 4.5% based on Casper exit-survey data. Comparatively, it is estimated that Black people represented 1.8% of applicants invited to admission interviews and 1.2% of admitted students in Quebec in 2019. Although data from different cohorts and data sources do not allow for direct comparisons, these numbers suggest that Black students applying to medical school are disproportionately rejected at the first step compared to non-Black students. Longitudinal data collection among medical school applicants will be necessary to monitor the situation. Further studies are required to pinpoint the factors contributing to this underrepresentation, to keep improving the equity of our selection processes.

Afin de remédier à la sous-représentation des étudiants noirs dans les facultés de médecine au Canada et de cibler les obstacles qu'ils rencontrent dans le processus de sélection, nous comparons les données du dernier recensement canadien avec celles d'un sondage réalisé à la suite d'un test de jugement situationnel (Casper) auprès de candidats ayant fait une demande d'admission dans un programme de doctorat en médecine et celles d'un sondage réalisé à la suite d'entretiens de sélection au Québec (Canada). La proportion de personnes noires âgées de 15 à 34 ans au Québec en 2016 était de 5,3 % à l'échelle de la province et de 8,2 % dans la région métropolitaine de Montréal. La proportion de cette population dans le bassin de candidats pour 2020 au Québec a été estimée à 4,5 % sur la base des données du sondage Casper. À titre de comparaison, on estime que les Noirs représentaient 1,8 % des candidats invités aux entrevues d'admission et 1,2 % des étudiants admis au Québec en 2019. Bien que les données pour les différentes cohortes, provenant de surcroît de sources différentes, ne permettent pas d'établir des comparaisons directes, ces chiffres suggèrent que les étudiants noirs qui demandent à être admis en médecine sont rejetés de manière disproportionnée à la première étape par rapport aux étudiants non noirs. Une collecte de données longitudinales parmi les candidats sera nécessaire pour suivre l'évolution de la situation, ainsi que d'autres études pour découvrir les facteurs qui contribuent à cette sous-représentation, notamment dans une visée d'amélioration de l'équité dans les processus de sélection.