Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study.

When determining the score given to candidates in multiple mini-interview (MMI) stations, raters have to translate a narrative judgment to an ordinal rating scale. When adding individual scores to calculate final ranking, it is generally presumed that the values of possible scores on the evaluation grid are separated by constant intervals, following a linear function, although this assumption is seldom validated with raters themselves. Inaccurate interval values could lead to systemic bias that could potentially distort candidates' final cumulative scores. The aim of this study was to establish rating scale values based on rater's intent, to validate these with an independent quantitative method, to explore their impact on final score, and to appraise their meaning according to experienced MMI interviewers. A 4-round consensus-group exercise was independently conducted with 42 MMI interviewers who were asked to determine relative values for the 6-point rating scale (from A to F) used in the Canadian integrated French MMI (IFMMI). In parallel, relative values were also calculated for each option of the scale by comparing the average scores concurrently given to the same individual in other stations every time that option was selected during three consecutive IFMMI years. Data from the same three cohorts was used to simulate the impact of using new score values on final rankings. Comments from the consensus group exercise were reviewed independently by two authors to explore raters' rationale for choosing specific values. Relative to the maximum (A = 100%) and minimum (F = 0%), experienced raters concluded to values of 86.7% (95% CI 86.3-87.1), 69.5% (68.9-70.1), 51.2% (50.6-51.8), and 29.3% (28.1-30.5), for scores of B, C, D and E respectively. The concurrent score approach was based on 43,412 IFMMI stations performed by 4345 medical school applicants. It provided quasi-identical values of 87.1% (82.4-91.5), 70.4% (66.1-74.7), 51.2% (47.1-55.3) and 31.8% (27.9-35.7), respectively. Qualitative analysis explained that while high scores are usually based on minor details of relatively low importance, low scores are usually attributed for more serious offenses and were assumed by the raters to carry more weight in the final score. Individual drop or increase in final MMI ranking with the use of new scale values ranged from - 21 to + 5 percentiles, with the average candidate changing by ± 1.4 percentiles. Consulting with experienced interviewers is a simple and effective approach to establish rating scale values that truly reflects raters' intent in MMI, thus improving the accuracy of the instrument and contributing to the general fairness of the process.

Impact of sociodemographic characteristics of applicants in multiple mini-interviews.

BACKGROUND: Multiple mini-interviews (MMI) are commonly used for medical school admission. This study aimed to assess if sociodemographic characteristics are associated with MMI performance, and how they may act as barriers or enablers to communication in MMI. METHODS: This mixed-method study combined data from a sociodemographic questionnaire, MMI scores, semi-structured interviews and focus groups with applicants and assessors. Quantitative and qualitative data were analyzed using multiple linear regression and a thematic framework analysis. RESULTS: 1099 applicants responded to the questionnaire. A regression model (R2 = 0.086) demonstrated that being age 25-29 (ß = 0.11, p = 0.001), female and a French-speaker (ß = 0.22, p = 0.003) were associated with better MMI scores. Having an Asian-born parent was associated with a lower score (ß = -0.12, p < 0.001). Candidates reporting a higher family income had higher MMI scores. In the qualitative data, participants discussed how maturity and financial support improved life experiences, how language could act as a barrier, and how ethnocultural differences could lead to misunderstandings. CONCLUSION: Age, gender, ethnicity, socioeconomic status and language seem to be associated with applicants' MMI scores because of perceived differences in communications skills and life experiences. Monitoring this association may provide guidance to improve fairness of MMI stations.

The Mini-Med School and Its Impact on Future Health Care Professionals' Attitudes toward Indigenous People.

Mini-Med Schools (MiMS) are an opportunity for health sciences and social work undergraduates to discuss health-related topics with Innu and Atikamekw youth in Canada. More than 500 undergraduates and 1,000 students have taken part in the project since its beginning in 2011. This study aims to assess the impact of both 1) MiMS's predeparture training and 2) the MiMS themselves on undergraduates' prejudices toward Indigenous peoples. Satisfaction of the undergraduates taking part in the activity was also assessed. Seventy-eight undergraduates were recruited and completed the Old-fashioned and Modern Prejudiced Attitudes Toward Aboriginals Scales (O-PATAS and M-PATAS) at baseline, after the pre-departure training, and after the MiMS. They also completed satisfaction surveys. This study shows a reduction of prejudices after participating to a MiMS, but no effect of a pre-departure training. The activities were overall appreciated by undergraduates and most of them would like to take part again in the MiMS.