Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews.

BACKGROUND: Multiple mini-interviews (MMI) are used to assess non-academic attributes for selection in medicine and other healthcare professions. It remains unclear if different MMI station formats (discussions, role-plays, collaboration) assess different dimensions. METHODS: Based on station formats of the 2018 and 2019 Integrated French MMI (IFMMI), which comprised five discussions, three role-plays and two collaboration stations, the authors performed confirmatory factor analysis (CFA) using the lavaan 0.6-5 R package and compared a one-factor solution to a three-factor solution for scores of the 2018 (n = 1438) and 2019 (n = 1440) cohorts of the IFMMI across three medical schools in Quebec, Canada. RESULTS: The three-factor solution was retained, with discussions, role-plays and collaboration stations all loading adequately with their scores. Furthermore, all three factors had moderate-to-high covariance (range 0.44 to 0.64). The model fit was also excellent with a Comparative fit index (CFI) of 0.983 (good if > 0.9), a Tucker Lewis index of 0.976 (good if > 0.95), a Standardized Root Mean Square Residual of 0.021 (good if < .08) and a Root Mean Square Error of 0.023 (good if < 0.08) for 2018 and similar results for 2019. In comparison, the single factor solution presented a lower fit (CFI = 0.819, TLI = 0.767, SRMR = 0.049 and RMSEA = 0.070). CONCLUSIONS: The IFMMI assessed three dimensions that were related to stations formats, a finding that was consistent across two cohorts. This suggests that different station formats may be assessing different skills, and has implications for the choice of appropriate reliability metrics and the interpretation of scores. Further studies should try to characterize the underlying constructs associated with each station format and look for differential predictive validity according to these formats.

Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study.

When determining the score given to candidates in multiple mini-interview (MMI) stations, raters have to translate a narrative judgment to an ordinal rating scale. When adding individual scores to calculate final ranking, it is generally presumed that the values of possible scores on the evaluation grid are separated by constant intervals, following a linear function, although this assumption is seldom validated with raters themselves. Inaccurate interval values could lead to systemic bias that could potentially distort candidates' final cumulative scores. The aim of this study was to establish rating scale values based on rater's intent, to validate these with an independent quantitative method, to explore their impact on final score, and to appraise their meaning according to experienced MMI interviewers. A 4-round consensus-group exercise was independently conducted with 42 MMI interviewers who were asked to determine relative values for the 6-point rating scale (from A to F) used in the Canadian integrated French MMI (IFMMI). In parallel, relative values were also calculated for each option of the scale by comparing the average scores concurrently given to the same individual in other stations every time that option was selected during three consecutive IFMMI years. Data from the same three cohorts was used to simulate the impact of using new score values on final rankings. Comments from the consensus group exercise were reviewed independently by two authors to explore raters' rationale for choosing specific values. Relative to the maximum (A = 100%) and minimum (F = 0%), experienced raters concluded to values of 86.7% (95% CI 86.3-87.1), 69.5% (68.9-70.1), 51.2% (50.6-51.8), and 29.3% (28.1-30.5), for scores of B, C, D and E respectively. The concurrent score approach was based on 43,412 IFMMI stations performed by 4345 medical school applicants. It provided quasi-identical values of 87.1% (82.4-91.5), 70.4% (66.1-74.7), 51.2% (47.1-55.3) and 31.8% (27.9-35.7), respectively. Qualitative analysis explained that while high scores are usually based on minor details of relatively low importance, low scores are usually attributed for more serious offenses and were assumed by the raters to carry more weight in the final score. Individual drop or increase in final MMI ranking with the use of new scale values ranged from - 21 to + 5 percentiles, with the average candidate changing by ± 1.4 percentiles. Consulting with experienced interviewers is a simple and effective approach to establish rating scale values that truly reflects raters' intent in MMI, thus improving the accuracy of the instrument and contributing to the general fairness of the process.

Impact of sociodemographic characteristics of applicants in multiple mini-interviews.

BACKGROUND: Multiple mini-interviews (MMI) are commonly used for medical school admission. This study aimed to assess if sociodemographic characteristics are associated with MMI performance, and how they may act as barriers or enablers to communication in MMI. METHODS: This mixed-method study combined data from a sociodemographic questionnaire, MMI scores, semi-structured interviews and focus groups with applicants and assessors. Quantitative and qualitative data were analyzed using multiple linear regression and a thematic framework analysis. RESULTS: 1099 applicants responded to the questionnaire. A regression model (R2 = 0.086) demonstrated that being age 25-29 (ß = 0.11, p = 0.001), female and a French-speaker (ß = 0.22, p = 0.003) were associated with better MMI scores. Having an Asian-born parent was associated with a lower score (ß = -0.12, p < 0.001). Candidates reporting a higher family income had higher MMI scores. In the qualitative data, participants discussed how maturity and financial support improved life experiences, how language could act as a barrier, and how ethnocultural differences could lead to misunderstandings. CONCLUSION: Age, gender, ethnicity, socioeconomic status and language seem to be associated with applicants' MMI scores because of perceived differences in communications skills and life experiences. Monitoring this association may provide guidance to improve fairness of MMI stations.

Clinical reasoning processes: unravelling complexity through graphical representation.

CONTEXT: Clinical reasoning is a core skill in medical practice, but remains notoriously difficult for students to grasp and teachers to nurture. To date, an accepted model that adequately captures the complexity of clinical reasoning processes does not exist. Knowledge-modelling software such as mot Plus (Modelling using Typified Objects [MOT]) may be exploited to generate models capable of unravelling some of this complexity. OBJECTIVES: This study was designed to create a comprehensive generic model of clinical reasoning processes that is intended for use by teachers and learners, and to provide data on the validity of the model. METHODS: Using a participatory action research method and the established modelling software (mot Plus), knowledge was extracted and entered into the model by a cognitician in a series of encounters with a group of experienced clinicians over more than 250 contact hours. The model was then refined through an iterative validation process involving the same group of doctors, after which other groups of clinicians were asked to solve a clinical problem involving simulated patients. RESULTS: A hierarchical model depicting the multifaceted processes of clinical reasoning was produced. Validation rounds suggested generalisability across disciplines and situations. CONCLUSIONS: The MOT model of clinical reasoning processes has potentially important applications for use within undergraduate and graduate medical curricula to inform teaching, learning and assessment. Specifically, it could be used to support curricular development because it can help to identify opportune moments for learning specific elements of clinical reasoning. It could also be used to precisely identify and remediate reasoning errors in students, residents and practising doctors with persistent difficulties in clinical reasoning.

Black students applying and admitted to medicine in the province of Quebec, Canada: what do we know so far?

To address the underrepresentation of Black students in medical schools in Canada and identify barriers in selection processes, we compare data from the latest Canadian census to that of an exit-survey conducted after a situational judgment test (Casper) among medical school applicants and from questionnaires done after selection interviews in Quebec, Canada. The proportion of Black people aged 15-34 years old in Quebec in 2016 was 5.3% province-wide and 8.2% in the Montreal metropolitan area. The proportion in the applicant pool for 2020 in Quebec was estimated to be 4.5% based on Casper exit-survey data. Comparatively, it is estimated that Black people represented 1.8% of applicants invited to admission interviews and 1.2% of admitted students in Quebec in 2019. Although data from different cohorts and data sources do not allow for direct comparisons, these numbers suggest that Black students applying to medical school are disproportionately rejected at the first step compared to non-Black students. Longitudinal data collection among medical school applicants will be necessary to monitor the situation. Further studies are required to pinpoint the factors contributing to this underrepresentation, to keep improving the equity of our selection processes.

Afin de remédier à la sous-représentation des étudiants noirs dans les facultés de médecine au Canada et de cibler les obstacles qu'ils rencontrent dans le processus de sélection, nous comparons les données du dernier recensement canadien avec celles d'un sondage réalisé à la suite d'un test de jugement situationnel (Casper) auprès de candidats ayant fait une demande d'admission dans un programme de doctorat en médecine et celles d'un sondage réalisé à la suite d'entretiens de sélection au Québec (Canada). La proportion de personnes noires âgées de 15 à 34 ans au Québec en 2016 était de 5,3 % à l'échelle de la province et de 8,2 % dans la région métropolitaine de Montréal. La proportion de cette population dans le bassin de candidats pour 2020 au Québec a été estimée à 4,5 % sur la base des données du sondage Casper. À titre de comparaison, on estime que les Noirs représentaient 1,8 % des candidats invités aux entrevues d'admission et 1,2 % des étudiants admis au Québec en 2019. Bien que les données pour les différentes cohortes, provenant de surcroît de sources différentes, ne permettent pas d'établir des comparaisons directes, ces chiffres suggèrent que les étudiants noirs qui demandent à être admis en médecine sont rejetés de manière disproportionnée à la première étape par rapport aux étudiants non noirs. Une collecte de données longitudinales parmi les candidats sera nécessaire pour suivre l'évolution de la situation, ainsi que d'autres études pour découvrir les facteurs qui contribuent à cette sous-représentation, notamment dans une visée d'amélioration de l'équité dans les processus de sélection.