Accuracy of rating scale interval values used in multiple mini-interviews: a mixed methods study.

When determining the score given to candidates in multiple mini-interview (MMI) stations, raters have to translate a narrative judgment to an ordinal rating scale. When adding individual scores to calculate final ranking, it is generally presumed that the values of possible scores on the evaluation grid are separated by constant intervals, following a linear function, although this assumption is seldom validated with raters themselves. Inaccurate interval values could lead to systemic bias that could potentially distort candidates' final cumulative scores. The aim of this study was to establish rating scale values based on rater's intent, to validate these with an independent quantitative method, to explore their impact on final score, and to appraise their meaning according to experienced MMI interviewers. A 4-round consensus-group exercise was independently conducted with 42 MMI interviewers who were asked to determine relative values for the 6-point rating scale (from A to F) used in the Canadian integrated French MMI (IFMMI). In parallel, relative values were also calculated for each option of the scale by comparing the average scores concurrently given to the same individual in other stations every time that option was selected during three consecutive IFMMI years. Data from the same three cohorts was used to simulate the impact of using new score values on final rankings. Comments from the consensus group exercise were reviewed independently by two authors to explore raters' rationale for choosing specific values. Relative to the maximum (A = 100%) and minimum (F = 0%), experienced raters concluded to values of 86.7% (95% CI 86.3-87.1), 69.5% (68.9-70.1), 51.2% (50.6-51.8), and 29.3% (28.1-30.5), for scores of B, C, D and E respectively. The concurrent score approach was based on 43,412 IFMMI stations performed by 4345 medical school applicants. It provided quasi-identical values of 87.1% (82.4-91.5), 70.4% (66.1-74.7), 51.2% (47.1-55.3) and 31.8% (27.9-35.7), respectively. Qualitative analysis explained that while high scores are usually based on minor details of relatively low importance, low scores are usually attributed for more serious offenses and were assumed by the raters to carry more weight in the final score. Individual drop or increase in final MMI ranking with the use of new scale values ranged from - 21 to + 5 percentiles, with the average candidate changing by ± 1.4 percentiles. Consulting with experienced interviewers is a simple and effective approach to establish rating scale values that truly reflects raters' intent in MMI, thus improving the accuracy of the instrument and contributing to the general fairness of the process.

Development and Validation of a Behavioural Index for Adaptation to High Summer Temperatures among Urban Dwellers.

One of the consequences of climate change is the growing number of extreme weather events, including heat waves, which have substantial impacts on the health of populations. From a public health standpoint, it is vital to ensure that people can adapt to high heat, particularly in cities where heat islands abound. Identifying indicators to include in a parsimonious index would help better differentiate individuals who adapt well to heat from those who do not adapt as well. This study aimed at developing and validating a summer heat adaptation index for residents of the 10 largest cities in the province of Québec, Canada. A sample of 2000 adults in 2015 and 1030 adults in 2016 completed a telephone questionnaire addressing their adoption (or non-adoption) of behaviours recommended by public health agencies to protect themselves during periods of high temperature, and their perceptions of how high summer heat affects their mental and physical health. Item analysis, confirmatory factor analysis, multiple correspondence analysis, measurement invariance analyses and criterion-validity analyses were used to develop a 12-behaviour heat adaptation index for distinguishing between individuals who adapt well to high temperatures and those who do not adapt as well. The results indicated that the measurement and the factor structure of the index were invariant (equivalent) across the two independent samples of participants who completed the questionnaire at different times one year apart, an important prerequisite for unambiguous interpretation of index scores across groups and over time. The results also showed that individuals who perceived more adverse effects on their physical or mental health adopted more preventive behaviours during periods of high temperatures and humidity conditions compared to those who felt lesser or no effects. This study thus presents support for the validity of the index that could be used in future studies to monitor preventive behaviours adoption during summer periods of high temperature.