Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

Situational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used.

A checklist that enhances the provision of education during insulin initiation simulation: a randomized controlled trial.

AIM: The study tested the hypothesis that doctors using an insulin information checklist during simulated insulin initiation would impart more information regarding insulin use. METHODS: A total of 128 simulations were conducted. Doctors (n = 64) were recruited from practitioners recently completing internship (n = 19) and those established in primary care (n = 45). Both groups of doctors were strata randomized to control (n = 32) and intervention groups (n = 32), so that each group contained equal numbers. Doctors in each group experienced two identical simulations of insulin initiation with an intervening period of 10 min. Doctors in the intervention arm were introduced to an insulin initiation checklist, which they reviewed independently and utilized in the second simulation. Trained assessors captured the provision of education in 21 predefined educational areas. Differences in the change of the total education provided between the first and second simulations were assessed using linear regression. RESULTS: The difference in the mean change of education provided between the first and second simulations within the 21 educational areas for the control and treatment groups was 9.7 [95% confidence interval (CI): 8.8-11.1, P < 0.001] - an increase of 46.2%. The difference for the 15 areas relevant to pen use was 7.3 (95% CI: 6.2-8.4, P < 0.001) - an increase of 51.6%. CONCLUSIONS: The checklist resulted in doctors providing significantly more education applicable to syringe and insulin pen routes of insulin administration during simulations. Further research is needed on the checklist's impact on healthcare professionals and patient outcomes in the clinical context. (Clinical Trials Registry No: NCT02266303).

Can Multiple Mini Interviews work in an Irish setting? A feasibility study.

Multiple Mini Interview (MMI) is a new selection tool for medical school applicants. Developed at McMaster University in 2004 it comprises a series of interview stations designed to measure performance across a range of competencies including communication skills, team work, and ethical reasoning. In September 2012, 109 First Year Medical students underwent the MMI. It consisted of 10 stations. The median total score, out of 150, was 100 (min 63, max 129). Cronbach Alphas for the 10 individual stations range from 0.74 to 0.80. Overall Cronbach Alpha of MMI items was 0.78. Staff and student feedback was positive. The outline cost per student was estimated at Euro 145. This study demonstrates that it is feasible to hold a MMI with acceptable levels of reliability and stakeholder approval in an Irish setting. Further work is ongoing to establish the concurrent and predictive validity of MMI in this cohort of medica students.