The MISTELS program to measure technical skill in laparoscopic surgery : evidence for reliability.

BACKGROUND: The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) is a series of five tasks with an objective scoring system. The purpose of this study was to estimate the interrater and test-retest reliability of the MISTELS metrics and to assess their internal consistency. METHODS: To determine interrater reliability, two trained observers scored 10 subjects, either live or on tape. Test-retest reliability was assessed by having 12 subjects perform two tests, the second immediately following the first. Interrater and test-retest reliability were assessed using intraclass correlation coefficients. Internal consistency between tasks was estimated using Cronbach's alpha. RESULTS: The interrater and test-retest reliabilities for the total scores were both excellent at 0.998 [95% confidence interval (CI), 0.985-1.00] and 0.892 (95% CI, 0.665-0.968), respectively. Cronbach's alpha for the first assessment of the test-retest was 0.86. CONCLUSIONS: The MISTELS metrics have excellent reliability, which exceeds the threshold level of 0.8 required for high-stakes evaluations. These findings support the use of MISTELS for evaluation in many different settings, including residency training programs.

Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system.

BACKGROUND: The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) was developed to assess laparoscopic skills and to score them objectively. This system has been described previously. The purpose of the current study was to determine a pass/fail threshold. METHODS: In this study, 165 individuals were tested and grouped according to their clinical competency in laparoscopic surgery. The noncompetent group consisted of medical students and surgical residents in their first 2 years of training (n = 83). The competent group consisted of chief general surgical residents in their last year of training, laparoscopy fellows, and practicing laparoscopic surgeons (n = 82). The Mann-Whitney U test was used to evaluate differences in task performance between the two groups. RESULTS: There was a significant difference in total scores and individual MISTELS task scores between the noncompetent and competent laparoscopic surgeons (189 vs 372.5; p <0.0001). By setting specific pass/fail total score thresholds (cutoff scores), competent surgeons can be discriminated from noncompetent surgeons. CONCLUSION: An objective pass/fail evaluation can be given to individuals tested with the MISTELS system.