Support vector machines improve the accuracy of evaluation for the performance of laparoscopic training tasks.

ABSTRACT

BACKGROUND: Despite technological advances in the tracking of surgical motions, automatic evaluation of laparoscopic skills remains remote. A new method is proposed that combines multiple discrete motion analysis metrics. This new method is compared with previously proposed metric combination methods and shown to provide greater ability for classifying novice and expert surgeons. METHODS: For this study, 30 participants (four experts and 26 novices) performed 696 trials of three training tasks peg transfer, pass rope, and cap needle. Instrument motions were recorded and reduced to four metrics. Three methods of combining metrics into a prediction of surgical competency (summed-ratios, z-score normalization, and support vector machine [SVM]) were compared. The comparison was based on the area under the receiver operating characteristic curve (AUC) and the predictive accuracy with a previously unseen validation data set. RESULTS: For all three tasks, the SVM method was superior in terms of both AUC and predictive accuracy with the validation set. The SVM method resulted in AUCs of 0.968, 0.952, and 0.970 for the three tasks compared respectively with 0.958, 0.899, and 0.884 for the next best method (weighted z-normalization). The SVM method correctly predicted 93.7, 91.3, and 90.0% of the subjects' competencies, whereas the weighted z-normalization respectively predicted 86.6, 79.3, and 75.7% accurately (p < 0.002). CONCLUSIONS: The findings show that an SVM-based analysis provides more accurate predictions of competency at laparoscopic training tasks than previous analysis techniques. An SVM approach to competency evaluation should be considered for computerized laparoscopic performance evaluation systems.