Skill-level classification and performance evaluation for endoscopic sleeve gastroplasty.

BACKGROUND: We previously developed grading metrics for quantitative performance measurement for simulated endoscopic sleeve gastroplasty (ESG) to create a scalar reference to classify subjects into experts and novices. In this work, we used synthetic data generation and expanded our skill level analysis using machine learning techniques. METHODS: We used the synthetic data generation algorithm SMOTE to expand and balance our dataset of seven actual simulated ESG procedures using synthetic data. We performed optimization to seek optimum metrics to classify experts and novices by identifying the most critical and distinctive sub-tasks. We used support vector machine (SVM), AdaBoost, K-nearest neighbors (KNN) Kernel Fisher discriminant analysis (KFDA), random forest, and decision tree classifiers to classify surgeons as experts or novices after grading. Furthermore, we used an optimization model to create weights for each task and separate the clusters by maximizing the distance between the expert and novice scores. RESULTS: We split our dataset into a training set of 15 samples and a testing dataset of five samples. We put this dataset through six classifiers, SVM, KFDA, AdaBoost, KNN, random forest, and decision tree, resulting in 0.94, 0.94, 1.00, 1.00, 1.00, and 1.00 accuracy, respectively, for training and 1.00 accuracy for the testing results for SVM and AdaBoost. Our optimization model maximized the distance between the expert and novice groups from 2 to 53.72. CONCLUSION: This paper shows that feature reduction, in combination with classification algorithms such as SVM and KNN, can be used in tandem to classify endoscopists as experts or novices based on their results recorded using our grading metrics. Furthermore, this work introduces a non-linear constraint optimization to separate the two clusters and find the most important tasks using weights.

Objective metrics for hand-sewn bowel anastomoses can differentiate novice from expert surgeons.

BACKGROUND: Assessing performance automatically in a virtual reality trainer or from recorded videos is advantageous but needs validated objective metrics. The purpose of this study is to obtain expert consensus and validate task-specific metrics developed for assessing performance in double-layered end-to-end anastomosis. MATERIALS AND METHODS: Subjects were recruited into expert (PGY 4-5, colorectal surgery residents, and attendings) and novice (PGY 1-3) groups. Weighted average scores of experts for each metric item, completion time, and the total scores computed using global and task-specific metrics were computed for assessment. RESULTS: A total of 43 expert surgeons rated our task-specific metric items with weighted averages ranging from 3.33 to 4.5 on a 5-point Likert scale. A total of 20 subjects (10 novices and 10 experts) participated in validation study. The novice group completed the task significantly more slowly than the experienced group (37.67 ± 7.09 vs 25.47 ± 7.82 min, p = 0.001). In addition, both the global rating scale (23.47 ± 4.28 vs 28.3 ± 3.85, p = 0.016) and the task-specific metrics showed a significant difference in performance between the two groups (38.77 ± 2.83 vs 42.58 ± 4.56 p = 0.027) following partial least-squares (PLS) regression. Furthermore, PLS regression showed that only two metric items (Stay suture tension and Tool handling) could reliably differentiate the performance between the groups (20.41 ± 2.42 vs 24.28 ± 4.09 vs, p = 0.037). CONCLUSIONS: Our study shows that our task-specific metrics have significant discriminant validity and can be used to evaluate the technical skills for this procedure.

Hierarchical task analysis of endoscopic sleeve gastroplasty.

BACKGROUND: Endoscopic sleeve gastroplasty (ESG) is a minimally invasive endoscopic weight loss procedure used to treat obesity. The long-term goal of this project is to develop a Virtual Bariatric Endoscopy (ViBE) simulator for training and assessment of the ESG procedure. The objectives of this current work are to: (a) perform a task analysis of ESG and (b) create metrics to be validated in the created simulator. METHODS: We performed a hierarchical task analysis (HTA) by identifying the significant tasks of the ESG procedure. We created the HTA to show the breakdown and connection of the tasks of the procedure. Utilizing the HTA and input from ESG experts, performance metrics were derived for objective measurement of the ESG procedure. Three blinded video raters analyzed seven recorded ESG procedures according to the proposed performance metrics. RESULTS: Based on the seven videos, there was a positive correlation between total task times and total performance scores (R = 0.886, P = 0.008). Endoscopists expert were found to be more skilled in reducing the area of the stomach compared to endoscopists novice (34.6% reduction versus 9.4% reduction, P = 0.01). The mean novice performance score was significantly lower than the mean expert performance score (34.7 vs. 23.8, P = 0.047). The inter-rater reliability test showed a perfect agreement among three raters for all tasks except for the suturing task. The suturing task had a significant agreement (Inter-rater Correlation = 0.84, Cronbach's alpha = 0.88). Suturing was determined to be a critical task that is positively correlated with the total score (R = 0.962, P = 0.0005). CONCLUSION: The task analysis and metrics development are critical for the development of the ViBE simulator. This preliminary assessment demonstrates that the performance metrics provide an accurate assessment of the endoscopist's performance. Further validation testing and refinement of the performance metrics are anticipated.

Endoscopic sleeve gastroplasty: stomach location and task classification for evaluation using artificial intelligence.

PURPOSE: We have previously developed grading metrics to objectively measure endoscopist performance in endoscopic sleeve gastroplasty (ESG). One of our primary goals is to automate the process of measuring performance. To achieve this goal, the repeated task being performed (grasping or suturing) and the location of the endoscopic suturing device in the stomach (Incisura, Anterior Wall, Greater Curvature, or Posterior Wall) need to be accurately recorded. METHODS: For this study, we populated our dataset using screenshots and video clips from experts carrying out the ESG procedure on ex vivo porcine specimens. Data augmentation was used to enlarge our dataset, and synthetic minority oversampling (SMOTE) to balance it. We performed stomach localization for parts of the stomach and task classification using deep learning for images and computer vision for videos. RESULTS: Classifying the stomach's location from the endoscope without SMOTE for images resulted in 89% and 84% testing and validation accuracy, respectively. For classifying the location of the stomach from the endoscope with SMOTE, the accuracies were 97% and 90% for images, while for videos, the accuracies were 99% and 98% for testing and validation, respectively. For task classification, the accuracies were 97% and 89% for images, while for videos, the accuracies were 100% for both testing and validation, respectively. CONCLUSION: We classified the four different stomach parts manipulated during the ESG procedure with 97% training accuracy and classified two repeated tasks with 99% training accuracy with images. We also classified the four parts of the stomach with a 99% training accuracy and two repeated tasks with a 100% training accuracy with video frames. This work will be essential in automating feedback mechanisms for learners in ESG.