Objective evaluation of laparoscopic surgical skills in wet lab training based on motion analysis and machine learning.

BACKGROUND: Our aim was to build a skill assessment system, providing objective feedback to trainees based on the motion metrics of laparoscopic surgical instruments. METHODS: Participants performed tissue dissection around the aorta (tissue dissection task) and renal parenchymal closure (parenchymal-suturing task), using swine organs in a box trainer under a motion capture (Mocap) system. Two experts assessed the recorded movies, according to the formula of global operative assessment of laparoscopic skills (GOALS: score range, 5-25), and the mean scores were utilized as objective variables in the regression analyses. The correlations between mean GOALS scores and Mocap metrics were evaluated, and potential Mocap metrics with a Spearman's rank correlation coefficient value exceeding 0.4 were selected for each GOALS item estimation. Four regression algorithms, support vector regression (SVR), principal component analysis (PCA)-SVR, ridge regression, and partial least squares regression, were utilized for automatic GOALS estimation. Model validation was conducted by nested and repeated k-fold cross validation, and the mean absolute error (MAE) was calculated to evaluate the accuracy of each regression model. RESULTS: Forty-five urologic, 9 gastroenterological, and 3 gynecologic surgeons, 4 junior residents, and 9 medical students participated in the training. In both tasks, a positive correlation was observed between the speed-related parameters (e.g., velocity, velocity range, acceleration, jerk) and mean GOALS scores, with a negative correlation between the efficiency-related parameters (e.g., task time, path length, number of opening/closing operations) and mean GOALS scores. Among the 4 algorithms, SVR showed the highest accuracy in the tissue dissection task ([Formula: see text]), and PCA-SVR in the parenchymal-suturing task ([Formula: see text]), based on 100 iterations of the validation process of automatic GOALS estimation. CONCLUSION: We developed a machine learning-based GOALS scoring system in wet lab training, with an error of approximately 1-2 points for the total score, and motion metrics that were explainable to trainees. Our future challenges are the further improvement of onsite GOALS feedback, exploring the educational benefit of our model and building an efficient training program.

Mental workload during endoscopic sinus surgery is associated with surgeons' skill levels.

Introduction: Surgeons' mental workload during endoscopic sinus surgery (ESS) has not been fully evaluated. The assessment was challenging due to the great diversity of each patient's anatomy and the consequence variety of surgical difficulties. In this study, we examined the mental workload of surgeons with various surgical skill levels during ESS under the standardized condition provided by novel-designed 3D sinus models. Materials and methods: Forty-seven participants performed a high-fidelity ESS simulation with 3D-printed sinus models. Surgeons' mental workload was assessed with the national aeronautics and space administration-task load index (NASA-TLX). Associations between the total and subscales score of NASA-TLX and surgical skill index, including the board certification status, the number of experienced ESS cases, and the objective structured assessment of technical skills (OSATS), were analyzed. In addition, 10 registrars repeated the simulation surgery, and their NASA-TLX score was compared before and after the repetitive training. Results: The total NASA-TLX score was significantly associated with OSATS score (p = 0.0001). Primary component analysis classified the surgeons' mental burden into three different categories: (1) the skill-level-dependent factors (temporal demand, effort, and performance), (2) the skill-level-independent factors (mental and physical demand), and (3) frustration. After the repetitive training, the skill-level-dependent factors were alleviated (temporal demand; z = -2.3664, p = 0.0091, effort; z = -2.1704, p = 0.0346, and performance; z = -2.5992, p = 0.0017), the independent factors were increased (mental demand; z = -2.5992, p = 0.0023 and physical demand; z = -2.2509, p = 0.0213), and frustration did not change (p = 0.3625). Conclusion: Some of the mental workload during ESS is associated with surgical skill level and alleviated with repetitive training. However, other aspects remain a burden or could worsen even when surgeons have gained surgical experience. Routine assessment of registrars' mental burdens would be necessary during surgical training to sustain their mental health.

Repetitive simulation training with novel 3D-printed sinus models for functional endoscopic sinus surgeries.

Background: The purpose of this study was to find a utility of a newly developed 3D-printed sinus model and to evaluate the educational benefit of simulation training with the models for functional endoscopic sinus surgery (FESS). Material and methods: Forty-seven otolaryngologists were categorized as experts (board-certified physicians with ≥200 experiences of FESS, n = 9), intermediates (board-certified physicians with <200 experiences of FESS, n = 19), and novices (registrars, n = 19). They performed FESS simulation training on 3D-printed models manufactured from DICOM images of computed tomography (CT) scan of real patients. Their surgical performance was assessed with the objective structured assessment of technical skills (OSATS) score and dissection quality evaluated radiologically with a postdissection CT scan. First we evaluated the face, content, and constructive values. Second we evaluated the educational benefit of the training. Ten novices underwent training (training group) and their outcomes were compared to the remaining novices without training (control group). The training group performed cadaveric FESS surgeries before and after the repetitive training. Results: The feedback from experts revealed high face and content value of the 3D-printed models. Experts, intermediates, and novices demonstrated statistical differences in their OSATS scores (74.7 ± 3.6, 58.3 ± 10.1, and 43.1 ± 11.1, respectively, p < .001), and dissection quality (81.1 ± 13.1, 93.7 ± 15.1, and 126.4 ± 25.2, respectively, p < .001). The training group improved their OSATS score (41.1 ± 8.0 to 61.1 ± 6.9, p < .001) and dissection quality (122.1 ± 22.2 to 90.9 ± 10.3, p = .013), while the control group not. After training, 80% of novices with no prior FESS experiences completed surgeries on cadaver sinuses. Conclusion: Repeated training using the models revealed an initial learning curve in novices, which was confirmed in cadaveric mock FESS surgeries. Level of evidence: N/A.

Automatic assessment of laparoscopic surgical skill competence based on motion metrics.

The purpose of this study was to characterize the motion features of surgical devices associated with laparoscopic surgical competency and build an automatic skill-credential system in porcine cadaver organ simulation training. Participants performed tissue dissection around the aorta, dividing vascular pedicles after applying Hem-o-lok (tissue dissection task) and parenchymal closure of the kidney (suturing task). Movements of surgical devices were tracked by a motion capture (Mocap) system, and Mocap-metrics were compared according to the level of surgical experience (experts: ≥50 laparoscopic surgeries, intermediates: 10-49, novices: 0-9), using the Kruskal-Wallis test and principal component analysis (PCA). Three machine-learning algorithms: support vector machine (SVM), PCA-SVM, and gradient boosting decision tree (GBDT), were utilized for discrimination of the surgical experience level. The accuracy of each model was evaluated by nested and repeated k-fold cross-validation. A total of 32 experts, 18 intermediates, and 20 novices participated in the present study. PCA revealed that efficiency-related metrics (e.g., path length) significantly contributed to PC 1 in both tasks. Regarding PC 2, speed-related metrics (e.g., velocity, acceleration, jerk) of right-hand devices largely contributed to the tissue dissection task, while those of left-hand devices did in the suturing task. Regarding the three-group discrimination, in the tissue dissection task, the GBDT method was superior to the other methods (median accuracy: 68.6%). In the suturing task, SVM and PCA-SVM methods were superior to the GBDT method (57.4 and 58.4%, respectively). Regarding the two-group discrimination (experts vs. intermediates/novices), the GBDT method resulted in a median accuracy of 72.9% in the tissue dissection task, and, in the suturing task, the PCA-SVM method resulted in a median accuracy of 69.2%. Overall, the mocap-based credential system using machine-learning classifiers provides a correct judgment rate of around 70% (two-group discrimination). Together with motion analysis and wet-lab training, simulation training could be a practical method for objectively assessing the surgical competence of trainees.