Eye gaze metrics for skill assessment and feedback in kidney stone surgery.

PURPOSE: Surgical skill assessment is essential for safe operations. In endoscopic kidney stone surgery, surgeons must perform a highly skill-dependent mental mapping from the pre-operative scan to the intraoperative endoscope image. Poor mental mapping can lead to incomplete exploration of the kidney and high reoperation rates. Yet there are few objective ways to evaluate competency. We propose to use unobtrusive eye-gaze measurements in the task space to evaluate skill and provide feedback. METHODS: We capture the surgeons' eye gaze on the surgical monitor with the Microsoft Hololens 2. To enable stable and accurate gaze detection, we develop a calibration algorithm to refine the eye tracking of the Hololens. In addition, we use a QR code to locate the eye gaze on the surgical monitor. We then run a user study with three expert and three novice surgeons. Each surgeon is tasked to locate three needles representing kidney stones in three different kidney phantoms. RESULTS: We find that experts have more focused gaze patterns. They complete the task faster, have smaller total gaze area, and the gaze fewer times outside the area of interest. While fixation to non-fixation ratio did not show significant difference in our findings, tracking the ratio over time shows different patterns between novices and experts. CONCLUSION: We show that a non-negligible difference holds between novice and expert surgeons' gaze metrics in kidney stone identification in phantoms. Expert surgeons demonstrate more targeted gaze throughout a trial, indicating their higher level of proficiency. To improve the skill acquisition process for novice surgeons, we suggest providing sub-task specific feedback. This approach presents an objective and non-invasive method to assess surgical competence.

Robot-assisted ex vivo neobladder reconstruction: preliminary results of surgical skill evaluation.

PURPOSE: Advanced developments in the medical field have gradually increased the public demand for surgical skill evaluation. However, this assessment always depends on the direct observation of experienced surgeons, which is time-consuming and variable. The introduction of robot-assisted surgery provides a new possibility for this evaluation paradigm. This paper aims at evaluating surgeon performance automatically with novel evaluation metrics based on different surgical data. METHODS: Urologists ([Formula: see text]) from a hospital were requested to perform a simplified neobladder reconstruction on an ex vivo setup twice with different camera modalities ([Formula: see text]) randomly. They were divided into novices and experts ([Formula: see text], respectively) according to their experience in robot-assisted surgeries. Different performance metrics ([Formula: see text]) are proposed to achieve the surgical skill evaluation, considering both instruments and endoscope. Also, nonparametric tests are adopted to check if there are significant differences when evaluating surgeons performance. RESULTS: When grouping according to four stages of neobladder reconstruction, statistically significant differences can be appreciated in phase 1 ([Formula: see text]) and phase 2 ([Formula: see text]) with normalized time-related metrics and camera movement-related metrics, respectively. On the other hand, considering experience grouping shows that both metrics are able to highlight statistically significant differences between novice and expert performances in the control protocol. It also shows that the camera-related performance of experts is significantly different ([Formula: see text]) when handling the endoscope manually and when it is automatic. CONCLUSION: Surgical skill evaluation, using the approach in this paper, can effectively measure surgical procedures of surgeons with different experience. Preliminary results demonstrate that different surgical data can be fully utilized to improve the reliability of surgical evaluation. It also demonstrates its versatility and potential in the quantitative assessment of various surgical operations.

Objective assessment of robotic surgical skills: review of literature and future directions.

BACKGROUND: Evaluation of robotic surgical skill has become increasingly important as robotic approaches to common surgeries become more widely utilized. However, evaluation of these currently lacks standardization. In this paper, we aimed to review the literature on robotic surgical skill evaluation. METHODS: A review of literature on robotic surgical skill evaluation was performed and representative literature presented over the past ten years. RESULTS: The study of reliability and validity in robotic surgical evaluation shows two main assessment categories: manual and automatic. Manual assessments have been shown to be valid but typically are time consuming and costly. Automatic evaluation and simulation are similarly valid and simpler to implement. Initial reports on evaluation of skill using artificial intelligence platforms show validity. Few data on evaluation methods of surgical skill connect directly to patient outcomes. CONCLUSION: As evaluation in surgery begins to incorporate robotic skills, a simultaneous shift from manual to automatic evaluation may occur given the ease of implementation of these technologies. Robotic platforms offer the unique benefit of providing more objective data streams including kinematic data which allows for precise instrument tracking in the operative field. Such data streams will likely incrementally be implemented in performance evaluations. Similarly, with advances in artificial intelligence, machine evaluation of human technical skill will likely form the next wave of surgical evaluation.

How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery.

INTRODUCTION: A surgeon's technical skills are an important factor in delivering optimal patient care. Most existing methods to estimate technical skills remain subjective and resource intensive. Robotic-assisted surgery (RAS) provides a unique opportunity to develop objective metrics using key elements of intraoperative surgeon behavior which can be captured unobtrusively, such as instrument positions and button presses. Recent studies have shown that objective metrics based on these data (referred to as objective performance indicators [OPIs]) correlate to select clinical outcomes during robotic-assisted radical prostatectomy. However, the current OPIs remain difficult to interpret directly and, therefore, to use within structured feedback to improve surgical efficiencies. METHODS: We analyzed kinematic and event data from da Vinci surgical systems (Intuitive Surgical, Inc., Sunnyvale, CA, USA) to calculate values that can summarize the use of robotic instruments, referred to as OPIs. These indicators were mapped to broader technical skill categories of established training protocols. A data-driven approach was then applied to further sub-select OPIs that distinguish skill for each technical skill category within each training task. This subset of OPIs was used to build a set of logistic regression classifiers that predict the probability of expertise in that skill to identify targeted improvement and practice. The final, proposed feedback using OPIs was based on the coefficients of the logistic regression model to highlight specific actions that can be taken to improve. RESULTS: We determine that for the majority of skills, only a small subset of OPIs (2-10) are required to achieve the highest model accuracies (80-95%) for estimating technical skills within clinical-like tasks on a porcine model. The majority of the skill models have similar accuracy as models predicting overall expertise for a task (80-98%). Skill models can divide a prediction into interpretable categories for simpler, targeted feedback. CONCLUSION: We define and validate a methodology to create interpretable metrics for key technical skills during clinical-like tasks when performing RAS. Using this framework for evaluating technical skills, we believe that surgical trainees can better understand both what can be improved and how to improve.

Comparative analysis of surgical processes for image-guided endoscopic sinus surgery.

PURPOSE: This study proposes a method to analyze surgical performance by modeling, aligning, and comparing surgical processes. This method is intended to serve as a means to support the enhancement of surgical skills for endoscopic sinus surgeries (ESSs). We focus on surgical navigation systems used in image-guided ESSs and aim to construct a comparative analysis method for surgical processes based on the information about the surgical instruments motion obtained from the navigation system. METHODS: The proposed method consists of the following three parts: quantification of surgical features, modeling of surgical processes, and alignment and comparison of surgical process models (SPMs). First, we defined time-series parameters using the navigation-based surgical data. Second, we created SPMs by applying the defined parameters and the relative positional information of the instruments to the patient's anatomy. Third, we constructed a method to align and compare SPMs based on dynamic time warping with barycenter averaging. RESULTS: The proposed method was validated on a dataset containing surgical data obtained by an optical tracking system from 14 clinical ESS cases. We evaluated the validity of the comparative analysis by aligning and comparing SPMs between experts and residents. The validation results suggested that the proposed method could achieve proper alignment of the SPMs and clarify the differences in surgical processes between experts and residents. CONCLUSION: We developed a method to enable a time-series comparative analysis of surgical processes based on the surgical data from the navigation system. This method can allow surgeons to identify differences between their procedures and reference procedures such as experts' procedures.

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery.

PURPOSE: With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. METHODS: We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels. RESULTS: We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1-3 second window, without needing an observation of entire training trial. CONCLUSION: This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

Predicting surgical skill from the first N seconds of a task: value over task time using the isogony principle.

PURPOSE: Most evaluations of surgical workflow or surgeon skill use simple, descriptive statistics (e.g., time) across whole procedures, thereby deemphasizing critical steps and potentially obscuring critical inefficiencies or skill deficiencies. In this work, we examine off-line, temporal clustering methods that chunk training procedures into clinically relevant surgical tasks or steps during robot-assisted surgery. METHODS: Features calculated from the isogony principle are used to train four common machine learning algorithms from dry-lab laparoscopic data gathered from three common training exercises. These models are used to predict the binary or ternary skill level of a surgeon. K-fold and leave-one-user-out cross-validation are used to assess the accuracy of the generated models. RESULTS: It is shown that the proposed scalar features can be trained to create 2-class and 3-class classification models that map to fundamentals of laparoscopic surgery skill level with median 85 and 63% accuracy in cross-validation, respectively, for the targeted dataset. Also, it is shown that the 2-class models can discern class at 90% of best-case mean accuracy with only 8 s of data from the start of the task. CONCLUSION: Novice and expert skill levels of unobserved trials can be discerned using a state vector machine trained with parameters based on the isogony principle. The accuracy of this classification comes within 90% of the classification accuracy from observing the full trial within 10 s of task initiation on average.

The minimally acceptable classification criterion for surgical skill: intent vectors and separability of raw motion data.

PURPOSE: Minimally invasive surgery requires objective methods for skill evaluation and training. This work presents the minimally acceptable classification (MAC) criterion for computational surgery: Given an obvious novice and an obvious expert, a surgical skill evaluation classifier must yield 100% accuracy. We propose that a rigorous motion analysis algorithm must meet this minimal benchmark in order to justify its cost and use. METHODS: We use this benchmark to investigate two concepts: First, how separable is raw, multidimensional dry laboratory laparoscopic motion data between obvious novices and obvious experts? We utilized information theoretic techniques to analytically address this. Second, we examined the use of intent vectors to classify surgical skill using three FLS tasks. RESULTS: We found that raw motion data alone are not sufficient to classify skill level; however, the intent vector approach is successful in classifying surgical skill level for certain tasks according to the MAC criterion. For a pattern cutting task, this approach yields 100% accuracy in leave-one-user-out cross-validation. CONCLUSION: Compared to prior art, the intent vector approach provides a generalized method to assess laparoscopic surgical skill using basic motion segments and passes the MAC criterion for some but not all FLS tasks.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room.

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Effect of soft-tissue impingement on range of motion during posterior approach Total Hip Arthroplasty: an in vivo measurement study.

PURPOSE: While implant impingement and bony impingement have been recognized as causes of poor outcomes in total hip arthroplasty (THA), reports of soft-tissue impingement are rare. To clarify the issue, the effect of anterior capsule resection on hip range of motion (ROM) was quantitatively measured in vivo during posterior approach THA using a CT-based hip navigation system. MATERIALS AND METHODS: For 47 patients (51 hips), hip ROM was measured intraoperatively before and after resection of the anterior hip capsule, and the difference was compared. RESULTS: Resection of the anterior hip capsule brought about an average 6° increase of ROM in the direction of flexion with internal rotation and did not markedly change ROM in other directions. CONCLUSIONS: During THA through a posterior approach, soft-tissue impingement by the anterior hip capsule can occur. Clinically, we expect that resection of the anterior hip capsule can reduce the risk of posterior instability without increasing the risk of anterior instability.

Design and implementation of a proficiency-based, structured endoscopy course for medical students applying for a surgical specialty.

BACKGROUND: Surgical simulation is becoming increasingly important in surgical education. Despite the important work done on simulators, simulator model development, and simulator assessment methodologies, there is a need for development of integrated simulators in the curriculum. In this paper, we describe the design of our evidence-based preclinical training program for medical students applying for a surgical career at the Centre for Surgical Technologies. METHODS: Twenty-two students participated in this training program. During their final months as medical students, they received structured, proficiency-based endoscopy training. The total amount of mentored training was 18 hours and the training was organized into three training blocks. The first block focused on psychomotor training, the second block focused on laparoscopic stitching and suturing, and the third block on laparoscopic dissection techniques and hemostasis. Deliberate practice was allowed and students had to show proficiency before proceeding to the next training block. Students' psychomotor abilities were tested before the course and after each training block. At the beginning of their careers as surgical registrars, their performance on a laparoscopic suturing task was compared with that of registrars from the previous year who did not have this training course. Student opinions about this course were evaluated using a visual analog scale. RESULTS: All students rated the training course as useful and their psychomotor abilities improved markedly. All students performed deliberate practice, and those who participated in this course scored significantly (P < 0.0001) better on the laparoscopic suturing task than first year registrars who did not participate in this course. CONCLUSION: Organization of a structured preclinical training program in laparoscopy for final year medical students is feasible, attractive, and successful.