FAST skill assessment from kinematics data using convolutional neural networks.

PURPOSE: FAST is a point of care ultrasound study that evaluates for the presence of free fluid, typically hemoperitoneum in trauma patients. FAST is an essential skill for Emergency Physicians. Thus, it requires objective evaluation tools that can reduce the necessity of direct observation for proficiency assessment. In this work, we use deep neural networks to automatically assess operators' FAST skills. METHODS: We propose a deep convolutional neural network for FAST proficiency assessment based on motion data. Prior work has shown that operators demonstrate different domain-specific dexterity metrics that can distinguish novices, intermediates, and experts. Therefore, we augment our dataset with this domain knowledge and employ fine-tuning to improve the model's classification capabilities. Our model, however, does not require specific points of interest (POIs) to be defined for scanning. RESULTS: The results show that the proposed deep convolutional neural network can classify FAST proficiency with 87.5% accuracy and 0.884, 0.886, 0.247 sensitivity for novices, intermediates, and experts, respectively. It demonstrates the potential of using kinematics data as an input in FAST skill assessment tasks. We also show that the proposed domain-specific features and region fine-tuning increase the model's classification accuracy and sensitivity. CONCLUSIONS: Variations in probe motion at different learning stages can be derived from kinematics data. These variations can be used for automatic and objective skill assessment without prior identification of clinical POIs. The proposed approach can improve the quality and objectivity of FAST proficiency evaluation. Furthermore, skill assessment combining ultrasound images and kinematics data can provide a more rigorous and diversified evaluation than using ultrasound images alone.

Reconstructing the nasal septum from instrument motion during septoplasty surgery.

Purpose: Surgery involves modifying anatomy to achieve a goal. Reconstructing anatomy can facilitate surgical care through surgical planning, real-time decision support, or anticipating outcomes. Tool motion is a rich source of data that can be used to quantify anatomy. Our work develops and validates a method for reconstructing the nasal septum from unstructured motion of the Cottle elevator during the elevation phase of septoplasty surgery, without need to explicitly delineate the surface of the septum. Approach: The proposed method uses iterative closest point registration to initially register a template septum to the tool motion. Subsequently, statistical shape modeling with iterative most likely oriented point registration is used to fit the reconstructed septum to Cottle tip position and orientation during flap elevation. Regularization of the shape model and transformation is incorporated. The proposed methods were validated on 10 septoplasty surgeries performed on cadavers by operators of varying experience level. Preoperative CT images of the cadaver septums were segmented as ground truth. Results: We estimated reconstruction error as the difference between the projections of the Cottle tip onto the surface of the reconstructed septum and the ground-truth septum segmented from the CT image. We found translational differences of 2.74 ( 2.06 - 2.81 ) mm and a rotational differences of 8.95 ( 7.11 - 10.55 ) deg between the reconstructed septum and the ground-truth septum [median (interquartile range)], given the optimal regularization parameters. Conclusions: Accurate reconstruction of the nasal septum can be achieved from tool tracking data during septoplasty surgery on cadavers. This enables understanding of the septal anatomy without need for traditional medical imaging. This result may be used to facilitate surgical planning, intraoperative care, or skills assessment.

Machine learning methods for automated technical skills assessment with instructional feedback in ultrasound-guided interventions.

OBJECTIVE: Currently, there is a worldwide shift toward competency-based medical education. This necessitates the use of automated skills assessment methods during self-guided interventions training. Making assessment methods that are transparent and configurable will allow assessment to be interpreted into instructional feedback. The purpose of this work is to develop and validate skills assessment methods in ultrasound-guided interventions that are transparent and configurable. METHODS: We implemented a method based upon decision trees and a method based upon fuzzy inference systems for technical skills assessment. Subsequently, we validated these methods for their ability to predict scores of operators on a 25-point global rating scale in ultrasound-guided needle insertions and their ability to provide useful feedback for training. RESULTS: Decision tree and fuzzy rule-based assessment performed comparably to state-of-the-art assessment methods. They produced median errors (on a 25-point scale) of 1.7 and 1.8 for in-plane insertions and 1.5 and 3.0 for out-of-plane insertions, respectively. In addition, these methods provided feedback that was useful for trainee learning. Decision tree assessment produced feedback with median usefulness 7 out of 7; fuzzy rule-based assessment produced feedback with median usefulness 6 out of 7. CONCLUSION: Transparent and configurable assessment methods are comparable to the state of the art and, in addition, can provide useful feedback. This demonstrates their value in self-guided interventions training curricula.

Surgery Tutor for Computational Assessment of Technical Proficiency in Soft-Tissue Tumor Resection in a Simulated Setting.

BACKGROUND: In competency-based medical education, progression between milestones requires reliable and valid methods of assessment. Surgery Tutor is an open-source motion tracking platform developed to objectively assess technical proficiency during open soft-tissue tumor resections in a simulated setting. The objective of our study was to provide evidence in support of construct validity of the scores obtained by Surgery Tutor. We hypothesized that Surgery Tutor would discriminate between novice, intermediate, and experienced operators. METHODS: Thirty participants were assigned to novice, intermediate, or experienced groups, based on the number of prior soft-tissue resections performed. Each participant resected 2 palpable and 2 nonpalpable lesions from a soft-tissue phantom. Surgery Tutor was used to track hand and instrument motions, number of tumor breaches, and time to perform each resection. Mass of excised specimens and margin status were also recorded. RESULTS: Surgery Tutor scores demonstrated "moderate" to "good" internal structure (test-retest reliability) for novice, intermediate, and experienced groups (interclass correlation coefficient = 0.596, 0.569, 0.737; p < 0.001). Evidence in support of construct validity (consequences) was demonstrated by comparing scores of novice, intermediate, and experienced participantsfor number of hand and instrument motions (690 ± 190, 597 ± 169, 469 ± 110; p < 0.001), number of tumor breaches (29 ± 34, 16 ± 11, 9 ± 6; p < 0.001), time per resection (677 ± 331 seconds, 561 ± 210 seconds, 449 ± 148 seconds; p < 0.001), mass of completely excised specimens (22 ± 7g, 21 ± 11g, 17 ± 6 g; p = 0.035), and rate of positive margin (68%, 50%, 28%; p < 0.001). There was "strong" and "moderate" relationships between motion scores and Objective Structured Assessment of Technical Skill scores, and time per resection and Objective Structured Assessment of Technical Skill scores respectively (r = -0.60, p < 0.001; r = -0.54, p < 0.001). CONCLUSION: Surgery Tutor scores demonstrate evidenceof construct validity with regards to good internal structure, consequences, and relationship to other variables in the assessment of technical proficiency duringopen soft-tissue tumor resections in a simulated setting. Utilization of Surgery Tutor can provide formative feedback and objective assessment of surgical proficiency in a simulated setting.

Self-guided training for deep brain stimulation planning using objective assessment.

OBJECTIVE: Deep brain stimulation (DBS) is an increasingly common treatment for neurodegenerative diseases. Neurosurgeons must have thorough procedural, anatomical, and functional knowledge to plan electrode trajectories and thus ensure treatment efficacy and patient safety. Developing this knowledge requires extensive training. We propose a training approach with objective assessment of neurosurgeon proficiency in DBS planning. METHODS: To assess proficiency, we propose analyzing both the viability of the planned trajectory and the manner in which the operator arrived at the trajectory. To improve understanding, we suggest a self-guided training course for DBS planning using real-time feedback. To validate the proposed measures of proficiency and training course, two experts and six novices followed the training course, and we monitored their proficiency measures throughout. RESULTS: At baseline, experts planned higher quality trajectories and did so more efficiently. As novices progressed through the training course, their proficiency measures increased significantly, trending toward expert measures. CONCLUSION: We developed and validated measures which reliably discriminate proficiency levels. These measures are integrated into a training course, which quantitatively improves trainee performance. The proposed training course can be used to improve trainees' proficiency, and the quantitative measures allow trainees' progress to be monitored.

Objective assessment of colonoscope manipulation skills in colonoscopy training.

OBJECTIVE: Manipulation of the colonoscope is a technical challenge for novice clinicians which is best learned in a simulated environment. It involves the coordination of scope tip steering with scope insertion, using a rotated image as reference. The purpose of this work is to develop and validate a system which objectively assesses colonoscopy technical skills proficiency in an arbitrary training environment, allowing novices to assess their technical proficiency prior to real patient encounters. METHODS: We implemented a motion tracking setup to objectively analyze and assess the way operators perform colonoscopies, including an analysis of wrist and elbow joint motions. Subsequently, we conducted a validation study to verify whether our motion analysis could discriminate novice colonoscopists from experts. Participants navigated a wooden bench-top model using a standard colonoscope while their motions were tracked. RESULTS: The developed motion tracking setup allowed colonoscopists of varying levels of proficiency to have their colonoscope manipulation assessed, and was able to be operated by a trained non-technical operator. Novice operators had significantly greater median times (101.5 vs. 31.5 s) and number of hand movements (62.0 vs. 21.5) than experts. Experts, however, spent a significantly greater proportion of time in extreme ranges of wrist and elbow joint motion than novices. CONCLUSION: We have developed and implemented a hand and joint motion analysis system that is able to discriminate novices from experts based on objective measures of motion. These metrics could, thus, serve as proxies for technical proficiency during training.