Robust hand tracking for surgical telestration.

PURPOSE: As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs. METHODS: We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching. RESULTS: According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied. CONCLUSION: Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.

Development of the open-source dose calculation and optimization toolkit matRad.

PURPOSE: We report on the development of the open-source cross-platform radiation treatment planning toolkit matRad and its comparison against validated treatment planning systems. The toolkit enables three-dimensional intensity-modulated radiation therapy treatment planning for photons, scanned protons and scanned carbon ions. METHODS: matRad is entirely written in Matlab and is freely available online. It re-implements well-established algorithms employing a modular and sequential software design to model the entire treatment planning workflow. It comprises core functionalities to import DICOM data, to calculate and optimize dose as well as a graphical user interface for visualization. matRad dose calculation algorithms (for carbon ions this also includes the computation of the relative biological effect) are compared against dose calculation results originating from clinically approved treatment planning systems. RESULTS: We observe three-dimensional γ-analysis pass rates ≥ 99.67% for all three radiation modalities utilizing a distance to agreement of 2 mm and a dose difference criterion of 2%. The computational efficiency of matRad is evaluated in a treatment planning study considering three different treatment scenarios for every radiation modality. For photons, we measure total run times of 145 s-1260 s for dose calculation and fluence optimization combined considering 4-72 beam orientations and 2608-13597 beamlets. For charged particles, we measure total run times of 63 s-993 s for dose calculation and fluence optimization combined considering 9963-45574 pencil beams. Using a CT and dose grid resolution of 0.3 cm3 requires a memory consumption of 1.59 GB-9.07 GB and 0.29 GB-17.94 GB for photons and charged particles, respectively. CONCLUSION: The dosimetric accuracy, computational performance and open-source character of matRad encourages a future application of matRad for both educational and research purposes.