Universal inverse modeling of point spread functions for SMLM localization and microscope characterization.

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Deep learning enables fast and dense single-molecule localization with high accuracy.

Single-molecule localization microscopy (SMLM) has had remarkable success in imaging cellular structures with nanometer resolution, but standard analysis algorithms require sparse emitters, which limits imaging speed and labeling density. Here, we overcome this major limitation using deep learning. We developed DECODE (deep context dependent), a computational tool that can localize single emitters at high density in three dimensions with highest accuracy for a large range of imaging modalities and conditions. In a public software benchmark competition, it outperformed all other fitters on 12 out of 12 datasets when comparing both detection accuracy and localization error, often by a substantial margin. DECODE allowed us to acquire fast dynamic live-cell SMLM data with reduced light exposure and to image microtubules at ultra-high labeling density. Packaged for simple installation and use, DECODE will enable many laboratories to reduce imaging times and increase localization density in SMLM.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization.

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

Robust hand tracking for surgical telestration.

PURPOSE: As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs. METHODS: We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching. RESULTS: According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied. CONCLUSION: Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.

Development of the open-source dose calculation and optimization toolkit matRad.

PURPOSE: We report on the development of the open-source cross-platform radiation treatment planning toolkit matRad and its comparison against validated treatment planning systems. The toolkit enables three-dimensional intensity-modulated radiation therapy treatment planning for photons, scanned protons and scanned carbon ions. METHODS: matRad is entirely written in Matlab and is freely available online. It re-implements well-established algorithms employing a modular and sequential software design to model the entire treatment planning workflow. It comprises core functionalities to import DICOM data, to calculate and optimize dose as well as a graphical user interface for visualization. matRad dose calculation algorithms (for carbon ions this also includes the computation of the relative biological effect) are compared against dose calculation results originating from clinically approved treatment planning systems. RESULTS: We observe three-dimensional γ-analysis pass rates ≥ 99.67% for all three radiation modalities utilizing a distance to agreement of 2 mm and a dose difference criterion of 2%. The computational efficiency of matRad is evaluated in a treatment planning study considering three different treatment scenarios for every radiation modality. For photons, we measure total run times of 145 s-1260 s for dose calculation and fluence optimization combined considering 4-72 beam orientations and 2608-13597 beamlets. For charged particles, we measure total run times of 63 s-993 s for dose calculation and fluence optimization combined considering 9963-45574 pencil beams. Using a CT and dose grid resolution of 0.3 cm3 requires a memory consumption of 1.59 GB-9.07 GB and 0.29 GB-17.94 GB for photons and charged particles, respectively. CONCLUSION: The dosimetric accuracy, computational performance and open-source character of matRad encourages a future application of matRad for both educational and research purposes.