Deep learning-based recognition of key anatomical structures during robot-assisted minimally invasive esophagectomy.

OBJECTIVE: To develop a deep learning algorithm for anatomy recognition in thoracoscopic video frames from robot-assisted minimally invasive esophagectomy (RAMIE) procedures using deep learning. BACKGROUND: RAMIE is a complex operation with substantial perioperative morbidity and a considerable learning curve. Automatic anatomy recognition may improve surgical orientation and recognition of anatomical structures and might contribute to reducing morbidity or learning curves. Studies regarding anatomy recognition in complex surgical procedures are currently lacking. METHODS: Eighty-three videos of consecutive RAMIE procedures between 2018 and 2022 were retrospectively collected at University Medical Center Utrecht. A surgical PhD candidate and an expert surgeon annotated the azygos vein and vena cava, aorta, and right lung on 1050 thoracoscopic frames. 850 frames were used for training of a convolutional neural network (CNN) to segment the anatomical structures. The remaining 200 frames of the dataset were used for testing the CNN. The Dice and 95% Hausdorff distance (95HD) were calculated to assess algorithm accuracy. RESULTS: The median Dice of the algorithm was 0.79 (IQR = 0.20) for segmentation of the azygos vein and/or vena cava. A median Dice coefficient of 0.74 (IQR = 0.86) and 0.89 (IQR = 0.30) were obtained for segmentation of the aorta and lung, respectively. Inference time was 0.026 s (39 Hz). The prediction of the deep learning algorithm was compared with the expert surgeon annotations, showing an accuracy measured in median Dice of 0.70 (IQR = 0.19), 0.88 (IQR = 0.07), and 0.90 (0.10) for the vena cava and/or azygos vein, aorta, and lung, respectively. CONCLUSION: This study shows that deep learning-based semantic segmentation has potential for anatomy recognition in RAMIE video frames. The inference time of the algorithm facilitated real-time anatomy recognition. Clinical applicability should be assessed in prospective clinical studies.

Reconstruction of the Corticospinal Tract in Patients with Motor-Eloquent High-Grade Gliomas Using Multilevel Fiber Tractography Combined with Functional Motor Cortex Mapping.

BACKGROUND AND PURPOSE: Tractography of the corticospinal tract is paramount to presurgical planning and guidance of intraoperative resection in patients with motor-eloquent gliomas. It is well-known that DTI-based tractography as the most frequently used technique has relevant shortcomings, particularly for resolving complex fiber architecture. The purpose of this study was to evaluate multilevel fiber tractography combined with functional motor cortex mapping in comparison with conventional deterministic tractography algorithms. MATERIALS AND METHODS: Thirty-one patients (mean age, 61.5 [SD, 12.2] years) with motor-eloquent high-grade gliomas underwent MR imaging with DWI (TR/TE = 5000/78 ms, voxel size = 2 × 2 × 2 mm3, 1 volume at b = 0 s/mm2, 32 volumes at b = 1000 s/mm2). DTI, constrained spherical deconvolution, and multilevel fiber tractography-based reconstruction of the corticospinal tract within the tumor-affected hemispheres were performed. The functional motor cortex was enclosed by navigated transcranial magnetic stimulation motor mapping before tumor resection and used for seeding. A range of angular deviation and fractional anisotropy thresholds (for DTI) was tested. RESULTS: For all investigated thresholds, multilevel fiber tractography achieved the highest mean coverage of the motor maps (eg, angular threshold = 60°; multilevel/constrained spherical deconvolution/DTI, 25% anisotropy threshold = 71.8%, 22.6%, and 11.7%) and the most extensive corticospinal tract reconstructions (eg, angular threshold = 60°; multilevel/constrained spherical deconvolution/DTI, 25% anisotropy threshold = 26,485 mm3, 6308 mm3, and 4270 mm3). CONCLUSIONS: Multilevel fiber tractography may improve the coverage of the motor cortex by corticospinal tract fibers compared with conventional deterministic algorithms. Thus, it could provide a more detailed and complete visualization of corticospinal tract architecture, particularly by visualizing fiber trajectories with acute angles that might be of high relevance in patients with gliomas and distorted anatomy.

Computer-aided anatomy recognition in intrathoracic and -abdominal surgery: a systematic review.

BACKGROUND: Minimally invasive surgery is complex and associated with substantial learning curves. Computer-aided anatomy recognition, such as artificial intelligence-based algorithms, may improve anatomical orientation, prevent tissue injury, and improve learning curves. The study objective was to provide a comprehensive overview of current literature on the accuracy of anatomy recognition algorithms in intrathoracic and -abdominal surgery. METHODS: This systematic review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline. Pubmed, Embase, and IEEE Xplore were searched for original studies up until January 2022 on computer-aided anatomy recognition, without requiring intraoperative imaging or calibration equipment. Extracted features included surgical procedure, study population and design, algorithm type, pre-training methods, pre- and post-processing methods, data augmentation, anatomy annotation, training data, testing data, model validation strategy, goal of the algorithm, target anatomical structure, accuracy, and inference time. RESULTS: After full-text screening, 23 out of 7124 articles were included. Included studies showed a wide diversity, with six possible recognition tasks in 15 different surgical procedures, and 14 different accuracy measures used. Risk of bias in the included studies was high, especially regarding patient selection and annotation of the reference standard. Dice and intersection over union (IoU) scores of the algorithms ranged from 0.50 to 0.98 and from 74 to 98%, respectively, for various anatomy recognition tasks. High-accuracy algorithms were typically trained using larger datasets annotated by expert surgeons and focused on less-complex anatomy. Some of the high-accuracy algorithms were developed using pre-training and data augmentation. CONCLUSIONS: The accuracy of included anatomy recognition algorithms varied substantially, ranging from moderate to good. Solid comparison between algorithms was complicated by the wide variety of applied methodology, target anatomical structures, and reported accuracy measures. Computer-aided intraoperative anatomy recognition is an upcoming research discipline, but still at its infancy. Larger datasets and methodological guidelines are required to improve accuracy and clinical applicability in future research. TRIAL REGISTRATION: PROSPERO registration number: CRD42021264226.

Fast contour propagation for MR-guided prostate radiotherapy using convolutional neural networks.

PURPOSE: To quickly and automatically propagate organ contours from pretreatment to fraction images in magnetic resonance (MR)-guided prostate external-beam radiotherapy. METHODS: Five prostate cancer patients underwent 20 fractions of image-guided external-beam radiotherapy on a 1.5 T MR-Linac system. For each patient, a pretreatment T2-weighted three-dimensional (3D) MR imaging (MRI) scan was used to delineate the clinical target volume (CTV) contours. The same scan was repeated during each fraction, with the CTV contour being manually adapted if necessary. A convolutional neural network (CNN) was trained for combined image registration and contour propagation. The network estimated the propagated contour and a deformation field between the two input images. The training set consisted of a synthetically generated ground truth of randomly deformed images and prostate segmentations. We performed a leave-one-out cross-validation on the five patients and propagated the prostate segmentations from the pretreatment to the fraction scans. Three variants of the CNN, aimed at investigating supervision based on optimizing segmentation overlap, optimizing the registration, and a combination of the two were compared to results of the open-source deformable registration software package Elastix. RESULTS: The neural networks trained on segmentation overlap or the combined objective achieved significantly better Hausdorff distances between predicted and ground truth contours than Elastix, at the much faster registration speed of 0.5 s. The CNN variant trained to optimize both the prostate overlap and deformation field, and the variant trained to only maximize the prostate overlap, produced the best propagation results. CONCLUSIONS: A CNN trained on maximizing prostate overlap and minimizing registration errors provides a fast and accurate method for deformable contour propagation for prostate MR-guided radiotherapy.

Evaluation of motion correction for clinical dynamic contrast enhanced MRI of the liver.

Motion correction of 4D dynamic contrast enhanced MRI (DCE-MRI) series is required for diagnostic evaluation of liver lesions. The registration, however, is a challenging task, owing to rapid changes in image appearance. In this study, two different registration approaches are compared; a conventional pairwise method applying mutual information as metric and a groupwise method applying a principal component analysis based metric, introduced by Huizinga et al (2016). The pairwise method transforms the individual 3D images one by one to a reference image, whereas the groupwise registration method computes the metric on all the images simultaneously, exploiting the temporal information, and transforms all 3D images to a common space. The performance of the two registration methods was evaluated using 70 clinical 4D DCE-MRI series with the focus on the liver. The evaluation was based on the smoothness of the time intensity curves in lesions, lesion volume change after deformation and the smoothness of spatial deformation. Furthermore, the visual quality of subtraction images (pre-contrast image subtracted from the post contrast images) before and after registration was rated by two observers. Both registration methods improved the alignment of the DCE-MRI images in comparison to the non-corrected series. Furthermore, the groupwise method achieved better temporal alignment with smoother spatial deformations than the pairwise method. The quality of the subtraction images was graded satisfactory in 32% of the cases without registration and in 77% and 80% of the cases after pairwise and groupwise registration, respectively. In conclusion, the groupwise registration method outperforms the pairwise registration method and achieves clinically satisfying results. Registration leads to improved subtraction images.

Multimodality infarct identification for optimal image-guided intramyocardial cell injections.

BACKGROUND: Intramyocardial cell injections in the context of cardiac regenerative therapy can currently be performed using electromechanical mapping (EMM) provided by the NOGA®XP catheter injection system. The gold standard technique to determine infarct size and location, however, is late gadolinium enhanced magnetic resonance imaging (LGE-MRI). In this article we describe a practical and accurate technique to co-register LGE-MRI and NOGA®XP datasets during the injection procedures to ultimately perform image-guided injections to the border zone of the infarct determined by LGE-MRI. MATERIALS AND METHODS: LGE-MRI and EMM were obtained in three pigs with chronic myocardial infarction. MRI and EMM datasets were registered using the in-house developed 3D CartBox image registration toolbox consisting of three steps: 1) landmark registration, 2) surface registration, and 3) manual optimization. The apex and the coronary ostia were used as landmarks. RESULTS: Image registration was successful in all datasets, and resulted in a mean registration error of 3.22 ± 1.86 mm between the MRI surface mesh and EMM points. Visual assessment revealed that the locations and the transmural extent of the infarctions measured by LGE-MRI only partly overlap with the infarct areas identified by the EMM parameters. CONCLUSIONS: The 3D CartBox image registration toolbox enables registration of EMM on pre-procedurally acquired MRI during the catheter injection procedure. This allows the operator to perform real-time image-guided cell injections into the border zone of the infarct as assessed by LGE-MRI. The 3D CartBox thereby enables, for the first time, standardisation of the injection location for cardiac regenerative therapy.

Registration of structurally dissimilar images in MRI-based brachytherapy.

A serious challenge in image registration is the accurate alignment of two images in which a certain structure is present in only one of the two. Such topological changes are problematic for conventional non-rigid registration algorithms. We propose to incorporate in a conventional free-form registration framework a geometrical penalty term that minimizes the volume of the missing structure in one image. We demonstrate our method on cervical MR images for brachytherapy. The intrapatient registration problem involves one image in which a therapy applicator is present and one in which it is not. By including the penalty term, a substantial improvement in the surface distance to the gold standard anatomical position and the residual volume of the applicator void are obtained. Registration of neighboring structures, i.e. the rectum and the bladder is generally improved as well, albeit to a lesser degree.

Expert-driven label fusion in multi-atlas-based segmentation of the prostate using weighted atlases.

PURPOSE: Automated segmentation is required for radiotherapy treatment planning, and multi-atlas methods are frequently used for this purpose. The combination of multiple intermediate results from multi-atlas segmentation into a single segmentation map can be achieved by label fusion. A method that includes expert knowledge in the label fusion phase of multi-atlas-based segmentation was developed. The method was tested by application to prostate segmentation, and the accuracy was compared to standard techniques. METHODS: The selective and iterative method for performance level estimation (SIMPLE) algorithm for label fusion was modified with a weight map given by an expert that indicates the importance of each region in the evaluation of segmentation results. Voxel-based weights specified by an expert when performing the label fusion step in atlas-based segmentation were introduced into the modified SIMPLE algorithm. These weights incorporate expert knowledge on accuracy requirements in different regions of a segmentation. Using this knowledge, segmentation accuracy in regions known to be important can be improved by sacrificing segmentation accuracy in less important regions. Contextual information such as the presence of vulnerable tissue is then used in the segmentation process. This method using weight maps to fine-tune the result of multi-atlas-based segmentation was tested using a set of 146 atlas images consisting of an MR image of the lower abdomen and a prostate segmentation. Each image served as a target in a set of leave-one-out experiments. These experiments were repeated for a weight map derived from the clinical practice in our hospital. RESULTS: The segmentation accuracy increased 6 % in regions that border vulnerable tissue using expert-specified voxel-based weight maps. This was achieved at the cost of a 4 % decrease in accuracy in less clinically relevant regions. CONCLUSION: The inclusion of expert knowledge in a multi-atlas-based segmentation procedure was shown to be feasible for prostate segmentation. This method allows an expert to ensure that automatic segmentation is most accurate in critical regions. This improved local accuracy can increase the practical value of automatic segmentation.

SU-E-J-95: Towards Optimum Boundary Conditions for Biomechanical Model Based Deformable Registration Using Intensity Based Image Matching for Prostate Correlative Pathology.

PURPOSE: Deformable registration of histology to MRI is an essential tool to validate in vivo prostate cancer imaging. However, direct registration of histology to in vivo MR is prone to error due to geometric differences between the tissue sections and the in vivo imaging planes. To increase the accuracy of registration, an ex vivo high resolution MRI is acquired to compensate for the direct registration difficulties. A novel intensity-based deformable registration algorithm based on local variation in image intensities is proposed to register the histology to ex vivo MRI of prostatectomy specimens. METHODS: Four sets of ex vivo MR and whole mount pathology images from four patients were used to investigate and validate the technique. In addition, 9 synthetically deformed ex vivo MR images were used. The standard deviation in local windows within the images was calculated to generate intermediate images based on both MR and histology. The intermediate images were registered using the Drop package (Munich, Germany). To further increase the accuracy, a final refinement of the registration was performed using Drop with a finer B-spline rid. The registration parameters were tuned to achieve a visually acceptable registration. Magnitude of Differences (MOD) and Angular Error (AE) were used to validate the synthetic data, and the Target Registration Error (TRE) of manually indicated landmarks was used for the clinical data. RESULTS: MOD of 0.6mm and AE of 8.3 degrees showed the efficacy of using intermediate images, compared to 0.8mm and 10.0 degrees achieved with Drop without the intermediate images. The average mean±std TRE among the four patients was 1.0±0.6 mm using the proposed method compared to 1.6±1.1 mm using Elastix (Utrecht, The Netherlands). CONCLUSIONS: An intensity-based deformable registration algorithm which uses intermediate images was evaluated on prostatectomy specimens and synthetically deformed clinical data, indicating improvement in overall accuracy and robustness. OICR, Terry Fox Ultrasound for Cancer Therapy. Dr. Brock is a Cancer Care Ontario Research Chair in Cancer Imaging and has financial interest in deformable registration technology through the licensing of Morfeus to RaySearch Laboratories.

Registration of 2D x-ray images to 3D MRI by generating pseudo-CT data.

Spatial and soft tissue information provided by magnetic resonance imaging can be very valuable during image-guided procedures, where usually only real-time two-dimensional (2D) x-ray images are available. Registration of 2D x-ray images to three-dimensional (3D) magnetic resonance imaging (MRI) data, acquired prior to the procedure, can provide optimal information to guide the procedure. However, registering x-ray images to MRI data is not a trivial task because of their fundamental difference in tissue contrast. This paper presents a technique that generates pseudo-computed tomography (CT) data from multi-spectral MRI acquisitions which is sufficiently similar to real CT data to enable registration of x-ray to MRI with comparable accuracy as registration of x-ray to CT. The method is based on a k-nearest-neighbors (kNN)-regression strategy which labels voxels of MRI data with CT Hounsfield Units. The regression method uses multi-spectral MRI intensities and intensity gradients as features to discriminate between various tissue types. The efficacy of using pseudo-CT data for registration of x-ray to MRI was tested on ex vivo animal data. 2D-3D registration experiments using CT and pseudo-CT data of multiple subjects were performed with a commonly used 2D-3D registration algorithm. On average, the median target registration error for registration of two x-ray images to MRI data was approximately 1 mm larger than for x-ray to CT registration. The authors have shown that pseudo-CT data generated from multi-spectral MRI facilitate registration of MRI to x-ray images. From the experiments it could be concluded that the accuracy achieved was comparable to that of registering x-ray images to CT data.

Semi-automatic construction of reference standards for evaluation of image registration.

Quantitative evaluation of image registration algorithms is a difficult and under-addressed issue due to the lack of a reference standard in most registration problems. In this work a method is presented whereby detailed reference standard data may be constructed in an efficient semi-automatic fashion. A well-distributed set of n landmarks is detected fully automatically in one scan of a pair to be registered. Using a custom-designed interface, observers define corresponding anatomic locations in the second scan for a specified subset of s of these landmarks. The remaining n-s landmarks are matched fully automatically by a thin-plate-spline based system using the s manual landmark correspondences to model the relationship between the scans. The method is applied to 47 pairs of temporal thoracic CT scans, three pairs of brain MR scans and five thoracic CT datasets with synthetic deformations. Interobserver differences are used to demonstrate the accuracy of the matched points. The utility of the reference standard data as a tool in evaluating registration is shown by the comparison of six sets of registration results on the 47 pairs of thoracic CT data.

Robust initialization of 2D-3D image registration using the projection-slice theorem and phase correlation.

PURPOSE: The image registration literature comprises many methods for 2D-3D registration for which accuracy has been established in a variety of applications. However, clinical application is limited by a small capture range. Initial offsets outside the capture range of a registration method will not converge to a successful registration. Previously reported capture ranges, defined as the 95% success range, are in the order of 4-11 mm mean target registration error. In this article, a relatively computationally inexpensive and robust estimation method is proposed with the objective to enlarge the capture range. METHODS: The method uses the projection-slice theorem in combination with phase correlation in order to estimate the transform parameters, which provides an initialization of the subsequent registration procedure. RESULTS: The feasibility of the method was evaluated by experiments using digitally reconstructed radiographs generated from in vivo 3D-RX data. With these experiments it was shown that the projection-slice theorem provides successful estimates of the rotational transform parameters for perspective projections and in case of translational offsets. The method was further tested on ex vivo ovine x-ray data. In 95% of the cases, the method yielded successful estimates for initial mean target registration errors up to 19.5 mm. Finally, the method was evaluated as an initialization method for an intensity-based 2D-3D registration method. The uninitialized and initialized registration experiments had success rates of 28.8% and 68.6%, respectively. CONCLUSIONS: The authors have shown that the initialization method based on the projection-slice theorem and phase correlation yields adequate initializations for existing registration methods, thereby substantially enlarging the capture range of these methods.

Semi-automatic reference standard construction for quantitative evaluation of lung CT registration.

An algorithm is presented for the efficient semi-automatic construction of a detailed reference standard for registration in thoracic CT. A well-distributed set of 100 landmarks is detected fully automatically in one scan of a pair to be registered. Using a custom-designed interface, observers locate corresponding anatomic locations in the second scan. The manual annotations are used to learn the relationship between the scans and after approximately twenty manual marks the remaining points are matched automatically. Inter-observer differences demonstrate the accuracy of the matching and the applicability of the reference standard is demonstrated on two different sets of registration results over 19 CT scan pairs.

Automated measurement of brain and white matter lesion volume in type 2 diabetes mellitus.

AIMS/HYPOTHESIS: Type 2 diabetes mellitus has been associated with brain atrophy and cognitive decline, but the association with ischaemic white matter lesions is unclear. Previous neuroimaging studies have mainly used semiquantitative rating scales to measure atrophy and white matter lesions (WMLs). In this study we used an automated segmentation technique to investigate the association of type 2 diabetes, several diabetes-related risk factors and cognition with cerebral tissue and WML volumes. SUBJECTS AND METHODS: Magnetic resonance images of 99 patients with type 2 diabetes and 46 control participants from a population-based sample were segmented using a k-nearest neighbour classifier trained on ten manually segmented data sets. White matter, grey matter, lateral ventricles, cerebrospinal fluid not including lateral ventricles, and WML volumes were assessed. Analyses were adjusted for age, sex, level of education and intracranial volume. RESULTS: Type 2 diabetes was associated with a smaller volume of grey matter (-21.8 ml; 95% CI -34.2, -9.4) and with larger lateral ventricle volume (7.1 ml; 95% CI 2.3, 12.0) and with larger white matter lesion volume (56.5%; 95% CI 4.0, 135.8), whereas white matter volume was not affected. In separate analyses for men and women, the effects of diabetes were only significant in women. CONCLUSIONS/INTERPRETATION: The combination of atrophy with larger WML volume indicates that type 2 diabetes is associated with mixed pathology in the brain. The observed sex differences were unexpected and need to be addressed in further studies.

Normalized mutual information based registration using k-means clustering and shading correction.

In this paper the influence of intensity clustering and shading correction on mutual information based image registration is studied. Instead of the generally used equidistant re-binning, we use k-means clustering in order to achieve a more natural binning of the intensity distribution. Secondly, image inhomogeneities occurring notably in MR images can have adverse effects on the registration. We use a shading correction method in order to reduce these effects. The method is validated on clinical MR, CT and PET images, as well as synthetic MR images. It is shown that by employing clustering with inhomogeneity correction the number of misregistrations is reduced without loss of accuracy thus increasing robustness as compared to the standard non-inhomogeneity corrected and equidistant binning based registration.

Registration, segmentation, and visualization of multimodal brain images.

This paper gives an overview of the studies performed at our institute over the last decade on the processing and visualization of brain images, in the context of international developments in the field. The focus is on multimodal image registration and multimodal visualization, while segmentation is touched upon as a preprocessing step for visualization. The state-of-the-art in these areas is discussed and suggestions for future research are given.

Image registration by maximization of combined mutual information and gradient information.

Mutual information has developed into an accurate measure for rigid and affine monomodality and multimodality image registration. The robustness of the measure is questionable, however. A possible reason for this is the absence of spatial information in the measure. The present paper proposes to include spatial information by combining mutual information with a term based on the image gradient of the images to be registered. The gradient term not only seeks to align locations of high gradient magnitude, but also aims for a similar orientation of the gradients at these locations. Results of combining both standard mutual information as well as a normalized measure are presented for rigid registration of three-dimensional clinical images [magnetic resonance (MR), computed tomography (CT), and positron emission tomography (PET)]. The results indicate that the combined measures yield a better registration function does mutual information or normalized mutual information per se. The registration functions are less sensitive to low sampling resolution, do not contain incorrect global maxima that are sometimes found in the mutual information function, and interpolation-induced local minima can be reduced. These characteristics yield the promise of more robust registration measures. The accuracy of the combined measures is similar to that of mutual information-based methods.