Spatiotemporal Deep Learning-Based Cine Loop Quality Filter for Handheld Point-of-Care Echocardiography.

The reliability of automated image interpretation of point-of-care (POC) echocardiography scans depends on the quality of the acquired ultrasound data. This work reports on the development and validation of spatiotemporal deep learning models to assess the suitability of input ultrasound cine loops collected using a handheld echocardiography device for processing by an automated quantification algorithm (e.g. ejection fraction estimation). POC echocardiograms (n=885 DICOM cine loops from 175 patients) from two sites were collected using a handheld ultrasound device and annotated for image quality at the frame-level. Attributes of high-quality frames for left ventricular (LV) quantification included a temporally-stable LV, reasonable coverage of LV borders, and good contrast between the borders and chamber. Attributes of low-quality frames included temporal instability of the LV and/or imaging artifacts (e.g., lack of contrast, haze, reverberation, acoustic shadowing). Three different neural network architectures were investigated - (a) frame-level convolutional neural network (CNN) which operates on individual echo frames (VectorCNN), (b) single-stream sequence-level CNN which operates on a sequence of echo frames (VectorCNN+LSTM) and (c) two-stream sequence-level CNNs which operate on a sequence of echo and optical flow frames (VectorCNN+LSTM+Average, VectorCNN+LSTM+MinMax, and VectorCNN+LSTM+ConvPool). Evaluation on a sequestered test dataset containing 76 DICOM cine loops with 16,914 frames showed that VectorCNN+LSTM can effectively utilize both spatial and temporal information to regress the quality of an input frame (accuracy: 0.925, sensitivity = 0.860, specificity = 0.952), compared to the frame-level VectorCNN that only utilizes spatial information in that frame (accuracy: 0.903, sensitivity = 0.791, specificity = 0.949). Furthermore, an independent sample t-test indicated that the cine loops classified to be of adequate quality by the VectorCNN+LSTM model had a statistically significant lower bias in the automatically estimated EF (mean bias = - 3.73 ± 7.46 %, versus a clinically obtained reference EF) compared to the loops classified as inadequate (mean bias = -15.92 ± 12.17 %) (p = 0.007). Thus, cine loop stratification using the proposed spatiotemporal CNN model improves the reliability of automated point-of-care echocardiography image interpretation.

Echocardiographic Detection of Regional Wall Motion Abnormalities Using Artificial Intelligence Compared to Human Readers.

BACKGROUND: Although regional wall motion abnormality (RWMA) detection is foundational to transthoracic echocardiography, current methods are prone to interobserver variability. We aimed to develop a deep learning (DL) model for RWMA assessment and compare it to expert and novice readers. METHODS: We used 15,746 transthoracic echocardiography studies-including 25,529 apical videos-which were split into training, validation, and test datasets. A convolutional neural network was trained and validated using apical 2-, 3-, and 4-chamber videos to predict the presence of RWMA in 7 regions defined by coronary perfusion territories, using the ground truth derived from clinical transthoracic echocardiography reports. Within the test cohort, DL model accuracy was compared to 6 expert and 3 novice readers using F1 score evaluation, with the ground truth of RWMA defined by expert readers. Significance between the DL model and novices was assessed using the permutation test. RESULTS: Within the test cohort, the DL model accurately identified any RWMA with an area under the curve of 0.96 (0.92-0.98). The mean F1 scores of the experts and the DL model were numerically similar for 6 of 7 regions: anterior (86 vs 84), anterolateral (80 vs 74), inferolateral (83 vs 87), inferoseptal (86 vs 86), apical (88 vs 87), inferior (79 vs 81), and any RWMA (90 vs 94), respectively, while in the anteroseptal region, the F1 score of the DL model was lower than the experts (75 vs 89). Using F1 scores, the DL model outperformed both novices 1 (P = .002) and 2 (P = .02) for the detection of any RWMA. CONCLUSIONS: Deep learning provides accurate detection of RWMA, which was comparable to experts and outperformed a majority of novices. Deep learning may improve the efficiency of RWMA assessment and serve as a teaching tool for novices.

Digital reference objects for evaluating algorithm performance in MR image formation.

PURPOSE: Digital Reference Objects (DROs) are mathematical phantoms that can serve as a basis for evaluating MR image quality (IQ) in an objective way. Their main purpose is to facilitate the establishment of fully automated and perfectly reproducible IQ metrics to objectively compare different algorithms in MR image formation in a standardized manner. They also allow to re-build parts of standard phantoms. METHODS: We sample DROs directly in k-space, using analytical formulas for the continuous Fourier transform of primitive shapes. We demonstrate this DRO approach by applying a state-of-the-art CNN-based denoising algorithm that is robust to varying noise levels to noisy images of the resolution section of the well-known ACR phantom for IQ assessment, reconstructed from both measured and simulated k-space data. RESULTS: Applying the CNN-based denoising algorithm to the measured and simulated version of the ACR phantom resolution section produced virtually identical results, as confirmed by visual and quantitative comparison. CONCLUSIONS: DROs can help guide technology selection during the development of new algorithms in MR image formation, e.g., via deep learning. This could be an important step towards reproducible MR image formation.

In vivo detection of head and neck tumors by hyperspectral imaging combined with deep learning methods.

Currently, there are no fast and accurate screening methods available for head and neck cancer, the eighth most common tumor entity. For this study, we used hyperspectral imaging, an imaging technique for quantitative and objective surface analysis, combined with deep learning methods for automated tissue classification. As part of a prospective clinical observational study, hyperspectral datasets of laryngeal, hypopharyngeal and oropharyngeal mucosa were recorded in 98 patients before surgery in vivo. We established an automated data interpretation pathway that can classify the tissue into healthy and tumorous using convolutional neural networks with 2D spatial or 3D spatio-spectral convolutions combined with a state-of-the-art Densenet architecture. Using 24 patients for testing, our 3D spatio-spectral Densenet classification method achieves an average accuracy of 81%, a sensitivity of 83% and a specificity of 79%.

Confocal Laser Microscopy for in vivo Intraoperative Application: Diagnostic Accuracy of Investigator and Machine Learning Strategies.

BACKGROUND: Confocal laser microscopy (CLM) is one of the optical techniques that are promising methods of intraoperative in vivo real-time tissue examination based on tissue fluorescence. However, surgeons might struggle interpreting CLM images intraoperatively due to different tissue characteristics of different tissue pathologies in clinical reality. Deep learning techniques enable fast and consistent image analysis and might support intraoperative image interpretation. The objective of this study was to analyze the diagnostic accuracy of newly trained observers in the evaluation of normal colon and peritoneal tissue and colon cancer and metastasis, respectively, and to compare it with that of convolutional neural networks (CNNs). METHODS: Two hundred representative CLM images of the normal and malignant colon and peritoneal tissue were evaluated by newly trained observers (surgeons and pathologists) and CNNs (VGG-16 and Densenet121), respectively, based on tissue dignity. The primary endpoint was the correct detection of the normal and cancer/metastasis tissue measured by sensitivity and specificity of both groups. Additionally, positive predictive values (PPVs) and negative predictive values (NPVs) were calculated for the newly trained observer group. The interobserver variability of dignity evaluation was calculated using kappa statistic. The F1-score and area under the curve (AUC) were used to evaluate the performance of image recognition of the CNNs' training scenarios. RESULTS: Sensitivity and specificity ranged between 0.55 and 1.0 (pathologists: 0.66-0.97; surgeons: 0.55-1.0) and between 0.65 and 0.96 (pathologists: 0.68-0.93; surgeons: 0.65-0.96), respectively. PPVs were 0.75 and 0.90 in the pathologists' group and 0.73-0.96 in the surgeons' group, respectively. NPVs were 0.73 and 0.96 for pathologists' and between 0.66 and 1.00 for surgeons' tissue analysis. The overall interobserver variability was 0.54. Depending on the training scenario, cancer/metastasis tissue was classified with an AUC of 0.77-0.88 by VGG-16 and 0.85-0.89 by Densenet121. Transfer learning improved performance over training from scratch. CONCLUSIONS: Newly trained investigators are able to learn CLM images features and interpretation rapidly, regardless of their clinical experience. Heterogeneity in tissue diagnosis and a moderate interobserver variability reflect the clinical reality more realistic. CNNs provide comparable diagnostic results as clinical observers and could improve surgeons' intraoperative tissue assessment.

Fully automated longitudinal segmentation of new or enlarged multiple sclerosis lesions using 3D convolutional neural networks.

The quantification of new or enlarged lesions from follow-up MRI scans is an important surrogate of clinical disease activity in patients with multiple sclerosis (MS). Not only is manual segmentation time consuming, but inter-rater variability is high. Currently, only a few fully automated methods are available. We address this gap in the field by employing a 3D convolutional neural network (CNN) with encoder-decoder architecture for fully automatic longitudinal lesion segmentation. Input data consist of two fluid attenuated inversion recovery (FLAIR) images (baseline and follow-up) per patient. Each image is entered into the encoder and the feature maps are concatenated and then fed into the decoder. The output is a 3D mask indicating new or enlarged lesions (compared to the baseline scan). The proposed method was trained on 1809 single point and 1444 longitudinal patient data sets and then validated on 185 independent longitudinal data sets from two different scanners. From the two validation data sets, manual segmentations were available from three experienced raters, respectively. The performance of the proposed method was compared to the open source Lesion Segmentation Toolbox (LST), which is a current state-of-art longitudinal lesion segmentation method. The mean lesion-wise inter-rater sensitivity was 62%, while the mean inter-rater number of false positive (FP) findings was 0.41 lesions per case. The two validated algorithms showed a mean sensitivity of 60% (CNN), 46% (LST) and a mean FP of 0.48 (CNN), 1.86 (LST) per case. Sensitivity and number of FP were not significantly different (p < 0.05) between the CNN and manual raters. New or enlarged lesions counted by the CNN algorithm appeared to be comparable with manual expert ratings. The proposed algorithm seems to outperform currently available approaches, particularly LST. The high inter-rater variability in case of manual segmentation indicates the complexity of identifying new or enlarged lesions. An automated CNN-based approach can quickly provide an independent and deterministic assessment of new or enlarged lesions from baseline to follow-up scans with acceptable reliability.

Multiple sclerosis lesion activity segmentation with attention-guided two-path CNNs.

Multiple sclerosis is an inflammatory autoimmune demyelinating disease that is characterized by lesions in the central nervous system. Typically, magnetic resonance imaging (MRI) is used for tracking disease progression. Automatic image processing methods can be used to segment lesions and derive quantitative lesion parameters. So far, methods have focused on lesion segmentation for individual MRI scans. However, for monitoring disease progression, lesion activity in terms of new and enlarging lesions between two time points is a crucial biomarker. For this problem, several classic methods have been proposed, e.g., using difference volumes. Despite their success for single-volume lesion segmentation, deep learning approaches are still rare for lesion activity segmentation. In this work, convolutional neural networks (CNNs) are studied for lesion activity segmentation from two time points. For this task, CNNs are designed and evaluated that combine the information from two points in different ways. In particular, two-path architectures with attention-guided interactions are proposed that enable effective information exchange between the two time point's processing paths. It is demonstrated that deep learning-based methods outperform classic approaches and it is shown that attention-guided interactions significantly improve performance. Furthermore, the attention modules produce plausible attention maps that have a masking effect that suppresses old, irrelevant lesions. A lesion-wise false positive rate of 26.4% is achieved at a true positive rate of 74.2%, which is not significantly different from the interrater performance.

Deep learning with 4D spatio-temporal data representations for OCT-based force estimation.

Estimating the forces acting between instruments and tissue is a challenging problem for robot-assisted minimally-invasive surgery. Recently, numerous vision-based methods have been proposed to replace electro-mechanical approaches. Moreover, optical coherence tomography (OCT) and deep learning have been used for estimating forces based on deformation observed in volumetric image data. The method demonstrated the advantage of deep learning with 3D volumetric data over 2D depth images for force estimation. In this work, we extend the problem of deep learning-based force estimation to 4D spatio-temporal data with streams of 3D OCT volumes. For this purpose, we design and evaluate several methods extending spatio-temporal deep learning to 4D which is largely unexplored so far. Furthermore, we provide an in-depth analysis of multi-dimensional image data representations for force estimation, comparing our 4D approach to previous, lower-dimensional methods. Also, we analyze the effect of temporal information and we study the prediction of short-term future force values, which could facilitate safety features. For our 4D force estimation architectures, we find that efficient decoupling of spatial and temporal processing is advantageous. We show that using 4D spatio-temporal data outperforms all previously used data representations with a mean absolute error of 10.7 mN. We find that temporal information is valuable for force estimation and we demonstrate the feasibility of force prediction.

Towards an Optical Biopsy during Visceral Surgical Interventions.

BACKGROUND: Cancer will replace cardiovascular diseases as the most frequent cause of death. Therefore, the goals of cancer treatment are prevention strategies and early detection by cancer screening and ideal stage therapy. From an oncological point of view, complete tumor resection is a significant prognostic factor. Optical coherence tomography (OCT) and confocal laser microscopy (CLM) are two techniques that have the potential to complement intraoperative frozen section analysis as in vivo and real-time optical biopsies. SUMMARY: In this review we present both procedures and review the progress of evaluation for intraoperative application in visceral surgery. For visceral surgery, there are promising studies evaluating OCT and CLM; however, application during routine visceral surgical interventions is still lacking. KEY MESSAGE: OCT and CLM are not competing but complementary approaches of tissue analysis to intraoperative frozen section analysis. Although intraoperative application of OCT and CLM is at an early stage, they are two promising techniques of intraoperative in vivo and real-time tissue examination. Additionally, deep learning strategies provide a significant supplement for automated tissue detection.

Spatio-temporal deep learning methods for motion estimation using 4D OCT image data.

PURPOSE: Localizing structures and estimating the motion of a specific target region are common problems for navigation during surgical interventions. Optical coherence tomography (OCT) is an imaging modality with a high spatial and temporal resolution that has been used for intraoperative imaging and also for motion estimation, for example, in the context of ophthalmic surgery or cochleostomy. Recently, motion estimation between a template and a moving OCT image has been studied with deep learning methods to overcome the shortcomings of conventional, feature-based methods. METHODS: We investigate whether using a temporal stream of OCT image volumes can improve deep learning-based motion estimation performance. For this purpose, we design and evaluate several 3D and 4D deep learning methods and we propose a new deep learning approach. Also, we propose a temporal regularization strategy at the model output. RESULTS: Using a tissue dataset without additional markers, our deep learning methods using 4D data outperform previous approaches. The best performing 4D architecture achieves an correlation coefficient (aCC) of 98.58% compared to 85.0% of a previous 3D deep learning method. Also, our temporal regularization strategy at the output further improves 4D model performance to an aCC of 99.06%. In particular, our 4D method works well for larger motion and is robust toward image rotations and motion distortions. CONCLUSIONS: We propose 4D spatio-temporal deep learning for OCT-based motion estimation. On a tissue dataset, we find that using 4D information for the model input improves performance while maintaining reasonable inference times. Our regularization strategy demonstrates that additional temporal information is also beneficial at the model output.

Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data.

In this paper, we describe our method for the ISIC 2019 Skin Lesion Classification Challenge. The challenge comes with two tasks. For task 1, skin lesions have to be classified based on dermoscopic images. For task 2, dermoscopic images and additional patient meta data are used. Our deep learning-based method achieved first place for both tasks. The are several problems we address with our method. First, there is an unknown class in the test set which we cover with a data-driven approach. Second, there is a severe class imbalance that we address with loss balancing. Third, there are images with different resolutions which motivates two different cropping strategies and multi-crop evaluation. Last, there is patient meta data available which we incorporate with a dense neural network branch. • We address skin lesion classification with an ensemble of deep learning models including EfficientNets, SENet, and ResNeXt WSL, selected by a search strategy. • We rely on multiple model input resolutions and employ two cropping strategies for training. We counter severe class imbalance with a loss balancing approach. • We predict an additional, unknown class with a data-driven approach and we make use of patient meta data with an additional input branch.

Skin Lesion Classification Using CNNs With Patch-Based Attention and Diagnosis-Guided Loss Weighting.

OBJECTIVE: This paper addresses two key problems of skin lesion classification. The first problem is the effective use of high-resolution images with pretrained standard architectures for image classification. The second problem is the high-class imbalance encountered in real-world multi-class datasets. METHODS: To use high-resolution images, we propose a novel patch-based attention architecture that provides global context between small, high-resolution patches. We modify three pretrained architectures and study the performance of patch-based attention. To counter class imbalance problems, we compare oversampling, balanced batch sampling, and class-specific loss weighting. Additionally, we propose a novel diagnosis-guided loss weighting method that takes the method used for ground-truth annotation into account. RESULTS: Our patch-based attention mechanism outperforms previous methods and improves the mean sensitivity by [Formula: see text]. Class balancing significantly improves the mean sensitivity and we show that our diagnosis-guided loss weighting method improves the mean sensitivity by [Formula: see text] over normal loss balancing. CONCLUSION: The novel patch-based attention mechanism can be integrated into pretrained architectures and provides global context between local patches while outperforming other patch-based methods. Hence, pretrained architectures can be readily used with high-resolution images without downsampling. The new diagnosis-guided loss weighting method outperforms other methods and allows for effective training when facing class imbalance. SIGNIFICANCE: The proposed methods improve automatic skin lesion classification. They can be extended to other clinical applications where high-resolution image data and class imbalance are relevant.

Spatio-temporal deep learning models for tip force estimation during needle insertion.

PURPOSE: Precise placement of needles is a challenge in a number of clinical applications such as brachytherapy or biopsy. Forces acting at the needle cause tissue deformation and needle deflection which in turn may lead to misplacement or injury. Hence, a number of approaches to estimate the forces at the needle have been proposed. Yet, integrating sensors into the needle tip is challenging and a careful calibration is required to obtain good force estimates. METHODS: We describe a fiber-optic needle tip force sensor design using a single OCT fiber for measurement. The fiber images the deformation of an epoxy layer placed below the needle tip which results in a stream of 1D depth profiles. We study different deep learning approaches to facilitate calibration between this spatio-temporal image data and the related forces. In particular, we propose a novel convGRU-CNN architecture for simultaneous spatial and temporal data processing. RESULTS: The needle can be adapted to different operating ranges by changing the stiffness of the epoxy layer. Likewise, calibration can be adapted by training the deep learning models. Our novel convGRU-CNN architecture results in the lowest mean absolute error of [Formula: see text] and a cross-correlation coefficient of 0.9997 and clearly outperforms the other methods. Ex vivo experiments in human prostate tissue demonstrate the needle's application. CONCLUSIONS: Our OCT-based fiber-optic sensor presents a viable alternative for needle tip force estimation. The results indicate that the rich spatio-temporal information included in the stream of images showing the deformation throughout the epoxy layer can be effectively used by deep learning models. Particularly, we demonstrate that the convGRU-CNN architecture performs favorably, making it a promising approach for other spatio-temporal learning problems.

Deep transfer learning methods for colon cancer classification in confocal laser microscopy images.

PURPOSE: The gold standard for colorectal cancer metastases detection in the peritoneum is histological evaluation of a removed tissue sample. For feedback during interventions, real-time in vivo imaging with confocal laser microscopy has been proposed for differentiation of benign and malignant tissue by manual expert evaluation. Automatic image classification could improve the surgical workflow further by providing immediate feedback. METHODS: We analyze the feasibility of classifying tissue from confocal laser microscopy in the colon and peritoneum. For this purpose, we adopt both classical and state-of-the-art convolutional neural networks to directly learn from the images. As the available dataset is small, we investigate several transfer learning strategies including partial freezing variants and full fine-tuning. We address the distinction of different tissue types, as well as benign and malignant tissue. RESULTS: We present a thorough analysis of transfer learning strategies for colorectal cancer with confocal laser microscopy. In the peritoneum, metastases are classified with an AUC of 97.1, and in the colon the primarius is classified with an AUC of 73.1. In general, transfer learning substantially improves performance over training from scratch. We find that the optimal transfer learning strategy differs for models and classification tasks. CONCLUSIONS: We demonstrate that convolutional neural networks and transfer learning can be used to identify cancer tissue with confocal laser microscopy. We show that there is no generally optimal transfer learning strategy and model as well as task-specific engineering is required. Given the high performance for the peritoneum, even with a small dataset, application for intraoperative decision support could be feasible.

Quantitative Analysis of 3D Artery Volume Reconstructions Using Biplane Angiography and Intravascular OCT Imaging.

Diameter and volume are frequently used parameters for cardiovascular diagnosis, e.g., to identify a stenosis of the coronary arteries. Intra-vascular OCT imaging has a high spatial resolution and promises accurate estimates of the vessel diameter. However, the actual images are reconstructed from A-scans relative to the catheter tip and imaging is subject to rotational artifacts. We study the impact of different volume reconstruction approaches on the accuracy of the vessel shape estimate. Using X-ray angiography we obtain the 3D vessel centerline and the 3D catheter trajectory, and we propose to align the A-scans using both. For comparison we consider reconstruction along a straight line and along the centerline. All methods are evaluated based on an experimental setup using a clinical angiography system and a vessel phantom with known shape. Our results illustrate potential pitfalls in the estimation of the vessel shape, particularly when the vessel is curved. We demonstrate that the conventional reconstruction approaches may result in an overestimate of the cross-section and that the proposed approach results in a good shape agreement in general and for curver segments, with DICE coefficients of approximately 0.96 and 0.98, respectively.

Automatic Plaque Detection in IVOCT Pullbacks Using Convolutional Neural Networks.

Coronary heart disease is a common cause of death despite being preventable. To treat the underlying plaque deposits in the arterial walls, intravascular optical coherence tomography can be used by experts to detect and characterize the lesions. In clinical routine, hundreds of images are acquired for each patient, which require automatic plaque detection for fast and accurate decision support. So far, automatic approaches rely on classic machine learning methods and deep learning solutions have rarely been studied. Given the success of deep learning methods with other imaging modalities, a thorough understanding of deep learning-based plaque detection for future clinical decision support systems is required. We address this issue with a new data set consisting of in vivo patient images labeled by three trained experts. Using this data set, we employ the state-of-the-art deep learning models that directly learn plaque classification from the images. For improved performance, we study different transfer learning approaches. Furthermore, we investigate the use of Cartesian and polar image representations and employ data augmentation techniques tailored to each representation. We fuse both representations in a multi-path architecture for more effective feature exploitation. Last, we address the challenge of plaque differentiation in addition to detection. Overall, we find that our combined model performs best with an accuracy of 91.7%, a sensitivity of 90.9%, and a specificity of 92.4%. Our results indicate that building a deep learning-based clinical decision support system for plaque detection is feasible.

Force estimation from OCT volumes using 3D CNNs.

PURPOSE: Estimating the interaction forces of instruments and tissue is of interest, particularly to provide haptic feedback during robot-assisted minimally invasive interventions. Different approaches based on external and integrated force sensors have been proposed. These are hampered by friction, sensor size, and sterilizability. We investigate a novel approach to estimate the force vector directly from optical coherence tomography image volumes. METHODS: We introduce a novel Siamese 3D CNN architecture. The network takes an undeformed reference volume and a deformed sample volume as an input and outputs the three components of the force vector. We employ a deep residual architecture with bottlenecks for increased efficiency. We compare the Siamese approach to methods using difference volumes and two-dimensional projections. Data were generated using a robotic setup to obtain ground-truth force vectors for silicon tissue phantoms as well as porcine tissue. RESULTS: Our method achieves a mean average error of [Formula: see text] when estimating the force vector. Our novel Siamese 3D CNN architecture outperforms single-path methods that achieve a mean average error of [Formula: see text]. Moreover, the use of volume data leads to significantly higher performance compared to processing only surface information which achieves a mean average error of [Formula: see text]. Based on the tissue dataset, our methods shows good generalization in between different subjects. CONCLUSIONS: We propose a novel image-based force estimation method using optical coherence tomography. We illustrate that capturing the deformation of subsurface structures substantially improves force estimation. Our approach can provide accurate force estimates in surgical setups when using intraoperative optical coherence tomography.

A deep learning approach for pose estimation from volumetric OCT data.

Tracking the pose of instruments is a central problem in image-guided surgery. For microscopic scenarios, optical coherence tomography (OCT) is increasingly used as an imaging modality. OCT is suitable for accurate pose estimation due to its micrometer range resolution and volumetric field of view. However, OCT image processing is challenging due to speckle noise and reflection artifacts in addition to the images' 3D nature. We address pose estimation from OCT volume data with a new deep learning-based tracking framework. For this purpose, we design a new 3D convolutional neural network (CNN) architecture to directly predict the 6D pose of a small marker geometry from OCT volumes. We use a hexapod robot to automatically acquire labeled data points which we use to train 3D CNN architectures for multi-output regression. We use this setup to provide an in-depth analysis on deep learning-based pose estimation from volumes. Specifically, we demonstrate that exploiting volume information for pose estimation yields higher accuracy than relying on 2D representations with depth information. Supporting this observation, we provide quantitative and qualitative results that 3D CNNs effectively exploit the depth structure of marker objects. Regarding the deep learning aspect, we present efficient design principles for 3D CNNs, making use of insights from the 2D deep learning community. In particular, we present Inception3D as a new architecture which performs best for our application. We show that our deep learning approach reaches errors at our ground-truth label's resolution. We achieve a mean average error of 14.89 ±â€¯9.3 µm and 0.096 ±â€¯0.072° for position and orientation learning, respectively.