AI-Dentify: deep learning for proximal caries detection on bitewing x-ray - HUNT4 Oral Health Study.

BACKGROUND: Dental caries diagnosis requires the manual inspection of diagnostic bitewing images of the patient, followed by a visual inspection and probing of the identified dental pieces with potential lesions. Yet the use of artificial intelligence, and in particular deep-learning, has the potential to aid in the diagnosis by providing a quick and informative analysis of the bitewing images. METHODS: A dataset of 13,887 bitewings from the HUNT4 Oral Health Study were annotated individually by six different experts, and used to train three different object detection deep-learning architectures: RetinaNet (ResNet50), YOLOv5 (M size), and EfficientDet (D0 and D1 sizes). A consensus dataset of 197 images, annotated jointly by the same six dental clinicians, was used for evaluation. A five-fold cross validation scheme was used to evaluate the performance of the AI models. RESULTS: The trained models show an increase in average precision and F1-score, and decrease of false negative rate, with respect to the dental clinicians. When compared against the dental clinicians, the YOLOv5 model shows the largest improvement, reporting 0.647 mean average precision, 0.548 mean F1-score, and 0.149 mean false negative rate. Whereas the best annotators on each of these metrics reported 0.299, 0.495, and 0.164 respectively. CONCLUSION: Deep-learning models have shown the potential to assist dental professionals in the diagnosis of caries. Yet, the task remains challenging due to the artifacts natural to the bitewing images.

Accuracy of instrument tip position using fiber optic shape sensing for navigated bronchoscopy.

The purpose of this study was to evaluate the accuracy of a method for estimating the tip position of a fiber optic shape-sensing (FOSS) integrated instrument being inserted through a bronchoscope. A modified guidewire with a multicore optical fiber was inserted into the working channel of a custom-made catheter with three electromagnetic (EM) sensors. The displacement between the instruments was manually set, and a point-based method was applied to match the position of the EM sensors to corresponding points on the shape. The accuracy was evaluated in a realistic bronchial model. An additional EM sensor was used to sample the tip of the guidewire, and the absolute deviation between this position and the estimated tip position was calculated. For small displacements between the tip of the FOSS integrated tool and the catheter, the median deviation in estimated tip position was ≤5 mm. For larger displacements, deviations exceeding 10 mm were observed. The deviations increased when the shape sensor had sharp curvatures relative to more straight shapes. The method works well for clinically relevant displacements of a biopsy tool from the bronchoscope tip, and when the path to the lesion has limited curvatures. However, improvements must be made to our configuration before pursuing further clinical testing.

Correction: Teacher-student approach for lung tumor segmentation from mixed-supervised datasets.

[This corrects the article DOI: 10.1371/journal.pone.0266147.].

Prediction of guidewire-induced aortic deformations during EVAR: a finite element and in vitro study.

Introduction and aims: During an Endovascular Aneurysm Repair (EVAR) procedure a stiff guidewire is inserted from the iliac arteries. This induces significant deformations on the vasculature, thus, affecting the pre-operative planning, and the accuracy of image fusion. The aim of the present work is to predict the guidewire induced deformations using a finite element approach validated through experiments with patient-specific additive manufactured models. The numerical approach herein developed could improve the pre-operative planning and the intra-operative navigation. Material and methods: The physical models used for the experiments in the hybrid operating room, were manufactured from the segmentations of pre-operative Computed Tomography (CT) angiographies. The finite element analyses (FEA) were performed with LS-DYNA Explicit. The material properties used in finite element analyses were obtained by uniaxial tensile tests. The experimental deformed configurations of the aorta were compared to those obtained from FEA. Three models, obtained from Computed Tomography acquisitions, were investigated in the present work: A) without intraluminal thrombus (ILT), B) with ILT, C) with ILT and calcifications. Results and discussion: A good agreement was found between the experimental and the computational studies. The average error between the final in vitro vs. in silico aortic configurations, i.e., when the guidewire is fully inserted, are equal to 1.17, 1.22 and 1.40 mm, respectively, for Models A, B and C. The increasing trend in values of deformations from Model A to Model C was noticed both experimentally and numerically. The presented validated computational approach in combination with a tracking technology of the endovascular devices may be used to obtain the intra-operative configuration of the vessels and devices prior to the procedure, thus limiting the radiation exposure and the contrast agent dose.

Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation.

PURPOSE: This study aims to explore training strategies to improve convolutional neural network-based image-to-image deformable registration for abdominal imaging. METHODS: Different training strategies, loss functions, and transfer learning schemes were considered. Furthermore, an augmentation layer which generates artificial training image pairs on-the-fly was proposed, in addition to a loss layer that enables dynamic loss weighting. RESULTS: Guiding registration using segmentations in the training step proved beneficial for deep-learning-based image registration. Finetuning the pretrained model from the brain MRI dataset to the abdominal CT dataset further improved performance on the latter application, removing the need for a large dataset to yield satisfactory performance. Dynamic loss weighting also marginally improved performance, all without impacting inference runtime. CONCLUSION: Using simple concepts, we improved the performance of a commonly used deep image registration architecture, VoxelMorph. In future work, our framework, DDMR, should be validated on different datasets to further assess its value.

Deep learning for image-based liver analysis - A comprehensive review focusing on malignant lesions.

Deep learning-based methods, in particular, convolutional neural networks and fully convolutional networks are now widely used in the medical image analysis domain. The scope of this review focuses on the analysis using deep learning of focal liver lesions, with a special interest in hepatocellular carcinoma and metastatic cancer; and structures like the parenchyma or the vascular system. Here, we address several neural network architectures used for analyzing the anatomical structures and lesions in the liver from various imaging modalities such as computed tomography, magnetic resonance imaging and ultrasound. Image analysis tasks like segmentation, object detection and classification for the liver, liver vessels and liver lesions are discussed. Based on the qualitative search, 91 papers were filtered out for the survey, including journal publications and conference proceedings. The papers reviewed in this work are grouped into eight categories based on the methodologies used. By comparing the evaluation metrics, hybrid models performed better for both the liver and the lesion segmentation tasks, ensemble classifiers performed better for the vessel segmentation tasks and combined approach performed better for both the lesion classification and detection tasks. The performance was measured based on the Dice score for the segmentation, and accuracy for the classification and detection tasks, which are the most commonly used metrics.

A novel clip-on device for electromagnetic tracking in endobronchial ultrasound bronchoscopy.

INTRODUCTION: The established method for assessment of mediastinal and hilar lymph nodes is endobronchial ultrasound bronchoscopy (EBUS) with needle aspirations. Previously, we presented an electromagnetic navigation platform for this purpose. There were several issues with the permanent electromagnetic tracking (EMT) sensor attachment on the tip of the experimental EBUS bronchoscope. The purpose was to develop a device for on-site attachment of the EMT sensor. MATERIAL AND METHODS: A clip-on EMT sensor attachment device was 3D-printed in Ultem™ and attached to an EBUS bronchoscope. A specially designed ultrasound probe calibration adapter was developed for on-site and quick probe calibration. Navigation accuracy was studied using a wire cross water phantom and clinical feasibility was tested in a healthy volunteer. RESULTS: The device attached to the EBUS bronchoscope increased its diameter from 6.9 mm to 9.5 mm. Average preclinical navigation accuracy was 3.9 mm after adapter calibration. The maneuvering of the bronchoscope examining a healthy volunteer was adequate without harming the respiratory epithelium, and the device stayed firmly attached. CONCLUSION: Development, calibration and testing of a clip-on EMT sensor attachment device for EBUS bronchoscopy was successfully demonstrated. Acceptable accuracy results were obtained, and the device is ready to be tested in patient studies.

Ultrasound-based navigation for open liver surgery using active liver tracking.

PURPOSE: Despite extensive preoperative imaging, intraoperative localization of liver lesions after systemic treatment can be challenging. Therefore, an image-guided navigation setup is explored that links preoperative diagnostic scans and 3D models to intraoperative ultrasound (US), enabling overlay of detailed diagnostic images on intraoperative US. Aim of this study is to assess the workflow and accuracy of such a navigation system which compensates for liver motion. METHODS: Electromagnetic (EM) tracking was used for organ tracking and movement of the transducer. After laparotomy, a sensor was attached to the liver surface while the EM-tracked US transducer enabled image acquisition and landmark digitization. Landmarks surrounding the lesion were selected during patient-specific preoperative 3D planning and identified for registration during surgery. Endpoints were accuracy and additional times of the investigative steps. Accuracy was computed at the center of the target lesion. RESULTS: In total, 22 navigated procedures were performed. Navigation provided useful visualization of preoperative 3D models and their overlay on US imaging. Landmark-based registration resulted in a mean fiducial registration error of 10.3 ± 4.3 mm, and a mean target registration error of 8.5 ± 4.2 mm. Navigation was available after an average of 12.7 min. CONCLUSION: We developed a navigation method combining ultrasound with active liver tracking for organ motion compensation, with an accuracy below 10 mm. Fixation of the liver sensor near the target lesion compensates for local movement and contributes to improved reliability during navigation. This represents an important step forward in providing surgical navigation throughout the procedure. TRIAL REGISTRATION: This study is registered in the Netherlands Trial Register (number NL7951).

Teacher-student approach for lung tumor segmentation from mixed-supervised datasets.

PURPOSE: Cancer is among the leading causes of death in the developed world, and lung cancer is the most lethal type. Early detection is crucial for better prognosis, but can be resource intensive to achieve. Automating tasks such as lung tumor localization and segmentation in radiological images can free valuable time for radiologists and other clinical personnel. Convolutional neural networks may be suited for such tasks, but require substantial amounts of labeled data to train. Obtaining labeled data is a challenge, especially in the medical domain. METHODS: This paper investigates the use of a teacher-student design to utilize datasets with different types of supervision to train an automatic model performing pulmonary tumor segmentation on computed tomography images. The framework consists of two models: the student that performs end-to-end automatic tumor segmentation and the teacher that supplies the student additional pseudo-annotated data during training. RESULTS: Using only a small proportion of semantically labeled data and a large number of bounding box annotated data, we achieved competitive performance using a teacher-student design. Models trained on larger amounts of semantic annotations did not perform better than those trained on teacher-annotated data. Our model trained on a small number of semantically labeled data achieved a mean dice similarity coefficient of 71.0 on the MSD Lung dataset. CONCLUSIONS: Our results demonstrate the potential of utilizing teacher-student designs to reduce the annotation load, as less supervised annotation schemes may be performed, without any real degradation in segmentation accuracy.

Can effective pedagogy be ensured in minimally invasive surgery e-learning?

INTRODUCTION: Effectiveness of e-learning diminishes without the support of a pedagogical model to guide its use. In minimally invasive surgery (MIS), this has been reported as a limitation when technology is used to deliver contents without a sound pedagogical background. MATERIAL AND METHODS: We describe how a generic pedagogical model, the 3D pedagogy framework, can be used for setting learning outcomes and activities in e-learning platforms focused on MIS cognitive skills. A demonstrator course on Nissen fundoplication was developed following the model step-by-step in the MISTELA learning platform. Course design was informed by Kolb's Experiential learning model. Content validation was performed by 13 MIS experts. RESULTS: Ten experts agreed on the suitability of content structuring done according to the pedagogical model. All experts agreed that the course provides means to assess the intended learning outcomes. CONCLUSIONS: This work showcases how a general-purpose e-learning framework can be accommodated to the needs of MIS training without limiting the course designers' pedagogical approach. Key advances for its success include: (1) proving the validity of the model in the wider scope of MIS skills and (2) raising awareness amongst stakeholders on the need of developing training plans with explicit, rather than assumed, pedagogical foundations. Abbreviations: MIS: minimally invasive surgery; TEL: technology enhanced learning.

Block-matching-based registration to evaluate ultrasound visibility of percutaneous needles in liver-mimicking phantoms.

PURPOSE: To present a novel methodical approach to compare visibility of percutaneous needles in ultrasound images. METHODS: A motor-driven rotation platform was used to gradually change the needle angle while capturing image data. Data analysis was automated using block-matching-based registration, with a tracking and refinement step. Every 25 frames, a Hough transform was used to improve needle alignments after large rotations. The method was demonstrated by comparing three commercial needles (14G radiofrequency ablation, RFA; 18G Trocar; 22G Chiba) and six prototype needles with different sizes, materials, and surface conditions (polished, sand-blasted, and kerfed), within polyvinyl alcohol phantom tissue and ex vivo bovine liver models. For each needle and angle, a contrast-to-noise ratio (CNR) was determined to quantify visibility. CNR values are presented as a function of needle type and insertion angle. In addition, the normalized area under the (CNR-angle) curve was used as a summary metric to compare needles. RESULTS: In phantom tissue, the first kerfed needle design had the largest normalized area of visibility and the polished 1 mm diameter stainless steel needle the smallest (0.704 ± 0.199 vs. 0.154 ± 0.027, p < 0.01). In the ex vivo model, the second kerfed needle design had the largest normalized area of visibility, and the sand-blasted stainless steel needle the smallest (0.470 ± 0.190 vs. 0.127 ± 0.047, p < 0.001). As expected, the analysis showed needle visibility peaks at orthogonal insertion angles. For acute or obtuse angles, needle visibility was similar or reduced. Overall, the variability in needle visibility was considerably higher in livers. CONCLUSION: The best overall visibility was found with kerfed needles and the commercial RFA needle. The presented methodical approach to quantify ultrasound visibility allows comparisons of (echogenic) needles, as well as other technological innovations aiming to improve ultrasound visibility of percutaneous needles, such as coatings, material treatments, and beam steering approaches.

Can a Dinosaur Think? Implementation of Artificial Intelligence in Extracorporeal Shock Wave Lithotripsy.

BACKGROUND: Extracorporeal shock wave lithotripsy (ESWL) of kidney stones is losing ground to more expensive and invasive endoscopic treatments. OBJECTIVE: This proof-of-concept project was initiated to develop artificial intelligence (AI)-augmented ESWL and to investigate the potential for machine learning to improve the efficacy of ESWL. DESIGN SETTING AND PARTICIPANTS: Two-dimensional ultrasound videos were captured during ESWL treatments from an inline ultrasound device with a video grabber. An observer annotated 23 212 images from 11 patients as either in or out of focus. The median hit rate was calculated on a patient level via bootstrapping. A convolutional neural network with U-Net architecture was trained on 57 ultrasound images with delineated kidney stones from the same patients annotated by a second observer. We tested U-Net on the ultrasound images annotated by the first observer. Cross-validation with a training set of nine patients, a validation set of one patient, and a test set of one patient was performed. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS: Classical metrics describing classifier performance were calculated, together with an estimation of how the algorithm would affect shock wave hit rate. RESULTS AND LIMITATIONS: The median hit rate for standard ESWL was 55.2% (95% confidence interval [CI] 43.2-67.3%). The performance metrics for U-Net were accuracy 63.9%, sensitivity 56.0%, specificity 74.7%, positive predictive value 75.3%, negative predictive value 55.2%, Youden's J statistic 30.7%, no-information rate 58.0%, and Cohen's κ 0.2931. The algorithm reduced total mishits by 67.1%. The main limitation is that this is a proof-of-concept study involving only 11 patients. CONCLUSIONS: Our calculated ESWL hit rate of 55.2% (95% CI 43.2-67.3%) supports findings from earlier research. We have demonstrated that a machine learning algorithm trained on just 11 patients increases the hit rate to 75.3% and reduces mishits by 67.1%. When U-Net is trained on more and higher-quality annotations, even better results can be expected. PATIENT SUMMARY: Kidney stones can be treated by applying shockwaves to the outside of the body. Ultrasound scans of the kidney are used to guide the machine delivering the shockwaves, but the shockwaves can still miss the stone. We used artificial intelligence to improve the accuracy in hitting the stone being treated.

An open electromagnetic tracking framework applied to targeted liver tumour ablation.

PURPOSE: Electromagnetic tracking is a core platform technology in the navigation and visualisation of image-guided procedures. The technology provides high tracking accuracy in non-line-of-sight environments, allowing instrument navigation in locations where optical tracking is not feasible. EMT can be beneficial in applications such as percutaneous radiofrequency ablation for the treatment of hepatic lesions where the needle tip may be obscured due to difficult liver environments (e.g subcutaneous fat or ablation artefacts). Advances in the field of EMT include novel methods of improving tracking system accuracy, precision and error compensation capabilities, though such system-level improvements cannot be readily incorporated in current therapy applications due to the 'blackbox' nature of commercial tracking solving algorithms. METHODS: This paper defines a software framework to allow novel EMT designs, and improvements become part of the global design process for image-guided interventions. An exemplary framework is implemented in the Python programming language and demonstrated with the open-source Anser EMT system. The framework is applied in the preclinical setting though targeted liver ablation therapy on an animal model. RESULTS: The developed framework was tested with the Anser EMT electromagnetic tracking platform. Liver tumour targeting was performed using the tracking framework with the CustusX navigation platform using commercially available electromagnetically tracked needles. Ablation of two tumours was performed with a commercially available ablation system. Necropsy of the tumours indicated ablations within 5 mm of the tumours. CONCLUSIONS: An open-source framework for electromagnetic tracking was presented and effectively demonstrated in the preclinical setting. We believe that this framework provides a structure for future advancement in EMT system in and customised instrument design.

Semantic segmentation and detection of mediastinal lymph nodes and anatomical structures in CT data for lung cancer staging.

PURPOSE: Accurate lung cancer diagnosis is crucial to select the best course of action for treating the patient. From a simple chest CT volume, it is necessary to identify whether the cancer has spread to nearby lymph nodes or not. It is equally important to know precisely where each malignant lymph node is with respect to the surrounding anatomical structures and the airways. In this paper, we introduce a new data-set containing annotations of fifteen different anatomical structures in the mediastinal area, including lymph nodes of varying sizes. We present a 2D pipeline for semantic segmentation and instance detection of anatomical structures and potentially malignant lymph nodes in the mediastinal area. METHODS: We propose a 2D pipeline combining the strengths of U-Net for pixel-wise segmentation using a loss function dealing with data imbalance and Mask R-CNN providing instance detection and improved pixel-wise segmentation within bounding boxes. A final stage performs pixel-wise labels refinement and 3D instance detection using a tracking approach along the slicing dimension. Detected instances are represented by a 3D pixel-wise mask, bounding volume, and centroid position. RESULTS: We validated our approach following a fivefold cross-validation over our new data-set of fifteen lung cancer patients. For the semantic segmentation task, we reach an average Dice score of 76% over all fifteen anatomical structures. For the lymph node instance detection task, we reach 75% recall for 9 false positives per patient, with an average centroid position estimation error of 3 mm in each dimension. CONCLUSION: Fusing 2D networks' results increases pixel-wise segmentation results while enabling good instance detection. Better leveraging of the 3D information and station mapping for the detected lymph nodes are the next steps.

Using the CustusX toolkit to create an image guided bronchoscopy application: Fraxinus.

PURPOSE: The aim of this paper is to show how a specialized planning and guidance application called Fraxinus, can be built on top of the CustusX platform (www.custusx.org), which is an open source image-guided intervention software platform. Fraxinus has been customized to meet the clinical needs in navigated bronchoscopy. METHODS: The application requirements for Fraxinus were defined in close collaboration between research scientists, software developers and clinicians (pulmonologists), and built on top of CustusX. Its superbuild system downloads specific versions of the required libraries and builds them for the application in question, including the selected plugins. New functionality is easily added through the plugin framework. The build process enables the creation of specialized applications, adding additional documentation and custom configurations. The toolkit's libraries offer building blocks for image-guided applications. An iterative development process was applied, where the clinicians would test and provide feedback during the entire process. RESULTS: Fraxinus has been developed and is released as an open source planning and guidance application built on top of CustusX. It is highly specialized for bronchoscopy. The proposed workflow is adapted to the different steps in this procedure. The user interface of CustusX has been modified to enhance information, quality assurance and user friendliness with the intention to increase the overall yield for the patient. As the workflow of the procedure is relatively constant, some actions are predicted and automatically performed by the application, according to the requirements from the clinicians. CONCLUSIONS: The CustusX platform facilitates development of new and specialized applications. The toolkit supports the process and makes important extension and injection points available for customization.

A Methodical Quantification of Needle Visibility and Echogenicity in Ultrasound Images.

During ultrasound-guided percutaneous interventions, needle localization can be a challenge. To increase needle visibility, enhancements of both the imaging methods and the needle surface properties have been investigated. However, a methodical approach to compare potential solutions is currently unavailable. The work described here involves automated image acquisition, analysis and reporting techniques to collect large amounts of data efficiently, delineate relevant factors and communicate effects. Data processing included filtering, line fitting and image intensity analysis steps. Foreground and background image samples were used to compute a contrast-to-noise ratio or a signal ratio. The approach was evaluated in a comparative study of commercially available and custom-made needles. Varied parameters included needle material, diameter and surface roughness. The shafts with kerfed patterns and the trocar and chiba tips performed best. The approach enabled an intuitive polar depiction of needle visibility in ultrasound images for a large range of insertion angles.

Pulmonologist evaluation on new CT visualization for guidance to lung lesions during bronchoscopy.

OBJECTIVE: Endoluminal visualization in virtual and video bronchoscopy lacks information about the surrounding structures, and the traditional 2 D axial, coronal and sagittal CT views can be difficult to interpret. To address this challenge, we previously introduced a novel visualization technique, Anchored to Centerline Curved Surface, for navigated bronchoscopy. The current study compares the ACCuSurf to the standard ACS CT views as planning and guiding tools in a phantom study. MATERIAL AND METHODS: Bronchoscope operators navigated in physical phantom guided by virtual realistic image data constructed by fusion of CT dataset of phantom and anonymized patient CT data. We marked four different target positions within the virtual image data and gave 12 pulmonologists the task to navigate, with either ACCuSurf or ACS as guidance, to the corresponding targets in the physical phantom. RESULTS: Using ACCuSurf reduced the planning time and increased the grade of successful navigation significantly compared to ACS. CONCLUSION: The phantom setup with virtual patient image data proved realistic according to the pulmonologists. ACCuSurf proved superior to ACS regarding planning time and navigation success grading. Improvements on visualisation or display techniques may consequently improve both planning and navigated bronchoscopy and thus contribute to more precise lung diagnostics.

Peripheral tumour targeting using open-source virtual bronchoscopy with electromagnetic tracking: a multi-user pre-clinical study.

Objectives: The goal was to demonstrate the utility of open-source tracking and visualisation tools in the targeting of lung cancer.Material and methods: The study demonstrates the first deployment of the Anser electromagnetic (EM) tracking system with the CustusX image-guided interventional research platform to navigate using an endobronchial catheter to injected tumour targets. Live animal investigations validated the deployment and targeting of peripheral tumour models using an innovative tumour marking routine.Results: Novel tumour model deployment was successfully achieved at all eight target sites across two live animal investigations without pneumothorax. Virtual bronchoscopy with tracking successfully guided the tracked catheter to 2-12 mm from the target tumour site. Deployment of a novel marker was achieved at all eight sites providing a reliable measure of targeting accuracy. Targeting accuracy within 10 mm was achieved in 7/8 sites and in all cases, the virtual target distance at marker deployment was within the range subsequently measured with x-ray.Conclusions: Endobronchial targeting of peripheral airway targets is feasible using existing open-source technology. Notwithstanding the shortcomings of current commercial platforms, technological improvements in EM tracking and registration accuracy fostered by open-source technology may provide the impetus for widespread clinical uptake of electromagnetic navigation in bronchoscopy.

Laboratory test of Single Landmark registration method for ultrasound-based navigation in laparoscopy using an open-source platform.

PURPOSE: Test the feasibility of the novel Single Landmark image-to-patient registration method for use in the operating room for future clinical trials. The algorithm is implemented in the open-source platform CustusX, a computer-aided intervention research platform dedicated to intraoperative navigation and ultrasound, with an interface for laparoscopic ultrasound probes. METHODS: The Single Landmark method is compared to fiducial landmark on an IOUSFAN (Kyoto Kagaku Co., Ltd., Japan) soft tissue abdominal phantom and T2 magnetic resonance scans of it. RESULTS: The experiments show that the accuracy of the Single Landmark registration is good close to the registered point, increasing with the distance from this point (12.4 mm error at 60 mm away from the registered point). In this point, the registration accuracy is mainly dominated by the accuracy of the user when clicking on the ultrasound image. In the presented set-up, the time required to perform the Single Landmark registration is 40% less than for the FLRM. CONCLUSION: The Single Landmark registration is suitable for being integrated in a laparoscopic workflow. The statistical analysis shows robustness against translational displacements of the patient and improvements in terms of time. The proposed method allows the clinician to accurately register lesions intraoperatively by clicking on these in the ultrasound image provided by the ultrasound transducer. The Single Landmark registration method can be further combined with other more accurate registration approaches improving the registration at relevant points defined by the clinicians.