Automatic caries detection in bitewing radiographs-Part II: experimental comparison.

OBJECTIVE: The objective of this study was to compare the detection of caries in bitewing radiographs by multiple dentists with an automatic method and to evaluate the detection performance in the absence of a reliable ground truth. MATERIALS AND METHODS: Four experts and three novices marked caries using bounding boxes in 100 bitewing radiographs. The same dataset was processed by an automatic object detection deep learning method. All annotators were compared in terms of the number of errors and intersection over union (IoU) using pairwise comparisons, with respect to the consensus standard, and with respect to the annotator of the training dataset of the automatic method. RESULTS: The number of lesions marked by experts in 100 images varied between 241 and 425. Pairwise comparisons showed that the automatic method outperformed all dentists except the original annotator in the mean number of errors, while being among the best in terms of IoU. With respect to a consensus standard, the performance of the automatic method was best in terms of the number of errors and slightly below average in terms of IoU. Compared with the original annotator, the automatic method had the highest IoU and only one expert made fewer errors. CONCLUSIONS: The automatic method consistently outperformed novices and performed as well as highly experienced dentists. CLINICAL SIGNIFICANCE: The consensus in caries detection between experts is low. An automatic method based on deep learning can improve both the accuracy and repeatability of caries detection, providing a useful second opinion even for very experienced dentists.

Fully automated imaging protocol independent system for pituitary adenoma segmentation: a convolutional neural network-based model on sparsely annotated MRI.

This study aims to develop a fully automated imaging protocol independent system for pituitary adenoma segmentation from magnetic resonance imaging (MRI) scans that can work without user interaction and evaluate its accuracy and utility for clinical applications. We trained two independent artificial neural networks on MRI scans of 394 patients. The scans were acquired according to various imaging protocols over the course of 11 years on 1.5T and 3T MRI systems. The segmentation model assigned a class label to each input pixel (pituitary adenoma, internal carotid artery, normal pituitary gland, background). The slice segmentation model classified slices as clinically relevant (structures of interest in slice) or irrelevant (anterior or posterior to sella turcica). We used MRI data of another 99 patients to evaluate the performance of the model during training. We validated the model on a prospective cohort of 28 patients, Dice coefficients of 0.910, 0.719, and 0.240 for tumour, internal carotid artery, and normal gland labels, respectively, were achieved. The slice selection model achieved 82.5% accuracy, 88.7% sensitivity, 76.7% specificity, and an AUC of 0.904. A human expert rated 71.4% of the segmentation results as accurate, 21.4% as slightly inaccurate, and 7.1% as coarsely inaccurate. Our model achieved good results comparable with recent works of other authors on the largest dataset to date and generalized well for various imaging protocols. We discussed future clinical applications, and their considerations. Models and frameworks for clinical use have yet to be developed and evaluated.

Automatic caries detection in bitewing radiographs: part I-deep learning.

OBJECTIVE: The aim of this work was to assemble a large annotated dataset of bitewing radiographs and to use convolutional neural networks to automate the detection of dental caries in bitewing radiographs with human-level performance. MATERIALS AND METHODS: A dataset of 3989 bitewing radiographs was created, and 7257 carious lesions were annotated using minimal bounding boxes. The dataset was then divided into 3 parts for the training (70%), validation (15%), and testing (15%) of multiple object detection convolutional neural networks (CNN). The tested CNN architectures included YOLOv5, Faster R-CNN, RetinaNet, and EfficientDet. To further improve the detection performance, model ensembling was used, and nested predictions were removed during post-processing. The models were compared in terms of the [Formula: see text] score and average precision (AP) with various thresholds of the intersection over union (IoU). RESULTS: The twelve tested architectures had [Formula: see text] scores of 0.72-0.76. Their performance was improved by ensembling which increased the [Formula: see text] score to 0.79-0.80. The best-performing ensemble detected caries with the precision of 0.83, recall of 0.77, [Formula: see text], and AP of 0.86 at IoU=0.5. Small carious lesions were predicted with slightly lower accuracy (AP 0.82) than medium or large lesions (AP 0.88). CONCLUSIONS: The trained ensemble of object detection CNNs detected caries with satisfactory accuracy and performed at least as well as experienced dentists (see companion paper, Part II). The performance on small lesions was likely limited by inconsistencies in the training dataset. CLINICAL SIGNIFICANCE: Caries can be automatically detected using convolutional neural networks. However, detecting incipient carious lesions remains challenging.

Automated segmentation of a motion mask to preserve sliding motion in deformable registration of thoracic CT.

PURPOSE: Deformable registration generally relies on the assumption that the sought spatial transformation is smooth. Yet, breathing motion involves sliding of the lung with respect to the chest wall, causing a discontinuity in the motion field, and the smoothness assumption can lead to poor matching accuracy. In response, alternative registration methods have been proposed, several of which rely on prior segmentations. We propose an original method for automatically extracting a particular segmentation, called a motion mask, from a CT image of the thorax. METHODS: The motion mask separates moving from less-moving regions, conveniently allowing simultaneous estimation of their motion, while providing an interface where sliding occurs. The sought segmentation is subanatomical and based on physiological considerations, rather than organ boundaries. We therefore first extract clear anatomical features from the image, with respect to which the mask is defined. Level sets are then used to obtain smooth surfaces interpolating these features. The resulting procedure comes down to a monitored level set segmentation of binary label images. The method was applied to sixteen inhale-exhale image pairs. To illustrate the suitability of the motion masks, they were used during deformable registration of the thorax. RESULTS: For all patients, the obtained motion masks complied with the physiological requirements and were consistent with respect to patient anatomy between inhale and exhale. Registration using the motion mask resulted in higher matching accuracy for all patients, and the improvement was statistically significant. Registration performance was comparable to that obtained using lung masks when considering the entire lung region, but the use of motion masks led to significantly better matching near the diaphragm and mediastinum, for the bony anatomy and for the trachea. The use of the masks was shown to facilitate the registration, allowing to reduce the complexity of the spatial transformation considerably, while maintaining matching accuracy. CONCLUSIONS: We proposed an automated segmentation method for obtaining motion masks, capable of facilitating deformable registration of the thorax. The use of motion masks during registration leads to matching accuracies comparable to the use of lung masks for the lung region but motion masks are more suitable when registering the entire thorax.

Characterization of drug effects on cell cultures from phase-contrast microscopy images.

In this work, we classify chemotherapeutic agents (topoisomerase inhibitors) based on their effect on U-2 OS cells. We use phase-contrast microscopy images, which are faster and easier to obtain than fluorescence images and support live cell imaging. We use a convolutional neural network (CNN) trained end-to-end directly on the input images without requiring for manual segmentations or any other auxiliary data. Our method can distinguish between tested cytotoxic drugs with an accuracy of 98%, provided that their mechanism of action differs, outperforming previous work. The results are even better when substance-specific concentrations are used. We show the benefit of sharing the extracted features over all classes (drugs). Finally, a 2D visualization of these features reveals clusters, which correspond well to known class labels, suggesting the possible use of our methodology for drug discovery application in analyzing new, unseen drugs.

The Effect of Primary Aldosteronism on Carotid Artery Texture in Ultrasound Images.

Primary aldosteronism (PA) is the most frequent cause of secondary hypertension. Early diagnoses of PA are essential to avoid the long-term negative effects of elevated aldosterone concentration on the cardiovascular and renal system. In this work, we study the texture of the carotid artery vessel wall from longitudinal ultrasound images in order to automatically distinguish between PA and essential hypertension (EH). The texture is characterized using 140 Haralick and 10 wavelet features evaluated in a region of interest in the vessel wall, followed by the XGBoost classifier. Carotid ultrasound studies were carried out on 33 patients aged 42-72 years with PA, 52 patients with EH, and 33 normotensive controls. For the most clinically relevant task of distinguishing PA and EH classes, we achieved a classification accuracy of 73% as assessed by a leave-one-out procedure. This result is promising even compared to the 57% prediction accuracy using clinical characteristics alone or 63% accuracy using a combination of clinical characteristics and intima-media thickness (IMT) parameters. If the accuracy is improved and the method incorporated into standard clinical procedures, this could eventually lead to an improvement in the early diagnosis of PA and consequently improve the clinical outcome for these patients in future.

Spatiotemporal motion estimation for respiratory-correlated imaging of the lungs.

PURPOSE: Four-dimensional computed tomography (4D CT) can provide patient-specific motion information for radiotherapy planning and delivery. Motion estimation in 4D CT is challenging due to the reduced image quality and the presence of artifacts. We aim to improve the robustness of deformable registration applied to respiratory-correlated imaging of the lungs, by using a global problem formulation and pursuing a restrictive parametrization for the spatiotemporal deformation model. METHODS: A spatial transformation based on free-form deformations was extended to the temporal domain, by explicitly modeling the trajectory using a cyclic temporal model based on B-splines. A global registration criterion allowed to consider the entire image sequence simultaneously and enforce the temporal coherence of the deformation throughout the respiratory cycle. To ensure a parametrization capable of capturing the dynamics of respiratory motion, a prestudy was performed on the temporal dimension separately. The temporal parameters were tuned by fitting them to diaphragm motion data acquired for a large patient group. Suitable properties were retained and applied to spatiotemporal registration of 4D CT data. Registration results were validated using large sets of landmarks and compared to consecutive spatial registrations. To illustrate the benefit of the spatiotemporal approach, we also assessed the performance in the presence of motion-induced artifacts. RESULTS: Cubic B-splines gave better or similar fitting results as lower orders and were selected because of their inherently stronger regularization. The fitting and registration errors increased gradually with the temporal control point spacing, representing a trade-off between achievable accuracy and sensitivity to noise and artifacts. A piecewise smooth trajectory model, allowing for a discontinuous change of speed at end-inhale, was found most suitable to account for the sudden changes of motion at this breathing phase. The spatiotemporal modeling allowed a reduction of the number of parameters of 45%, while maintaining registration accuracy within 0.1 mm. The approach reduced the sensitivity to artifacts. CONCLUSIONS: Spatiotemporal registration can provide accurate motion estimation for 4D CT and improves the robustness to artifacts.

ANHIR: Automatic Non-Rigid Histological Image Registration Challenge.

Automatic Non-rigid Histological Image Registration (ANHIR) challenge was organized to compare the performance of image registration algorithms on several kinds of microscopy histology images in a fair and independent manner. We have assembled 8 datasets, containing 355 images with 18 different stains, resulting in 481 image pairs to be registered. Registration accuracy was evaluated using manually placed landmarks. In total, 256 teams registered for the challenge, 10 submitted the results, and 6 participated in the workshop. Here, we present the results of 7 well-performing methods from the challenge together with 6 well-known existing methods. The best methods used coarse but robust initial alignment, followed by non-rigid registration, used multiresolution, and were carefully tuned for the data at hand. They outperformed off-the-shelf methods, mostly by being more robust. The best methods could successfully register over 98% of all landmarks and their mean landmark registration accuracy (TRE) was 0.44% of the image diagonal. The challenge remains open to submissions and all images are available for download.

2-D locally regularized tissue strain estimation from radio-frequency ultrasound images: theoretical developments and results on experimental data.

In this paper, a 2-D locally regularized strain estimation method for imaging deformation of soft biological tissues from radio-frequency (RF) ultrasound (US) data is introduced. Contrary to most 2-D techniques that model the compression-induced local displacement as a 2-D shift, our algorithm also considers a local scaling factor in the axial direction. This direction-dependent model of tissue motion and deformation is induced by the highly anisotropic resolution of RF US images. Optimal parameters are computed through the constrained maximization of a similarity criterion defined as the normalized correlation coefficient. Its value at the solution is then used as an indicator of estimation reliability, the probability of correct estimation increasing with the correlation value. In case of correlation loss, the estimation integrates an additional constraint, imposing local continuity within displacement and strain fields. Using local scaling factors and regularization increase the method's robustness with regard to decorrelation noise, resulting in a wider range of precise measurements. Results on simulated US data from a mechanically homogeneous medium subjected to successive uniaxial loadings demonstrate that our method is theoretically able to accurately estimate strains up to 17%. Experimental strain images of phantom and cut specimens of bovine liver clearly show the harder inclusions.

Reconstructing Evolving Tree Structures in Time Lapse Sequences by Enforcing Time-Consistency.

We propose a novel approach to reconstructing curvilinear tree structures evolving over time, such as road networks in 2D aerial images or neural structures in 3D microscopy stacks acquired in vivo. To enforce temporal consistency, we simultaneously process all images in a sequence, as opposed to reconstructing structures of interest in each image independently. We formulate the problem as a Quadratic Mixed Integer Program and demonstrate the additional robustness that comes from using all available visual clues at once, instead of working frame by frame. Furthermore, when the linear structures undergo local changes over time, our approach automatically detects them.

Geometric Graph Matching Using Monte Carlo Tree Search.

We present an efficient matching method for generalized geometric graphs. Such graphs consist of vertices in space connected by curves and can represent many real world structures such as road networks in remote sensing, or vessel networks in medical imaging. Graph matching can be used for very fast and possibly multimodal registration of images of these structures. We formulate the matching problem as a single player game solved using Monte Carlo Tree Search, which automatically balances exploring new possible matches and extending existing matches. Our method can handle partial matches, topological differences, geometrical distortion, does not use appearance information and does not require an initial alignment. Moreover, our method is very efficient-it can match graphs with thousands of nodes, which is an order of magnitude better than the best competing method, and the matching only takes a few seconds.

Left ventricle Hermite-based segmentation.

In recent years, computed tomography (CT) has become a standard technique in cardiac imaging because it provides detailed information that may facilitate the diagnosis of the conditions that interfere with correct heart function. However, CT-based cardiac diagnosis requires manual segmentation of heart cavities, which is a difficult and time-consuming task. Thus, in this paper, we propose a novel technique to segment endocardium and epicardium boundaries based on a 2D approach. The proposal computes relevant information of the left ventricle and its adjacent structures using the Hermite transform. The novelty of the work is that the information is combined with active shape models and level sets to improve the segmentation. Our database consists of mid-third slices selected from 28 volumes manually segmented by expert physicians. The segmentation is assessed using Dice coefficient and Hausdorff distance. In addition, we introduce a novel metric called Ray Feature error to evaluate our method. The results show that the proposal accurately discriminates cardiac tissue. Thus, it may be a useful tool for supporting heart disease diagnosis and tailoring treatments.

Generalized head models for MEG/EEG: boundary element method beyond nested volumes.

Accurate geometrical models of the head are necessary for solving the forward and inverse problems of magneto- and electro-encephalography (MEG/EEG). Boundary element methods (BEMs) require a geometrical model describing the interfaces between different tissue types. Classically, head models with a nested volume topology have been used. In this paper, we demonstrate how this constraint can be relaxed, allowing us to model more realistic head topologies. We describe the symmetric BEM for this new model. The symmetric BEM formulation uses both potentials and currents at the interfaces as unknowns and is in general more accurate than the alternative double-layer formulation.

Fully automated classification of bone marrow infiltration in low-dose CT of patients with multiple myeloma based on probabilistic density model and supervised learning.

This paper presents a fully automated method for the identification of bone marrow infiltration in femurs in low-dose CT of patients with multiple myeloma. We automatically find the femurs and the bone marrow within them. In the next step, we create a probabilistic, spatially dependent density model of normal tissue. At test time, we detect unexpectedly high density voxels which may be related to bone marrow infiltration, as outliers to this model. Based on a set of global, aggregated features representing all detections from one femur, we classify the subjects as being either healthy or not. This method was validated on a dataset of 127 subjects with ground truth created from a consensus of two expert radiologists, obtaining an AUC of 0.996 for the task of distinguishing healthy controls and patients with bone marrow infiltration. To the best of our knowledge, no other automatic image-based method for this task has been published before.

Automated Analysis of Microscopic Images of Isolated Pancreatic Islets.

Clinical islet transplantation programs rely on the capacities of individual centers to quantify isolated islets. Current computer-assisted methods require input from human operators. Here we describe two machine learning algorithms for islet quantification: the trainable islet algorithm (TIA) and the nontrainable purity algorithm (NPA). These algorithms automatically segment pancreatic islets and exocrine tissue on microscopic images in order to count individual islets and calculate islet volume and purity. References for islet counts and volumes were generated by the fully manual segmentation (FMS) method, which was validated against the internal DNA standard. References for islet purity were generated via the expert visual assessment (EVA) method, which was validated against the FMS method. The TIA is intended to automatically evaluate micrographs of isolated islets from future donors after being trained on micrographs from a limited number of past donors. Its training ability was first evaluated on 46 images from four donors. The pixel-to-pixel comparison, binary statistics, and islet DNA concentration indicated that the TIA was successfully trained, regardless of the color differences of the original images. Next, the TIA trained on the four donors was validated on an additional 36 images from nine independent donors. The TIA was fast (67 s/image), correlated very well with the FMS method (R2=1.00 and 0.92 for islet volume and islet count, respectively), and had small REs (0.06 and 0.07 for islet volume and islet count, respectively). Validation of the NPA against the EVA method using 70 images from 12 donors revealed that the NPA had a reasonable speed (69 s/image), had an acceptable RE (0.14), and correlated well with the EVA method (R2=0.88). Our results demonstrate that a fully automated analysis of clinical-grade micrographs of isolated pancreatic islets is feasible. The algorithms described herein will be freely available as a Fiji platform plugin.

Spatio-temporal nonrigid registration for ultrasound cardiac motion estimation.

We propose a new spatio-temporal elastic registration algorithm for motion reconstruction from a series of images. The specific application is to estimate displacement fields from two-dimensional ultrasound sequences of the heart. The basic idea is to find a spatio-temporal deformation field that effectively compensates for the motion by minimizing a difference with respect to a reference frame. The key feature of our method is the use of a semi-local spatio-temporal parametric model for the deformation using splines, and the reformulation of the registration task as a global optimization problem. The scale of the spline model controls the smoothness of the displacement field. Our algorithm uses a multiresolution optimization strategy to obtain a higher speed and robustness. We evaluated the accuracy of our algorithm using a synthetic sequence generated with an ultrasound simulation package, together with a realistic cardiac motion model. We compared our new global multiframe approach with a previous method based on pairwise registration of consecutive frames to demonstrate the benefits of introducing temporal consistency. Finally, we applied the algorithm to the regional analysis of the left ventricle. Displacement and strain parameters were evaluated showing significant differences between the normal and pathological segments, thereby illustrating the clinical applicability of our method.

A common formalism for the integral formulations of the forward EEG problem.

The forward electroencephalography (EEG) problem involves finding a potential V from the Poisson equation inverted Delta x (sigma inverted Delta V) f, in which f represents electrical sources in the brain, and sigma the conductivity of the head tissues. In the piecewise constant conductivity head model, this can be accomplished by the boundary element method (BEM) using a suitable integral formulation. Most previous work uses the same integral formulation, corresponding to a double-layer potential. In this paper we present a conceptual framework based on a well-known theorem (Theorem 1) that characterizes harmonic functions defined on the complement of a bounded smooth surface. This theorem says that such harmonic functions are completely defined by their values and those of their normal derivatives on this surface. It allows us to cast the previous BEM approaches in a unified setting and to develop two new approaches corresponding to different ways of exploiting the same theorem. Specifically, we first present a dual approach which involves a single-layer potential. Then, we propose a symmetric formulation, which combines single- and double-layer potentials, and which is new to the field of EEG, although it has been applied to other problems in electromagnetism. The three methods have been evaluated numerically using a spherical geometry with known analytical solution, and the symmetric formulation achieves a significantly higher accuracy than the alternative methods. Additionally, we present results with realistically shaped meshes. Beside providing a better understanding of the foundations of BEM methods, our approach appears to lead also to more efficient algorithms.

Fast multipole acceleration of the MEG/EEG boundary element method.

The accurate solution of the forward electrostatic problem is an essential first step before solving the inverse problem of magneto- and electroencephalography (MEG/EEG). The symmetric Galerkin boundary element method is accurate but cannot be used for very large problems because of its computational complexity and memory requirements. We describe a fast multipole-based acceleration for the symmetric boundary element method (BEM). It creates a hierarchical structure of the elements and approximates far interactions using spherical harmonics expansions. The accelerated method is shown to be as accurate as the direct method, yet for large problems it is both faster and more economical in terms of memory consumption.

Non-Rigid Graph Registration Using Active Testing Search.

We present a new approach for matching sets of branching curvilinear structures that form graphs embedded in R2 or R3 and may be subject to deformations. Unlike earlier methods, ours does not rely on local appearance similarity nor does require a good initial alignment. Furthermore, it can cope with non-linear deformations, topological differences, and partial graphs. To handle arbitrary non-linear deformations, we use Gaussian process regressions to represent the geometrical mapping relating the two graphs. In the absence of appearance information, we iteratively establish correspondences between points, update the mapping accordingly, and use it to estimate where to find the most likely correspondences that will be used in the next step. To make the computation tractable for large graphs, the set of new potential matches considered at each iteration is not selected at random as with many RANSAC-based algorithms. Instead, we introduce a so-called Active Testing Search strategy that performs a priority search to favor the most likely matches and speed-up the process. We demonstrate the effectiveness of our approach first on synthetic cases and then on angiography data, retinal fundus images, and microscopy image stacks acquired at very different resolutions.

Fast parametric elastic image registration.

We present an algorithm for fast elastic multidimensional intensity-based image registration with a parametric model of the deformation. It is fully automatic in its default mode of operation. In the case of hard real-world problems, it is capable of accepting expert hints in the form of soft landmark constraints. Much fewer landmarks are needed and the results are far superior compared to pure landmark registration. Particular attention has been paid to the factors influencing the speed of this algorithm. The B-spline deformation model is shown to be computationally more efficient than other alternatives. The algorithm has been successfully used for several two-dimensional (2-D) and three-dimensional (3-D) registration tasks in the medical domain, involving MRI, SPECT, CT, and ultrasound image modalities. We also present experiments in a controlled environment, permitting an exact evaluation of the registration accuracy. Test deformations are generated automatically using a random hierarchical fractional wavelet-based generator.