Automatic Detection of Periapical Osteolytic Lesions on Cone-beam Computed Tomography Using Deep Convolutional Neuronal Networks.

INTRODUCTION: Cone-beam computed tomography (CBCT) is an essential diagnostic tool in oral radiology. Radiolucent periapical lesions (PALs) represent the most frequent jaw lesions. However, the description, interpretation, and documentation of radiological findings, especially incidental findings, are time-consuming and resource-intensive, requiring a high degree of expertise. To improve quality, dentists may use artificial intelligence in the form of deep learning tools. This study was conducted to develop and validate a deep convolutional neuronal network for the automated detection of osteolytic PALs in CBCT data sets. METHODS: CBCT data sets from routine clinical operations (maxilla, mandible, or both) performed from January to October 2020 were retrospectively screened and selected. A 2-step approach was used for automatic PAL detection. First, tooth localization and identification were performed using the SpatialConfiguration-Net based on heatmap regression. Second, binary segmentation of lesions was performed using a modified U-Net architecture. A total of 144 CBCT images were used to train and test the networks. The method was evaluated using the 4-fold cross-validation technique. RESULTS: The success detection rate of the tooth localization network ranged between 72.6% and 97.3%, whereas the sensitivity and specificity values of lesion detection were 97.1% and 88.0%, respectively. CONCLUSIONS: Although PALs showed variations in appearance, size, and shape in the CBCT data set and a high imbalance existed between teeth with and without PALs, the proposed fully automated method provided excellent results compared with related literature.

Deep Metric Learning with BIER: Boosting Independent Embeddings Robustly.

Learning similarity functions between image pairs with deep neural networks yields highly correlated activations of embeddings. In this work, we show how to improve the robustness of such embeddings by exploiting the independence within ensembles. To this end, we divide the last embedding layer of a deep network into an embedding ensemble and formulate the task of training this ensemble as an online gradient boosting problem. Each learner receives a reweighted training sample from the previous learners. Further, we propose two loss functions which increase the diversity in our ensemble. These loss functions can be applied either for weight initialization or during training. Together, our contributions leverage large embedding sizes more effectively by significantly reducing correlation of the embedding and consequently increase retrieval accuracy of the embedding. Our method works with any differentiable loss function and does not introduce any additional parameters during test time. We evaluate our metric learning method on image retrieval tasks and show that it improves over state-of-the-art methods on the CUB-200-2011, Cars-196, Stanford Online Products, In-Shop Clothes Retrieval and VehicleID datasets. Therefore, our findings suggest that by dividing deep networks at the end into several smaller and diverse networks, we can significantly reduce overfitting.

Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks.

Differently to semantic segmentation, instance segmentation assigns unique labels to each individual instance of the same object class. In this work, we propose a novel recurrent fully convolutional network architecture for tracking such instance segmentations over time, which is highly relevant, e.g., in biomedical applications involving cell growth and migration. Our network architecture incorporates convolutional gated recurrent units (ConvGRU) into a stacked hourglass network to utilize temporal information, e.g., from microscopy videos. Moreover, we train our network with a novel embedding loss based on cosine similarities, such that the network predicts unique embeddings for every instance throughout videos, even in the presence of dynamic structural changes due to mitosis of cells. To create the final tracked instance segmentations, the pixel-wise embeddings are clustered among subsequent video frames by using the mean shift algorithm. After showing the performance of the instance segmentation on a static in-house dataset of muscle fibers from H&E-stained microscopy images, we also evaluate our proposed recurrent stacked hourglass network regarding instance segmentation and tracking performance on six datasets from the ISBI celltracking challenge, where it delivers state-of-the-art results.

Integrating spatial configuration into heatmap regression based CNNs for landmark localization.

In many medical image analysis applications, only a limited amount of training data is available due to the costs of image acquisition and the large manual annotation effort required from experts. Training recent state-of-the-art machine learning methods like convolutional neural networks (CNNs) from small datasets is a challenging task. In this work on anatomical landmark localization, we propose a CNN architecture that learns to split the localization task into two simpler sub-problems, reducing the overall need for large training datasets. Our fully convolutional SpatialConfiguration-Net (SCN) learns this simplification due to multiplying the heatmap predictions of its two components and by training the network in an end-to-end manner. Thus, the SCN dedicates one component to locally accurate but ambiguous candidate predictions, while the other component improves robustness to ambiguities by incorporating the spatial configuration of landmarks. In our extensive experimental evaluation, we show that the proposed SCN outperforms related methods in terms of landmark localization error on a variety of size-limited 2D and 3D landmark localization datasets, i.e., hand radiographs, lateral cephalograms, hand MRIs, and spine CTs.

Integrated computer-aided forensic case analysis, presentation, and documentation based on multimodal 3D data.

Three-dimensional (3D) crime scene documentation using 3D scanners and medical imaging modalities like computed tomography (CT) and magnetic resonance imaging (MRI) are increasingly applied in forensic casework. Together with digital photography, these modalities enable comprehensive and non-invasive recording of forensically relevant information regarding injuries/pathologies inside the body and on its surface. Furthermore, it is possible to capture traces and items at crime scenes. Such digitally secured evidence has the potential to similarly increase case understanding by forensic experts and non-experts in court. Unlike photographs and 3D surface models, images from CT and MRI are not self-explanatory. Their interpretation and understanding requires radiological knowledge. Findings in tomography data must not only be revealed, but should also be jointly studied with all the 2D and 3D data available in order to clarify spatial interrelations and to optimally exploit the data at hand. This is technically challenging due to the heterogeneous data representations including volumetric data, polygonal 3D models, and images. This paper presents a novel computer-aided forensic toolbox providing tools to support the analysis, documentation, annotation, and illustration of forensic cases using heterogeneous digital data. Conjoint visualization of data from different modalities in their native form and efficient tools to visually extract and emphasize findings help experts to reveal unrecognized correlations and thereby enhance their case understanding. Moreover, the 3D case illustrations created for case analysis represent an efficient means to convey the insights gained from case analysis to forensic non-experts involved in court proceedings like jurists and laymen. The capability of the presented approach in the context of case analysis, its potential to speed up legal procedures and to ultimately enhance legal certainty is demonstrated by introducing a number of representative forensic cases.

Towards automatic bone age estimation from MRI: localization of 3D anatomical landmarks.

Bone age estimation (BAE) is an important procedure in forensic practice which recently has seen a shift in attention from X-ray to MRI based imaging. To automate BAE from MRI, localization of the joints between hand bones is a crucial first step, which is challenging due to anatomical variations, different poses and repeating structures within the hand. We propose a landmark localization algorithm using multiple random regression forests, first analyzing the shape of the hand from information of the whole image, thus implicitly modeling the global landmark configuration, followed by a refinement based on more local information to increase prediction accuracy. We are able to clearly outperform related approaches on our dataset of 60 T1-weighted MR images, achieving a mean landmark localization error of 1.4 ± 1.5mm, while having only 0.25% outliers with an error greater than 10mm.

Fully automatic bone age estimation from left hand MR images.

There has recently been an increased demand in bone age estimation (BAE) of living individuals and human remains in legal medicine applications. A severe drawback of established BAE techniques based on X-ray images is radiation exposure, since many countries prohibit scanning involving ionizing radiation without diagnostic reasons. We propose a completely automated method for BAE based on volumetric hand MRI images. On our database of 56 male caucasian subjects between 13 and 19 years, we are able to estimate the subjects age with a mean difference of 0.85 ± 0.58 years compared to the chronological age, which is in line with radiologist results using established radiographic methods. We see this work as a promising first step towards a novel MRI based bone age estimation system, with the key benefits of lacking exposure to ionizing radiation and higher accuracy due to exploitation of volumetric data.

Intuitive presentation of clinical forensic data using anonymous and person-specific 3D reference manikins.

The increasing use of CT/MR devices in forensic analysis motivates the need to present forensic findings from different sources in an intuitive reference visualization, with the aim of combining 3D volumetric images along with digital photographs of external findings into a 3D computer graphics model. This model allows a comprehensive presentation of forensic findings in court and enables comparative evaluation studies correlating data sources. The goal of this work was to investigate different methods to generate anonymous and patient-specific 3D models which may be used as reference visualizations. The issue of registering 3D volumetric as well as 2D photographic data to such 3D models is addressed to provide an intuitive context for injury documentation from arbitrary modalities. We present an image processing and visualization work-flow, discuss the major parts of this work-flow, compare the different investigated reference models, and show a number of cases studies that underline the suitability of the proposed work-flow for presenting forensically relevant information in 3D visualizations.

Structured Labels in Random Forests for Semantic Labelling and Object Detection.

Ensembles of randomized decision trees, known as Random Forests, have become a valuable machine learning tool for addressing many computer vision problems. Despite their popularity, few works have tried to exploit contextual and structural information in random forests in order to improve their performance. In this paper, we propose a simple and effective way to integrate contextual information in random forests, which is typically reflected in the structured output space of complex problems like semantic image labelling. Our paper has several contributions: We show how random forests can be augmented with structured label information and be used to deliver structured low-level predictions. The learning task is carried out by employing a novel split function evaluation criterion that exploits the joint distribution observed in the structured label space. This allows the forest to learn typical label transitions between object classes and avoid locally implausible label configurations. We provide two approaches for integrating the structured output predictions obtained at a local level from the forest into a concise, global, semantic labelling. We integrate our new ideas also in the Hough-forest framework with the view of exploiting contextual information at the classification level to improve the performance on the task of object detection. Finally, we provide experimental evidence for the effectiveness of our approach on different tasks: Semantic image labelling on the challenging MSRCv2 and CamVid databases, reconstruction of occluded handwritten Chinese characters on the Kaist database and pedestrian detection on the TU Darmstadt databases.

Global localization of 3D anatomical structures by pre-filtered Hough forests and discrete optimization.

The accurate localization of anatomical landmarks is a challenging task, often solved by domain specific approaches. We propose a method for the automatic localization of landmarks in complex, repetitive anatomical structures. The key idea is to combine three steps: (1) a classifier for pre-filtering anatomical landmark positions that (2) are refined through a Hough regression model, together with (3) a parts-based model of the global landmark topology to select the final landmark positions. During training landmarks are annotated in a set of example volumes. A classifier learns local landmark appearance, and Hough regressors are trained to aggregate neighborhood information to a precise landmark coordinate position. A non-parametric geometric model encodes the spatial relationships between the landmarks and derives a topology which connects mutually predictive landmarks. During the global search we classify all voxels in the query volume, and perform regression-based agglomeration of landmark probabilities to highly accurate and specific candidate points at potential landmark locations. We encode the candidates' weights together with the conformity of the connecting edges to the learnt geometric model in a Markov Random Field (MRF). By solving the corresponding discrete optimization problem, the most probable location for each model landmark is found in the query volume. We show that this approach is able to consistently localize the model landmarks despite the complex and repetitive character of the anatomical structures on three challenging data sets (hand radiographs, hand CTs, and whole body CTs), with a median localization error of 0.80 mm, 1.19 mm and 2.71 mm, respectively.

Forensic-case analysis: from 3D imaging to interactive visualization.

An interactive framework prepares raw computed-tomography and magnetic-resonance-imaging scans for courtroom presentations. The framework makes use of combined computer graphics and computer vision techniques to enable a forensic case analysis workflow.

Neural process reconstruction from sparse user scribbles.

We present a novel semi-automatic method for segmenting neural processes in large, highly anisotropic EM (electron microscopy) image stacks. Our method takes advantage of sparse scribble annotations provided by the user to guide a 3D variational segmentation model, thereby allowing our method to globally optimally enforce 3D geometric constraints on the segmentation. Moreover, we leverage a novel algorithm for propagating segmentation constraints through the image stack via optimal volumetric pathways, thereby allowing our method to compute highly accurate 3D segmentations from very sparse user input. We evaluate our method by reconstructing 16 neural processes in a 1024 x 1024 x 50 nanometer-scale EM image stack of a mouse hippocampus. We demonstrate that, on average, our method is 68% more accurate than previous state-of-the-art semi-automatic methods.

Rapid Communication with a "P300" Matrix Speller Using Electrocorticographic Signals (ECoG).

A brain-computer interface (BCI) can provide a non-muscular communication channel to severely disabled people. One particular realization of a BCI is the P300 matrix speller that was originally described by Farwell and Donchin (1988). This speller uses event-related potentials (ERPs) that include the P300 ERP. All previous online studies of the P300 matrix speller used scalp-recorded electroencephalography (EEG) and were limited in their communication performance to only a few characters per minute. In our study, we investigated the feasibility of using electrocorticographic (ECoG) signals for online operation of the matrix speller, and determined associated spelling rates. We used the matrix speller that is implemented in the BCI2000 system. This speller used ECoG signals that were recorded from frontal, parietal, and occipital areas in one subject. This subject spelled a total of 444 characters in online experiments. The results showed that the subject sustained a rate of 17 characters/min (i.e., 69 bits/min), and achieved a peak rate of 22 characters/min (i.e., 113 bits/min). Detailed analysis of the results suggests that ERPs over visual areas (i.e., visual evoked potentials) contribute significantly to the performance of the matrix speller BCI system. Our results also point to potential reasons for the apparent advantages in spelling performance of ECoG compared to EEG. Thus, with additional verification in more subjects, these results may further extend the communication options for people with serious neuromuscular disabilities.

Stereoscopic motion analysis in densely packed clusters: 3D analysis of the shimmering behaviour in Giant honey bees.

BACKGROUND: The detailed interpretation of mass phenomena such as human escape panic or swarm behaviour in birds, fish and insects requires detailed analysis of the 3D movements of individual participants. Here, we describe the adaptation of a 3D stereoscopic imaging method to measure the positional coordinates of individual agents in densely packed clusters. The method was applied to study behavioural aspects of shimmering in Giant honeybees, a collective defence behaviour that deters predatory wasps by visual cues, whereby individual bees flip their abdomen upwards in a split second, producing Mexican wave-like patterns. RESULTS: Stereoscopic imaging provided non-invasive, automated, simultaneous, in-situ 3D measurements of hundreds of bees on the nest surface regarding their thoracic position and orientation of the body length axis. Segmentation was the basis for the stereo matching, which defined correspondences of individual bees in pairs of stereo images. Stereo-matched "agent bees" were re-identified in subsequent frames by the tracking procedure and triangulated into real-world coordinates. These algorithms were required to calculate the three spatial motion components (dx: horizontal, dy: vertical and dz: towards and from the comb) of individual bees over time. CONCLUSIONS: The method enables the assessment of the 3D positions of individual Giant honeybees, which is not possible with single-view cameras. The method can be applied to distinguish at the individual bee level active movements of the thoraces produced by abdominal flipping from passive motions generated by the moving bee curtain. The data provide evidence that the z-deflections of thoraces are potential cues for colony-intrinsic communication. The method helps to understand the phenomenon of collective decision-making through mechanoceptive synchronization and to associate shimmering with the principles of wave propagation. With further, minor modifications, the method could be used to study aspects of other mass phenomena that involve active and passive movements of individual agents in densely packed clusters.

Localization and trajectory reconstruction in surveillance cameras with nonoverlapping views.

This paper proposes a method that localizes two surveillance cameras and simultaneously reconstructs object trajectories in 3D space. The method is an extension of the Direct Reference Plane method, which formulates the localization and the reconstruction as a system of linear equations that is globally solvable by Singular Value Decomposition. The method's assumptions are static synchronized cameras, smooth trajectories, known camera internal parameters, and the rotation between the cameras in a world coordinate system. The paper describes the method in the context of self-calibrating cameras, where the internal parameters and the rotation can be jointly obtained assuming a man-made scene with orthogonal structures. Experiments with synthetic and real--image data show that the method can recover the camera centers with an error less than half a meter even in the presence of a 4 meter gap between the fields of view.

DT-MRI based computation of collagen fiber deformation in human articular cartilage: a feasibility study.

Accurate techniques for simulating the deformation of soft biological tissues are an increasingly valuable tool in many areas of biomechanical analysis and medical image computing. To model the complex morphology and response of articular cartilage, a hyperviscoelastic (dispersed) fiber-reinforced constitutive model is employed to complete two specimen-specific finite element (FE) simulations of an indentation experiment, with and without considering fiber dispersion. Ultra-high field Diffusion Tensor Magnetic Resonance Imaging (17.6 T DT-MRI) is performed on a specimen of human articular cartilage before and after indentation to approximately 20% compression. Based on this DT-MRI data, we detail a novel FE approach to determine the geometry (edge detection from first eigenvalue), the meshing (semi-automated smoothing of DTI measurement voxels), and the fiber structural input (estimated principal fiber direction and dispersion). The global and fiber fabric deformations of both the un-dispersed and dispersed fiber models provide a satisfactory match to that estimated experimentally. In both simulations, the fiber fabric in the superficial and middle zones becomes more aligned with the articular surface, although the dispersed model appears more consistent with the literature. In the future, a multi-disciplinary combination of DT-MRI and numerical simulation will allow the functional state of articular cartilage to be determined in vivo.

Segmentation of interwoven 3d tubular tree structures utilizing shape priors and graph cuts.

The segmentation of tubular tree structures like vessel systems in volumetric datasets is of vital interest for many medical applications. We present a novel approach that allows to simultaneously separate and segment multiple interwoven tubular tree structures. The algorithm consists of two main processing steps. First, the tree structures are identified and corresponding shape priors are generated by using a bottom-up identification of tubular objects combined with a top-down grouping of these objects into complete tree structures. The grouping step allows us to separate interwoven trees and to handle local disturbances. Second, the generated shape priors are utilized for the intrinsic segmentation of the different tubular systems to avoid leakage or undersegmentation in locally disturbed regions. We have evaluated our method on phantom and different clinical CT datasets and demonstrated its ability to correctly obtain/separate different tree structures, accurately determine the surface of tubular tree structures, and robustly handle noise, disturbances (e.g., tumors), and deviations from cylindrical tube shapes like for example aneurysms.

A phenomenological approach toward patient-specific computational modeling of articular cartilage including collagen fiber tracking.

To model the cartilage morphology and the material response, a phenomenological and patient-specific simulation approach incorporating the collagen fiber fabric is proposed. Cartilage tissue response is nearly isochoric and time-dependent under physiological pressure levels. Hence, a viscoelastic constitutive model capable of reproducing finite strains is employed, while the time-dependent deformation change is purely isochoric. The model incorporates seven material parameters, which all have a physical interpretation. To calibrate the model and facilitate further analysis, five human cartilage specimens underwent a number of tests. A series of magnetic resonance imaging (MRI) sequences is taken, next the cartilage surface is imaged, then mechanical indentation tests are completed at 2-7 different locations per sample, resulting in force/displacement data over time, and finally, the underlying bone surface is imaged. Imaging and mechanical testing are performed with a custom-built robotics-based testing device. Stereo reconstruction of the cartilage and subchondral bone surface is employed, which, together with the proposed constitutive model, led to specimen-specific finite element simulations of the mechanical indentation tests. The force-time response of 23 such indentation experiment simulations is optimized to estimate the mean material parameters and corresponding standard deviations. The model is capable of reproducing the deformation behavior of human articular cartilage in the physiological loading domain, as demonstrated by the good agreement between the experiment and numerical results (R(2)=0.95+/-0.03, mean+/-standard deviation of force-time response for 23 indentation tests). To address validation, a sevenfold cross-validation experiment is performed on the 21 experiments representing healthy cartilage. To quantify the predictive error, the mean of the absolute force differences and Pearson's correlation coefficient are both calculated. Deviations in the mean absolute difference, normalized by the peak force, range from 4% to 90%, with 40+/-25% (M+/-SD). The correlation coefficients across all predictions have a minimum of 0.939, and a maximum of 0.993 with 0.975+/-0.013 (M+/-SD), which demonstrates an excellent match of the decay characteristics. A novel feature of the proposed method is 3D sample-specific numerical tracking of the fiber fabric deformation under general loading. This feature is demonstrated by comparing the estimated fiber fabric deformation with recently published experimental data determined by diffusion tensor MRI. The proposed approach is efficient enough to enable large-scale 3D contact simulations of knee joint loading in simulations with accurate joint geometries.

A practical procedure for real-time functional mapping of eloquent cortex using electrocorticographic signals in humans.

Functional mapping of eloquent cortex is often necessary prior to invasive brain surgery, but current techniques that derive this mapping have important limitations. In this article, we demonstrate the first comprehensive evaluation of a rapid, robust, and practical mapping system that uses passive recordings of electrocorticographic signals. This mapping procedure is based on the BCI2000 and SIGFRIED technologies that we have been developing over the past several years. In our study, we evaluated 10 patients with epilepsy from four different institutions and compared the results of our procedure with the results derived using electrical cortical stimulation (ECS) mapping. The results show that our procedure derives a functional motor cortical map in only a few minutes. They also show a substantial concurrence with the results derived using ECS mapping. Specifically, compared with ECS maps, a next-neighbor evaluation showed no false negatives, and only 0.46 and 1.10% false positives for hand and tongue maps, respectively. In summary, we demonstrate the first comprehensive evaluation of a practical and robust mapping procedure that could become a new tool for planning of invasive brain surgeries.