Influence of training and expertise on deep neural network attention and human attention during a medical image classification task.

In many different domains, experts can make complex decisions after glancing very briefly at an image. However, the perceptual mechanisms underlying expert performance are still largely unknown. Recently, several machine learning algorithms have been shown to outperform human experts in specific tasks. But these algorithms often behave as black boxes and their information processing pipeline remains unknown. This lack of transparency and interpretability is highly problematic in applications involving human lives, such as health care. One way to "open the black box" is to compute an artificial attention map from the model, which highlights the pixels of the input image that contributed the most to the model decision. In this work, we directly compare human visual attention to machine visual attention when performing the same visual task. We have designed a medical diagnosis task involving the detection of lesions in small bowel endoscopic images. We collected eye movements from novices and gastroenterologist experts while they classified medical images according to their relevance for Crohn's disease diagnosis. We trained three state-of-the-art deep learning models on our carefully labeled dataset. Both humans and machine performed the same task. We extracted artificial attention with six different post hoc methods. We show that the model attention maps are significantly closer to human expert attention maps than to novices', especially for pathological images. As the model gets trained and its performance gets closer to the human experts, the similarity between model and human attention increases. Through the understanding of the similarities between the visual decision-making process of human experts and deep neural networks, we hope to inform both the training of new doctors and the architecture of new algorithms.

Influence of inputs for bone lesion segmentation in longitudinal 18F-FDG PET/CT imaging studies.

In metastatic breast cancer, bone metastases are prevalent and associated with multiple complications. Assessing their response to treatment is therefore crucial. Most deep learning methods segment or detect lesions on a single acquisition while only a few focus on longitudinal studies. In this work, 45 patients with baseline (BL) and follow-up (FU) images recruited in the context of the EPICUREseinmeta study were analyzed. The aim was to determine if a network trained for a particular timepoint can generalize well to another one, and to explore different improvement strategies. Four networks based on the same 3D U-Net framework to segment bone lesions on BL and FU images were trained with different strategies and compared. These four networks were trained 1) only with BL images 2) only with FU images 3) with both BL and FU images 4) only with FU images but with BL images and bone lesion segmentations registered as input channels. With the obtained segmentations, we computed the PET Bone Index (PBI) which assesses the bone metastases burden of patients and we analyzed its potential for treatment response evaluation. Dice scores of 0.53, 0.55, 0.59 and 0.62 were respectively obtained on FU acquisitions. The under-performance of the first and third networks may be explained by the lower SUV uptake due to treatment response in FU images compared to BL images. The fourth network gives better results than the second network showing that the addition of BL PET images and bone lesion segmentations as prior knowledge has its importance. With an AUC of 0.86, the difference of PBI between two acquisitions could be used to assess treatment response. Clinical relevance- To assess the response to treatment of bone metastases, it is crucial to detect and segment them on several acquisitions from a same patient. We proposed a completely automatic method to detect and segment these metastases on longitudinal 18F-FDG PET/CT images in the context of metastatic breast cancer. We also proposed an automatic PBI to quantitatively assess the evolution of the bone metastases burden of patient and to automatically evaluate their response to treatment.

A time-lapse embryo dataset for morphokinetic parameter prediction.

One of the most common treatments for infertile couples is In Vitro Fertilization (IVF). It consists of controlled ovarian hyperstimulation, followed by ovum pickup, fertilization, and embryo culture for 2-6 days under controlled environmental conditions, leading to intrauterine transfer or freezing of embryos identified as having a good implantation potential by embryologists. To allow continuous monitoring of embryo development, Time-lapse imaging incubators (TLI) were first released in the IVF market around 2010. This time-lapse technology provides a dynamic overview of embryonic in vitro development by taking photographs of each embryo at regular intervals throughout its development. TLI appears to be the most promising solution to improve embryo quality assessment methods, and subsequently the clinical efficiency of IVF. In particular, the unprecedented high volume of high-quality images produced by TLI systems has already been leveraged using modern Artificial Intelligence (AI) methods, like deep learning (DL). An important limitation to the development of AI-based solutions for IVF is the absence of a public reference dataset to train and evaluate deep learning (DL) models. In this work, we describe a fully annotated dataset of 704 TLI videos of developing embryos with all 7 focal planes available, for a total of 2,4M images. Of note, we propose highly detailed annotations with 16 different development phases, including early cell division phases, but also late cell divisions, phases after morulation, and very early phases, which have never been used before. This is the first public dataset that will allow the community to evaluate morphokinetic models and the first step towards deep learning-powered IVF. We postulate that this dataset will help improve the overall performance of DL approaches on time-lapse videos of embryo development, ultimately benefiting infertile patients with improved clinical success rates.

Applications of artificial intelligence and machine learning for the hip and knee surgeon: current state and implications for the future.

BACKGROUND: Artificial Intelligence (AI)/Machine Learning (ML) applications have been proven efficient to improve diagnosis, to stratify risk, and to predict outcomes in many respective medical specialties, including in orthopaedics. CHALLENGES AND DISCUSSION: Regarding hip and knee reconstruction surgery, AI/ML have not made it yet to clinical practice. In this review, we present sound AI/ML applications in the field of hip and knee degenerative disease and reconstruction. From osteoarthritis (OA) diagnosis and prediction of its advancement, clinical decision-making, identification of hip and knee implants to prediction of clinical outcome and complications following a reconstruction procedure of these joints, we report how AI/ML systems could facilitate data-driven personalized care for our patients.

Multi-expert annotation of Crohn's disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network.

Background and study aims Computer-aided diagnostic tools using deep neural networks are efficient for detection of lesions in endoscopy but require a huge number of images. The impact of the quality of annotation has not been tested yet. Here we describe a multi-expert annotated dataset of images extracted from capsules from Crohn's disease patients and the impact of the quality of annotations on the accuracy of a recurrent attention neural network. Methods Images of capsule were annotated by a reader first and then reviewed by three experts in inflammatory bowel disease. Concordance analysis between experts was evaluated by Fleiss' kappa and all the discordant images were, again, read by all the endoscopists to obtain a consensus annotation. A recurrent attention neural network developed for the study was tested before and after the consensus annotation. Available neural networks (ResNet and VGGNet) were also tested under the same conditions. Results The final dataset included 3498 images with 2124 non-pathological (60.7 %), 1360 pathological (38.9 %), and 14 (0.4 %) inconclusive. Agreement of the experts was good for distinguishing pathological and non-pathological images with a kappa of 0.79 ( P  < 0.0001). The accuracy of our classifier and the available neural networks increased after the consensus annotation with a precision of 93.7 %, sensitivity of 93 %, and specificity of 95 %. Conclusions The accuracy of the neural network increased with improved annotations, suggesting that the number of images needed for the development of these systems could be diminished using a well-designed dataset.

Automatic Segmentation of Metastatic Breast Cancer Lesions on 18F-FDG PET/CT Longitudinal Acquisitions for Treatment Response Assessment.

Metastatic breast cancer patients receive lifelong medication and are regularly monitored for disease progression. The aim of this work was to (1) propose networks to segment breast cancer metastatic lesions on longitudinal whole-body PET/CT and (2) extract imaging biomarkers from the segmentations and evaluate their potential to determine treatment response. Baseline and follow-up PET/CT images of 60 patients from the EPICUREseinmeta study were used to train two deep-learning models to segment breast cancer metastatic lesions: One for baseline images and one for follow-up images. From the automatic segmentations, four imaging biomarkers were computed and evaluated: SULpeak, Total Lesion Glycolysis (TLG), PET Bone Index (PBI) and PET Liver Index (PLI). The first network obtained a mean Dice score of 0.66 on baseline acquisitions. The second network obtained a mean Dice score of 0.58 on follow-up acquisitions. SULpeak, with a 32% decrease between baseline and follow-up, was the biomarker best able to assess patients' response (sensitivity 87%, specificity 87%), followed by TLG (43% decrease, sensitivity 73%, specificity 81%) and PBI (8% decrease, sensitivity 69%, specificity 69%). Our networks constitute promising tools for the automatic segmentation of lesions in patients with metastatic breast cancer allowing treatment response assessment with several biomarkers.

Recovering missing slices of the discrete Fourier transform using Ghosts.

The discrete Fourier transform (DFT) underpins the solution to many inverse problems commonly possessing missing or unmeasured frequency information. This incomplete coverage of the Fourier space always produces systematic artifacts called Ghosts. In this paper, a fast and exact method for deconvolving cyclic artifacts caused by missing slices of the DFT using redundant image regions is presented. The slices discussed here originate from the exact partitioning of the Discrete Fourier Transform (DFT) space, under the projective Discrete Radon Transform, called the discrete Fourier slice theorem. The method has a computational complexity of O(n log(2) n) (for an n=N×N image) and is constructed from a new cyclic theory of Ghosts. This theory is also shown to unify several aspects of work done on Ghosts over the past three decades. This paper concludes with an application to fast, exact, non-iterative image reconstruction from a highly asymmetric set of rational angle projections that give rise to sets of sparse slices within the DFT.

Joint source-channel coding: secured and progressive transmission of compressed medical images on the Internet.

The joint source-channel coding system proposed in this paper has two aims: lossless compression with a progressive mode and the integrity of medical data, which takes into account the priorities of the image and the properties of a network with no guaranteed quality of service. In this context, the use of scalable coding, locally adapted resolution (LAR) and a discrete and exact Radon transform, known as the Mojette transform, meets this twofold requirement. In this paper, details of this joint coding implementation are provided as well as a performance evaluation with respect to the reference CALIC coding and to unequal error protection using Reed-Solomon codes.