Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.

Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, appearance, and location makes the detection of polyps challenging. Moreover, colonoscopy surveillance and removal of polyps are highly operator-dependent procedures and occur in a highly complex organ topology. There exists a high missed detection rate and incomplete removal of colonic polyps. To assist in clinical procedures and reduce missed rates, automated methods for detecting and segmenting polyps using machine learning have been achieved in past years. However, the major drawback in most of these methods is their ability to generalise to out-of-sample unseen datasets from different centres, populations, modalities, and acquisition systems. To test this hypothesis rigorously, we, together with expert gastroenterologists, curated a multi-centre and multi-population dataset acquired from six different colonoscopy systems and challenged the computational expert teams to develop robust automated detection and segmentation methods in a crowd-sourcing Endoscopic computer vision challenge. This work put forward rigorous generalisability tests and assesses the usability of devised deep learning methods in dynamic and actual clinical colonoscopy procedures. We analyse the results of four top performing teams for the detection task and five top performing teams for the segmentation task. Our analyses demonstrate that the top-ranking teams concentrated mainly on accuracy over the real-time performance required for clinical applicability. We further dissect the devised methods and provide an experiment-based hypothesis that reveals the need for improved generalisability to tackle diversity present in multi-centre datasets and routine clinical procedures.

Photometric and Monte-Carlo modeling unified approach for the calculation of spatially-resolved correction coefficients linking simulated and experimental diffuse reflectance spectra.

The estimation of skin optical properties by means of inverse problem solving from spatially resolved diffuse reflectance (SR-DR) spectra is one way to exploit the acquired clinical signals. This method requires the comparison between the experimental spectra collected with a medical device, and spectra generated by the photons transport numerical simulations. This comparison is usually limited to spectral shape due to the absence of intensity standardization of the experimental DR spectra. This study proposes to theoretically (using photometric calculation) and experimentally (from experimental spectra acquired on optical phantom) establish a corrective factor to obtain common intensity unit for experimental and simulated signals.

Implementation of data fusion to increase the efficiency of classification of precancerous skin states using in vivo bimodal spectroscopic technique.

This study presents the results of the classification of diffuse reflectance (DR) spectra and multiexcitation autofluorescence (AF) spectra that were collected in vivo from precancerous and benign skin lesions at three different source detector separation (SDS) values. Spectra processing pipeline consisted of dimensionality reduction, which was performed using principal component analysis (PCA), followed by classification step using such methods as support vector machine (SVM), multilayered perceptron (MLP), linear discriminant analysis (LDA), and random forest (RF). In order to increase the efficiency of lesion classification, several data fusion methods were applied to the classification results: majority voting, stacking, and manual optimization of weights. The results of the study showed that in most of cases the use of data fusion methods increased the average multiclass classification accuracy from 2% up to 4%. The highest accuracy of multiclass classification was obtained using the manual optimization of weights and reached 94.41%.

A multi-centre polyp detection and segmentation dataset for generalisability assessment.

Polyps in the colon are widely known cancer precursors identified by colonoscopy. Whilst most polyps are benign, the polyp's number, size and surface structure are linked to the risk of colon cancer. Several methods have been developed to automate polyp detection and segmentation. However, the main issue is that they are not tested rigorously on a large multicentre purpose-built dataset, one reason being the lack of a comprehensive public dataset. As a result, the developed methods may not generalise to different population datasets. To this extent, we have curated a dataset from six unique centres incorporating more than 300 patients. The dataset includes both single frame and sequence data with 3762 annotated polyp labels with precise delineation of polyp boundaries verified by six senior gastroenterologists. To our knowledge, this is the most comprehensive detection and pixel-level segmentation dataset (referred to as PolypGen) curated by a team of computational scientists and expert gastroenterologists. The paper provides insight into data construction and annotation strategies, quality assurance, and technical validation.

Evaluation and understanding of automated urinary stone recognition methods.

OBJECTIVE: To assess the potential of automated machine-learning methods for recognizing urinary stones in endoscopy. MATERIALS AND METHODS: Surface and section images of 123 urinary calculi (109 ex vivo and 14 in vivo stones) were acquired using ureteroscopes. The stones were more than 85% 'pure'. Six classes of urolithiasis were represented: Groups I (calcium oxalate monohydrate, whewellite), II (calcium oxalate dihydrate, weddellite), III (uric acid), IV (brushite and struvite stones), and V (cystine). The automated stone recognition methods that were developed for this study followed two types of approach: shallow classification methods and deep-learning-based methods. Their sensitivity, specificity and positive predictive value (PPV) were evaluated by simultaneously using stone surface and section images to classify them into one of the main morphological groups (subgroups were not considered in this study). RESULTS: Using shallow methods (based on texture and colour criteria), relatively high sensitivity, specificity and PPV for the six classes were attained: 91%, 90% and 89%, respectively, for whewellite; 99%, 98% and 99% for weddellite; 88%, 89% and 88% for uric acid; 91%, 89% and 90% for struvite; 99%, 99% and 99% for cystine; and 94%, 98% and 99% for brushite. Using deep-learning methods, the sensitivity, specificity and PPV for each of the classes were as follows: 99%, 98% and 97% for whewellite; 98%, 98% and 98% for weddellite; 97%, 98% and 98% for uric acid; 97%, 97% and 96% for struvite; 99%, 99% and 99% for cystine; and 94%, 97% and 98% for brushite. CONCLUSION: Endoscopic stone recognition is challenging, and few urologists have sufficient expertise to achieve a diagnosis performance comparable to morpho-constitutional analysis. This work is a proof of concept that artificial intelligence could be a solution, with promising results achieved for pure stones. Further studies on a larger panel of stones (pure and mixed) are needed to further develop these methods.

Optical biopsy mapping on endoscopic image mosaics with a marker-free probe.

Gastric cancer is the second leading cause of cancer-related deaths worldwide. Early diagnosis significantly increases the chances of survival; therefore, improved assisted exploration and screening techniques are necessary. Previously, we made use of an augmented multi-spectral endoscope by inserting an optical probe into the instrumentation channel. However, the limited field of view and the lack of markings left by optical biopsies on the tissue complicate the navigation and revisit of the suspect areas probed in-vivo. In this contribution two innovative tools are introduced to significantly increase the traceability and monitoring of patients in clinical practice: (i) video mosaicing to build a more comprehensive and panoramic view of large gastric areas; (ii) optical biopsy targeting and registration with the endoscopic images. The proposed optical flow-based mosaicing technique selects images that minimize texture discontinuities and is robust despite the lack of texture and illumination variations. The optical biopsy targeting is based on automatic tracking of a free-marker probe in the endoscopic view using deep learning to dynamically estimate its pose during exploration. The accuracy of pose estimation is sufficient to ensure a precise overlapping of the standard white-light color image and the hyperspectral probe image, assuming that the small target area of the organ is almost flat. This allows the mapping of all spatio-temporally tracked biopsy sites onto the panoramic mosaic. Experimental validations are carried out from videos acquired on patients in hospital. The proposed technique is purely software-based and therefore easily integrable into clinical practice. It is also generic and compatible to any imaging modality that connects to a fiberscope.

Assessing deep learning methods for the identification of kidney stones in endoscopic images.

Knowing the type (i.e., the biochemical composition) of kidney stones is crucial to prevent relapses with an appropriate treatment. During ureteroscopies, kidney stones are fragmented, extracted from the urinary tract, and their composition is determined using a morpho-constitutional analysis. This procedure is time-consuming (the morpho-constitutional analysis results are only available after several weeks) and tedious (the fragment extraction lasts up to an hour). Identifying the kidney stone type only with the in-vivo endoscopic images would allow for the dusting of the fragments and eneable early treatments, while the morpho-constitutional analysis is ready. Only few contributions dealing with the in vivo identification of kidney stones have been published. This paper discusses and compares five classification methods including deep convolutional neural networks (DCNN)-based approaches and traditional (non DCNN-based) ones. Even if the best method is a DCCN approach with a precision and recall of 98% and 97% over four classes, this contribution shows that an XGBoost classifier exploiting well-chosen feature vectors can closely approach the performances of DCNN classifiers for a medical application with a limited number of annotated data.

Anatomical 3D Modeling Using IR Sensors and Radiometric Processing Based on Structure from Motion: Towards a Tool for the Diabetic Foot Diagnosis.

Medical infrared thermography has proven to be a complementary procedure to physiological disorders, such as the diabetic foot. However, the technique remains essentially based on 2D images that display partial anatomy. In this context, a 3D thermal model provides improved visualization and faster inspection. This paper presents a 3D reconstruction method associated with temperature information. The proposed solution is based on a Structure from Motion and Multi-view Stereo approach, exploiting a set of multimodal merged images. The infrared images were obtained by automatically processing the radiometric data to remove thermal interferences, segment the RoI, enhance false-color contrast, and for multimodal co-registration under a controlled environment and a ∆T < 2.6% between the RoI and thermal interferences. The geometric verification accuracy was 77% ± 2%. Moreover, a normalized error was adjusted per sample based on a linear model to compensate for the curvature emissivity (error ≈ 10% near to 90°). The 3D models were displayed with temperature information and interaction controls to observe any point of view. The temperature sidebar values were assigned with information retrieved only from the RoI. The results have proven the feasibility of the 3D multimodal construction to be used as a promising tool in the diagnosis of diabetic foot.

Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy.

The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the robustness of deep learning methods applied to the gastrointestinal tract organs as they can be confused with tissue of interest. EndoCV2020 challenges are designed to address research questions in these remits. In this paper, we present a summary of methods developed by the top 17 teams and provide an objective comparison of state-of-the-art methods and methods designed by the participants for two sub-challenges: i) artefact detection and segmentation (EAD2020), and ii) disease detection and segmentation (EDD2020). Multi-center, multi-organ, multi-class, and multi-modal clinical endoscopy datasets were compiled for both EAD2020 and EDD2020 sub-challenges. The out-of-sample generalization ability of detection algorithms was also evaluated. Whilst most teams focused on accuracy improvements, only a few methods hold credibility for clinical usability. The best performing teams provided solutions to tackle class imbalance, and variabilities in size, origin, modality and occurrences by exploring data augmentation, data fusion, and optimal class thresholding techniques.

Towards an automated classification method for ureteroscopic kidney stone images using ensemble learning.

Urolithiasis is a common disease around the world and its incidence has been growing every year. There are various diagnosis techniques based on kidney stone identification aiming to find the formation cause. However, most of them are time consuming, tedious and expensive. The accuracy of the diagnosis is crucial for the prescription of an appropriate treatment that can eliminate the stones and diminish future relapses. This paper presents two effective supervised learning methods to automate and improve the accuracy of the classification of kidney stones; as well as a dataset consisting of kidney stone images captured with ureteroscopes. In the proposed methods, the image features that are visually exploited by urologists to distinguish the type of kidney stones are analyzed and encoded as vectors. Then, the classification is performed on these feature vectors through Random Forest and ensemble K Nearest Neighbor classifiers. The overall classification accuracy obtained was 89%, outperforming previous methods by more than 10%. The details of the classifier implementation, as well as their performance and accuracy, are presented and discussed. Finally, future work and improvements are proposed.

Two diagnostic criteria of optical spectroscopy for bladder tumor detection: Clinical study using 5-ALA induced fluorescence and mathematical modeling.

BACKGROUND: The study proposes to improve bladder cancer diagnosis by photodynamic diagnosis (PDD) using red-light excitation (632.8 nm) of 5-ALA induced-protoporphyrin IX. Employing 9 patients' bladders, two types of signals were used to improve diagnostic accuracy for malignancy and we also present numerical modeling of the scattering coefficient to provide biological explanation of the results obtained. METHODS: Two modalities of bladder cancer spectral diagnosis are presented: conventional PDD and intensity assessment of the diffusely reflected laser light by fiber-optic spectroscopy. Experiments are done in clinical conditions and as a series of numerical simulations. RESULTS: High-grade cancerous bladder tissues display twice a higher relative fluorescence intensity (mean value 1, n = 9) than healthy (0.39, n = 9), dysplastic (0.44, n = 5) tissues and CIS (0.39, n = 2). The laser back-scattering signal allows to discriminate most effectively high-grade cancerous and dysplastic tissues from normal. Numerical modeling of diffuse reflectance spectra reveals that spectral behavior of the back-scattered light depends on both, nuclear size and nuclear density of tumoral cells. CONCLUSIONS: Unlike the fluorescence signal, where its value is higher in the case of pathological tissues, the tendency of the laser signal to, both, decrease or increase in comparison with the signal from normal urothelium, should be perceived as a sign towards neoplasm. Numerical simulation reveals that such a double-analysis at a multiwavelength mode potentially may be used to provide diagnostic accuracy.

An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy.

We present a comprehensive analysis of the submissions to the first edition of the Endoscopy Artefact Detection challenge (EAD). Using crowd-sourcing, this initiative is a step towards understanding the limitations of existing state-of-the-art computer vision methods applied to endoscopy and promoting the development of new approaches suitable for clinical translation. Endoscopy is a routine imaging technique for the detection, diagnosis and treatment of diseases in hollow-organs; the esophagus, stomach, colon, uterus and the bladder. However the nature of these organs prevent imaged tissues to be free of imaging artefacts such as bubbles, pixel saturation, organ specularity and debris, all of which pose substantial challenges for any quantitative analysis. Consequently, the potential for improved clinical outcomes through quantitative assessment of abnormal mucosal surface observed in endoscopy videos is presently not realized accurately. The EAD challenge promotes awareness of and addresses this key bottleneck problem by investigating methods that can accurately classify, localize and segment artefacts in endoscopy frames as critical prerequisite tasks. Using a diverse curated multi-institutional, multi-modality, multi-organ dataset of video frames, the accuracy and performance of 23 algorithms were objectively ranked for artefact detection and segmentation. The ability of methods to generalize to unseen datasets was also evaluated. The best performing methods (top 15%) propose deep learning strategies to reconcile variabilities in artefact appearance with respect to size, modality, occurrence and organ type. However, no single method outperformed across all tasks. Detailed analyses reveal the shortcomings of current training strategies and highlight the need for developing new optimal metrics to accurately quantify the clinical applicability of methods.

Clinical evaluation of a device providing simultaneous white-light and fluorescence video streams as well as panoramic imaging during fluorescence assisted-transurethral resection of bladder cancer.

The current clinical study is aimed at evaluating the clinical relevance of an innovative device (called CyPaM2 device) that for the first time provides urologists with (i) a panoramic image of the bladder inner wall within the surgery time, and with (ii) a simultaneous (bimodal) display of fluorescence and white-light video streams during the fluorescence assisted-transurethral resection of bladder cancers procedure. The clinical relevance of this CyPaM2 device was evaluated on 10 patients according to three criteria (image quality, fluorescent lesions detection relevance, and ergonomics) compared with a reference medical device. Innovative features displayed by the CyPaM2 device were evaluated without any possible comparison: (i) simultaneous bimodal display of white-light and fluorescence video streams, (ii) remote light control, and (iii) time delay for the panoramic image building. The results highlight the progress to achieve in order to obtain a fully mature device ready for commercialization and the relevance of the innovative features proposed by the CyPaM2 device confirming their interest.

Source separation approach for the analysis of spatially resolved multiply excited autofluorescence spectra during optical clearing of ex vivo skin.

Spatially resolved multiply excited autofluorescence spectroscopy is a valuable optical biopsy technique to investigate skin UV-visible optical properties in vivo in clinics. However, it provides bulk fluorescence signals from which the individual endogenous fluorophore contributions need to be disentangled. Skin optical clearing allows for increasing tissue transparency, thus providing access to more accurate in-depth information. The aim of the present contribution was to study the time changes in skin spatially resolved and multiply excited autofluorescence spectra during skin optical clearing. The latter spectra were acquired on an ex vivo human skin strip lying on a fluorescent gel substrate during 37 minutes of the optical clearing process of a topically applied sucrose-based solution. A Non Negative Matrix Factorization-based blind source separation approach was proposed to unmix skin tissue intrinsic fluorophore contributions and to analyze the time evolution of this mixing throughout the optical clearing process. This spectral unmixing exploited the multidimensionality of the acquired data, i.e., spectra resolved in five excitation wavelengths, four source-to-detector separations, and eight measurement times. Best fitting results between experimental and estimated spectra were obtained for optimal numbers of 3 and 4 sources. These estimated spectral sources exhibited common identifiable shapes of fluorescence emission spectra related to the fluorescent gel substrate and to known skin intrinsic fluorophores matching namely dermis collagen/elastin and epidermis flavins. The time analysis of the fluorophore contributions allowed us to highlight how the clearing process towards the deepest skin layers impacts skin autofluorescence through time, namely with a strongest contribution to the bulk autofluorescence signal of dermis collagen (respectively epidermis flavins) fluorescence at shortest (respectively longest) excitation wavelengths and longest (respectively shortest) source-to-detector separations.

Spatially and spectrally resolved particle swarm optimization for precise optical property estimation using diffuse-reflectance spectroscopy.

This paper presents a new approach to estimate optical properties (absorption and scattering coefficients µa and µs) of biological tissues from spatially-resolved spectroscopy measurements. A Particle Swarm Optimization (PSO)-based algorithm was implemented and firstly modified to deal with spatial and spectral resolutions of the data, and to solve the corresponding inverse problem. Secondly, the optimization was improved by fitting exponential decays to the two best points among all clusters of the "particles" randomly distributed all over the parameter space (µs, µa) of possible solutions. The consequent acceleration of all the groups of particles to the "best" curve leads to significant error decrease in the optical property estimation. The study analyzes the estimated optical property error as a function of the various PSO parameter combinations, and several performance criteria such as the cost-function error and the number of iterations in the algorithms proposed. The final one led to error values between ground truth and estimated values of µs and µa less than 6%.

Optical flow with structure information for epithelial image mosaicing.

Mosaicing of biological tissue surfaces is challenging due to the weak image textures. This contribution presents a mosaicing algorithm based on a robust and accurate variational optical flow scheme. A Riesz pyramid based multiscale approach aims at overcoming the "flattening-out" problem at coarser levels. Moreover, the structure information present in images of epithelial surfaces is incorporated into the data-term to improve the algorithm robustness. The algorithm accuracy is first assessed with simulated sequences and then used for mosaicing standard clinical endoscopic data.

Validation of a digital mammographic unit model for an objective and highly automated clinical image quality assessment.

In mammography, image quality assessment has to be directly related to breast cancer indicator (e.g. microcalcifications) detectability. Recently, we proposed an X-ray source/digital detector (XRS/DD) model leading to such an assessment. This model simulates very realistic contrast-detail phantom (CDMAM) images leading to gold disc (representing microcalcifications) detectability thresholds that are very close to those of real images taken under the simulated acquisition conditions. The detection step was performed with a mathematical observer. The aim of this contribution is to include human observers into the disc detection process in real and virtual images to validate the simulation framework based on the XRS/DD model. Mathematical criteria (contrast-detail curves, image quality factor, etc.) are used to assess and to compare, from the statistical point of view, the cancer indicator detectability in real and virtual images. The quantitative results given in this paper show that the images simulated by the XRS/DD model are useful for image quality assessment in the case of all studied exposure conditions using either human or automated scoring. Also, this paper confirms that with the XRS/DD model the image quality assessment can be automated and the whole time of the procedure can be drastically reduced. Compared to standard quality assessment methods, the number of images to be acquired is divided by a factor of eight.

Computation of realistic virtual phantom images for an objective lesion detectability assessment in digital mammography.

Image quality assessment is required for an optimal use of mammographic units. On the one hand, there are objective image quality assessment methods based on the measurement of technical parameters such as modulation transfer function (MTF), noise power spectrum (NPS) or detection quantum efficiency (DQE) describing performances of digital detectors. These parameters are, however, without direct relationship with lesion detectability in clinical practice. On the other hand, there are image quality assessment methods involving time consuming procedures, but presenting a direct relationship with lesion detectability. This contribution describes an X-ray source/digital detector model leading to the simulation of virtual contrast-detail phantom (CDMAM) images. The virtual image computation method requires the acquisition of only few real images and allows for an objective image quality assessment presenting a direct relationship with lesion detectability. The transfer function of the proposed model takes as input physical parameters (MTF* and noise) measured under clinical conditions on mammographic units. As presented in this contribution, MTF* is a modified MTF taking into account the effects due to X-ray scatter in the breast and magnification. Results obtained with the structural similarity index prove that the simulated images are quite realistic in terms of contrast and noise. Tests using contrast detail curves highlight the fact that the simulated and real images lead to very similar data quality in terms of lesion detectability. Finally, various statistical tests show that quality factors computed for both the simulated images and the real images are very close for the two data sets.

Mosaicing of bladder endoscopic image sequences: distortion calibration and registration algorithm.

Cancers located on the internal wall of bladders can be detected in image sequences acquired with endoscopes. The clinical diagnosis and follow-up can be facilitated by building a unique panoramic image of the bladder with the images acquired from different viewpoints. This process, called image mosaicing, consists of two steps. In the first step, consecutive images are pairwise registered to find the local transformation matrices linking geometrically consecutive images. In the second step, all images are placed in a common and global coordinate system. In this contribution, a mutual information-based similarity measure and a stochastic gradient optimization method were implemented in the registration process. However, the images have to be preprocessed in order to register the data in a robust way. Thus, a simple correction method of the distortions affecting endoscopic images is presented. After the placement of all images in the global coordinate system, the parameters of the local transformation matrices are all adjusted to improve the visual aspect of the panoramic images. Phantoms are used to evaluate the global mosaicing accuracy and the limits of the registration algorithm. The mean distances between ground truth positions in the mosaiced image range typically in 1-3 pixels. Results given for in vivo patient data illustrate the ability of the algorithm to give coherent panoramic images in the case of bladders.