Artificial intelligence and endo-histo-omics: new dimensions of precision endoscopy and histology in inflammatory bowel disease.

Integrating artificial intelligence into inflammatory bowel disease (IBD) has the potential to revolutionise clinical practice and research. Artificial intelligence harnesses advanced algorithms to deliver accurate assessments of IBD endoscopy and histology, offering precise evaluations of disease activity, standardised scoring, and outcome prediction. Furthermore, artificial intelligence offers the potential for a holistic endo-histo-omics approach by interlacing and harmonising endoscopy, histology, and omics data towards precision medicine. The emerging applications of artificial intelligence could pave the way for personalised medicine in IBD, offering patient stratification for the most beneficial therapy with minimal risk. Although artificial intelligence holds promise, challenges remain, including data quality, standardisation, reproducibility, scarcity of randomised controlled trials, clinical implementation, ethical concerns, legal liability, and regulatory issues. The development of standardised guidelines and interdisciplinary collaboration, including policy makers and regulatory agencies, is crucial for addressing these challenges and advancing artificial intelligence in IBD clinical practice and trials.

MICIL: Multiple-Instance Class-Incremental Learning for skin cancer whole slide images.

Artificial intelligence (AI) agents encounter the problem of catastrophic forgetting when they are trained in sequentially with new data batches. This issue poses a barrier to the implementation of AI-based models in tasks that involve ongoing evolution, such as cancer prediction. Moreover, whole slide images (WSI) play a crucial role in cancer management, and their automated analysis has become increasingly popular in assisting pathologists during the diagnosis process. Incremental learning (IL) techniques aim to develop algorithms capable of retaining previously acquired information while also acquiring new insights to predict future data. Deep IL techniques need to address the challenges posed by the gigapixel scale of WSIs, which often necessitates the use of multiple instance learning (MIL) frameworks. In this paper, we introduce an IL algorithm tailored for analyzing WSIs within a MIL paradigm. The proposed Multiple Instance Class-Incremental Learning (MICIL) algorithm combines MIL with class-IL for the first time, allowing for the incremental prediction of multiple skin cancer subtypes from WSIs within a class-IL scenario. Our framework incorporates knowledge distillation and data rehearsal, along with a novel embedding-level distillation, aiming to preserve the latent space at the aggregated WSI level. Results demonstrate the algorithm's effectiveness in addressing the challenge of balancing IL-specific metrics, such as intransigence and forgetting, and solving the plasticity-stability dilemma.

Annotation protocol and crowdsourcing multiple instance learning classification of skin histological images: The CR-AI4SkIN dataset.

Digital Pathology (DP) has experienced a significant growth in recent years and has become an essential tool for diagnosing and prognosis of tumors. The availability of Whole Slide Images (WSIs) and the implementation of Deep Learning (DL) algorithms have paved the way for the appearance of Artificial Intelligence (AI) systems that support the diagnosis process. These systems require extensive and varied data for their training to be successful. However, creating labeled datasets in histopathology is laborious and time-consuming. We have developed a crowdsourcing-multiple instance labeling/learning protocol that is applied to the creation and use of the CR-AI4SkIN dataset.2 CR-AI4SkIN contains 271 WSIs of 7 Cutaneous Spindle Cell (CSC) neoplasms with expert and non-expert labels at region and WSI levels. It is the first dataset of these types of neoplasms made available. The regions selected by the experts are used to learn an automatic extractor of Regions of Interest (ROIs) from WSIs. To produce the embedding of each WSI, the representations of patches within the ROIs are obtained using a contrastive learning method, and then combined. Finally, they are fed to a Gaussian process-based crowdsourcing classifier, which utilizes the noisy non-expert WSI labels. We validate our crowdsourcing-multiple instance learning method in the CR-AI4SkIN dataset, addressing a binary classification problem (malign vs. benign). The proposed method obtains an F1 score of 0.7911 on the test set, outperforming three widely used aggregation methods for crowdsourcing tasks. Furthermore, our crowdsourcing method also outperforms the supervised model with expert labels on the test set (F1-score = 0.6035). The promising results support the proposed crowdsourcing multiple instance learning annotation protocol. It also validates the automatic extraction of interest regions and the use of contrastive embedding and Gaussian process classification to perform crowdsourcing classification tasks.

Durable 3D murine ex vivo retina glaucoma models for optical coherence tomography.

Durable and standardized phantoms with optical properties similar to native healthy and disease-like biological tissues are essential tools for the development, performance testing, calibration and comparison of label-free high-resolution optical coherence tomography (HR-OCT) systems. Available phantoms are based on artificial materials and reflect thus only partially ocular properties. To address this limitation, we have performed investigations on the establishment of durable tissue phantoms from ex vivo mouse retina for enhanced reproduction of in vivo structure and complexity. In a proof-of-concept study, we explored the establishment of durable 3D models from dissected mouse eyes that reproduce the properties of normal retina structures and tissue with glaucoma-like layer thickness alterations. We explored different sectioning and preparation procedures for embedding normal and N-methyl-D-aspartate (NMDA)-treated mouse retina in transparent gel matrices and epoxy resins, to generate durable three-dimensional tissue models. Sample quality and reproducibility were quantified by thickness determination of the generated layered structures utilizing computer-assisted segmentation of OCT B-scans that were acquired with a commercial HR-OCT system at a central wavelength of 905 nm and analyzed with custom build software. Our results show that the generated 3D models feature thin biological layers close to current OCT resolution limits and glaucoma-like tissue alterations that are suitable for reliable HR-OCT performance characterization. The comparison of data from resin-embedded tissue with native murine retina in gels demonstrates that by utilization of appropriate preparation protocols, highly stable samples with layered structures equivalent to native tissues can be fabricated. The experimental data demonstrate our concept as a promising approach toward the fabrication of durable biological 3D models suitable for high-resolution OCT system performance characterization supporting the development of optimized instruments for ophthalmology applications.

A Spitzoid Tumor dataset with clinical metadata and Whole Slide Images for Deep Learning models.

Spitzoid tumors (ST) are a group of melanocytic tumors of high diagnostic complexity. Since 1948, when Sophie Spitz first described them, the diagnostic uncertainty remains until now, especially in the intermediate category known as Spitz tumor of unknown malignant potential (STUMP) or atypical Spitz tumor. Studies developing deep learning (DL) models to diagnose melanocytic tumors using whole slide imaging (WSI) are scarce, and few used ST for analysis, excluding STUMP. To address this gap, we introduce SOPHIE: the first ST dataset with WSIs, including labels as benign, malignant, and atypical tumors, along with the clinical information of each patient. Additionally, we explain two DL models implemented as validation examples using this database.

ProGleason-GAN: Conditional progressive growing GAN for prostatic cancer Gleason grade patch synthesis.

BACKGROUND AND OBJECTIVE: Prostate cancer is one of the most common diseases affecting men. The main diagnostic and prognostic reference tool is the Gleason scoring system. An expert pathologist assigns a Gleason grade to a sample of prostate tissue. As this process is very time-consuming, some artificial intelligence applications were developed to automatize it. The training process is often confronted with insufficient and unbalanced databases which affect the generalisability of the models. Therefore, the aim of this work is to develop a generative deep learning model capable of synthesising patches of any selected Gleason grade to perform data augmentation on unbalanced data and test the improvement of classification models. METHODOLOGY: The methodology proposed in this work consists of a conditional Progressive Growing GAN (ProGleason-GAN) capable of synthesising prostate histopathological tissue patches by selecting the desired Gleason Grade cancer pattern in the synthetic sample. The conditional Gleason Grade information is introduced into the model through the embedding layers, so there is no need to add a term to the Wasserstein loss function. We used minibatch standard deviation and pixel normalisation to improve the performance and stability of the training process. RESULTS: The reality assessment of the synthetic samples was performed with the Frechet Inception Distance (FID). We obtained an FID metric of 88.85 for non-cancerous patterns, 81.86 for GG3, 49.32 for GG4 and 108.69 for GG5 after post-processing stain normalisation. In addition, a group of expert pathologists was selected to perform an external validation of the proposed framework. Finally, the application of our proposed framework improved the classification results in SICAPv2 dataset, proving its effectiveness as a data augmentation method. CONCLUSIONS: ProGleason-GAN approach combined with a stain normalisation post-processing provides state-of-the-art results regarding Frechet's Inception Distance. This model can synthesise samples of non-cancerous patterns, GG3, GG4 or GG5. The inclusion of conditional information about the Gleason grade during the training process allows the model to select the cancerous pattern in a synthetic sample. The proposed framework can be used as a data augmentation method.

Labeling confidence for uncertainty-aware histology image classification.

Deep learning-based models applied to digital pathology require large, curated datasets with high-quality (HQ) annotations to perform correctly. In many cases, recruiting expert pathologists to annotate large databases is not feasible, and it is necessary to collect additional labeled data with varying label qualities, e.g., pathologists-in-training (henceforth, non-expert annotators). Learning from datasets with noisy labels is more challenging in medical applications since medical imaging datasets tend to have instance-dependent noise and suffer from high inter/intra-observer variability. In this paper, we design an uncertainty-driven labeling strategy with which we generate soft labels from 10 non-expert annotators for multi-class skin cancer classification. Based on this soft annotation, we propose an uncertainty estimation-based framework to handle these noisy labels. This framework is based on a novel formulation using a dual-branch min-max entropy calibration to penalize inexact labels during the training. Comprehensive experiments demonstrate the promising performance of our labeling strategy. Results show a consistent improvement by using soft labels with standard cross-entropy loss during training (∼4.0% F1-score) and increases when calibrating the model with the proposed min-max entropy calibration (∼6.6% F1-score). These improvements are produced at negligible cost, both in terms of annotation and calculation.

Artificial Intelligence Enabled Histological Prediction of Remission or Activity and Clinical Outcomes in Ulcerative Colitis.

BACKGROUND & AIMS: Microscopic inflammation has significant prognostic value in ulcerative colitis (UC); however, its assessment is complex with high interobserver variability. We aimed to develop and validate an artificial intelligence (AI) computer-aided diagnosis system to evaluate UC biopsies and predict prognosis. METHODS: A total of 535 digitalized biopsies (273 patients) were graded according to the PICaSSO Histologic Remission Index (PHRI), Robarts, and Nancy Histological Index. A convolutional neural network classifier was trained to distinguish remission from activity on a subset of 118 biopsies, calibrated on 42 and tested on 375. The model was additionally tested to predict the corresponding endoscopic assessment and occurrence of flares at 12 months. The system output was compared with human assessment. Diagnostic performance was reported as sensitivity, specificity, prognostic prediction through Kaplan-Meier, and hazard ratios of flares between active and remission groups. We externally validated the model in 154 biopsies (58 patients) with similar characteristics but more histologically active patients. RESULTS: The system distinguished histological activity/remission with sensitivity and specificity of 89% and 85% (PHRI), 94% and 76% (Robarts Histological Index), and 89% and 79% (Nancy Histological Index). The model predicted the corresponding endoscopic remission/activity with 79% and 82% accuracy for UC endoscopic index of severity and Paddington International virtual ChromoendoScopy ScOre, respectively. The hazard ratio for disease flare-up between histological activity/remission groups according to pathologist-assessed PHRI was 3.56, and 4.64 for AI-assessed PHRI. Both histology and outcome prediction were confirmed in the external validation cohort. CONCLUSION: We developed and validated an AI model that distinguishes histologic remission/activity in biopsies of UC and predicts flare-ups. This can expedite, standardize, and enhance histologic assessment in practice and trials.

A virtual chromoendoscopy artificial intelligence system to detect endoscopic and histologic activity/remission and predict clinical outcomes in ulcerative colitis.

BACKGROUND: Endoscopic and histological remission (ER, HR) are therapeutic targets in ulcerative colitis (UC). Virtual chromoendoscopy (VCE) improves endoscopic assessment and the prediction of histology; however, interobserver variability limits standardized endoscopic assessment. We aimed to develop an artificial intelligence (AI) tool to distinguish ER/activity, and predict histology and risk of flare from white-light endoscopy (WLE) and VCE videos. METHODS: 1090 endoscopic videos (67 280 frames) from 283 patients were used to develop a convolutional neural network (CNN). UC endoscopic activity was graded by experts using the Ulcerative Colitis Endoscopic Index of Severity (UCEIS) and Paddington International virtual ChromoendoScopy ScOre (PICaSSO). The CNN was trained to distinguish ER/activity on endoscopy videos, and retrained to predict HR/activity, defined according to multiple indices, and predict outcome; CNN and human agreement was measured. RESULTS: The AI system detected ER (UCEIS ≤ 1) in WLE videos with 72 % sensitivity, 87 % specificity, and an area under the receiver operating characteristic curve (AUROC) of 0.85; for detection of ER in VCE videos (PICaSSO ≤ 3), the sensitivity was 79 %, specificity 95 %, and the AUROC 0.94. The prediction of HR was similar between WLE and VCE videos (accuracies ranging from 80 % to 85 %). The model's stratification of risk of flare was similar to that of physician-assessed endoscopy scores. CONCLUSIONS: Our system accurately distinguished ER/activity and predicted HR and clinical outcome from colonoscopy videos. This is the first computer model developed to detect inflammation/healing on VCE using the PICaSSO and the first computer tool to provide endoscopic, histologic, and clinical assessment.

Constrained multiple instance learning for ulcerative colitis prediction using histological images.

BACKGROUND AND OBJECTIVE: Ulcerative colitis (UC) is an inflammatory bowel disease (IBD) affecting the colon and the rectum characterized by a remitting-relapsing course. To detect mucosal inflammation associated with UC, histology is considered the most stringent criteria. In turn, histologic remission (HR) correlates with improved clinical outcomes and has been recently recognized as a desirable treatment target. The leading biomarker for assessing histologic remission is the presence or absence of neutrophils. Therefore, the finding of this cell in specific colon structures indicates that the patient has UC activity. However, no previous studies based on deep learning have been developed to identify UC based on neutrophils detection using whole-slide images (WSI). METHODS: The methodological core of this work is a novel multiple instance learning (MIL) framework with location constraints able to determine the presence of UC activity using WSI. In particular, we put forward an effective way to introduce constraints about positive instances to effectively explore additional weakly supervised information that is easy to obtain and enjoy a significant boost to the learning process. In addition, we propose a new weighted embedding to enlarge the relevance of the positive instances. RESULTS: Extensive experiments on a multi-center dataset of colon and rectum WSIs, PICASSO-MIL, demonstrate that using the location information we can improve considerably the results at WSI-level. In comparison with prior MIL settings, our method allows for 10% improvements in bag-level accuracy. CONCLUSION: Our model, which introduces a new form of constraints, surpass the results achieved from current state-of-the-art methods that focus on the MIL paradigm. Our method can be applied to other histological concerns where the morphological features determining a positive WSI are tiny and similar to others in the image.

PICaSSO Histologic Remission Index (PHRI) in ulcerative colitis: development of a novel simplified histological score for monitoring mucosal healing and predicting clinical outcomes and its applicability in an artificial intelligence system.

Histological remission is evolving as an important treatment target in UC. We aimed to develop a simple histological index, aligned to endoscopy, correlated with clinical outcomes, and suited to apply to an artificial intelligence (AI) system to evaluate inflammatory activity. METHODS: Using a set of 614 biopsies from 307 patients with UC enrolled into a prospective multicentre study, we developed the Paddington International virtual ChromoendoScopy ScOre (PICaSSO) Histologic Remission Index (PHRI). Agreement with multiple other histological indices and validation for inter-reader reproducibility were assessed. Finally, to implement PHRI into a computer-aided diagnosis system, we trained and tested a novel deep learning strategy based on a CNN architecture to detect neutrophils, calculate PHRI and identify active from quiescent UC using a subset of 138 biopsies. RESULTS: PHRI is strongly correlated with endoscopic scores (Mayo Endoscopic Score and UC Endoscopic Index of Severity and PICaSSO) and with clinical outcomes (hospitalisation, colectomy and initiation or changes in medical therapy due to UC flare-up). A PHRI score of 1 could accurately stratify patients' risk of adverse outcomes (hospitalisation, colectomy and treatment optimisation due to flare-up) within 12 months. Our inter-reader agreement was high (intraclass correlation 0.84). Our preliminary AI algorithm differentiated active from quiescent UC with 78% sensitivity, 91.7% specificity and 86% accuracy. CONCLUSIONS: PHRI is a simple histological index in UC, and it exhibits the highest correlation with endoscopic activity and clinical outcomes. A PHRI-based AI system was accurate in predicting histological remission.

Automatic Segmentation of the Retinal Nerve Fiber Layer by Means of Mathematical Morphology and Deformable Models in 2D Optical Coherence Tomography Imaging.

Glaucoma is a neurodegenerative disease process that leads to progressive damage of the optic nerve to produce visual impairment and blindness. Spectral-domain OCT technology enables peripapillary circular scans of the retina and the measurement of the thickness of the retinal nerve fiber layer (RNFL) for the assessment of the disease status or progression in glaucoma patients. This paper describes a new approach to segment and measure the retinal nerve fiber layer in peripapillary OCT images. The proposed method consists of two stages. In the first one, morphological operators robustly detect the coarse location of the layer boundaries, despite the speckle noise and diverse artifacts in the OCT image. In the second stage, deformable models are initialized with the results of the previous stage to perform a fine segmentation of the boundaries, providing an accurate measurement of the entire RNFL. The results of the RNFL segmentation were qualitatively assessed by ophthalmologists, and the measurements of the thickness of the RNFL were quantitatively compared with those provided by the OCT inbuilt software as well as the state-of-the-art methods.

An attention-based weakly supervised framework for spitzoid melanocytic lesion diagnosis in whole slide images.

Melanoma is an aggressive neoplasm responsible for the majority of deaths from skin cancer. Specifically, spitzoid melanocytic tumors are one of the most challenging melanocytic lesions due to their ambiguous morphological features. The gold standard for its diagnosis and prognosis is the analysis of skin biopsies. In this process, dermatopathologists visualize skin histology slides under a microscope, in a highly time-consuming and subjective task. In the last years, computer-aided diagnosis (CAD) systems have emerged as a promising tool that could support pathologists in daily clinical practice. Nevertheless, no automatic CAD systems have yet been proposed for the analysis of spitzoid lesions. Regarding common melanoma, no system allows both the selection of the tumor region and the prediction of the benign or malignant form in the diagnosis. Motivated by this, we propose a novel end-to-end weakly supervised deep learning model, based on inductive transfer learning with an improved convolutional neural network (CNN) to refine the embedding features of the latent space. The framework is composed of a source model in charge of finding the tumor patch-level patterns, and a target model focuses on the specific diagnosis of a biopsy. The latter retrains the backbone of the source model through a multiple instance learning workflow to obtain the biopsy-level scoring. To evaluate the performance of the proposed methods, we performed extensive experiments on a private skin database with spitzoid lesions. Test results achieved an accuracy of 0.9231 and 0.80 for the source and the target models, respectively. In addition, the heat map findings are directly in line with the clinicians' medical decision and even highlight, in some cases, patterns of interest that were overlooked by the pathologist.

Circumpapillary OCT-focused hybrid learning for glaucoma grading using tailored prototypical neural networks.

Glaucoma is one of the leading causes of blindness worldwide and Optical Coherence Tomography (OCT) is the quintessential imaging technique for its detection. Unlike most of the state-of-the-art studies focused on glaucoma detection, in this paper, we propose, for the first time, a novel framework for glaucoma grading using raw circumpapillary B-scans. In particular, we set out a new OCT-based hybrid network which combines hand-driven and deep learning algorithms. An OCT-specific descriptor is proposed to extract hand-crafted features related to the retinal nerve fibre layer (RNFL). In parallel, an innovative CNN is developed using skip-connections to include tailored residual and attention modules to refine the automatic features of the latent space. The proposed architecture is used as a backbone to conduct a novel few-shot learning based on static and dynamic prototypical networks. The k-shot paradigm is redefined giving rise to a supervised end-to-end system which provides substantial improvements discriminating between healthy, early and advanced glaucoma samples. The training and evaluation processes of the dynamic prototypical network are addressed from two fused databases acquired via Heidelberg Spectralis system. Validation and testing results reach a categorical accuracy of 0.9459 and 0.8788 for glaucoma grading, respectively. Besides, the high performance reported by the proposed model for glaucoma detection deserves a special mention. The findings from the class activation maps are directly in line with the clinicians' opinion since the heatmaps pointed out the RNFL as the most relevant structure for glaucoma diagnosis.

Retinal layer segmentation in rodent OCT images: Local intensity profiles & fully convolutional neural networks.

BACKGROUND AND OBJECTIVE: Optical coherence tomography (OCT) is a useful technique to monitor retinal layer state both in humans and animal models. Automated OCT analysis in rats is of great relevance to study possible toxic effect of drugs and other treatments before human trials. In this paper, two different approaches to detect the most significant retinal layers in a rat OCT image are presented. METHODS: One approach is based on a combination of local horizontal intensity profiles along with a new proposed variant of watershed transformation and the other is built upon an encoder-decoder convolutional network architecture. RESULTS: After a wide validation, an averaged absolute distance error of 3.77 ± 2.59 and 1.90 ± 0.91 µm is achieved by both approaches, respectively, on a batch of the rat OCT database. After a second test of the deep-learning-based method using an unseen batch of the database, an averaged absolute distance error of 2.67 ± 1.25 µm is obtained. The rat OCT database used in this paper is made publicly available to facilitate further comparisons. CONCLUSIONS: Based on the obtained results, it was demonstrated the competitiveness of the first approach since outperforms the commercial Insight image segmentation software (Phoenix Research Labs) as well as its utility to generate labelled images for validation purposes speeding significantly up the ground truth generation process. Regarding the second approach, the deep-learning-based method improves the results achieved by the more conventional method and also by other state-of-the-art techniques. In addition, it was verified that the results of the proposed network can be generalized to new rat OCT images.

Automatic Segmentation of Epidermis and Hair Follicles in Optical Coherence Tomography Images of Normal Skin by Convolutional Neural Networks.

Optical coherence tomography (OCT) is a well-established bedside imaging modality that allows analysis of skin structures in a non-invasive way. Automated OCT analysis of skin layers is of great relevance to study dermatological diseases. In this paper, an approach to detect the epidermal layer along with the follicular structures in healthy human OCT images is presented. To the best of the authors' knowledge, the approach presented in this paper is the only epidermis detection algorithm that segments the pilosebaceous unit, which is of importance in the progression of several skin disorders such as folliculitis, acne, lupus erythematosus, and basal cell carcinoma. The proposed approach is composed of two main stages. The first stage is a Convolutional Neural Network based on U-Net architecture. The second stage is a robust post-processing composed by a Savitzky-Golay filter and Fourier Domain Filtering to fully define the borders belonging to the hair follicles. After validation, an average Dice of 0.83 ± 0.06 and a thickness error of 10.25 µm is obtained on 270 human skin OCT images. Based on these results, the proposed method outperforms other state-of-the-art methods for epidermis segmentation. It demonstrates that the proposed image segmentation method successfully detects the epidermal region in a fully automatic way in addition to defining the follicular skin structures as main novelty.