Foundation models in gastrointestinal endoscopic AI: Impact of architecture, pre-training approach and data efficiency.

Pre-training deep learning models with large data sets of natural images, such as ImageNet, has become the standard for endoscopic image analysis. This approach is generally superior to training from scratch, due to the scarcity of high-quality medical imagery and labels. However, it is still unknown whether the learned features on natural imagery provide an optimal starting point for the downstream medical endoscopic imaging tasks. Intuitively, pre-training with imagery closer to the target domain could lead to better-suited feature representations. This study evaluates whether leveraging in-domain pre-training in gastrointestinal endoscopic image analysis has potential benefits compared to pre-training on natural images. To this end, we present a dataset comprising of 5,014,174 gastrointestinal endoscopic images from eight different medical centers (GastroNet-5M), and exploit self-supervised learning with SimCLRv2, MoCov2 and DINO to learn relevant features for in-domain downstream tasks. The learned features are compared to features learned on natural images derived with multiple methods, and variable amounts of data and/or labels (e.g. Billion-scale semi-weakly supervised learning and supervised learning on ImageNet-21k). The effects of the evaluation is performed on five downstream data sets, particularly designed for a variety of gastrointestinal tasks, for example, GIANA for angiodyplsia detection and Kvasir-SEG for polyp segmentation. The findings indicate that self-supervised domain-specific pre-training, specifically using the DINO framework, results into better performing models compared to any supervised pre-training on natural images. On the ResNet50 and Vision-Transformer-small architectures, utilizing self-supervised in-domain pre-training with DINO leads to an average performance boost of 1.63% and 4.62%, respectively, on the downstream datasets. This improvement is measured against the best performance achieved through pre-training on natural images within any of the evaluated frameworks. Moreover, the in-domain pre-trained models also exhibit increased robustness against distortion perturbations (noise, contrast, blur, etc.), where the in-domain pre-trained ResNet50 and Vision-Transformer-small with DINO achieved on average 1.28% and 3.55% higher on the performance metrics, compared to the best performance found for pre-trained models on natural images. Overall, this study highlights the importance of in-domain pre-training for improving the generic nature, scalability and performance of deep learning for medical image analysis. The GastroNet-5M pre-trained weights are made publicly available in our repository: huggingface.co/tgwboers/GastroNet-5M_Pretrained_Weights.

Linked color imaging improves identification of early gastric cancer lesions by expert and non-expert endoscopists.

BACKGROUND AND AIMS: Early gastric cancer (EGC) lesions are often subtle and endoscopically poorly visible. The aim of this study is to evaluate the additive effect of linked color imaging (LCI) next to white-light endoscopy (WLE) for identification of EGC, when assessed by expert and non-expert endoscopists. METHODS: Forty EGC cases were visualized in corresponding WLE and LCI images. Endoscopists evaluated the cases in 3 assessment phases: Phase 1: WLE images only; Phase 2: LCI images only; Phase 3: WLE and LCI images side-to-side. First, 3 expert endoscopists delineated all cases. A high level of agreement between the expert delineations corresponded with a high AND/OR ratio. Subsequently, 62 non-experts indicated their preferred biopsy location. Outcomes of the study are as follows: (1) difference in expert AND/OR ratio; (2) accuracy of biopsy placement by non-expert endoscopists; and (3) preference of imaging modality by non-expert endoscopists. RESULTS: Quantitative agreement between experts increased significantly when LCI was available (0.58 vs. 0.46, p = 0.007). This increase was more apparent for the more challenging cases (0.21 vs. 0.47, p < 0.001). Non-experts placed the biopsy mark more accurately with LCI (82.3% vs. 87.2%, p < 0.001). Again this increase was more profound for the more challenging cases (70.4% vs. 83.4%, p < 0.001). Non-experts indicated to prefer LCI over WLE. CONCLUSION: The addition of LCI next to WLE improves visualization of EGC. Experts reach higher consensus on discrimination between neoplasia and inflammation when using LCI. Non-experts improve their targeted biopsy placement with the use of LCI. LCI therefore appears to be a useful tool for identification of EGC.

Multi-stage domain-specific pretraining for improved detection and localization of Barrett's neoplasia: A comprehensive clinically validated study.

Patients suffering from Barrett's Esophagus (BE) are at an increased risk of developing esophageal adenocarcinoma and early detection is crucial for a good prognosis. To aid the endoscopists with the early detection for this preliminary stage of esophageal cancer, this work concentrates on the development and extensive evaluation of a state-of-the-art computer-aided classification and localization algorithm for dysplastic lesions in BE. To this end, we have employed a large-scale endoscopic data set, consisting of 494,355 images, in combination with a novel semi-supervised learning algorithm to pretrain several instances of the proposed neural network architecture. Next, several Barrett-specific data sets that are increasingly closer to the target domain with significantly more data compared to other related work, were used in a multi-stage transfer learning strategy. Additionally, the algorithm was evaluated on two prospectively gathered external test sets and compared against 53 medical professionals. Finally, the model was also evaluated in a live setting without interfering with the current biopsy protocol. Results from the performed experiments show that the proposed model improves on the state-of-the-art on all measured metrics. More specifically, compared to the best performing state-of-the-art model, the specificity is improved by more than 20% points while simultaneously preserving high sensitivity and reducing the false positive rate substantially. Our algorithm yields similar scores on the localization metrics, where the intersection of all experts is correctly indicated in approximately 92% of the cases. Furthermore, the live pilot study shows great performance in a clinical setting with a patient level accuracy, sensitivity, and specificity of 90%. Finally, the proposed algorithm outperforms each individual medical expert by at least 5% and the average assessor by more than 10% over all assessor groups with respect to accuracy.

Machine learning in GI endoscopy: practical guidance in how to interpret a novel field.

There has been a vast increase in GI literature focused on the use of machine learning in endoscopy. The relative novelty of this field poses a challenge for reviewers and readers of GI journals. To appreciate scientific quality and novelty of machine learning studies, understanding of the technical basis and commonly used techniques is required. Clinicians often lack this technical background, while machine learning experts may be unfamiliar with clinical relevance and implications for daily practice. Therefore, there is an increasing need for a multidisciplinary, international evaluation on how to perform high-quality machine learning research in endoscopy. This review aims to provide guidance for readers and reviewers of peer-reviewed GI journals to allow critical appraisal of the most relevant quality requirements of machine learning studies. The paper provides an overview of common trends and their potential pitfalls and proposes comprehensive quality requirements in six overarching themes: terminology, data, algorithm description, experimental setup, interpretation of results and machine learning in clinical practice.

Deep principal dimension encoding for the classification of early neoplasia in Barrett's Esophagus with volumetric laser endomicroscopy.

Barrett cancer is a treatable disease when detected at an early stage. However, current screening protocols are often not effective at finding the disease early. Volumetric Laser Endomicroscopy (VLE) is a promising new imaging tool for finding dysplasia in Barrett's esophagus (BE) at an early stage, by acquiring cross-sectional images of the microscopic structure of BE up to 3-mm deep. However, interpretation of VLE scans is difficult for medical doctors due to both the size and subtlety of the gray-scale data. Therefore, algorithms that can accurately find cancerous regions are very valuable for the interpretation of VLE data. In this study, we propose a fully-automatic multi-step Computer-Aided Detection (CAD) algorithm that optimally leverages the effectiveness of deep learning strategies by encoding the principal dimension in VLE data. Additionally, we show that combining the encoded dimensions with conventional machine learning techniques further improves results while maintaining interpretability. Furthermore, we train and validate our algorithm on a new histopathologically validated set of in-vivo VLE snapshots. Additionally, an independent test set is used to assess the performance of the model. Finally, we compare the performance of our algorithm against previous state-of-the-art systems. With the encoded principal dimension, we obtain an Area Under the Curve (AUC) and F1 score of 0.93 and 87.4% on the test set respectively. We show this is a significant improvement compared to the state-of-the-art of 0.89 and 83.1%, respectively, thereby demonstrating the effectiveness of our approach.

The Argos project: The development of a computer-aided detection system to improve detection of Barrett's neoplasia on white light endoscopy.

Background: Computer-aided detection (CAD) systems might assist endoscopists in the recognition of Barrett's neoplasia. Aim: To develop a CAD system using endoscopic images of Barrett's neoplasia. Methods: White light endoscopy (WLE) overview images of 40 neoplastic Barrett's lesions and 20 non-dysplastic Barret's oesophagus (NDBO) patients were prospectively collected. Experts delineated all neoplastic images.The overlap area of at least four delineations was labelled as the 'sweet spot'. The area with at least one delineation was labelled as the 'soft spot'. The CAD system was trained on colour and texture features. Positive features were taken from the sweet spot and negative features from NDBO images. Performance was evaluated using leave-one-out cross-validation. Outcome parameters were diagnostic accuracy of the CAD system per image, and localization of the expert soft spot by CAD delineation (localization score) and its indication of preferred biopsy location (red-flag indication score). Results: Accuracy, sensitivity and specificity for detection were 92, 95 and 85%, respectively. The system localized and red-flagged the soft spot in 100 and 90%, respectively. Conclusion: This uniquely trained and validated CAD system detected and localized early Barrett's neoplasia on WLE images with high accuracy. This is an important step towards real-time automated detection of Barrett's neoplasia.