Central Reading of Ulcerative Colitis Clinical Trial Videos Using Neural Networks.

BACKGROUND AND AIMS: Endoscopic disease activity scoring in ulcerative colitis (UC) is useful in clinical practice but done infrequently. It is required in clinical trials, where it is expensive and slow because human central readers are needed. A machine learning algorithm automating the process could elevate clinical care and facilitate clinical research. Prior work using single-institution databases and endoscopic still images has been promising. METHODS: Seven hundred and ninety-five full-length endoscopy videos were prospectively collected from a phase 2 trial of mirikizumab with 249 patients from 14 countries, totaling 19.5 million image frames. Expert central readers assigned each full-length endoscopy videos 1 endoscopic Mayo score (eMS) and 1 Ulcerative Colitis Endoscopic Index of Severity (UCEIS) score. Initially, video data were cleaned and abnormality features extracted using convolutional neural networks. Subsequently, a recurrent neural network was trained on the features to predict eMS and UCEIS from individual full-length endoscopy videos. RESULTS: The primary metric to assess the performance of the recurrent neural network model was quadratic weighted kappa (QWK) comparing the agreement of the machine-read endoscopy score with the human central reader score. QWK progressively penalizes disagreements that exceed 1 level. The model's agreement metric was excellent, with a QWK of 0.844 (95% confidence interval, 0.787-0.901) for eMS and 0.855 (95% confidence interval, 0.80-0.91) for UCEIS. CONCLUSIONS: We found that a deep learning algorithm can be trained to predict levels of UC severity from full-length endoscopy videos. Our data set was prospectively collected in a multinational clinical trial, videos rather than still images were used, UCEIS and eMS were reported, and machine learning algorithm performance metrics met or exceeded those previously published for UC severity scores.

Proceedings from the First Global Artificial Intelligence in Gastroenterology and Endoscopy Summit.

BACKGROUND AND AIMS: Artificial intelligence (AI), specifically deep learning, offers the potential to enhance the field of GI endoscopy in areas ranging from lesion detection and classification to quality metrics and documentation. Progress in this field will be measured by whether AI implementation can lead to improved patient outcomes and more efficient clinical workflow for GI endoscopists. The aims of this article are to report the findings of a multidisciplinary group of experts focusing on issues in AI research and applications related to gastroenterology and endoscopy, to review the current status of the field, and to produce recommendations for investigators developing and studying new AI technologies for gastroenterology. METHODS: A multidisciplinary meeting was held on September 28, 2019, bringing together academic, industry, and regulatory experts in diverse fields including gastroenterology, computer and imaging sciences, machine learning, computer vision, U.S. Food and Drug Administration, and the National Institutes of Health. Recent and ongoing studies in gastroenterology and current technology in AI were presented and discussed, key gaps in knowledge were identified, and recommendations were made for research that would have the highest impact in making advances and implementation in the field of AI to gastroenterology. RESULTS: There was a consensus that AI will transform the field of gastroenterology, particularly endoscopy and image interpretation. Powered by advanced machine learning algorithms, the use of computer vision in endoscopy has the potential to result in better prediction and treatment outcomes for patients with gastroenterology disorders and cancer. Large libraries of endoscopic images, "EndoNet," will be important to facilitate development and application of AI systems. The regulatory environment for implementation of AI systems is evolving, but common outcomes such as colon polyp detection have been highlighted as potential clinical trial endpoints. Other threshold outcomes will be important, as well as clarity on iterative improvement of clinical systems. CONCLUSIONS: Gastroenterology is a prime candidate for early adoption of AI. AI is rapidly moving from an experimental phase to a clinical implementation phase in gastroenterology. It is anticipated that the implementation of AI in gastroenterology over the next decade will have a significant and positive impact on patient care and clinical workflows. Ongoing collaboration among gastroenterologists, industry experts, and regulatory agencies will be important to ensure that progress is rapid and clinically meaningful. However, several constraints and areas will benefit from further exploration, including potential clinical applications, implementation, structure and governance, role of gastroenterologists, and potential impact of AI in gastroenterology.

Artificial intelligence using convolutional neural networks for real-time detection of early esophageal neoplasia in Barrett's esophagus (with video).

BACKGROUND AND AIMS: The visual detection of early esophageal neoplasia (high-grade dysplasia and T1 cancer) in Barrett's esophagus (BE) with white-light and virtual chromoendoscopy still remains challenging. The aim of this study was to assess whether a convolutional neural artificial intelligence network can aid in the recognition of early esophageal neoplasia in BE. METHODS: Nine hundred sixteen images from 65 patients of histology-proven early esophageal neoplasia in BE containing high-grade dysplasia or T1 cancer were collected. The area of neoplasia was masked using image annotation software. Nine hundred nineteen control images were collected of BE without high-grade dysplasia. A convolutional neural network (CNN) algorithm was pretrained on ImageNet and then fine-tuned with the goal of providing the correct binary classification of "dysplastic" or "nondysplastic." We developed an object detection algorithm that drew localization boxes around regions classified as dysplasia. RESULTS: The CNN analyzed 458 test images (225 dysplasia and 233 nondysplasia) and correctly detected early neoplasia with sensitivity of 96.4%, specificity of 94.2%, and accuracy of 95.4%. With regard to the object detection algorithm for all images in the validation set, the system was able to achieve a mean average precision of .7533 at an intersection over union of .3 CONCLUSIONS: In this pilot study, our artificial intelligence model was able to detect early esophageal neoplasia in BE images with high accuracy. In addition, the object detection algorithm was able to draw a localization box around the areas of dysplasia with high precision and at a speed that allows for real-time implementation.

Prediction of Polyp Pathology Using Convolutional Neural Networks Achieves "Resect and Discard" Thresholds.

OBJECTIVES: Reliable in situ diagnosis of diminutive (≤5 mm) colorectal polyps could allow for "resect and discard" and "diagnose and leave" strategies, resulting in $1 billion cost savings per year in the United States alone. Current methodologies have failed to consistently meet the Preservation and Incorporation of Valuable endoscopic Innovations (PIVIs) initiative thresholds. Convolutional neural networks (CNNs) have the potential to predict polyp pathology and achieve PIVI thresholds in real time. METHODS: We developed a CNN-based optical pathology (OP) model using Tensorflow and pretrained on ImageNet, capable of operating at 77 frames per second. A total of 6,223 images of unique colorectal polyps of known pathology, location, size, and light source (white light or narrow band imaging [NBI]) underwent 5-fold cross-training (80%) and validation (20%). Separate fresh validation was performed on 634 polyp images. Surveillance intervals were calculated, comparing OP with true pathology. RESULTS: In the original validation set, the negative predictive value for adenomas was 97% among diminutive rectum/rectosigmoid polyps. Results were independent of use of NBI or white light. Surveillance interval concordance comparing OP and true pathology was 93%. In the fresh validation set, the negative predictive value was 97% among diminutive polyps in the rectum and rectosigmoid and surveillance concordance was 94%. DISCUSSION: This study demonstrates the feasibility of in situ diagnosis of colorectal polyps using CNN. Our model exceeds PIVI thresholds for both "resect and discard" and "diagnose and leave" strategies independent of NBI use. Point-of-care adenoma detection rate and surveillance recommendations are potential added benefits.

DocBot: a novel clinical decision support algorithm.

DocBot is a web-based clinical decision support system (CDSS) that uses patient interaction and electronic health record analytics to assist medical practitioners with decision making. It consists of two distinct HTML interfaces: a preclinical form wherein a patient inputs symptomatic and demographic information, and an interface wherein a medical practitioner views patient information and analysis. DocBot comprises an improved software architecture that uses patient information, electronic health records, and etiologically relevant binary decision questions (stored in a knowledgebase) to provide medical practitioners with information including, but not limited to medical assessments, treatment plans, and specialist referrals.