ABSTRACT
Objective: Recent developments in artificial intelligence (AI) have positioned it to transform several stages of the clinical trial process. In this study, we explore the role of AI in clinical trial recruitment of individuals with geographic atrophy (GA), an advanced stage of age-related macular degeneration, amidst numerous ongoing clinical trials for this condition. Design: Cross-sectional study. Subjects: Retrospective dataset from the INSIGHT Health Data Research Hub at Moorfields Eye Hospital in London, United Kingdom, including 306 651 patients (602 826 eyes) with suspected retinal disease who underwent OCT imaging between January 1, 2008 and April 10, 2023. Methods: A deep learning model was trained on OCT scans to identify patients potentially eligible for GA trials, using AI-generated segmentations of retinal tissue. This method's efficacy was compared against a traditional keyword-based electronic health record (EHR) search. A clinical validation with fundus autofluorescence (FAF) images was performed to calculate the positive predictive value of this approach, by comparing AI predictions with expert assessments. Main Outcome Measures: The primary outcomes included the positive predictive value of AI in identifying trial-eligible patients, and the secondary outcome was the intraclass correlation between GA areas segmented on FAF by experts and AI-segmented OCT scans. Results: The AI system shortlisted a larger number of eligible patients with greater precision (1139, positive predictive value: 63%; 95% confidence interval [CI]: 54%-71%) compared with the EHR search (693, positive predictive value: 40%; 95% CI: 39%-42%). A combined AI-EHR approach identified 604 eligible patients with a positive predictive value of 86% (95% CI: 79%-92%). Intraclass correlation of GA area segmented on FAF versus AI-segmented area on OCT was 0.77 (95% CI: 0.68-0.84) for cases meeting trial criteria. The AI also adjusts to the distinct imaging criteria from several clinical trials, generating tailored shortlists ranging from 438 to 1817 patients. Conclusions: This study demonstrates the potential for AI in facilitating automated prescreening for clinical trials in GA, enabling site feasibility assessments, data-driven protocol design, and cost reduction. Once treatments are available, similar AI systems could also be used to identify individuals who may benefit from treatment. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
ABSTRACT
Importance: Vision-language models (VLMs) are a novel artificial intelligence technology capable of processing image and text inputs. While demonstrating strong generalist capabilities, their performance in ophthalmology has not been extensively studied. Objective: To assess the performance of the Gemini Pro VLM in expert-level tasks for macular diseases from optical coherence tomography (OCT) scans. Design, Setting, and Participants: This was a cross-sectional diagnostic accuracy study evaluating a generalist VLM on ophthalmology-specific tasks using the open-source Optical Coherence Tomography Image Database. The dataset included OCT B-scans from 50 unique patients: healthy individuals and those with macular hole, diabetic macular edema, central serous chorioretinopathy, and age-related macular degeneration. Each OCT scan was labeled for 10 key pathological features, referral recommendations, and treatments. The images were captured using a Cirrus high definition OCT machine (Carl Zeiss Meditec) at Sankara Nethralaya Eye Hospital, Chennai, India, and the dataset was published in December 2018. Image acquisition dates were not specified. Exposures: Gemini Pro, using a standard prompt to extract structured responses on December 15, 2023. Main Outcomes and Measures: The primary outcome was model responses compared against expert labels, calculating F1 scores for each pathological feature. Secondary outcomes included accuracy in diagnosis, referral urgency, and treatment recommendation. The model's internal concordance was evaluated by measuring the alignment between referral and treatment recommendations, independent of diagnostic accuracy. Results: The mean F1 score was 10.7% (95% CI, 2.4-19.2). Measurable F1 scores were obtained for macular hole (36.4%; 95% CI, 0-71.4), pigment epithelial detachment (26.1%; 95% CI, 0-46.2), subretinal hyperreflective material (24.0%; 95% CI, 0-45.2), and subretinal fluid (20.0%; 95% CI, 0-45.5). A correct diagnosis was achieved in 17 of 50 cases (34%; 95% CI, 22-48). Referral recommendations varied: 28 of 50 were correct (56%; 95% CI, 42-70), 10 of 50 were overcautious (20%; 95% CI, 10-32), and 12 of 50 were undercautious (24%; 95% CI, 12-36). Referral and treatment concordance were very high, with 48 of 50 (96%; 95 % CI, 90-100) and 48 of 49 (98%; 95% CI, 94-100) correct answers, respectively. Conclusions and Relevance: In this study, a generalist VLM demonstrated limited vision capabilities for feature detection and management of macular disease. However, it showed low self-contradiction, suggesting strong language capabilities. As VLMs continue to improve, validating their performance on large benchmarking datasets will help ascertain their potential in ophthalmology.