The impact of artificial intelligence on retinal disease management: Vision Academy retinal expert consensus.

PURPOSE OF REVIEW: The aim of this review is to define the "state-of-the-art" in artificial intelligence (AI)-enabled devices that support the management of retinal conditions and to provide Vision Academy recommendations on the topic. RECENT FINDINGS: Most of the AI models described in the literature have not been approved for disease management purposes by regulatory authorities. These new technologies are promising as they may be able to provide personalized treatments as well as a personalized risk score for various retinal diseases. However, several issues still need to be addressed, such as the lack of a common regulatory pathway and a lack of clarity regarding the applicability of AI-enabled medical devices in different populations. SUMMARY: It is likely that current clinical practice will need to change following the application of AI-enabled medical devices. These devices are likely to have an impact on the management of retinal disease. However, a consensus needs to be reached to ensure they are safe and effective for the overall population.

Current status and practical considerations of artificial intelligence use in screening and diagnosing retinal diseases: Vision Academy retinal expert consensus.

PURPOSE OF REVIEW: The application of artificial intelligence (AI) technologies in screening and diagnosing retinal diseases may play an important role in telemedicine and has potential to shape modern healthcare ecosystems, including within ophthalmology. RECENT FINDINGS: In this article, we examine the latest publications relevant to AI in retinal disease and discuss the currently available algorithms. We summarize four key requirements underlining the successful application of AI algorithms in real-world practice: processing massive data; practicability of an AI model in ophthalmology; policy compliance and the regulatory environment; and balancing profit and cost when developing and maintaining AI models. SUMMARY: The Vision Academy recognizes the advantages and disadvantages of AI-based technologies and gives insightful recommendations for future directions.

Building trust in real-world data: lessons from INSIGHT, the UK's health data research hub for eye health and oculomics.

PURPOSE OF REVIEW: In this review, we consider the challenges of creating a trusted resource for real-world data in ophthalmology, based on our experience of establishing INSIGHT, the UK's Health Data Research Hub for Eye Health and Oculomics. RECENT FINDINGS: The INSIGHT Health Data Research Hub maximizes the benefits and impact of historical, patient-level UK National Health Service (NHS) electronic health record data, including images, through making it research-ready including curation and anonymisation. It is built around a shared 'north star' of enabling research for patient benefit. INSIGHT has worked to establish patient and public trust in the concept and delivery of INSIGHT, with efficient and robust governance processes that support safe and secure access to data for researchers. By linking to systemic data, there is an opportunity for discovery of novel ophthalmic biomarkers of systemic diseases ('oculomics'). Datasets that provide a representation of the whole population are an important tool to address the increasingly recognized threat of health data poverty. SUMMARY: Enabling efficient, safe access to routinely collected clinical data is a substantial undertaking, especially when this includes imaging modalities, but provides an exceptional resource for research. Research and innovation built on inclusive real-world data is an important tool in ensuring that discoveries and technologies of the future may not only favour selected groups, but also work for all patients.

Detecting Algorithmic Errors and Patient Harms for AI-Enabled Medical Devices in Randomized Controlled Trials: Protocol for a Systematic Review.

BACKGROUND: Artificial intelligence (AI) medical devices have the potential to transform existing clinical workflows and ultimately improve patient outcomes. AI medical devices have shown potential for a range of clinical tasks such as diagnostics, prognostics, and therapeutic decision-making such as drug dosing. There is, however, an urgent need to ensure that these technologies remain safe for all populations. Recent literature demonstrates the need for rigorous performance error analysis to identify issues such as algorithmic encoding of spurious correlations (eg, protected characteristics) or specific failure modes that may lead to patient harm. Guidelines for reporting on studies that evaluate AI medical devices require the mention of performance error analysis; however, there is still a lack of understanding around how performance errors should be analyzed in clinical studies, and what harms authors should aim to detect and report. OBJECTIVE: This systematic review will assess the frequency and severity of AI errors and adverse events (AEs) in randomized controlled trials (RCTs) investigating AI medical devices as interventions in clinical settings. The review will also explore how performance errors are analyzed including whether the analysis includes the investigation of subgroup-level outcomes. METHODS: This systematic review will identify and select RCTs assessing AI medical devices. Search strategies will be deployed in MEDLINE (Ovid), Embase (Ovid), Cochrane CENTRAL, and clinical trial registries to identify relevant papers. RCTs identified in bibliographic databases will be cross-referenced with clinical trial registries. The primary outcomes of interest are the frequency and severity of AI errors, patient harms, and reported AEs. Quality assessment of RCTs will be based on version 2 of the Cochrane risk-of-bias tool (RoB2). Data analysis will include a comparison of error rates and patient harms between study arms, and a meta-analysis of the rates of patient harm in control versus intervention arms will be conducted if appropriate. RESULTS: The project was registered on PROSPERO in February 2023. Preliminary searches have been completed and the search strategy has been designed in consultation with an information specialist and methodologist. Title and abstract screening started in September 2023. Full-text screening is ongoing and data collection and analysis began in April 2024. CONCLUSIONS: Evaluations of AI medical devices have shown promising results; however, reporting of studies has been variable. Detection, analysis, and reporting of performance errors and patient harms is vital to robustly assess the safety of AI medical devices in RCTs. Scoping searches have illustrated that the reporting of harms is variable, often with no mention of AEs. The findings of this systematic review will identify the frequency and severity of AI performance errors and patient harms and generate insights into how errors should be analyzed to account for both overall and subgroup performance. TRIAL REGISTRATION: PROSPERO CRD42023387747; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=387747. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/51614.

AI as a Medical Device for Ophthalmic Imaging in Europe, Australia, and the United States: Protocol for a Systematic Scoping Review of Regulated Devices.

BACKGROUND: Artificial intelligence as a medical device (AIaMD) has the potential to transform many aspects of ophthalmic care, such as improving accuracy and speed of diagnosis, addressing capacity issues in high-volume areas such as screening, and detecting novel biomarkers of systemic disease in the eye (oculomics). In order to ensure that such tools are safe for the target population and achieve their intended purpose, it is important that these AIaMD have adequate clinical evaluation to support any regulatory decision. Currently, the evidential requirements for regulatory approval are less clear for AIaMD compared to more established interventions such as drugs or medical devices. There is therefore value in understanding the level of evidence that underpins AIaMD currently on the market, as a step toward identifying what the best practices might be in this area. In this systematic scoping review, we will focus on AIaMD that contributes to clinical decision-making (relating to screening, diagnosis, prognosis, and treatment) in the context of ophthalmic imaging. OBJECTIVE: This study aims to identify regulator-approved AIaMD for ophthalmic imaging in Europe, Australia, and the United States; report the characteristics of these devices and their regulatory approvals; and report the available evidence underpinning these AIaMD. METHODS: The Food and Drug Administration (United States), the Australian Register of Therapeutic Goods (Australia), the Medicines and Healthcare products Regulatory Agency (United Kingdom), and the European Database on Medical Devices (European Union) regulatory databases will be searched for ophthalmic imaging AIaMD through a snowballing approach. PubMed and clinical trial registries will be systematically searched, and manufacturers will be directly contacted for studies investigating the effectiveness of eligible AIaMD. Preliminary regulatory database searches, evidence searches, screening, data extraction, and methodological quality assessment will be undertaken by 2 independent review authors and arbitrated by a third at each stage of the process. RESULTS: Preliminary searches were conducted in February 2023. Data extraction, data synthesis, and assessment of methodological quality commenced in October 2023. The review is on track to be completed and submitted for peer review by April 2024. CONCLUSIONS: This systematic review will provide greater clarity on ophthalmic imaging AIaMD that have achieved regulatory approval as well as the evidence that underpins them. This should help adopters understand the range of tools available and whether they can be safely incorporated into their clinical workflow, and it should also support developers in navigating regulatory approval more efficiently. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/52602.

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines.

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

Measuring Inflammation in the Vitreous and Retina: A Narrative Review.

Uveitis consists of a group of syndromes characterised by intraocular inflammation, accounting for up to 15% of visual loss in the western world and 10% worldwide. Assessment of intraocular inflammation has been limited to clinician-dependent, subjective grading. Developments in imaging technology, such as optical coherence tomography (OCT), have enabled the development of objective, quantitative measures of inflammatory activity. Important quantitative metrics including central macular thickness and vitreous signal intensity allow longitudinal monitoring of disease activity and can be used in conjunction with other imaging modalities enabling holistic assessment of ocular inflammation. Ongoing work into the validation of instrument-based measures alongside development of core outcome sets is crucial for standardisation of clinical trial endpoints and developing guidance for quantitative multi-modal imaging approaches. This review outlines methods of grading inflammation in the vitreous and retina, with a focus on the use of OCT as an objective measure of disease activity.

Reliability of Optical Coherence Tomography Angiography Retinal Blood Flow Analyses.

Purpose: Investigate the association between the optical coherence tomography angiography (OCTA) metrics derived from different analysis programs to understand the comparability of studies using these different approaches. Methods: Secondary analysis of a prospective observational study (March 2018-September 2021). Forty-four right eyes and 42 left eyes from 44 patients were included. Patients were either undergoing upper gastrointestinal surgery with a critical care stay planned or were already in the critical care unit with sepsis. OCTA scans were obtained in an ophthalmology department or critical care setting. Fourteen OCTA metrics were compared within and between the programs, and agreement was measured by Pearson's R coefficient and intraclass correlation coefficient. Results: Correlation was highest between all Heidelberg metrics and Fractalyse (all >0.84), and lowest between Matlab skeletonized or foveal avascular zone metrics and all other measures (e.g., skeletal fractal dimension and vessel density at -0.02). Agreement between eyes was moderate to excellent in all metrics (0.60-0.90). Conclusions: The significant variability between metrics and programs used for OCTA analysis demonstrates that they are not interchangeable and supports a recommendation for perfusion density metrics to be reported as standard. Translational Relevance: Agreement between different OCTA analyses is variable and not interchangeable. The high agreement between non-skeletonized vessel density metrics suggests that these should be routinely reported.

A Datasheet for the INSIGHT Birmingham, Solihull, and Black Country Diabetic Retinopathy Screening Dataset.

Purpose: Diabetic retinopathy (DR) is the most common microvascular complication associated with diabetes mellitus (DM), affecting approximately 40% of this patient population. Early detection of DR is vital to ensure monitoring of disease progression and prompt sight saving treatments as required. This article describes the data contained within the INSIGHT Birmingham, Solihull, and Black Country Diabetic Retinopathy Dataset. Design: Dataset descriptor for routinely collected eye screening data. Participants: All diabetic patients aged 12 years and older, attending annual digital retinal photography-based screening within the Birmingham, Solihull, and Black Country Eye Screening Programme. Methods: The INSIGHT Health Data Research Hub for Eye Health is a National Health Service (NHS)-led ophthalmic bioresource that provides researchers with safe access to anonymized, routinely collected data from contributing NHS hospitals to advance research for patient benefit. This report describes the INSIGHT Birmingham, Solihull, and Black Country DR Screening Dataset, a dataset of anonymized images and linked screening data derived from the United Kingdom's largest regional DR screening program. Main Outcome Measures: This dataset consists of routinely collected data from the eye screening program. The data primarily include retinal photographs with the associated DR grading data. Additional data such as corresponding demographic details, information regarding patients' diabetic status, and visual acuity data are also available. Further details regarding available data points are available in the supplementary information, in addition to the INSIGHT webpage included below. Results: At the time point of this analysis (December 31, 2019), the dataset comprised 6 202 161 images from 246 180 patients, with a dataset inception date of January 1, 2007. The dataset includes 1 360 547 grading episodes between R0M0 and R3M1. Conclusions: This dataset descriptor article summarizes the content of the dataset, how it has been curated, and what its potential uses are. Data are available through a structured application process for research studies that support discovery, clinical evidence analyses, and innovation in artificial intelligence technologies for patient benefit. Further information regarding the data repository and contact details can be found at https://www.insight.hdrhub.org/. Financial Disclosures: Proprietary or commercial disclosure may be found after the references.

OCT Assisted Quantification of Vitreous Inflammation in Uveitis.

Purpose: Vitreous haze (VH) is a key marker of inflammation in uveitis but limited by its subjectivity. Optical coherence tomography (OCT) has potential as an objective, noninvasive method for quantifying VH. We test the hypotheses that OCT can reliably quantify VH and the measurement is associated with slit-lamp based grading of VH. Methods: In this prospective study, participants underwent three repeated OCT macular scans to evaluate the within-eye reliability of the OCT vitreous intensity (VI). Association between OCT VI and clinical findings (including VH grade, phakic status, visual acuity [VA], anterior chamber cells, and macular thickness) were assessed. Results: One hundred nineteen participants were included (41 healthy participants, 32 patients with uveitis without VH, and 46 patients with uveitis with VH). Within-eye test reliability of OCT VI was high in healthy eyes and in all grades of VH (intraclass correlation coefficient [ICC] > 0.79). Average OCT VI was significantly different between healthy eyes and uveitic eyes without and uveitic eyes with VH, and was associated with increasing clinical VH grade (P < 0.05). OCT VI was significantly associated with VA, whereas clinical VH grading was not. Cataract was also associated with higher OCT VI (P = 0.03). Conclusions: OCT VI is a fast, noninvasive, objective, and automated method for measuring vitreous inflammation. It is associated with clinician grading of vitreous inflammation and VA, however, it can be affected by media opacities. Translational Relevance: OCT imaging for quantifying vitreous inflammation shows high within-eye repeatability and is associated with clinical grading of vitreous haze. OCT measurements are also associated with visual acuity but may be affected by structures anterior to the acquisition window, such as lens opacity and other anterior segment changes.

Stability of OCT and OCTA in the Intensive Therapy Unit Setting.

To assess the stability of retinal structure and blood flow measures over time and in different clinical settings using portable optical coherence tomography angiography (OCTA) as a potential biomarker of central perfusion in critical illness, 18 oesophagectomy patients completed retinal structure and blood flow measurements by portable OCT and OCTA in the eye clinic and intensive therapy unit (ITU) across three timepoints: (1) pre-operation in a clinic setting; (2) 24-48 h post-operation during ITU admission; and (3) seven days post-operation, if the patient was still admitted. Blood flow and macular structural measures were stable between the examination settings, with no consistent variation between pre- and post-operation scans, while retinal nerve fibre layer thickness increased in the post-operative scans (+2.31 µm, p = 0.001). Foveal avascular zone (FAZ) measurements were the most stable, with an intraclass correlation coefficient of up to 0.92 for right eye FAZ area. Blood flow and structural measures were lower in left eyes than right eyes. Retinal blood flow assessed in patients before and during an ITU stay using portable OCTA showed no systematic differences between the clinical settings. The stability of retinal blood flow measures suggests the potential for portable OCTA to provide clinically useful measures in ITU patients.

Instrument-based Tests for Measuring Anterior Chamber Cells in Uveitis: A Systematic Review.

PURPOSE: New instrument-based techniques for anterior chamber (AC) cell counting can offer automation and objectivity above clinician assessment. This review aims to identify such instruments and its correlation with clinician estimates. METHODS: Using standard systematic review methodology, we identified and tabulated the outcomes of studies reporting reliability and correlation between instrument-based measurements and clinician AC cell grading. RESULTS: From 3470 studies, 6 reported correlation between an instrument-based AC cell count to clinician grading. The two instruments were optical coherence tomography (OCT) and laser flare-cell photometry (LFCP). Correlation between clinician grading and LFCP was 0.66-0.87 and 0.06-0.97 between clinician grading and OCT. OCT volume scans demonstrated correlation between 0.75 and 0.78. Line scans in the middle AC demonstrated higher correlation (0.73-0.97) than in the inferior AC (0.06-0.56). CONCLUSION: AC cell count by OCT and LFP can achieve high levels of correlation with clinician grading, whilst offering additional advantages of speed, automation, and objectivity.

A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.

BACKGROUND: Deep learning offers considerable promise for medical diagnostics. We aimed to evaluate the diagnostic accuracy of deep learning algorithms versus health-care professionals in classifying diseases using medical imaging. METHODS: In this systematic review and meta-analysis, we searched Ovid-MEDLINE, Embase, Science Citation Index, and Conference Proceedings Citation Index for studies published from Jan 1, 2012, to June 6, 2019. Studies comparing the diagnostic performance of deep learning models and health-care professionals based on medical imaging, for any disease, were included. We excluded studies that used medical waveform data graphics material or investigated the accuracy of image segmentation rather than disease classification. We extracted binary diagnostic accuracy data and constructed contingency tables to derive the outcomes of interest: sensitivity and specificity. Studies undertaking an out-of-sample external validation were included in a meta-analysis, using a unified hierarchical model. This study is registered with PROSPERO, CRD42018091176. FINDINGS: Our search identified 31 587 studies, of which 82 (describing 147 patient cohorts) were included. 69 studies provided enough data to construct contingency tables, enabling calculation of test accuracy, with sensitivity ranging from 9·7% to 100·0% (mean 79·1%, SD 0·2) and specificity ranging from 38·9% to 100·0% (mean 88·3%, SD 0·1). An out-of-sample external validation was done in 25 studies, of which 14 made the comparison between deep learning models and health-care professionals in the same sample. Comparison of the performance between health-care professionals in these 14 studies, when restricting the analysis to the contingency table for each study reporting the highest accuracy, found a pooled sensitivity of 87·0% (95% CI 83·0-90·2) for deep learning models and 86·4% (79·9-91·0) for health-care professionals, and a pooled specificity of 92·5% (95% CI 85·1-96·4) for deep learning models and 90·5% (80·6-95·7) for health-care professionals. INTERPRETATION: Our review found the diagnostic performance of deep learning models to be equivalent to that of health-care professionals. However, a major finding of the review is that few studies presented externally validated results or compared the performance of deep learning models and health-care professionals using the same sample. Additionally, poor reporting is prevalent in deep learning studies, which limits reliable interpretation of the reported diagnostic accuracy. New reporting standards that address specific challenges of deep learning could improve future studies, enabling greater confidence in the results of future evaluations of this promising technology. FUNDING: None.