ARTIFICIAL INTELLIGENCE FOR OPTICAL COHERENCE TOMOGRAPHY ANGIOGRAPHY-BASED DISEASE ACTIVITY PREDICTION IN AGE-RELATED MACULAR DEGENERATION.

PURPOSE: The authors hypothesize that optical coherence tomography angiography (OCTA)-visualized vascular morphology may be a predictor of choroidal neovascularization status in age-related macular degeneration (AMD). The authors thus evaluated the use of artificial intelligence (AI) to predict different stages of AMD disease based on OCTA en face 2D projections scans. METHODS: Retrospective cross-sectional study based on collected 2D OCTA data from 310 high-resolution scans. Based on OCT B-scan fluid and clinical status, OCTA was classified as normal, dry AMD, wet AMD active, and wet AMD in remission with no signs of activity. Two human experts graded the same test set, and a consensus grading between two experts was used for the prediction of four categories. RESULTS: The AI can achieve 80.36% accuracy on a four-category grading task with 2D OCTA projections. The sensitivity of prediction by AI was 0.7857 (active), 0.7142 (remission), 0.9286 (dry AMD), and 0.9286 (normal) and the specificity was 0.9524, 0.9524, 0.9286, and 0.9524, respectively. The sensitivity of prediction by human experts was 0.4286 active choroidal neovascularization, 0.2143 remission, 0.8571 dry AMD, and 0.8571 normal with specificity of 0.7619, 0.9286, 0.7857, and 0.9762, respectively. The overall AI classification prediction was significantly better than the human (odds ratio = 1.95, P = 0.0021). CONCLUSION: These data show that choroidal neovascularization morphology can be used to predict disease activity by AI; longitudinal studies are needed to better understand the evolution of choroidal neovascularization and features that predict reactivation. Future studies will be able to evaluate the additional predicative value of OCTA on top of other imaging characteristics (i.e., fluid location on OCT B scans) to help predict response to treatment.

Ultra-wide field and new wide field composite retinal image registration with AI-enabled pipeline and 3D distortion correction algorithm.

PURPOSE: This study aimed to compare a new Artificial Intelligence (AI) method to conventional mathematical warping in accurately overlaying peripheral retinal vessels from two different imaging devices: confocal scanning laser ophthalmoscope (cSLO) wide-field images and SLO ultra-wide field images. METHODS: Images were captured using the Heidelberg Spectralis 55-degree field-of-view and Optos ultra-wide field. The conventional mathematical warping was performed using Random Sample Consensus-Sample and Consensus sets (RANSAC-SC). This was compared to an AI alignment algorithm based on a one-way forward registration procedure consisting of full Convolutional Neural Networks (CNNs) with Outlier Rejection (OR CNN), as well as an iterative 3D camera pose optimization process (OR CNN + Distortion Correction [DC]). Images were provided in a checkerboard pattern, and peripheral vessels were graded in four quadrants based on alignment to the adjacent box. RESULTS: A total of 660 boxes were analysed from 55 eyes. Dice scores were compared between the three methods (RANSAC-SC/OR CNN/OR CNN + DC): 0.3341/0.4665/4784 for fold 1-2 and 0.3315/0.4494/4596 for fold 2-1 in composite images. The images composed using the OR CNN + DC have a median rating of 4 (out of 5) versus 2 using RANSAC-SC. The odds of getting a higher grading level are 4.8 times higher using our OR CNN + DC than RANSAC-SC (p < 0.0001). CONCLUSION: Peripheral retinal vessel alignment performed better using our AI algorithm than RANSAC-SC. This may help improve co-localizing retinal anatomy and pathology with our algorithm.

Efficacy and accuracy of artificial intelligence to overlay multimodal images from different optical instruments in patients with retinitis pigmentosa.

BACKGROUND: Retinitis pigmentosa (RP) represents a group of progressive, genetically heterogenous blinding diseases. Recently, relationships between measures of retinal function and structure are needed to help identify outcome measures or biomarkers for clinical trials. The ability to align retinal multimodal images, taken on different platforms, will allow better understanding of this relationship. We investigate the efficacy of artificial intelligence (AI) in overlaying different multimodal retinal images in RP patients. METHODS: We overlayed infrared images from microperimetry on near-infra-red images from scanning laser ophthalmoscope and spectral domain optical coherence tomography in RP patients using manual alignment and AI. The AI adopted a two-step framework and was trained on a separate dataset. Manual alignment was performed using in-house software that allowed labelling of six key points located at vessel bifurcations. Manual overlay was considered successful if the distance between same key points on the overlayed images was ≤1/2°. RESULTS: Fifty-seven eyes of 32 patients were included in the analysis. AI was significantly more accurate and successful in aligning images compared to manual alignment as confirmed by linear mixed-effects modelling (p < 0.001). A receiver operating characteristic analysis, used to compute the area under the curve of the AI (0.991) and manual (0.835) Dice coefficients in relation to their respective 'truth' values, found AI significantly more accurate in the overlay (p < 0.001). CONCLUSION: AI was significantly more accurate than manual alignment in overlaying multimodal retinal imaging in RP patients and showed the potential to use AI algorithms for future multimodal clinical and research applications.

Accuracy and Time Comparison Between Side-by-Side and Artificial Intelligence Overlayed Images.

BACKGROUND AND OBJECTIVE: The purpose of this study was to evaluate the accuracy and the time to find a lesion, taken in different platforms, color fundus photographs and infrared scanning laser ophthalmoscope images, using the traditional side-by-side (SBS) colocalization technique to an artificial intelligence (AI)-assisted technique. PATIENTS AND METHODS: Fifty-three pathological lesions were studied in 11 eyes. Images were aligned using SBS and AI overlaid methods. The location of each color fundus lesion on the corresponding infrared scanning laser ophthalmoscope image was analyzed twice, one time for each method, on different days, for two specialists, in random order. The outcomes for each method were measured and recorded by an independent observer. RESULTS: The colocalization AI method was superior to the conventional in accuracy and time (P < .001), with a mean time to colocalize 37% faster. The error rate using AI was 0% compared with 18% in SBS measurements. CONCLUSIONS: AI permitted a more accurate and faster colocalization of pathologic lesions than the conventional method. [Ophthalmic Surg Lasers Imaging Retina 2023;54:108-113.].

Low-latency label-free image-activated cell sorting using fast deep learning and AI inferencing.

Classification and sorting of cells using image-activated cell sorting (IACS) systems can bring significant insight to biomedical sciences. Incorporating deep learning algorithms into IACS enables cell classification and isolation based on complex and human-vision uninterpretable morphological features within a heterogeneous cell population. However, the limited capabilities and complicated implementation of deep learning-assisted IACS systems reported to date hinder the adoption of the systems for a wide range of biomedical research. Here, we present image-activated cell sorting by applying fast deep learning algorithms to conduct cell sorting without labeling. The overall sorting latency, including signal processing and AI inferencing, is less than 3 ms, and the training time for the deep learning model is less than 30 min with a training dataset of 20,000 images. Both values set the record for IACS with sorting by AI inference. . We demonstrated our system performance through a 2-part polystyrene beads sorting experiment with 96.6% sorting purity, and a 3-part human leukocytes sorting experiment with 89.05% sorting purity for monocytes, 92.00% sorting purity for lymphocytes, and 98.24% sorting purity for granulocytes. The above performance was achieved with simple hardware containing only 1 FPGA, 1 PC and GPU, as a result of an optimized custom CNN UNet and efficient use of computing power. The system provides a compact, sterile, low-cost, label-free, and low-latency cell sorting solution based on real-time AI inferencing and fast training of the deep learning model.

Self-Supervised Rigid Registration for Multimodal Retinal Images.

The ability to accurately overlay one modality retinal image to another is critical in ophthalmology. Our previous framework achieved the state-of-the-art results for multimodal retinal image registration. However, it requires human-annotated labels due to the supervised approach of the previous work. In this paper, we propose a self-supervised multimodal retina registration method to alleviate the burdens of time and expense to prepare for training data, that is, aiming to automatically register multimodal retinal images without any human annotations. Specially, we focus on registering color fundus images with infrared reflectance and fluorescein angiography images, and compare registration results with several conventional and supervised and unsupervised deep learning methods. From the experimental results, the proposed self-supervised framework achieves a comparable accuracy comparing to the state-of-the-art supervised learning method in terms of registration accuracy and Dice coefficient.

A high-throughput technique to map cell images to cell positions using a 3D imaging flow cytometer.

We develop a high-throughput technique to relate positions of individual cells to their three-dimensional (3D) imaging features with single-cell resolution. The technique is particularly suitable for nonadherent cells where existing spatial biology methodologies relating cell properties to their positions in a solid tissue do not apply. Our design consists of two parts, as follows: recording 3D cell images at high throughput (500 to 1,000 cells/s) using a custom 3D imaging flow cytometer (3D-IFC) and dispensing cells in a first-in-first-out (FIFO) manner using a robotic cell placement platform (CPP). To prevent errors due to violations of the FIFO principle, we invented a method that uses marker beads and DNA sequencing software to detect errors. Experiments with human cancer cell lines demonstrate the feasibility of mapping 3D side scattering and fluorescent images, as well as two-dimensional (2D) transmission images of cells to their locations on the membrane filter for around 100,000 cells in less than 10 min. While the current work uses our specially designed 3D imaging flow cytometer to produce 3D cell images, our methodology can support other imaging modalities. The technology and method form a bridge between single-cell image analysis and single-cell molecular analysis.

JOINT MOTION CORRECTION AND 3D SEGMENTATION WITH GRAPH-ASSISTED NEURAL NETWORKS FOR RETINAL OCT.

Optical Coherence Tomography (OCT) is a widely used non-invasive high resolution 3D imaging technique for biological tissues and plays an important role in ophthalmology. OCT retinal layer segmentation is a fundamental image processing step for OCT-Angiography projection, and disease analysis. A major problem in retinal imaging is the motion artifacts introduced by involuntary eye movements. In this paper, we propose neural networks that jointly correct eye motion and retinal layer segmentation utilizing 3D OCT information, so that the segmentation among neighboring B-scans would be consistent. The experimental results show both visual and quantitative improvements by combining motion correction and 3D OCT layer segmentation comparing to conventional and deep-learning based 2D OCT layer segmentation.

Two-Step Registration on Multi-Modal Retinal Images via Deep Neural Networks.

Multi-modal retinal image registration plays an important role in the ophthalmological diagnosis process. The conventional methods lack robustness in aligning multi-modal images of various imaging qualities. Deep-learning methods have not been widely developed for this task, especially for the coarse-to-fine registration pipeline. To handle this task, we propose a two-step method based on deep convolutional networks, including a coarse alignment step and a fine alignment step. In the coarse alignment step, a global registration matrix is estimated by three sequentially connected networks for vessel segmentation, feature detection and description, and outlier rejection, respectively. In the fine alignment step, a deformable registration network is set up to find pixel-wise correspondence between a target image and a coarsely aligned image from the previous step to further improve the alignment accuracy. Particularly, an unsupervised learning framework is proposed to handle the difficulties of inconsistent modalities and lack of labeled training data for the fine alignment step. The proposed framework first changes multi-modal images into a same modality through modality transformers, and then adopts photometric consistency loss and smoothness loss to train the deformable registration network. The experimental results show that the proposed method achieves state-of-the-art results in Dice metrics and is more robust in challenging cases.

Perspective Distortion Correction for Multi-Modal Registration between Ultra-Widefield and Narrow-Angle Retinal Images.

Multi-modal retinal image registration between 2D Ultra-Widefield (UWF) and narrow-angle (NA) images has not been well-studied, since most existing methods mainly focus on NA image alignment. The stereographic projection model used in UWF imaging causes strong distortions in peripheral areas, which leads to inferior alignment quality. We propose a distortion correction method that remaps the UWF images based on estimated camera view points of NA images. In addition, we set up a CNN-based registration pipeline for UWF and NA images, which consists of the distortion correction method and three networks for vessel segmentation, feature detection and matching, and outlier rejection. Experimental results on our collected dataset shows the effectiveness of the proposed pipeline and the distortion correction method.

Robust Content-Adaptive Global Registration for Multimodal Retinal Images Using Weakly Supervised Deep-Learning Framework.

Multimodal retinal imaging plays an important role in ophthalmology. We propose a content-adaptive multimodal retinal image registration method in this paper that focuses on the globally coarse alignment and includes three weakly supervised neural networks for vessel segmentation, feature detection and description, and outlier rejection. We apply the proposed framework to register color fundus images with infrared reflectance and fluorescein angiography images, and compare it with several conventional and deep learning methods. Our proposed framework demonstrates a significant improvement in robustness and accuracy reflected by a higher success rate and Dice coefficient compared with other methods.

A Convolutional Neural Network Pipeline For Multi-Temporal Retinal Image Registration.

A sequence of images is usually captured to observe the change of health status in medical diagnosis. However, an image sequence taken over year usually suffers from severe deformation, making it time-consuming for physicians to match corresponding patterns. In this paper, we propose a coarse-to-fine pipeline for retinal image registration based on convolutional neural network. By leveraging the three components of the pipeline: feature matching, outlier rejection, and local registration, we recover the deformation and accurately align multi-temporal image sequences. Experimental results show that the proposed network is robust to severe deformation as well as illumination and contrast variations. With the proposed registration pipeline, the change of image patterns over time can be identified through visual analysis.

LEARNING TO CORRECT AXIAL MOTION IN OCT FOR 3D RETINAL IMAGING.

Optical Coherence Tomography (OCT) is a powerful technique for non-invasive 3D imaging of biological tissues at high resolution that has revolutionized retinal imaging. A major challenge in OCT imaging is the motion artifacts introduced by involuntary eye movements. In this paper, we propose a convolutional neural network that learns to correct axial motion in OCT based on a single volumetric scan. The proposed method is able to correct large motion, while preserving the overall curvature of the retina. The experimental results show significant improvements in visual quality as well as overall error compared to the conventional methods in both normal and disease cases.

Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images.

Purpose: The purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI). Methods: We collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image. Results: Our new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders (P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method. Conclusions: AI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks. Translational Relevance: The ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment.

Resource allocation for error resilient video coding over AWGN using optimization approach.

The number of slices for error resilient video coding is jointly optimized with 802.11a-like media access control and the physical layers with automatic repeat request and rate compatible punctured convolutional code over additive white gaussian noise channel as well as channel times allocation for time division multiple access. For error resilient video coding, the relation between the number of slices and coding efficiency is analyzed and formulated as a mathematical model. It is applied for the joint optimization problem, and the problem is solved by a convex optimization method such as the primal-dual decomposition method. We compare the performance of a video communication system which uses the optimal number of slices with one that codes a picture as one slice. From numerical examples, end-to-end distortion of utility functions can be significantly reduced with the optimal slices of a picture especially at low signal-to-noise ratio.

Iterative rate-distortion optimization of H.264 with constant bit rate constraint.

In this paper, we apply the primal-dual decomposition and subgradient projection methods to solve the rate-distortion optimization problem with the constant bit rate constraint. The primal decomposition method enables spatial or temporal prediction dependency within a group of picture (GOP) to be processed in the master primal problem. As a result, we can apply the dual decomposition to minimize independently the Lagrangian cost of all the MBs using the reference software model of H.264. Furthermore, the optimal Lagrange multiplier lambda* is iteratively derived from the solution of the dual problem. As an example, we derive the optimal bit allocation condition with the consideration of temporal prediction dependency among the pictures. Experimental results show that the proposed method achieves better performance than the reference software model of H.264 with rate control.