Development of an automated combined positive score prediction pipeline using artificial intelligence on multiplexed immunofluorescence images.

Immunotherapy targeting immune checkpoint proteins, such as programmed cell death ligand 1 (PD-L1), has shown impressive outcomes in many clinical trials but only 20%-40% of patients benefit from it. Utilizing Combined Positive Score (CPS) to evaluate PD-L1 expression in tumour biopsies to identify patients with the highest likelihood of responsiveness to anti-PD-1/PD-L1 therapy has been approved by the Food and Drug Administration for several solid tumour types. Current CPS workflow requires a pathologist to manually score the two-colour PD-L1 chromogenic immunohistochemistry image. Multiplex immunofluorescence (mIF) imaging reveals the expression of an increased number of immune markers in tumour biopsies and has been used extensively in immunotherapy research. Recent rapid progress of Artificial Intelligence (AI)-based imaging analysis, particularly Deep Learning, provides cost effective and high-quality solutions to healthcare. In this article, we propose an imaging pipeline that utilizes three-colour mIF images (DAPI, PD-L1, and Pan-cytokeratin) as input and predicts the CPS using AI techniques. Our novel pipeline is composed of three modules employing algorithms of image processing, machine learning, and deep learning techniques. The first module of quality check (QC) detects and removes the image regions contaminated with sectioning and staining artefacts. The QC module ensures that only image regions free of the three common artefacts are used for downstream analysis. The second module of nuclear segmentation uses deep learning to segment and count nuclei in the DAPI images wherein our specialized method can accurately separate touching nuclei. The third module of cell phenotyping calculates CPS by identifying and counting PD-L1 positive cells and tumour cells. These modules are data-efficient and require only few manual annotations for training purposes. Using tumour biopsies from a clinical trial, we found that the CPS from the AI-based models shows a high Spearman correlation (78%, p = 0.003) to the pathologist-scored CPS.

Spatial-context-aware RNA-sequence prediction from head and neck cancer histopathology images.

Molecular profiling of the tumor in addition to the histological tumor analysis can provide robust information for targeted cancer therapies. Often such data are not available for analysis due to processing delays, cost or inaccessibility. In this paper, we proposed a deep learning-based method to predict RNA-sequence expression (RNA-seq) from Hematoxylin and Eosin whole-slide images (H&E WSI) in head and neck cancer patients. Conventional methods utilize a patch-by-patch prediction and aggregation strategy to predict RNA-seq at a whole-slide level. However, these methods lose spatial-contextual relationships between patches that comprise morphology interactions crucial for predicting RNA-seq. We proposed a novel framework that employs a neural image compressor to preserve the spatial relationships between patches and generate a compressed representation of the whole-slide image, and a customized deep-learning regressor to predict RNA-seq from the compressed representation by learning both global and local features. We tested our proposed method on publicly available TCGA-HNSC dataset comprising 43 test patients for 10 oncogenes. Our experiments showed that the proposed method achieves a 4.12% higher mean correlation and predicts 6 out of 10 genes with better correlation than a state-of-the-art baseline method. Furthermore, we provided interpretability using pathway analysis of the best-predicted genes, and activation maps to highlight the regions in an H&E image that are the most salient of the RNA-seq prediction.Clinical relevance-The proposed method has the potential to discover genetic biomarkers directly from the histopathology images which could be used to pre-screen the patients before actual genetic testing thereby saving cost and time.

Dual Encoder Attention U-net for Nuclei Segmentation.

Nuclei segmentation in whole slide images (WSIs) stained with Hematoxylin and Eosin (H&E) dye, is a key step in computational pathology which aims to automate the laborious process of manual counting and segmentation. Nuclei segmentation is a challenging problem that involves challenges such as touching nuclei resolution, small-sized nuclei, size, and shape variations. With the advent of deep learning, convolution neural networks (CNNs) have shown a powerful ability to extract effective representations from microscopic H&E images. We propose a novel dual encoder Attention U-net (DEAU) deep learning architecture and pseudo hard attention gating mechanism, to enhance the attention to target instances. We added a new secondary encoder to the attention U-net to capture the best attention for a given input. Since H captures nuclei information, we propose a stain-separated H channel as input to the secondary encoder. The role of the secondary encoder is to transform attention prior to different spatial resolutions while learning significant attention information. The proposed DEAU performance was evaluated on three publicly available H&E data sets for nuclei segmentation from different research groups. Experimental results show that our approach outperforms other attention-based approaches for nuclei segmentation.

Blur-Robust Nuclei Segmentation for Immunofluorescence Images.

Automated nuclei segmentation from immunofluorescence (IF) microscopic image is a crucial first step in digital pathology. A lot of research has been devoted to develop novel nuclei segmentation algorithms to give high performance on good quality images. However, fewer methods were developed for poor-quality images like out-of-focus (blurry) data. In this work, we take a principled approach to study the performance of nuclei segmentation algorithms on out-of-focus images for different levels of blur. A deep learning encoder-decoder framework with a novel Y forked decoder is proposed here. The two fork ends are tied to segmentation and deblur output. The addition of a separate deblurring task in the training paradigm helps to regularize the network on blurry images. Our proposed method accurately predicts the instance nuclei segmentation on sharp as well as out-of-focus images. Additionally, predicted deblurred image provides interpretable insights to experts. Experimental analysis on the Human U2OS cells (out-of-focus) dataset shows that our algorithm is robust and outperforms the state-of-the-art methods.

Functional connectivity with cortical depth assessed by resting state fMRI of subregions of S1 in squirrel monkeys.

Whereas resting state blood oxygenation-level dependent (BOLD) functional MRI has been widely used to assess functional connectivity between cortical regions, the laminar specificity of such measures is poorly understood. This study aims to determine: (a) whether the resting state functional connectivity (rsFC) between two functionally related cortical regions varies with cortical depth, (b) the relationship between layer-resolved tactile stimulus-evoked activation pattern and interlayer rsFC pattern between two functionally distinct but related somatosensory areas 3b and 1, and (c) the effects of spatial resolution on rsFC measures. We examined the interlayer rsFC between areas 3b and 1 of squirrel monkeys under anesthesia using tactile stimulus-driven and resting state BOLD acquisitions at submillimeter resolution. Consistent with previous observations in the areas 3b and 1, we detected robust stimulus-evoked BOLD activations with foci were confined mainly to the upper layers (centered at 21% of the cortical depth). By carefully placing seeds in upper, middle, and lower layers of areas 3b and 1, we observed strong rsFC between upper and middle layers of these two areas. The layer-resolved activation patterns in areas 3b and 1 agree with their interlayer rsFC patterns, and are consistent with the known anatomical connections between layers. In summary, using BOLD rsFC pattern, we identified an interlayer interareal microcircuit that shows strong intrinsic functional connections between upper and middle layer areas 3b and 1. RsFC can be used as a robust invasive tool to probe interlayer corticocortical microcircuits.

Integration of resting-state FMRI and diffusion-weighted MRI connectivity analyses of the human brain: limitations and improvement.

BACKGROUND: Integration of functional connectivity analysis based on resting-state functional Magnetic Resonance Imaging (fMRI) and structural connectivity analysis based on Diffusion-Weighted Imaging (DWI) has shown great potential to improve understanding of the neural networks in the human brain. However, there are sensitivity and specificity-related interpretation issues that must be addressed. METHODS: We assessed the long-range functional and structural connections of the default-mode, attention, visual and motor networks on 25 healthy subjects. For each network, we first integrated these two analyses based on one common seed region. We then introduced a functional-assisted fiber tracking strategy, where seed regions were defined based on independent component analysis of the resting-state fMRI dataset. RESULTS: The single-seed based technique successfully identified the expected functional connections within these networks at both subject and group levels. However, the success rate of structural connectivity analysis showed a high level of variation among the subjects. The functional-assisted fiber tracking strategy highly improved the rate of successful fiber tracking. CONCLUSIONS: This fMRI/DWI integration study suggests that functional connectivity analysis might be a more sensitive and robust approach in understanding the connectivity between cortical regions, and can be used to improve DWI-based structural connectivity analysis.

Alzheimer's disease and amnestic mild cognitive impairment weaken connections within the default-mode network: a multi-modal imaging study.

We applied a multi-modal imaging approach to examine structural and functional alterations in the default-mode network (DMN) that are associated with Alzheimer's disease (AD) and amnestic mild cognitive impairment (aMCI), a transitional phase between healthy cognitive aging and dementia. Subjects included 10 patients with probable AD, 11 patients with aMCI, and 12 age- and education-matched normal controls (NC). Whole-brain resting-state functional, diffusion-weighted, and volumetric magnetic resonance imaging (MRI) data as well as 18F-fluorodeoxyglucose-based positron emission tomography (FDG-PET) data were acquired. We carried out resting-state functional MRI-based functional connectivity and diffusion MRI-based structural connectivity analyses using isthmus of the cingulate cortex (ICC) and the subjacent white matter as the seeds. Whole-brain group and region of interest-based analyses demonstrated that AD weakens the structural and functional connections between ICC and other regions within the DMN, consistent with regional reduction of metabolic activity and atrophy within the DMN. A progressive weakening trend of these connections was also observed from NC to aMCI and then AD, although significant differences between aMCI and the other two groups were not found. Overall, based on both FDG-PET and MRI results, the DMN appears to serve as a window to understanding structural and functional brain changes associated with AD and aMCI.

A diffusion gradient optimization framework for spinal cord diffusion tensor imaging.

The uncertainty in the estimation of diffusion model parameters in diffusion tensor imaging (DTI) can be reduced by optimally selecting the diffusion gradient directions utilizing some prior structural information. This is beneficial for spinal cord DTI, where the magnetic resonance images have low signal-to-noise ratio and thus high uncertainty in diffusion model parameter estimation. Presented is a gradient optimization scheme based on D-optimality, which reduces the overall estimation uncertainty by minimizing the Rician Cramer-Rao lower bound of the variance of the model parameter estimates. The tensor-based diffusion model for DTI is simplified to a four-parameter axisymmetric DTI model where diffusion transverse to the principal eigenvector of the tensor is assumed isotropic. Through simulations and experimental validation, we demonstrate that an optimized gradient scheme based on D-optimality is able to reduce the overall uncertainty in the estimation of diffusion model parameters for the cervical spinal cord and brain stem white matter tracts.