Image harmonization: A review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization.

Magnetic resonance imaging and computed tomography from multiple batches (e.g. sites, scanners, datasets, etc.) are increasingly used alongside complex downstream analyses to obtain new insights into the human brain. However, significant confounding due to batch-related technical variation, called batch effects, is present in this data; direct application of downstream analyses to the data may lead to biased results. Image harmonization methods seek to remove these batch effects and enable increased generalizability and reproducibility of downstream results. In this review, we describe and categorize current approaches in statistical and deep learning harmonization methods. We also describe current evaluation metrics used to assess harmonization methods and provide a standardized framework to evaluate newly-proposed methods for effective harmonization and preservation of biological information. Finally, we provide recommendations to end-users to advocate for more effective use of current methods and to methodologists to direct future efforts and accelerate development of the field.

Ensemble Approaches to Recognize Protected Health Information in Radiology Reports.

Natural language processing (NLP) techniques for electronic health records have shown great potential to improve the quality of medical care. The text of radiology reports frequently constitutes a large fraction of EHR data, and can provide valuable information about patients' diagnoses, medical history, and imaging findings. The lack of a major public repository for radiological reports severely limits the development, testing, and application of new NLP tools. De-identification of protected health information (PHI) presents a major challenge to building such repositories, as many automated tools for de-identification were trained or designed for clinical notes and do not perform sufficiently well to build a public database of radiology reports. We developed and evaluated six ensemble models based on three publically available de-identification tools: MIT de-id, NeuroNER, and Philter. A set of 1023 reports was set aside as the testing partition. Two individuals with medical training annotated the test set for PHI; differences were resolved by consensus. Ensemble methods included simple voting schemes (1-Vote, 2-Votes, and 3-Votes), a decision tree, a naïve Bayesian classifier, and Adaboost boosting. The 1-Vote ensemble achieved recall of 998 / 1043 (95.7%); the 3-Votes ensemble had precision of 1035 / 1043 (99.2%). F1 scores were: 93.4% for the decision tree, 71.2% for the naïve Bayesian classifier, and 87.5% for the boosting method. Basic voting algorithms and machine learning classifiers incorporating the predictions of multiple tools can outperform each tool acting alone in de-identifying radiology reports. Ensemble methods hold substantial potential to improve automated de-identification tools for radiology reports to make such reports more available for research use to improve patient care and outcomes.

3D printed vascular phantoms for high-resolution biophotonic image quality assessment via direct laser writing.

Fluorescence imaging techniques such as fluorescein angiography and fundus autofluorescence are often used to diagnose retinal pathologies; however, there are currently no standardized test methods for evaluating device performance. Here we present microstructured fluorescent phantoms fabricated using a submicron-scale three-dimensional printing technology, direct laser writing (DLW). We employ an in situ DLW technique to print 10 µm diameter microfluidic channels that support perfusions of fluorescent dyes. We then demonstrate how broadband photoresist fluorescence can be exploited to generate resolution targets and biomimetic models of retinal vasculature using standard DLW processes. The results indicate that these approaches show significant promise for generating better performance evaluation tools for fluorescence microscopy and imaging devices.

Multivariate testing and effect size measures for batch effect evaluation in radiomic features.

While precision medicine applications of radiomics analysis are promising, differences in image acquisition can cause "batch effects" that reduce reproducibility and affect downstream predictive analyses. Harmonization methods such as ComBat have been developed to correct these effects, but evaluation methods for quantifying batch effects are inconsistent. In this study, we propose the use of the multivariate statistical test PERMANOVA and the Robust Effect Size Index (RESI) to better quantify and characterize batch effects in radiomics data. We evaluate these methods in both simulated and real radiomics features extracted from full-field digital mammography (FFDM) data. PERMANOVA demonstrated higher power than standard univariate statistical testing, and RESI was able to interpretably quantify the effect size of site at extremely large sample sizes. These methods show promise as more powerful and interpretable methods for the detection and quantification of batch effects in radiomics studies.

Radiomic phenotyping of the lung parenchyma in a lung cancer screening cohort.

High-throughput extraction of radiomic features from low-dose CT scans can characterize the heterogeneity of the lung parenchyma and potentially aid in identifying subpopulations that may have higher risk of lung diseases, such as COPD, and lung cancer due to inflammation or obstruction of the airways. We aim to determine the feasibility of a lung radiomics phenotyping approach in a lung cancer screening cohort, while quantifying the effect of different CT reconstruction algorithms on phenotype robustness. We identified low-dose CT scans (n = 308) acquired with Siemens Healthineers scanners from patients who completed low-dose CT within our lung cancer screening program between 2015 and 2018 and had two different sets of image reconstructions kernel available (i.e., medium (I30f.), sharp (I50f.)) for the same acquisition. Following segmentation of the lung field, a total of 26 radiomic features were extracted from the entire 3D lung-field using a previously validated fully-automated lattice-based software pipeline, adapted for low-dose CT scans. The lattice in-house software was used to extract features including gray-level histogram, co-occurrence, and run-length descriptors. The lattice approach uses non-overlapping windows for traversing along pixels of images and calculates different features. Each feature was averaged for each scan within a range of lattice window sizes (W) of 4, 8 and 20 mm. The extracted imaging features from both datasets were harmonized to correct for differences in image acquisition parameters. Subsequently, unsupervised hierarchical clustering was applied on the extracted features to identify distinct phenotypic patterns of the lung parenchyma, where consensus clustering was used to identify the optimal number of clusters (K = 2). Differences between phenotypes for demographic and clinical covariates including sex, age, BMI, pack-years of smoking, Lung-RADS and cancer diagnosis were assessed for each phenotype cluster, and then compared across clusters for the two different CT reconstruction algorithms using the cluster entanglement metric, where a lower entanglement coefficient corresponds to good cluster alignment. Furthermore, an independent set of low-dose CT scans (n = 88) from patients with available pulmonary function data on lung obstruction were analyzed using the identified optimal clusters to assess associations to lung obstruction and validate the lung phenotyping paradigm. Heatmaps generated by radiomic features identified two distinct lung parenchymal phenotype patterns across different feature extraction window sizes, for both reconstruction algorithms (P < 0.05 with K = 2). Associations of radiomic-based clusters with clinical covariates showed significant differences for BMI and pack-years of smoking (P < 0.05) for both reconstruction kernels. Radiomic phenotype patterns were more similar across the two reconstructed kernels, when smaller window sizes (W = 4 and 8 mm) were used for radiomic feature extraction, as deemed by their entanglement coefficient. Validation of clustering approaches using cluster mapping for the independent sample with lung obstruction also showed two statistically significant phenotypes (P < 0.05) with significant difference for BMI and smoking pack-years. Radiomic analysis can be used to characterize lung parenchymal phenotypes from low-dose CT scans, which appear reproducible for different reconstruction kernels. Further work should seek to evaluate the effect of additional CT acquisition parameters and validate these phenotypes in characterizing lung cancer screening populations, to potentially better stratify disease patterns and cancer risk.

DeepComBat: A Statistically Motivated, Hyperparameter-Robust, Deep Learning Approach to Harmonization of Neuroimaging Data.

Neuroimaging data from multiple batches (i.e. acquisition sites, scanner manufacturer, datasets, etc.) are increasingly necessary to gain new insights into the human brain. However, multi-batch data, as well as extracted radiomic features, exhibit pronounced technical artifacts across batches. These batch effects introduce confounding into the data and can obscure biological effects of interest, decreasing the generalizability and reproducibility of findings. This is especially true when multi-batch data is used alongside complex downstream analysis models, such as machine learning methods. Image harmonization methods seeking to remove these batch effects are important for mitigating these issues; however, significant multivariate batch effects remain in the data following harmonization by current state-of-the-art statistical and deep learning methods. We present DeepCombat, a deep learning harmonization method based on a conditional variational autoencoder architecture and the ComBat harmonization model. DeepCombat learns and removes subject-level batch effects by accounting for the multivariate relationships between features. Additionally, DeepComBat relaxes a number of strong assumptions commonly made by previous deep learning harmonization methods and is empirically robust across a wide range of hyperparameter choices. We apply this method to neuroimaging data from a large cognitive-aging cohort and find that DeepCombat outperforms existing methods, as assessed by a battery of machine learning methods, in removing scanner effects from cortical thickness measurements while preserving biological heterogeneity. Additionally, DeepComBat provides a new perspective for statistically-motivated deep learning harmonization methods.

Automated machine learning for differentiation of hepatocellular carcinoma from intrahepatic cholangiocarcinoma on multiphasic MRI.

With modern management of primary liver cancer shifting towards non-invasive diagnostics, accurate tumor classification on medical imaging is increasingly critical for disease surveillance and appropriate targeting of therapy. Recent advancements in machine learning raise the possibility of automated tools that can accelerate workflow, enhance performance, and increase the accessibility of artificial intelligence to clinical researchers. We explore the use of an automated Tree-Based Optimization Tool that leverages a genetic programming algorithm for differentiation of the two common primary liver cancers on multiphasic MRI. Manual and automated analyses were performed to select an optimal machine learning model, with an accuracy of 73-75% (95% CI 0.59-0.85), sensitivity of 70-75% (95% CI 0.48-0.89), and specificity of 71-79% (95% CI 0.52-0.90) on manual optimization, and an accuracy of 73-75% (95% CI 0.59-0.85), sensitivity of 65-75% (95% CI 0.43-0.89) and specificity of 75-79% (95% CI 0.56-0.90) for automated machine learning. We found that automated machine learning performance was similar to that of manual optimization, and it could classify hepatocellular carcinoma and intrahepatic cholangiocarcinoma with an sensitivity and specificity comparable to that of radiologists. However, automated machine learning performance was poor on a subset of scans that met LI-RADS criteria for LR-M. Exploration of additional feature selection and classifier methods with automated machine learning to improve performance on LR-M cases as well as prospective validation in the clinical setting are needed prior to implementation.

Generalized ComBat harmonization methods for radiomic features with multi-modal distributions and multiple batch effects.

Radiomic features have a wide range of clinical applications, but variability due to image acquisition factors can affect their performance. The harmonization tool ComBat is a promising solution but is limited by inability to harmonize multimodal distributions, unknown imaging parameters, and multiple imaging parameters. In this study, we propose two methods for addressing these limitations. We propose a sequential method that allows for harmonization of radiomic features by multiple imaging parameters (Nested ComBat). We also employ a Gaussian Mixture Model (GMM)-based method (GMM ComBat) where scans are split into groupings based on the shape of the distribution used for harmonization as a batch effect and subsequent harmonization by a known imaging parameter. These two methods were evaluated on features extracted with CapTK and PyRadiomics from two public lung computed tomography datasets. We found that Nested ComBat exhibited similar performance to standard ComBat in reducing the percentage of features with statistically significant differences in distribution attributable to imaging parameters. GMM ComBat improved harmonization performance over standard ComBat (- 11%, - 10% for Lung3/CAPTK, Lung3/PyRadiomics harmonizing by kernel resolution). Features harmonized with a variant of the Nested method and the GMM split method demonstrated similar c-statistics and Kaplan-Meier curves when used in survival analyses.

Resampling and harmonization for mitigation of heterogeneity in image parameters of baseline scans.

Our study investigates the effects of heterogeneity in image parameters on the reproducibility of prognostic performance of models built using radiomic biomarkers. We compare the prognostic performance of models derived from the heterogeneity-mitigated features with that of models obtained from raw features, to assess whether reproducibility of prognostic scores improves upon application of our methods. We used two datasets: The Breast I-SPY1 dataset-Baseline DCE-MRI scans of 156 women with locally advanced breast cancer, treated with neoadjuvant chemotherapy, publicly available via The Cancer Imaging Archive (TCIA); The NSCLC IO dataset-Baseline CT scans of 107 patients with stage 4 non-small cell lung cancer (NSCLC), treated with pembrolizumab immunotherapy at our institution. Radiomic features (n = 102) are extracted from the tumor ROIs. We use a variety of resampling and harmonization scenarios to mitigate the heterogeneity in image parameters. The patients were divided into groups based on batch variables. For each group, the radiomic phenotypes are combined with the clinical covariates into a prognostic model. The performance of the groups is assessed using the c-statistic, derived from a Cox proportional hazards model fitted on all patients within a group. The heterogeneity-mitigation scenario (radiomic features, derived from images that have been resampled to minimum voxel spacing, are harmonized using the image acquisition parameters as batch variables) gave models with highest prognostic scores (for e.g., IO dataset; batch variable: high kernel resolution-c-score: 0.66). The prognostic performance of patient groups is not comparable in case of models built using non-heterogeneity mitigated features (for e.g., I-SPY1 dataset; batch variable: small pixel spacing-c-score: 0.54, large pixel spacing-c-score: 0.65). The prognostic performance of patient groups is closer in case of heterogeneity-mitigated scenarios (for e.g., scenario-harmonize by voxel spacing parameters: IO dataset; thin slice-c-score: 0.62, thick slice-c-score: 0.60). Our results indicate that accounting for heterogeneity in image parameters is important to obtain more reproducible prognostic scores, irrespective of image site or modality. For non-heterogeneity mitigated models, the prognostic scores are not comparable across patient groups divided based on batch variables. This study can be a step in the direction of constructing reproducible radiomic biomarkers, thus increasing their application in clinical decision making.

Improved generalized ComBat methods for harmonization of radiomic features.

Radiomic approaches in precision medicine are promising, but variation associated with image acquisition factors can result in severe biases and low generalizability. Multicenter datasets used in these studies are often heterogeneous in multiple imaging parameters and/or have missing information, resulting in multimodal radiomic feature distributions. ComBat is a promising harmonization tool, but it only harmonizes by single/known variables and assumes standardized input data are normally distributed. We propose a procedure that sequentially harmonizes for multiple batch effects in an optimized order, called OPNested ComBat. Furthermore, we propose to address bimodality by employing a Gaussian Mixture Model (GMM) grouping considered as either a batch variable (OPNested + GMM) or as a protected clinical covariate (OPNested - GMM). Methods were evaluated on features extracted with CapTK and PyRadiomics from two public lung computed tomography (CT) datasets. We found that OPNested ComBat improved harmonization performance over standard ComBat. OPNested + GMM ComBat exhibited the best harmonization performance but the lowest predictive performance, while OPNested - GMM ComBat showed poorer harmonization performance, but the highest predictive performance. Our findings emphasize that improved harmonization performance is no guarantee of improved predictive performance, and that these methods show promise for superior standardization of datasets heterogeneous in multiple or unknown imaging parameters and greater generalizability.

Radiomic Phenotypes for Improving Early Prediction of Survival in Stage III Non-Small Cell Lung Cancer Adenocarcinoma after Chemoradiation.

We evaluate radiomic phenotypes derived from CT scans as early predictors of overall survival (OS) after chemoradiation in stage III primary lung adenocarcinoma. We retrospectively analyzed 110 thoracic CT scans acquired between April 2012-October 2018. Patients received a median radiation dose of 66.6 Gy at 1.8 Gy/fraction delivered with proton (55.5%) and photon (44.5%) beam treatment, as well as concurrent chemotherapy (89%) with carboplatin-based (55.5%) and cisplatin-based (36.4%) doublets. A total of 56 death events were recorded. Using manual tumor segmentations, 107 radiomic features were extracted. Feature harmonization using ComBat was performed to mitigate image heterogeneity due to the presence or lack of intravenous contrast material and variability in CT scanner vendors. A binary radiomic phenotype to predict OS was derived through the unsupervised hierarchical clustering of the first principal components explaining 85% of the variance of the radiomic features. C-scores and likelihood ratio tests (LRT) were used to compare the performance of a baseline Cox model based on ECOG status and age, with a model integrating the radiomic phenotype with such clinical predictors. The model integrating the radiomic phenotype (C-score = 0.69, 95% CI = (0.62, 0.77)) significantly improved (p<0.005) upon the baseline model (C-score = 0.65, CI = (0.57, 0.73)). Our results suggest that harmonized radiomic phenotypes can significantly improve OS prediction in stage III NSCLC after chemoradiation.

Development of a robust radiomic biomarker of progression-free survival in advanced non-small cell lung cancer patients treated with first-line immunotherapy.

We aim to determine the feasibility of a novel radiomic biomarker that can integrate with other established clinical prognostic factors to predict progression-free survival (PFS) in patients with non-small cell lung cancer (NSCLC) undergoing first-line immunotherapy. Our study includes 107 patients with stage 4 NSCLC treated with pembrolizumab-based therapy (monotherapy: 30%, combination chemotherapy: 70%). The ITK-SNAP software was used for 3D tumor volume segmentation from pre-therapy CT scans. Radiomic features (n = 102) were extracted using the CaPTk software. Impact of heterogeneity introduced by image physical dimensions (voxel spacing parameters) and acquisition parameters (contrast enhancement and CT reconstruction kernel) was mitigated by resampling the images to the minimum voxel spacing parameters and harmonization by a nested ComBat technique. This technique was initialized with radiomic features, clinical factors of age, sex, race, PD-L1 expression, ECOG status, body mass index (BMI), smoking status, recurrence event and months of progression-free survival, and image acquisition parameters as batch variables. Two phenotypes were identified using unsupervised hierarchical clustering of harmonized features. Prognostic factors, including PDL1 expression, ECOG status, BMI and smoking status, were combined with radiomic phenotypes in Cox regression models of PFS and Kaplan Meier (KM) curve-fitting. Cox model based on clinical factors had a c-statistic of 0.57, which increased to 0.63 upon addition of phenotypes derived from harmonized features. There were statistically significant differences in survival outcomes stratified by clinical covariates, as measured by the log-rank test (p = 0.034), which improved upon addition of phenotypes (p = 0.00022). We found that mitigation of heterogeneity by image resampling and nested ComBat harmonization improves prognostic value of phenotypes, resulting in better prediction of PFS when added to other prognostic variables.

Morphological and functional characteristics of aging kidneys based on two-photon microscopy in vivo.

Age-related kidney disease, which is chronic and naturally occurring, is a general term for a set of heterogeneous disorders affecting kidney structures and characterized by a decline in renal function. Age-related renal insufficiency has important implications with regard to body homeostasis, drug toxicity and renal transplantation. In our study, two-photon microscopy was used to image kidney morphological and functional characteristics in an age-related rat model in vivo. The changes in morphology are analyzed based on autofluorescence and Hoechst 33342 labeling in rats with different ages. Structural parameters including renal tubular diameter, cell nuclei density, size and shape are studied and compared with Hematoxylin and Eosin histological analysis. Functional characteristics, such as blood flow, and glomerular filtration rate are studied with high-molecular weight (MW) 500-kDa dextran-fluorescein and low-MW 10-kDa dextran-rhodamine. Results indicate that morphology changes significantly and functional characteristics deteriorate with age. These parameters are potential indicators for evaluating age-related renal morphology and function changes. Combined analyses of these parameters could provide a quantitative, novel method for monitoring kidney diseases and/or therapeutic effects of kidney drugs.

Assessing metastatic potential of breast cancer cells based on EGFR dynamics.

Derailed transmembrane receptor trafficking could be a hallmark of tumorigenesis and increased tumor invasiveness, but receptor dynamics have not been used to differentiate metastatic cancer cells from less invasive ones. Using single-particle tracking techniques, we  developed a phenotyping asssay named Transmembrane Receptor Dynamics (TReD), studied the dynamics of epidermal growth factor receptor (EGFR) in seven breast epithelial cell lines and developed a phenotyping assay named Transmembrane Receptor Dynamics (TReD). Here we show a clear evidence that increased EGFR diffusivity and enlarged EGFR confinement size in the plasma membrane (PM) are correlated with the enhanced metastatic potential in these cell lines. By comparing the TReD results with the gene expression profiles, we found a clear negative correlation between the EGFR diffusivities and the breast cancer luminal differentiation scores (r = -0.75). Upon the induction of epithelial-mesenchymal transition (EMT), EGFR diffusivity significantly increased for the non-tumorigenic MCF10A (99%) and the non-invasive MCF7 (56%) cells, but not for the highly metastatic MDA-MB-231 cell. We believe that the reorganization of actin filaments during EMT modified the PM structures, causing the receptor dynamics to change. TReD can thus serve as a new biophysical marker to probe the metastatic potential of cancer cells and even to monitor the transition of metastasis.

Spatial EGFR Dynamics and Metastatic Phenotypes Modulated by Upregulated EphB2 and Src Pathways in Advanced Prostate Cancer.

Advanced prostate cancer is a very heterogeneous disease reflecting in diverse regulations of oncogenic signaling pathways. Aberrant spatial dynamics of epidermal growth factor receptor (EGFR) promote their dimerization and clustering, leading to constitutive activation in oncogenesis. The EphB2 and Src signaling pathways are associated with the reorganization of the cytoskeleton leading to malignancy, but their roles in regulating EGFR dynamics and activation are scarcely reported. Using single-particle tracking techniques, we found that highly phosphorylated EGFR in the advanced prostate cancer cell line, PC3, was associated with higher EGFR diffusivity, as compared with LNCaP and less aggressive DU145. The increased EGFR activation and biophysical dynamics were consistent with high proliferation, migration, and invasion. After performing single-cell RNA-seq on prostate cancer cell lines and circulating tumor cells from patients, we identified that upregulated gene expression in the EphB2 and Src pathways are associated with advanced malignancy. After dasatinib treatment or siRNA knockdowns of EphB2 or Src, the PC3 cells exhibited significantly lower EGFR dynamics, cell motility, and invasion. Partial inhibitory effects were also found in DU145 cells. The upregulation of parts of the EphB2 and Src pathways also predicts poor prognosis in the prostate cancer patient cohort of The Cancer Genome Atlas. Our results provide evidence that overexpression of the EphB2 and Src signaling pathways regulate EGFR dynamics and cellular aggressiveness in some advanced prostate cancer cells.

Quantitative evaluation of redox ratio and collagen characteristics during breast cancer chemotherapy using two-photon intrinsic imaging.

Preoperative neoadjuvant treatment in locally advanced breast cancer is recognized as an effective adjuvant therapy, as it improves treatment outcomes. However, the potential complications remain a threat, so there is an urgent clinical need to assess both the tumor response and changes in its microenvironment using non-invasive and precise identification techniques. Here, two-photon microscopy was employed to detect morphological alterations in breast cancer progression and recession throughout chemotherapy. The changes in structure were analyzed based on the autofluorescence and collagen of differing statuses. Parameters, including optical redox ratio, the ratio of second harmonic generation and auto-fluorescence signal, collagen density, and collagen shape orientation, were studied. Results indicate that these parameters are potential indicators for evaluating breast tumors and their microenvironment changes during progression and chemotherapy. Combined analyses of these parameters could provide a quantitative, novel method for monitoring tumor therapy.

3D mesoscopic fluorescence tomography for imaging micro-distribution of antibody-photon absorber conjugates during near infrared photoimmunotherapy in vivo.

As a novel low-side-effect cancer therapy, photo-immunotherapy (PIT) is based on conjugating monoclonal antibody (mAb) with a near-infrared (NIR) phthalocyanine dye IRDye700DX (IR 700). IR700 is not only fluorescent to be used as an imaging agent, but also phototoxic. When illuminating with NIR light, PIT can induce highly-selective cancer cell death while leaving most of tumor blood vessels unharmed, leading to an effect termed super-enhanced permeability and retention (SUPR), which can significantly improve the effectiveness of anti-cancer drug. Currently, the therapeutic effects of PIT are monitored using 2D macroscopic fluorescence reflectance imager, which lacks the resolution and depth information to reveal the 3D distribution of mAb-IR700. In the study, we applied a multi-modal optical imaging approach including high-resolution optical coherence tomography (OCT) and high-sensitivity fluorescence laminar optical tomography (FLOT), to provide 3D tumor micro-structure and micro-distribution of mAb-IR700 in the tumor simultaneously during PIT in situ and in vivo. The multi-wavelength FLOT can also provide the blood vessels morphology of the tumor. Thus, the 3D FLOT reconstructed images allow us to evaluate the IR700 fluorescence distribution change with respect to the blood vessels and at different tumor locations/depths non-invasively, thereby enabling evaluation of the therapeutic effects in vivo and optimization of treatment regimens accordingly. The mAb-IR700 can access more tumor areas after PIT treatment, which can be explained by increased vascular permeability immediately after NIR-PIT. Two-photon microscopy was also used to record the mAb-IR700 on the tumor surface near the blood vessels to verify the results.