A self-supervised framework for cross-modal search in histopathology archives using scale harmonization.

The exponential growth of data across various medical domains has generated a substantial demand for techniques to analyze multimodal big data. This demand is particularly pronounced in fields such as computational pathology due to the diverse nature of the tissue. Cross-modal retrieval aims to identify a common latent space where different modalities, such as image-text pairs, exhibit close alignment. The primary challenge, however, often lies in the representation of tissue features. While language models can be trained relatively easily, visual models frequently struggle due to the scarcity of labeled data. To address this issue, the innovative concept of harmonization has been introduced, extending the learning scheme distillation without supervision, known as DINO. The harmonization of scale refines the DINO paradigm through a novel patching approach, overcoming the complexities posed by gigapixel whole slide images in digital pathology. Experiments conducted on diverse datasets have demonstrated that the proposed approach significantly enhances cross-modal retrieval in tissue imaging. Moreover, it exhibits vast potential for other fields that rely on gigapixel imaging.

Biased data, biased AI: deep networks predict the acquisition site of TCGA images.

BACKGROUND: Deep learning models applied to healthcare applications including digital pathology have been increasing their scope and importance in recent years. Many of these models have been trained on The Cancer Genome Atlas (TCGA) atlas of digital images, or use it as a validation source. One crucial factor that seems to have been widely ignored is the internal bias that originates from the institutions that contributed WSIs to the TCGA dataset, and its effects on models trained on this dataset. METHODS: 8,579 paraffin-embedded, hematoxylin and eosin stained, digital slides were selected from the TCGA dataset. More than 140 medical institutions (acquisition sites) contributed to this dataset. Two deep neural networks (DenseNet121 and KimiaNet were used to extract deep features at 20× magnification. DenseNet was pre-trained on non-medical objects. KimiaNet has the same structure but trained for cancer type classification on TCGA images. The extracted deep features were later used to detect each slide's acquisition site, and also for slide representation in image search. RESULTS: DenseNet's deep features could distinguish acquisition sites with 70% accuracy whereas KimiaNet's deep features could reveal acquisition sites with more than 86% accuracy. These findings suggest that there are acquisition site specific patterns that could be picked up by deep neural networks. It has also been shown that these medically irrelevant patterns can interfere with other applications of deep learning in digital pathology, namely image search. This study shows that there are acquisition site specific patterns that can be used to identify tissue acquisition sites without any explicit training. Furthermore, it was observed that a model trained for cancer subtype classification has exploited such medically irrelevant patterns to classify cancer types. Digital scanner configuration and noise, tissue stain variation and artifacts, and source site patient demographics are among factors that likely account for the observed bias. Therefore, researchers should be cautious of such bias when using histopathology datasets for developing and training deep networks.

Generalization of vision pre-trained models for histopathology.

Out-of-distribution (OOD) generalization, especially for medical setups, is a key challenge in modern machine learning which has only recently received much attention. We investigate how different convolutional pre-trained models perform on OOD test data-that is data from domains that have not been seen during training-on histopathology repositories attributed to different trial sites. Different trial site repositories, pre-trained models, and image transformations are examined as specific aspects of pre-trained models. A comparison is also performed among models trained entirely from scratch (i.e., without pre-training) and models already pre-trained. The OOD performance of pre-trained models on natural images, i.e., (1) vanilla pre-trained ImageNet, (2) semi-supervised learning (SSL), and (3) semi-weakly-supervised learning (SWSL) models pre-trained on IG-1B-Targeted are examined in this study. In addition, the performance of a histopathology model (i.e., KimiaNet) trained on the most comprehensive histopathology dataset, i.e., TCGA, has also been studied. Although the performance of SSL and SWSL pre-trained models are conducive to better OOD performance in comparison to the vanilla ImageNet pre-trained model, the histopathology pre-trained model is still the best in overall. In terms of top-1 accuracy, we demonstrate that diversifying the images in the training using reasonable image transformations is effective to avoid learning shortcuts when the distribution shift is significant. In addition, XAI techniques-which aim to achieve high-quality human-understandable explanations of AI decisions-are leveraged for further investigations.

Bias reduction in representation of histopathology images using deep feature selection.

Appearing traces of bias in deep networks is a serious reliability issue which can play a significant role in ethics and generalization related concerns. Recent studies report that the deep features extracted from the histopathology images of The Cancer Genome Atlas (TCGA), the largest publicly available archive, are surprisingly able to accurately classify the whole slide images (WSIs) based on their acquisition site while these features are extracted to primarily discriminate cancer types. This is clear evidence that the utilized Deep Neural Networks (DNNs) unexpectedly detect the specific patterns of the source site, i.e, the hospital of origin, rather than histomorphologic patterns, a biased behavior resulting in degraded trust and generalization. This observation motivated us to propose a method to alleviate the destructive impact of hospital bias through a novel feature selection process. To this effect, we have proposed an evolutionary strategy to select a small set of optimal features to not only accurately represent the histological patterns of tissue samples but also to eliminate the features contributing to internal bias toward the institution. The defined objective function for an optimal subset selection of features is to minimize the accuracy of the model to classify the source institutions which is basically defined as a bias indicator. By the conducted experiments, the selected features extracted by the state-of-the-art network trained on TCGA images (i.e., the KimiaNet), considerably decreased the institutional bias, while improving the quality of features to discriminate the cancer types. In addition, the selected features could significantly improve the results of external validation compared to the entire set of features which has been negatively affected by bias. The proposed scheme is a model-independent approach which can be employed when it is possible to define a bias indicator as a participating objective in a feature selection process; even with unknown bias sources.

Evolutionary deep feature selection for compact representation of gigapixel images in digital pathology.

Despite the recent progress in Deep Neural Networks (DNNs) to characterize histopathology images, compactly representing a gigapixel whole-slide image (WSI) via salient features to enable computational pathology is still an urgent need and a significant challenge. In this paper, we propose a novel WSI characterization approach to represent, search and classify biopsy specimens using a compact feature vector (CFV) extracted from a multitude of deep feature vectors. Since the non-optimal design and training of deep networks may result in many irrelevant and redundant features and also cause computational bottlenecks, we proposed a low-cost stochastic method to optimize the output of pre-trained deep networks using evolutionary algorithms to generate a very small set of features to accurately represent each tissue/biopsy. The performance of the proposed method has been assessed using WSIs from the publicly available TCGA image data. In addition to acquiring a very compact representation (i.e., 11,000 times smaller than the initial set of features), the optimized features achieved 93% classification accuracy resulting in 11% improvement compared to the published benchmarks. The experimental results reveal that the proposed method can reliably select salient features of the biopsy sample. Furthermore, the proposed approach holds the potential to immensely facilitate the adoption of digital pathology by enabling a new generation of WSI representation for efficient storage and more user-friendly visualization.

Hospital-Agnostic Image Representation Learning in Digital Pathology.

Whole Slide Images (WSIs) in digital pathology are used to diagnose cancer subtypes. The difference in procedures to acquire WSIs at various trial sites gives rise to variability in the histopathology images, thus making consistent diagnosis challenging. These differences may stem from variability in image acquisition through multi-vendor scanners, variable acquisition parameters, and differences in staining procedure; as well, patient demographics may bias the glass slide batches before image acquisition. These variabilities are assumed to cause a domain shift in the images of different hospitals. It is crucial to overcome this domain shift because an ideal machine-learning model must be able to work on the diverse sources of images, independent of the acquisition center. A domain generalization technique is leveraged in this study to improve the generalization capability of a Deep Neural Network (DNN), to an unseen histopathology image set (i.e., from an unseen hospital/trial site) in the presence of domain shift. According to experimental results, the conventional supervisedlearning regime generalizes poorly to data collected from different hospitals. However, the proposed hospital-agnostic learning can improve the generalization considering the lowdimensional latent space representation visualization, and classification accuracy results.

Selection, Visualization, and Interpretation of Deep Features in Lung Adenocarcinoma and Squamous Cell Carcinoma.

Although deep learning networks applied to digital images have shown impressive results for many pathology-related tasks, their black-box approach and limitation in terms of interpretability are significant obstacles for their widespread clinical utility. This study investigates the visualization of deep features (DFs) to characterize two lung cancer subtypes, adenocarcinoma and squamous cell carcinoma. It demonstrates that a subset of DFs, called prominent DFs, can accurately distinguish these two cancer subtypes. Visualization of such individual DFs allows for a better understanding of histopathologic patterns at both the whole-slide and patch levels, and discrimination of these cancer types. These DFs were visualized at the whole slide image level through DF-specific heatmaps and at tissue patch level through the generation of activation maps. In addition, these prominent DFs can distinguish carcinomas of organs other than the lung. This framework may serve as a platform for evaluating the interpretability of any deep network for diagnostic decision making.

Classifying Pneumonia among Chest X-Rays Using Transfer Learning.

Chest radiography has become the modality of choice for diagnosing pneumonia. However, analyzing chest X-ray images may be tedious, time-consuming and requiring expert knowledge that might not be available in less-developed regions. therefore, computer-aided diagnosis systems are needed. Recently, many classification systems based on deep learning have been proposed. Despite their success, the high development cost for deep networks is still a hurdle for deployment. Deep transfer learning (or simply transfer learning) has the merit of reducing the development cost by borrowing architectures from trained models followed by slight fine-tuning of some layers. Nevertheless, whether deep transfer learning is effective over training from scratch in the medical setting remains a research question for many applications. In this work, we investigate the use of deep transfer learning to classify pneumonia among chest X-ray images. Experimental results demonstrated that, with slight fine-tuning, deep transfer learning brings performance advantage over training from scratch. Three models, ResNet-50, Inception V3 and DensetNet121, were trained separately through transfer learning and from scratch. The former can achieve a 4.1% to 52.5% larger area under the curve (AUC) than those obtained by the latter, suggesting the effectiveness of deep transfer learning for classifying pneumonia in chest X-ray images.

Evaluating the Predictability of Cancer Types from 536 Somatic Mutations: A New Dataset.

In this paper, we introduce a new dataset for cancer research containing somatic mutation states of 536 genes of the Cancer Gene Census (CGC). We used somatic mutation information from the Cancer Genome Atlas (TCGA) projects to create this dataset. As preliminary investigations, we employed machine learning techniques, including k-Nearest Neighbors, Decision Tree, Random Forest, and Artificial Neural Networks (ANNs) to evaluate the potential of these somatic mutations for classification of cancer types. We compared our models on accuracy, precision, recall, and F1-score. We observed that ANNs outperformed the other models with F1-score of 0.36 and overall classification accuracy of 40%, and precision ranging from 12% to 92% for different cancer types. The 40% accuracy is significantly higher than random guessing which would have resulted in 3% overall classification accuracy. Although the model has relatively low overall accuracy, it has an average classification specificity of 98%. The ANN achieved high precision scores (> 0.7) for 5 of the 33 cancer types. The introduced dataset can be used for research on TCGA data, such as survival analysis, histopathology image analysis and content-based image retrieval. The dataset is available online for download: https://kimialab.uwaterloo.ca/kimia/.

An evolutionary decomposition-based multi-objective feature selection for multi-label classification.

Data classification is a fundamental task in data mining. Within this field, the classification of multi-labeled data has been seriously considered in recent years. In such problems, each data entity can simultaneously belong to several categories. Multi-label classification is important because of many recent real-world applications in which each entity has more than one label. To improve the performance of multi-label classification, feature selection plays an important role. It involves identifying and removing irrelevant and redundant features that unnecessarily increase the dimensions of the search space for the classification problems. However, classification may fail with an extreme decrease in the number of relevant features. Thus, minimizing the number of features and maximizing the classification accuracy are two desirable but conflicting objectives in multi-label feature selection. In this article, we introduce a multi-objective optimization algorithm customized for selecting the features of multi-label data. The proposed algorithm is an enhanced variant of a decomposition-based multi-objective optimization approach, in which the multi-label feature selection problem is divided into single-objective subproblems that can be simultaneously solved using an evolutionary algorithm. This approach leads to accelerating the optimization process and finding more diverse feature subsets. The proposed method benefits from a local search operator to find better solutions for each subproblem. We also define a pool of genetic operators to generate new feature subsets based on old generation. To evaluate the performance of the proposed algorithm, we compare it with two other multi-objective feature selection approaches on eight real-world benchmark datasets that are commonly used for multi-label classification. The reported results of multi-objective method evaluation measures, such as hypervolume indicator and set coverage, illustrate an improvement in the results obtained by the proposed method. Moreover, the proposed method achieved better results in terms of classification accuracy with fewer features compared with state-of-the-art methods.

Sequential Registration-Based Segmentation of the Prostate Gland in MR Image Volumes.

Accurate and fast segmentation and volume estimation of the prostate gland in magnetic resonance (MR) images are necessary steps in the diagnosis, treatment, and monitoring of prostate cancer. This paper presents an algorithm for the prostate gland volume estimation based on the semi-automated segmentation of individual slices in T2-weighted MR image sequences. The proposed sequential registration-based segmentation (SRS) algorithm, which was inspired by the clinical workflow during medical image contouring, relies on inter-slice image registration and user interaction/correction to segment the prostate gland without the use of an anatomical atlas. It automatically generates contours for each slice using a registration algorithm, provided that the user edits and approves the marking in some previous slices. We conducted comprehensive experiments to measure the performance of the proposed algorithm using three registration methods (i.e., rigid, affine, and nonrigid). Five radiation oncologists participated in the study where they contoured the prostate MR (T2-weighted) images of 15 patients both manually and using the SRS algorithm. Compared to the manual segmentation, on average, the SRS algorithm reduced the contouring time by 62% (a speedup factor of 2.64×) while maintaining the segmentation accuracy at the same level as the intra-user agreement level (i.e., Dice similarity coefficient of 91 versus 90%). The proposed algorithm exploits the inter-slice similarity of volumetric MR image series to achieve highly accurate results while significantly reducing the contouring time.

Color separation in forensic image processing using interactive differential evolution.

Color separation is an image processing technique that has often been used in forensic applications to differentiate among variant colors and to remove unwanted image interference. This process can reveal important information such as covered text or fingerprints in forensic investigation procedures. However, several limitations prevent users from selecting the appropriate parameters pertaining to the desired and undesired colors. This study proposes the hybridization of an interactive differential evolution (IDE) and a color separation technique that no longer requires users to guess required control parameters. The IDE algorithm optimizes these parameters in an interactive manner by utilizing human visual judgment to uncover desired objects. A comprehensive experimental verification has been conducted on various sample test images, including heavily obscured texts, texts with subtle color variations, and fingerprint smudges. The advantage of IDE is apparent as it effectively optimizes the color separation parameters at a level indiscernible to the naked eyes.

Multi-resolution level sets with shape priors: a validation report for 2D segmentation of prostate gland in T2W MR images.

The level set approach to segmentation of medical images has received considerable attention in recent years. Evolving an initial contour to converge to anatomical boundaries of an organ or tumor is a very appealing method, especially when it is based on a well-defined mathematical foundation. However, one drawback of such evolving method is its high computation time. It is desirable to design and implement algorithms that are not only accurate and robust but also fast in execution. Bresson et al. have proposed a variational model using both boundary and region information as well as shape priors. The latter can be a significant factor in medical image analysis. In this work, we combine the variational model of level set with a multi-resolution approach to accelerate the processing. The question is whether a multi-resolution context can make the segmentation faster without affecting the accuracy. As well, we investigate the question whether a premature convergence, which happens in a much shorter time, would reduce accuracy. We examine multiple semiautomated configurations to segment the prostate gland in T2W MR images. Comprehensive experimentation is conducted using a data set of a 100 patients (1,235 images) to verify the effectiveness of the multi-resolution level set with shape priors. The results show that the convergence speed can be increased by a factor of ≈ 2.5 without affecting the segmentation accuracy. Furthermore, a premature convergence approach drastically increases the segmentation speed by a factor of ≈ 17.9.

Heterogeneous differential evolution for numerical optimization.

Differential evolution (DE) is a population-based stochastic search algorithm which has shown a good performance in solving many benchmarks and real-world optimization problems. Individuals in the standard DE, and most of its modifications, exhibit the same search characteristics because of the use of the same DE scheme. This paper proposes a simple and effective heterogeneous DE (HDE) to balance exploration and exploitation. In HDE, individuals are allowed to follow different search behaviors randomly selected from a DE scheme pool. Experiments are conducted on a comprehensive set of benchmark functions, including classical problems and shifted large-scale problems. The results show that heterogeneous DE achieves promising performance on a majority of the test problems.

Inter-slice bidirectional registration-based segmentation of the prostate gland in MR and CT image sequences.

PURPOSE: Accurate segmentation and volume estimation of the prostate gland in magnetic resonance (MR) and computed tomography (CT) images are necessary steps in diagnosis, treatment, and monitoring of prostate cancer. This paper presents an algorithm for the prostate gland volume estimation based on the semiautomated segmentation of individual slices in T2-weighted MR and CT image sequences. METHODS: The proposed Inter-Slice Bidirectional Registration-based Segmentation (iBRS) algorithm relies on interslice image registration of volume data to segment the prostate gland without the use of an anatomical atlas. It requires the user to mark only three slices in a given volume dataset, i.e., the first, middle, and last slices. Next, the proposed algorithm uses a registration algorithm to autosegment the remaining slices. We conducted comprehensive experiments to measure the performance of the proposed algorithm using three registration methods (i.e., rigid, affine, and nonrigid techniques). RESULTS: The results with the proposed technique were compared with manual marking using prostate MR and CT images from 117 patients. Manual marking was performed by an expert user for all 117 patients. The median accuracies for individual slices measured using the Dice similarity coefficient (DSC) were 92% and 91% for MR and CT images, respectively. The iBRS algorithm was also evaluated regarding user variability, which confirmed that the algorithm was robust to interuser variability when marking the prostate gland. CONCLUSIONS: The proposed algorithm exploits the interslice data redundancy of the images in a volume dataset of MR and CT images and eliminates the need for an atlas, minimizing the computational cost while producing highly accurate results which are robust to interuser variability.

Gaussian Bare-Bones Differential Evolution.

Differential evolution (DE) is a well-known algorithm for global optimization over continuous search spaces. However, choosing the optimal control parameters is a challenging task because they are problem oriented. In order to minimize the effects of the control parameters, a Gaussian bare-bones DE (GBDE) and its modified version (MGBDE) are proposed which are almost parameter free. To verify the performance of our approaches, 30 benchmark functions and two real-world problems are utilized. Conducted experiments indicate that the MGBDE performs significantly better than, or at least comparable to, several state-of-the-art DE variants and some existing bare-bones algorithms.