Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

The Cell Tracking Challenge: 10 years of objective benchmarking.

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

An objective comparison of cell-tracking algorithms.

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

CytoPacq: a web-interface for simulating multi-dimensional cell imaging.

MOTIVATION: Objective assessment of bioimage analysis methods is an essential step towards understanding their robustness and parameter sensitivity, calling for the availability of heterogeneous bioimage datasets accompanied by their reference annotations. Because manual annotations are known to be arduous, highly subjective and barely reproducible, numerous simulators have emerged over past decades, generating synthetic bioimage datasets complemented with inherent reference annotations. However, the installation and configuration of these tools generally constitutes a barrier to their widespread use. RESULTS: We present a modern, modular web-interface, CytoPacq, to facilitate the generation of synthetic benchmark datasets relevant for multi-dimensional cell imaging. CytoPacq poses a user-friendly graphical interface with contextual tooltips and currently allows a comfortable access to various cell simulation systems of fluorescence microscopy, which have already been recognized and used by the scientific community, in a straightforward and self-contained form. AVAILABILITY AND IMPLEMENTATION: CytoPacq is a publicly available online service running at https://cbia.fi.muni.cz/simulator. More information about it as well as examples of generated bioimage datasets are available directly through the web-interface. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Drug Penetration Analysis in 3D Cell Cultures Using Fiducial-Based Semiautomatic Coregistration of MALDI MSI and Immunofluorescence Images.

In this paper, we present an easy-to-follow procedure for the analysis of tissue sections from 3D cell cultures (spheroids) by matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI) and laser scanning confocal microscopy (LSCM). MALDI MSI was chosen to detect the distribution of the drug of interest, while fluorescence immunohistochemistry (IHC) followed by LSCM was used to localize the cells featuring specific markers of viability, proliferation, apoptosis and metastasis. The overlay of the mass spectrometry (MS) and IHC spheroid images, typically without any morphological features, required fiducial-based coregistration. The MALDI MSI protocol was optimized in terms of fiducial composition and antigen epitope preservation to allow MALDI MSI to be performed and directly followed by IHC analysis on exactly the same spheroid section. Once MS and IHC images were coregistered, the quantification of the MS and IHC signals was performed by an algorithm evaluating signal intensities along equidistant layers from the spheroid boundary to its center. This accurate colocalization of MS and IHC signals showed limited penetration of the clinically tested drug perifosine into spheroids during a 24 h period, revealing the fraction of proliferating and promigratory/proinvasive cells present in the perifosine-free areas, decrease of their abundance in the perifosine-positive regions, and distinguishing between apoptosis resulting from hypoxia/nutrient deprivation and drug exposure.

Quantitative Assessment of Anti-Cancer Drug Efficacy From Coregistered Mass Spectrometry and Fluorescence Microscopy Images of Multicellular Tumor Spheroids.

Spheroids-three-dimensional aggregates of cells grown from a cancer cell line-represent a model of living tissue for chemotherapy investigation. Distribution of chemotherapeutics in spheroid sections was determined using the matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI MSI). Proliferating or apoptotic cells were immunohistochemically labeled and visualized by laser scanning confocal fluorescence microscopy (LSCM). Drug efficacy was evaluated by comparing coregistered MALDI MSI and LSCM data of drug-treated spheroids with LSCM only data of untreated control spheroids. We developed a fiducial-based workflow for coregistration of low-resolution MALDI MS with high-resolution LSCM images. To allow comparison of drug and cell distribution between the drug-treated and untreated spheroids of different shapes or diameters, we introduced a common diffusion-related coordinate, the distance from the spheroid boundary. In a procedure referred to as "peeling", we correlated average drug distribution at a certain distance with the average reduction in the affected cells between the untreated and the treated spheroids. This novel approach makes it possible to differentiate between peripheral cells that died due to therapy and the innermost cells which died naturally. Two novel algorithms-for MALDI MS image denoising and for weighting of MALDI MSI and LSCM data by the presence of cell nuclei-are also presented.

Image-based surrogate biomarkers for molecular subtypes of colorectal cancer.

MOTIVATION: Whole genome expression profiling of large cohorts of different types of cancer led to the identification of distinct molecular subcategories (subtypes) that may partially explain the observed inter-tumoral heterogeneity. This is also the case of colorectal cancer (CRC) where several such categorizations have been proposed. Despite recent developments, the problem of subtype definition and recognition remains open, one of the causes being the intrinsic heterogeneity of each tumor, which is difficult to estimate from gene expression profiles. However, one of the observations of these studies indicates that there may be links between the dominant tumor morphology characteristics and the molecular subtypes. Benefiting from a large collection of CRC samples, comprising both gene expression and histopathology images, we investigated the possibility of building image-based classifiers able to predict the molecular subtypes. We employed deep convolutional neural networks for extracting local descriptors which were then used for constructing a dictionary-based representation of each tumor sample. A set of support vector machine classifiers were trained to solve different binary decision problems, their combined outputs being used to predict one of the five molecular subtypes. RESULTS: A hierarchical decomposition of the multi-class problem was obtained with an overall accuracy of 0.84 (95%CI=0.79-0.88). The predictions from the image-based classifier showed significant prognostic value similar to their molecular counterparts. CONTACT: popovici@iba.muni.cz. AVAILABILITY AND IMPLEMENTATION: Source code used for the image analysis is freely available from https://github.com/higex/qpath . SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Virtual cell imaging: A review on simulation methods employed in image cytometry.

The simulations of cells and microscope images thereof have been used to facilitate the development, selection, and validation of image analysis algorithms employed in cytometry as well as for modeling and understanding cell structure and dynamics beyond what is visible in the eyepiece. The simulation approaches vary from simple parametric models of specific cell components-especially shapes of cells and cell nuclei-to learning-based synthesis and multi-stage simulation models for complex scenes that simultaneously visualize multiple object types and incorporate various properties of the imaged objects and laws of image formation. This review covers advances in artificial digital cell generation at scales ranging from particles up to tissue synthesis and microscope image simulation methods, provides examples of the use of simulated images for various purposes ranging from subcellular object detection to cell tracking, and discusses how such simulators have been validated. Finally, the future possibilities and limitations of simulation-based validation are considered. © 2016 International Society for Advancement of Cytometry.

Challenges and Benchmarks in Bioimage Analysis.

Similar to the medical imaging community, the bioimaging community has recently realized the need to benchmark various image analysis methods to compare their performance and assess their suitability for specific applications. Challenges sponsored by prestigious conferences have proven to be an effective means of encouraging benchmarking and new algorithm development for a particular type of image data. Bioimage analysis challenges have recently complemented medical image analysis challenges, especially in the case of the International Symposium on Biomedical Imaging (ISBI). This review summarizes recent progress in this respect and describes the general process of designing a bioimage analysis benchmark or challenge, including the proper selection of datasets and evaluation metrics. It also presents examples of specific target applications and biological research tasks that have benefited from these challenges with respect to the performance of automatic image analysis methods that are crucial for the given task. Finally, available benchmarks and challenges in terms of common features, possible classification and implications drawn from the results are analysed.

A benchmark for comparison of cell tracking algorithms.

MOTIVATION: Automatic tracking of cells in multidimensional time-lapse fluorescence microscopy is an important task in many biomedical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this article, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. RESULTS: The main contributions of the challenge include the creation of a comprehensive video dataset repository and the definition of objective measures for comparison and ranking of the algorithms. With this benchmark, six algorithms covering a variety of segmentation and tracking paradigms have been compared and ranked based on their performance on both synthetic and real datasets. Given the diversity of the datasets, we do not declare a single winner of the challenge. Instead, we present and discuss the results for each individual dataset separately. AVAILABILITY AND IMPLEMENTATION: The challenge Web site (http://www.codesolorzano.com/celltrackingchallenge) provides access to the training and competition datasets, along with the ground truth of the training videos. It also provides access to Windows and Linux executable files of the evaluation software and most of the algorithms that competed in the challenge.

Performance and sensitivity evaluation of 3D spot detection methods in confocal microscopy.

Reliable 3D detection of diffraction-limited spots in fluorescence microscopy images is an important task in subcellular observation. Generally, fluorescence microscopy images are heavily degraded by noise and non-specifically stained background, making reliable detection a challenging task. In this work, we have studied the performance and parameter sensitivity of eight recent methods for 3D spot detection. The study is based on both 3D synthetic image data and 3D real confocal microscopy images. The synthetic images were generated using a simulator modeling the complete imaging setup, including the optical path as well as the image acquisition process. We studied the detection performance and parameter sensitivity under different noise levels and under the influence of uneven background signal. To evaluate the parameter sensitivity, we propose a novel measure based on the gradient magnitude of the F1 score. We measured the success rate of the individual methods for different types of the image data and found that the type of image degradation is an important factor. Using the F1 score and the newly proposed sensitivity measure, we found that the parameter sensitivity is not necessarily proportional to the success rate of a method. This also provided an explanation why the best performing method for synthetic data was outperformed by other methods when applied to the real microscopy images. On the basis of the results obtained, we conclude with the recommendation of the HDome method for data with relatively low variations in quality, or the Sorokin method for image sets in which the quality varies more. We also provide alternative recommendations for high-quality images, and for situations in which detailed parameter tuning might be deemed expensive.

Possibilities of Using Multi-b-value Diffusion Magnetic Resonance Imaging for Classification of Brain Lesions.

In contrast to conventional diffusion magnetic resonance imaging (MRI), multi-b-value diffusion MRI methods are able to separate the signal from free water, pseudo-diffusion, and non-Gaussian components of water molecule diffusion. These approaches can then be utilised in so-called intravoxel incoherent motion imaging and diffusion kurtosis imaging. Various parameters provided by these methods can describe additional characteristics of the tissue microstructure and potentially help in the diagnosis and classification of various pathological processes. In this review, we present the basic principles and methods of analysing multi-b-value diffusion imaging data and specifically focus on the known possibilities for its use in the diagnosis of brain lesions. We also suggest possible directions for further research.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Endonuclease G interacts with histone H2B and DNA topoisomerase II alpha during apoptosis.

Apoptosis is a natural form of cell death involved in many physiological changes in the cell. Defects in the process of apoptosis can lead to serious diseases. During some apoptotic pathways, proteins apoptosis-inducing factor (AIF) and endonuclease G (EndoG) are released from the mitochondria and they translocate into the cell nuclei, where they probably participate in chromatin degradation together with other nuclear proteins. Exact mechanism of EndoG activity in cell nucleus is still unknown. Some interacting partners like flap endonuclease 1, DNase I, and exonuclease III were already suggested, but also other interacting partners were proposed. We conducted a living-cell confocal fluorescence microscopy followed by an image analysis of fluorescence resonance energy transfer to analyze the possibility of protein interactions of EndoG with histone H2B and human DNA topoisomerase II alpha (TOPO2a). Our results show that EndoG interacts with both these proteins during apoptotic cell death. Therefore, we can conclude that EndoG and TOPO2a may actively participate in apoptotic chromatin degradation. The possible existence of a degradation complex consisting of EndoG and TOPO2a and possibly other proteins like AIF and cyclophilin A have yet to be investigated.

Knockdown of apoptosis-inducing factor disrupts function of respiratory complex I.

Recent findings suggest that apoptotic protein apoptosis-inducing factor (AIF) may also play an important non-apoptotic function inside mitochondria. AIF was proposed to be an important component of respiratory chain complex I that is the major producer of superoxide radical. The possible role of AIF is still controversial. Superoxide production could be used as a valuable measure of complex I function, because the majority of superoxide is produced there. Therefore, we employed superoxide-specific mitochondrial fluorescence dye for detection of superoxide production. We studied an impact of AIF knockdown on function of mitochondrial complex I by analyzing superoxide production in selected cell lines. Our results show that tumoral telomerase-positive (TP) AIF knockdown cell lines display significant increase in superoxide production in comparison to control cells, while a non-tumoral cell line and tumoral telomerase-negative cell lines with alternative lengthening of telomeres (ALT) show a decrease in superoxide production. According to these results, we can conclude that AIF knockdown disrupts function of complex I and therefore increases the superoxide production in mitochondria. The distinct effect of AIF depletion in various cell lines could result from recently discovered activity of telomerase in mitochondria of TP cancer cells, but this hypothesis needs further investigation.

Proliferation and differentiation potential of CD133+ and CD34+ populations from the bone marrow and mobilized peripheral blood.

CD34 is the most frequently used marker for the selection of cells for bone marrow (BM) transplantation. The use of CD133 as an alternative marker is an open research topic. The goal of this study was to evaluate the proliferation and differentiation potential for hematopoiesis (short and long term) of CD133+ and CD34+ populations from bone marrow and mobilized peripheral blood. Eight cell populations were compared: CD34+ and CD133+ cells from both the BM (CML Ph-, CML Ph+, and healthy volunteers) and mobilized peripheral blood cells. Multicolor flow cytometry and cultivation experiments were used to measure expression and differentiation of the individual populations. It was observed that the CD133+ BM population showed higher cell expansion. Another finding is that during a 6-day cultivation with 5(6)-carboxyfluorescein diacetate N-succinimidyl ester (CFSE), more cells remained in division D0 (non-dividing cells). There was a higher percentage of CD38- cells observed on the CD133+ BM population. It was also observed that the studied populations contained very similar but not the same pools of progenitors: erythroid, lymphoid, and myeloid. This was confirmed by CFU-GM and CFU-E experiments. The VEGFR antigen was used to monitor subpopulations of endothelial sinusoidal progenitors. The CD133+ BM population contained significantly more VEGFR+ cells. Our findings suggest that the CD133+ population from the BM shows better proliferation activity and a higher distribution of primitive progenitors than any other studied population.

Lactic Acidosis Interferes With Toxicity of Perifosine to Colorectal Cancer Spheroids: Multimodal Imaging Analysis.

Colorectal cancer (CRC) is a disease with constantly increasing incidence and high mortality. The treatment efficacy could be curtailed by drug resistance resulting from poor drug penetration into tumor tissue and the tumor-specific microenvironment, such as hypoxia and acidosis. Furthermore, CRC tumors can be exposed to different pH depending on the position in the intestinal tract. CRC tumors often share upregulation of the Akt signaling pathway. In this study, we investigated the role of external pH in control of cytotoxicity of perifosine, the Akt signaling pathway inhibitor, to CRC cells using 2D and 3D tumor models. In 3D settings, we employed an innovative strategy for simultaneous detection of spatial drug distribution and biological markers of proliferation/apoptosis using a combination of mass spectrometry imaging and immunohistochemistry. In 3D conditions, low and heterogeneous penetration of perifosine into the inner parts of the spheroids was observed. The depth of penetration depended on the treatment duration but not on the external pH. However, pH alteration in the tumor microenvironment affected the distribution of proliferation- and apoptosis-specific markers in the perifosine-treated spheroid. Accurate co-registration of perifosine distribution and biological response in the same spheroid section revealed dynamic changes in apoptotic and proliferative markers occurring not only in the perifosine-exposed cells, but also in the perifosine-free regions. Cytotoxicity of perifosine to both 2D and 3D cultures decreased in an acidic environment below pH 6.7. External pH affects cytotoxicity of the other Akt inhibitor, MK-2206, in a similar way. Our innovative approach for accurate determination of drug efficiency in 3D tumor tissue revealed that cytotoxicity of Akt inhibitors to CRC cells is strongly dependent on pH of the tumor microenvironment. Therefore, the effect of pH should be considered during the design and pre-clinical/clinical testing of the Akt-targeted cancer therapy.

BIAS: Transparent reporting of biomedical image analysis challenges.

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.