Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

The Cell Tracking Challenge: 10 years of objective benchmarking.

The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

Imbalanced classification for protein subcellular localization with multilabel oversampling.

MOTIVATION: Subcellular localization of human proteins is essential to comprehend their functions and roles in physiological processes, which in turn helps in diagnostic and prognostic studies of pathological conditions and impacts clinical decision-making. Since proteins reside at multiple locations at the same time and few subcellular locations host far more proteins than other locations, the computational task for their subcellular localization is to train a multilabel classifier while handling data imbalance. In imbalanced data, minority classes are underrepresented, thus leading to a heavy bias towards the majority classes and the degradation of predictive capability for the minority classes. Furthermore, data imbalance in multilabel settings is an even more complex problem due to the coexistence of majority and minority classes. RESULTS: Our studies reveal that based on the extent of concurrence of majority and minority classes, oversampling of minority samples through appropriate data augmentation techniques holds promising scope for boosting the classification performance for the minority classes. We measured the magnitude of data imbalance per class and the concurrence of majority and minority classes in the dataset. Based on the obtained values, we identified minority and medium classes, and a new oversampling method is proposed that includes non-linear mixup, geometric and colour transformations for data augmentation and a sampling approach to prepare minibatches. Performance evaluation on the Human Protein Atlas Kaggle challenge dataset shows that the proposed method is capable of achieving better predictions for minority classes than existing methods. AVAILABILITY AND IMPLEMENTATION: Data used in this study are available at https://www.kaggle.com/competitions/human-protein-atlas-image-classification/data. Source code is available at https://github.com/priyarana/Protein-subcellular-localisation-method. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Protein Phosphatase 2B Dual Function Facilitates Synaptic Integrity and Motor Learning.

Protein phosphatase 2B (PP2B) is critical for synaptic plasticity and learning, but the molecular mechanisms involved remain unclear. Here we identified different types of proteins that interact with PP2B, including various structural proteins of the postsynaptic densities (PSDs) of Purkinje cells (PCs) in mice. Deleting PP2B reduced expression of PSD proteins and the relative thickness of PSD at the parallel fiber to PC synapses, whereas reexpression of inactive PP2B partly restored the impaired distribution of nanoclusters of PSD proteins, together indicating a structural role of PP2B. In contrast, lateral mobility of surface glutamate receptors solely depended on PP2B phosphatase activity. Finally, the level of motor learning covaried with both the enzymatic and nonenzymatic functions of PP2B. Thus, PP2B controls synaptic function and learning both through its action as a phosphatase and as a structural protein that facilitates synapse integrity.SIGNIFICANCE STATEMENT Phosphatases are generally considered to serve their critical role in learning and memory through their enzymatic operations. Here, we show that protein phosphatase 2B (PP2B) interacts with structural proteins at the synapses of cerebellar Purkinje cells. Differentially manipulating the enzymatic and structural domains of PP2B leads to different phenotypes in cerebellar learning. We propose that PP2B is crucial for cerebellar learning via two complementary actions, an enzymatic and a structural operation.

Automatic improvement of deep learning-based cell segmentation in time-lapse microscopy by neural architecture search.

MOTIVATION: Live cell segmentation is a crucial step in biological image analysis and is also a challenging task because time-lapse microscopy cell sequences usually exhibit complex spatial structures and complicated temporal behaviors. In recent years, numerous deep learning-based methods have been proposed to tackle this task and obtained promising results. However, designing a network with excellent performance requires professional knowledge and expertise and is very time-consuming and labor-intensive. Recently emerged neural architecture search (NAS) methods hold great promise in eliminating these disadvantages, because they can automatically search an optimal network for the task. RESULTS: We propose a novel NAS-based solution for deep learning-based cell segmentation in time-lapse microscopy images. Different from current NAS methods, we propose (i) jointly searching non-repeatable micro architectures to construct the macro network for exploring greater NAS potential and better performance and (ii) defining a specific search space suitable for the live cell segmentation task, including the incorporation of a convolutional long short-term memory network for exploring the temporal information in time-lapse sequences. Comprehensive evaluations on the 2D datasets from the cell tracking challenge demonstrate the competitiveness of the proposed method compared to the state of the art. The experimental results show that the method is capable of achieving more consistent top performance across all ten datasets than the other challenge methods. AVAILABILITYAND IMPLEMENTATION: The executable files of the proposed method as well as configurations for each dataset used in the presented experiments will be available for non-commercial purposes from https://github.com/291498346/nas_cellseg. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Deep Learning in Diverse Intelligent Sensor Based Systems.

Deep learning has become a predominant method for solving data analysis problems in virtually all fields of science and engineering. The increasing complexity and the large volume of data collected by diverse sensor systems have spurred the development of deep learning methods and have fundamentally transformed the way the data are acquired, processed, analyzed, and interpreted. With the rapid development of deep learning technology and its ever-increasing range of successful applications across diverse sensor systems, there is an urgent need to provide a comprehensive investigation of deep learning in this domain from a holistic view. This survey paper aims to contribute to this by systematically investigating deep learning models/methods and their applications across diverse sensor systems. It also provides a comprehensive summary of deep learning implementation tips and links to tutorials, open-source codes, and pretrained models, which can serve as an excellent self-contained reference for deep learning practitioners and those seeking to innovate deep learning in this space. In addition, this paper provides insights into research topics in diverse sensor systems where deep learning has not yet been well-developed, and highlights challenges and future opportunities. This survey serves as a catalyst to accelerate the application and transformation of deep learning in diverse sensor systems.

Deep-learning method for data association in particle tracking.

MOTIVATION: Biological studies of dynamic processes in living cells often require accurate particle tracking as a first step toward quantitative analysis. Although many particle tracking methods have been developed for this purpose, they are typically based on prior assumptions about the particle dynamics, and/or they involve careful tuning of various algorithm parameters by the user for each application. This may make existing methods difficult to apply by non-expert users and to a broader range of tracking problems. Recent advances in deep-learning techniques hold great promise in eliminating these disadvantages, as they can learn how to optimally track particles from example data. RESULTS: Here, we present a deep-learning-based method for the data association stage of particle tracking. The proposed method uses convolutional neural networks and long short-term memory networks to extract relevant dynamics features and predict the motion of a particle and the cost of linking detected particles from one time point to the next. Comprehensive evaluations on datasets from the particle tracking challenge demonstrate the competitiveness of the proposed deep-learning method compared to the state of the art. Additional tests on real-time-lapse fluorescence microscopy images of various types of intracellular particles show the method performs comparably with human experts. AVAILABILITY AND IMPLEMENTATION: The software code implementing the proposed method as well as a description of how to obtain the test data used in the presented experiments will be available for non-commercial purposes from https://github.com/yoyohoho0221/pt_linking. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Deep learning methods for automatic segmentation of lower leg muscles and bones from MRI scans of children with and without cerebral palsy.

Cerebral palsy is a neurological condition that is known to affect muscle growth. Detailed investigations of muscle growth require segmentation of muscles from MRI scans, which is typically done manually. In this study, we evaluated the performance of 2D, 3D, and hybrid deep learning models for automatic segmentation of 11 lower leg muscles and two bones from MRI scans of children with and without cerebral palsy. All six models were trained and evaluated on manually segmented T1 -weighted MRI scans of the lower legs of 20 children, six of whom had cerebral palsy. The segmentation results were assessed using the median Dice similarity coefficient (DSC), average symmetric surface distance (ASSD), and volume error (VError) of all 13 labels of every scan. The best performance was achieved by H-DenseUNet, a hybrid model (DSC 0.90, ASSD 0.5 mm, and VError 2.6 cm3 ). The performance was equivalent to the inter-rater performance of manual segmentation (DSC 0.89, ASSD 0.6 mm, and VError 3.3 cm3 ). Models trained with the Dice loss function outperformed models trained with the cross-entropy loss function. Near-optimal performance could be attained using only 11 scans for training. Segmentation performance was similar for scans of typically developing children (DSC 0.90, ASSD 0.5 mm, and VError 2.8 cm3 ) and children with cerebral palsy (DSC 0.85, ASSD 0.6 mm, and VError 2.4 cm3 ). These findings demonstrate the feasibility of fully automatic segmentation of individual muscles and bones from MRI scans of children with and without cerebral palsy.

An objective comparison of cell-tracking algorithms.

We present a combined report on the results of three editions of the Cell Tracking Challenge, an ongoing initiative aimed at promoting the development and objective evaluation of cell segmentation and tracking algorithms. With 21 participating algorithms and a data repository consisting of 13 data sets from various microscopy modalities, the challenge displays today's state-of-the-art methodology in the field. We analyzed the challenge results using performance measures for segmentation and tracking that rank all participating methods. We also analyzed the performance of all of the algorithms in terms of biological measures and practical usability. Although some methods scored high in all technical aspects, none obtained fully correct solutions. We found that methods that either take prior information into account using learning strategies or analyze cells in a global spatiotemporal video context performed better than other methods under the segmentation and tracking scenarios included in the challenge.

Automated neuron tracing using probability hypothesis density filtering.

Motivation: The functionality of neurons and their role in neuronal networks is tightly connected to the cell morphology. A fundamental problem in many neurobiological studies aiming to unravel this connection is the digital reconstruction of neuronal cell morphology from microscopic image data. Many methods have been developed for this, but they are far from perfect, and better methods are needed. Results: Here we present a new method for tracing neuron centerlines needed for full reconstruction. The method uses a fundamentally different approach than previous methods by considering neuron tracing as a Bayesian multi-object tracking problem. The problem is solved using probability hypothesis density filtering. Results of experiments on 2D and 3D fluorescence microscopy image datasets of real neurons indicate the proposed method performs comparably or even better than the state of the art. Availability and Implementation: Software implementing the proposed neuron tracing method was written in the Java programming language as a plugin for the ImageJ platform. Source code is freely available for non-commercial use at https://bitbucket.org/miroslavradojevic/phd . Contact: meijering@imagescience.org. Supplementary information: Supplementary data are available at Bioinformatics online.

Objective comparison of particle tracking methods.

Particle tracking is of key importance for quantitative analysis of intracellular dynamic processes from time-lapse microscopy image data. Because manually detecting and following large numbers of individual particles is not feasible, automated computational methods have been developed for these tasks by many groups. Aiming to perform an objective comparison of methods, we gathered the community and organized an open competition in which participating teams applied their own methods independently to a commonly defined data set including diverse scenarios. Performance was assessed using commonly defined measures. Although no single method performed best across all scenarios, the results revealed clear differences between the various approaches, leading to notable practical conclusions for users and developers.

Comprehensive single cell-resolution analysis of the role of chromatin regulators in early C. elegans embryogenesis.

Chromatin regulators are widely expressed proteins with diverse roles in gene expression, nuclear organization, cell cycle regulation, pluripotency, physiology and development, and are frequently mutated in human diseases such as cancer. Their inhibition often results in pleiotropic effects that are difficult to study using conventional approaches. We have developed a semi-automated nuclear tracking algorithm to quantify the divisions, movements and positions of all nuclei during the early development of Caenorhabditis elegans and have used it to systematically study the effects of inhibiting chromatin regulators. The resulting high dimensional datasets revealed that inhibition of multiple regulators, including F55A3.3 (encoding FACT subunit SUPT16H), lin-53 (RBBP4/7), rba-1 (RBBP4/7), set-16 (MLL2/3), hda-1 (HDAC1/2), swsn-7 (ARID2), and let-526 (ARID1A/1B) affected cell cycle progression and caused chromosome segregation defects. In contrast, inhibition of cir-1 (CIR1) accelerated cell division timing in specific cells of the AB lineage. The inhibition of RNA polymerase II also accelerated these division timings, suggesting that normal gene expression is required to delay cell cycle progression in multiple lineages in the early embryo. Quantitative analyses of the dataset suggested the existence of at least two functionally distinct SWI/SNF chromatin remodeling complex activities in the early embryo, and identified a redundant requirement for the egl-27 and lin-40 MTA orthologs in the development of endoderm and mesoderm lineages. Moreover, our dataset also revealed a characteristic rearrangement of chromatin to the nuclear periphery upon the inhibition of multiple general regulators of gene expression. Our systematic, comprehensive and quantitative datasets illustrate the power of single cell-resolution quantitative tracking and high dimensional phenotyping to investigate gene function. Furthermore, the results provide an overview of the functions of essential chromatin regulators during the early development of an animal.

Quantitative imaging of focal adhesion dynamics and their regulation by HGF and Rap1 signaling.

Cell migration is crucial in development, tissue repair and immunity and frequently aberrant in pathological processes including tumor metastasis. Focal adhesions (FAs) are integrin-based adhesion complexes that form the link between the cytoskeleton and the extracellular matrix and are thought to orchestrate cell migration. Understanding the regulation of FAs by (oncogenic) signaling pathways may identify strategies to target pathological cell migration. Here we describe the development of a robust FA tracker that enables the automatic, multi-parametric analysis of FA dynamics, morphology and composition from time-lapse image series generated by total internal reflection fluorescence (TIRF) microscopy. In control prostate carcinoma cells, this software recapitulates previous findings that relate morphological characteristics of FAs to their lifetime and their cellular location. We then investigated how FAs are altered when cell migration is induced by the metastasis-promoting hormone HGF and subsequently inhibited by activation of the small GTPase Rap1. We performed a detailed analysis of individual FA parameters, which identified FA size, sliding and intensity as primary targets of Rap1. HGF did not have strong effects on any of the FA parameters within the first hours of its addition. Subsequent Bayesian network inference (BNI), using all measured parameters as input, revealed little correlation between changes in cell migration and FA characteristics in this prostate carcinoma cell line. Instead BNI indicated a concerted coordination of cell size and FA parameters. Thus our results did not reveal a direct relation between the regulation of cell migration and the regulation of FA dynamics.

End-binding proteins sensitize microtubules to the action of microtubule-targeting agents.

Microtubule-targeting agents (MTAs) are widely used for treatment of cancer and other diseases, and a detailed understanding of the mechanism of their action is important for the development of improved microtubule-directed therapies. Although there is a large body of data on the interactions of different MTAs with purified tubulin and microtubules, much less is known about how the effects of MTAs are modulated by microtubule-associated proteins. Among the regulatory factors with a potential to have a strong impact on MTA activity are the microtubule plus end-tracking proteins, which control multiple aspects of microtubule dynamic instability. Here, we reconstituted microtubule dynamics in vitro to investigate the influence of end-binding proteins (EBs), the core components of the microtubule plus end-tracking protein machinery, on the effects that MTAs exert on microtubule plus-end growth. We found that EBs promote microtubule catastrophe induction in the presence of all MTAs tested. Analysis of microtubule growth times supported the view that catastrophes are microtubule age dependent. This analysis indicated that MTAs affect microtubule aging in multiple ways: destabilizing MTAs, such as colchicine and vinblastine, accelerate aging in an EB-dependent manner, whereas stabilizing MTAs, such as paclitaxel and peloruside A, induce not only catastrophes but also rescues and can reverse the aging process.

A benchmark for comparison of cell tracking algorithms.

MOTIVATION: Automatic tracking of cells in multidimensional time-lapse fluorescence microscopy is an important task in many biomedical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this article, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. RESULTS: The main contributions of the challenge include the creation of a comprehensive video dataset repository and the definition of objective measures for comparison and ranking of the algorithms. With this benchmark, six algorithms covering a variety of segmentation and tracking paradigms have been compared and ranked based on their performance on both synthetic and real datasets. Given the diversity of the datasets, we do not declare a single winner of the challenge. Instead, we present and discuss the results for each individual dataset separately. AVAILABILITY AND IMPLEMENTATION: The challenge Web site (http://www.codesolorzano.com/celltrackingchallenge) provides access to the training and competition datasets, along with the ground truth of the training videos. It also provides access to Windows and Linux executable files of the evaluation software and most of the algorithms that competed in the challenge.

Three-dimensional inversion recovery manganese-enhanced MRI of mouse brain using super-resolution reconstruction to visualize nuclei involved in higher brain function.

The visualization of activity in mouse brain using inversion recovery spin echo (IR-SE) manganese-enhanced MRI (MEMRI) provides unique contrast, but suffers from poor resolution in the slice-encoding direction. Super-resolution reconstruction (SRR) is a resolution-enhancing post-processing technique in which multiple low-resolution slice stacks are combined into a single volume of high isotropic resolution using computational methods. In this study, we investigated, first, whether SRR can improve the three-dimensional resolution of IR-SE MEMRI in the slice selection direction, whilst maintaining or improving the contrast-to-noise ratio of the two-dimensional slice stacks. Second, the contrast-to-noise ratio of SRR IR-SE MEMRI was compared with a conventional three-dimensional gradient echo (GE) acquisition. Quantitative experiments were performed on a phantom containing compartments of various manganese concentrations. The results showed that, with comparable scan times, the signal-to-noise ratio of three-dimensional GE acquisition is higher than that of SRR IR-SE MEMRI. However, the contrast-to-noise ratio between different compartments can be superior with SRR IR-SE MEMRI, depending on the chosen inversion time. In vivo experiments were performed in mice receiving manganese using an implanted osmotic pump. The results showed that SRR works well as a resolution-enhancing technique in IR-SE MEMRI experiments. In addition, the SRR image also shows a number of brain structures that are more clearly discernible from the surrounding tissues than in three-dimensional GE acquisition, including a number of nuclei with specific higher brain functions, such as memory, stress, anxiety and reward behavior.

Artificial intelligence in the diagnosis of glaucoma and neurodegenerative diseases.

Artificial Intelligence is a rapidly expanding field within computer science that encompasses the emulation of human intelligence by machines. Machine learning and deep learning - two primary data-driven pattern analysis approaches under the umbrella of artificial intelligence - has created considerable interest in the last few decades. The evolution of technology has resulted in a substantial amount of artificial intelligence research on ophthalmic and neurodegenerative disease diagnosis using retinal images. Various artificial intelligence-based techniques have been used for diagnostic purposes, including traditional machine learning, deep learning, and their combinations. Presented here is a review of the literature covering the last 10 years on this topic, discussing the use of artificial intelligence in analysing data from different modalities and their combinations for the diagnosis of glaucoma and neurodegenerative diseases. The performance of published artificial intelligence methods varies due to several factors, yet the results suggest that such methods can potentially facilitate clinical diagnosis. Generally, the accuracy of artificial intelligence-assisted diagnosis ranges from 67-98%, and the area under the sensitivity-specificity curve (AUC) ranges from 0.71-0.98, which outperforms typical human performance of 71.5% accuracy and 0.86 area under the curve. This indicates that artificial intelligence-based tools can provide clinicians with useful information that would assist in providing improved diagnosis. The review suggests that there is room for improvement of existing artificial intelligence-based models using retinal imaging modalities before they are incorporated into clinical practice.

LSKANet: Long Strip Kernel Attention Network for Robotic Surgical Scene Segmentation.

Surgical scene segmentation is a critical task in Robotic-assisted surgery. However, the complexity of the surgical scene, which mainly includes local feature similarity (e.g., between different anatomical tissues), intraoperative complex artifacts, and indistinguishable boundaries, poses significant challenges to accurate segmentation. To tackle these problems, we propose the Long Strip Kernel Attention network (LSKANet), including two well-designed modules named Dual-block Large Kernel Attention module (DLKA) and Multiscale Affinity Feature Fusion module (MAFF), which can implement precise segmentation of surgical images. Specifically, by introducing strip convolutions with different topologies (cascaded and parallel) in two blocks and a large kernel design, DLKA can make full use of region- and strip-like surgical features and extract both visual and structural information to reduce the false segmentation caused by local feature similarity. In MAFF, affinity matrices calculated from multiscale feature maps are applied as feature fusion weights, which helps to address the interference of artifacts by suppressing the activations of irrelevant regions. Besides, the hybrid loss with Boundary Guided Head (BGH) is proposed to help the network segment indistinguishable boundaries effectively. We evaluate the proposed LSKANet on three datasets with different surgical scenes. The experimental results show that our method achieves new state-of-the-art results on all three datasets with improvements of 2.6%, 1.4%, and 3.4% mIoU, respectively. Furthermore, our method is compatible with different backbones and can significantly increase their segmentation accuracy. Code is available at https://github.com/YubinHan73/LSKANet.

Brain Image Segmentation for Ultrascale Neuron Reconstruction via an Adaptive Dual-Task Learning Network.

Accurate morphological reconstruction of neurons in whole brain images is critical for brain science research. However, due to the wide range of whole brain imaging, uneven staining, and optical system fluctuations, there are significant differences in image properties between different regions of the ultrascale brain image, such as dramatically varying voxel intensities and inhomogeneous distribution of background noise, posing an enormous challenge to neuron reconstruction from whole brain images. In this paper, we propose an adaptive dual-task learning network (ADTL-Net) to quickly and accurately extract neuronal structures from ultrascale brain images. Specifically, this framework includes an External Features Classifier (EFC) and a Parameter Adaptive Segmentation Decoder (PASD), which share the same Multi-Scale Feature Encoder (MSFE). MSFE introduces an attention module named Channel Space Fusion Module (CSFM) to extract structure and intensity distribution features of neurons at different scales for addressing the problem of anisotropy in 3D space. Then, EFC is designed to classify these feature maps based on external features, such as foreground intensity distributions and image smoothness, and select specific PASD parameters to decode them of different classes to obtain accurate segmentation results. PASD contains multiple sets of parameters trained by different representative complex signal-to-noise distribution image blocks to handle various images more robustly. Experimental results prove that compared with other advanced segmentation methods for neuron reconstruction, the proposed method achieves state-of-the-art results in the task of neuron reconstruction from ultrascale brain images, with an improvement of about 49% in speed and 12% in F1 score.