PAIP 2020: Microsatellite instability prediction in colorectal cancer.

Microsatellite instability (MSI) refers to alterations in the length of simple repetitive genomic sequences. MSI status serves as a prognostic and predictive factor in colorectal cancer. The MSI-high status is a good prognostic factor in stage II/III cancer, and predicts a lack of benefit to adjuvant fluorouracil chemotherapy in stage II cancer but a good response to immunotherapy in stage IV cancer. Therefore, determining MSI status in patients with colorectal cancer is important for identifying the appropriate treatment protocol. In the Pathology Artificial Intelligence Platform (PAIP) 2020 challenge, artificial intelligence researchers were invited to predict MSI status based on colorectal cancer slide images. Participants were required to perform two tasks. The primary task was to classify a given slide image as belonging to either the MSI-high or the microsatellite-stable group. The second task was tumor area segmentation to avoid ties with the main task. A total of 210 of the 495 participants enrolled in the challenge downloaded the images, and 23 teams submitted their final results. Seven teams from the top 10 participants agreed to disclose their algorithms, most of which were convolutional neural network-based deep learning models, such as EfficientNet and UNet. The top-ranked system achieved the highest F1 score (0.9231). This paper summarizes the various methods used in the PAIP 2020 challenge. This paper supports the effectiveness of digital pathology for identifying the relationship between colorectal cancer and the MSI characteristics.

Biomolecular condensate assembly of nArgBP2 tunes its functionality to manifest the structural plasticity of dendritic spines.

nArgBP2, a candidate gene for intellectual disability, is a postsynaptic protein critical for dendritic spine development and morphogenesis, and its knockdown (KD) in developing neurons severely impairs spine-bearing excitatory synapse formation. Surprisingly, nArgBP2 KD in mature neurons did not cause morphological defects in the existing spines at rest, raising questions of how it functions in mature neurons. We found that unlike its inaction at rest, nArgBP2 KD completely inhibited the enlargement of dendritic spines during chemically induced long-term potentiation (cLTP) in mature neurons. We further found that nArgBP2 forms condensates in dendritic spines and that these condensates are dispersed by cLTP, which spatiotemporally coincides with spine head enlargement. Condensates with CaMKII phosphorylation-deficient mutant or CaMKII inhibition are neither dispersed nor accompanied by spine enlargement during cLTP. We found that nArgBP2 condensates in spines exhibited liquid-like properties, and in heterologous and in vitro expression systems, nArgBP2 undergoes liquid-liquid phase separation via multivalent intermolecular interactions between SH3 domains and proline-rich domains. It also forms coacervates with CaMKIIα, which is rapidly dissembled by calcium/CaMKIIα-dependent phosphorylation. We further showed that the interaction between nArgBP2 and WAVE1 competes with nArgBP2 phase separation and that blocking the nArgBP2-WAVE1 interaction prevents spine enlargement during cLTP. Together, our results suggest that nArgBP2 at rest is confined to the condensates but is released by CaMKIIα-mediated phosphorylation during synaptic plasticity, which regulates its timely interaction with WAVE1 to induce spine head enlargement in mature neurons.

RAMP: response-aware multi-task learning with contrastive regularization for cancer drug response prediction.

The accurate prediction of cancer drug sensitivity according to the multiomics profiles of individual patients is crucial for precision cancer medicine. However, the development of prediction models has been challenged by the complex crosstalk of input features and the resistance-dominant drug response information contained in public databases. In this study, we propose a novel multidrug response prediction framework, response-aware multitask prediction (RAMP), via a Bayesian neural network and restrict it by soft-supervised contrastive regularization. To utilize network embedding vectors as representation learning features for heterogeneous networks, we harness response-aware negative sampling, which applies cell line-drug response information to the training of network embeddings. RAMP overcomes the prediction accuracy limitation induced by the imbalance of trained response data based on the comprehensive selection and utilization of drug response features. When trained on the Genomics of Drug Sensitivity in Cancer dataset, RAMP achieved an area under the receiver operating characteristic curve > 89%, an area under the precision-recall curve > 59% and an $\textrm{F}_1$ score > 52% and outperformed previously developed methods on both balanced and imbalanced datasets. Furthermore, RAMP predicted many missing drug responses that were not included in the public databases. Our results showed that RAMP will be suitable for the high-throughput prediction of cancer drug sensitivity and will be useful for guiding cancer drug selection processes. The Python implementation for RAMP is available at https://github.com/hvcl/RAMP.

DXplorer: A Unified Visualization Framework for Interactive Dendritic Spine Analysis Using 3D Morphological Features.

Dendritic spines are dynamic, submicron-scale protrusions on neuronal dendrites that receive neuronal inputs. Morphological changes in the dendritic spine often reflect alterations in physiological conditions and are indicators of various neuropsychiatric conditions. However, owing to the highly dynamic and heterogeneous nature of spines, accurate measurement and objective analysis of spine morphology are major challenges in neuroscience research. Most conventional approaches for analyzing dendritic spines are based on two-dimensional (2D) images, which barely reflect the actual three-dimensional (3D) shapes. Although some recent studies have attempted to analyze spines with various 3D-based features, it is still difficult to objectively categorize and analyze spines based on 3D morphology. Here, we propose a unified visualization framework for an interactive 3D dendritic spine analysis system, DXplorer, that displays 3D rendering of spines and plots the high-dimensional features extracted from the 3D mesh of spines. With this system, users can perform the clustering of spines interactively and explore and analyze dendritic spines based on high-dimensional features. We propose a series of high-dimensional morphological features extracted from a 3D mesh of dendritic spines. In addition, an interactive machine learning classifier with visual exploration and user feedback using an interactive 3D mesh grid view ensures a more precise classification based on the spine phenotype. A user study and two case studies were conducted to quantitatively verify the performance and usability of the DXplorer. We demonstrate that the system performs the entire analytic process effectively and provides high-quality, accurate, and objective analysis.

Noise2Kernel: Adaptive Self-Supervised Blind Denoising Using a Dilated Convolutional Kernel Architecture.

With the advent of unsupervised learning, efficient training of a deep network for image denoising without pairs of noisy and clean images has become feasible. Most current unsupervised denoising methods are built on self-supervised loss with the assumption of zero-mean noise under the signal-independent condition, which causes brightness-shifting artifacts on unconventional noise statistics (i.e., different from commonly used noise models). Moreover, most blind denoising methods require a random masking scheme for training to ensure the invariance of the denoising process. In this study, we propose a dilated convolutional network that satisfies an invariant property, allowing efficient kernel-based training without random masking. We also propose an adaptive self-supervision loss to increase the tolerance for unconventional noise, which is specifically effective in removing salt-and-pepper or hybrid noise where prior knowledge of noise statistics is not readily available. We demonstrate the efficacy of the proposed method by comparing it with state-of-the-art denoising methods using various examples.

Quantitative three-dimensional image analysis of the superior canal after surgical plugging to treat superior semicircular canal dehiscence.

Surgical plugging to treat superior semicircular canal dehiscence (SCD) has been proven to impede the effect of the third mobile window, abating cochleovestibular symptoms. Knowledge of superior semicircular canal (SC)-plugging status has been proposed to serve as a guide for adjuvant treatment. Here, we investigated disturbances in the inner ear fluid space following SC plugging using a novel three-dimensional (3D) reconstruction-based method. This approach used a semi-automatic segmentation algorithm and a direct volume rendering method derived from conventional magnetic resonance images. The variable extents of filling defects at the sites of SC plugging and the positional relation of the defect to the ampulla and common crus were identified. The success group exhibited markedly reduced volumes following surgery, whereas the failure group displayed no changes in volume. These results indicate that the success or failure of SC plugging was related to 3D volume changes in the labyrinth fluid signal. Collectively, this study presents individualized SC-plugging statuses using a novel 3D reconstruction-based method and it facilitates future work regarding easy-to-measure 3D volume changes. This current technology also aids in the exploration of pathologic changes in various targets of interest.

ISCL: Interdependent Self-Cooperative Learning for Unpaired Image Denoising.

With the advent of advances in self-supervised learning, paired clean-noisy data are no longer required in deep learning-based image denoising. However, existing blind denoising methods still require the assumption with regard to noise characteristics, such as zero-mean noise distribution and pixel-wise noise-signal independence; this hinders wide adaptation of the method in the medical domain. On the other hand, unpaired learning can overcome limitations related to the assumption on noise characteristics, which makes it more feasible for collecting the training data in real-world scenarios. In this paper, we propose a novel image denoising scheme, Interdependent Self-Cooperative Learning (ISCL), that leverages unpaired learning by combining cyclic adversarial learning with self-supervised residual learning. Unlike the existing unpaired image denoising methods relying on matching data distributions in different domains, the two architectures in ISCL, designed for different tasks, complement each other and boost the learning process. To assess the performance of the proposed method, we conducted extensive experiments in various biomedical image degradation scenarios, such as noise caused by physical characteristics of electron microscopy (EM) devices (film and charging noise), and structural noise found in low-dose computer tomography (CT). We demonstrate that the image quality of our method is superior to conventional and current state-of-the-art deep learning-based unpaired image denoising methods.

Development and operation of a digital platform for sharing pathology image data.

BACKGROUND: Artificial intelligence (AI) research is highly dependent on the nature of the data available. With the steady increase of AI applications in the medical field, the demand for quality medical data is increasing significantly. We here describe the development of a platform for providing and sharing digital pathology data to AI researchers, and highlight challenges to overcome in operating a sustainable platform in conjunction with pathologists. METHODS: Over 3000 pathological slides from five organs (liver, colon, prostate, pancreas and biliary tract, and kidney) in histologically confirmed tumor cases by pathology departments at three hospitals were selected for the dataset. After digitalizing the slides, tumor areas were annotated and overlaid onto the images by pathologists as the ground truth for AI training. To reduce the pathologists' workload, AI-assisted annotation was established in collaboration with university AI teams. RESULTS: A web-based data sharing platform was developed to share massive pathological image data in 2019. This platform includes 3100 images, and 5 pre-processing algorithms for AI researchers to easily load images into their learning models. DISCUSSION: Due to different regulations among countries for privacy protection, when releasing internationally shared learning platforms, it is considered to be most prudent to obtain consent from patients during data acquisition. CONCLUSIONS: Despite limitations encountered during platform development and model training, the present medical image sharing platform can steadily fulfill the high demand of AI developers for quality data. This study is expected to help other researchers intending to generate similar platforms that are more effective and accessible in the future.

DeepHCS++: Bright-field to fluorescence microscopy image conversion using multi-task learning with adversarial losses for label-free high-content screening.

In this paper, we propose a novel microscopy image translation method for transforming a bright-field microscopy image into three different fluorescence images to observe the apoptosis, nuclei, and cytoplasm of cells, which visualize dead cells, nuclei of cells, and cytoplasm of cells, respectively. These biomarkers are commonly used in high-content drug screening to analyze drug response. The main contribution of the proposed work is the automatic generation of three fluorescence images from a conventional bright-field image; this can greatly reduce the time-consuming and laborious tissue preparation process and improve throughput of the screening process. Our proposed method uses only a single bright-field image and the corresponding fluorescence images as a set of image pairs for training an end-to-end deep convolutional neural network. By leveraging deep convolutional neural networks with a set of image pairs of bright-field and corresponding fluorescence images, our proposed method can produce synthetic fluorescence images comparable to real fluorescence microscopy images with high accuracy. Our proposed model uses multi-task learning with adversarial losses to generate more accurate and realistic microscopy images. We assess the efficacy of the proposed method using real bright-field and fluorescence microscopy image datasets from patient-driven samples of a glioblastoma, and validate the method's accuracy with various quality metrics including cell number correlation (CNC), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), cell viability correlation (CVC), error maps, and R2 correlation.

Distributed Interactive Visualization Using GPU-Optimized Spark.

With the advent of advances in imaging and computing technologies, large-scale data acquisition and processing have become commonplace in many science and engineering disciplines. Conventional workflows for large-scale data processing usually rely on in-house or commercial software that are designed for domain-specific computing tasks. Recent advances in MapReduce, which was originally developed for batch processing textual data via a simplified programming model of the map and reduce functions, have expanded its applications to more general tasks in big-data processing, such as scientific computing, and biomedical image processing. However, as shown in previous work, volume rendering and visualization using MapReduce is still considered challenging and impractical owing to the disk-based, batch-processing nature of its computing model. In this article, contrary to this common belief, we show that the MapReduce computing model can be effectively used for interactive visualization. Our proposed system is a novel extension of Spark, one of the most popular open-source MapReduce frameworks, which offers GPU-accelerated MapReduce computing. To minimize CPU-GPU communication and overcome slow, disk-based shuffle performance, the proposed system supports GPU in-memory caching and MPI-based direct communication between compute nodes. To allow for GPU-accelerated in-situ visualization using raster graphics in Spark, we leveraged the CUDA-OpenGL interoperability, resulting in faster processing speeds by several orders of magnitude compared to conventional MapReduce systems. We demonstrate the performance of our system via several volume processing and visualization tasks, such as direct volume rendering, iso-surface extraction, and numerical simulations with in-situ visualization.

PAIP 2019: Liver cancer segmentation challenge.

Pathology Artificial Intelligence Platform (PAIP) is a free research platform in support of pathological artificial intelligence (AI). The main goal of the platform is to construct a high-quality pathology learning data set that will allow greater accessibility. The PAIP Liver Cancer Segmentation Challenge, organized in conjunction with the Medical Image Computing and Computer Assisted Intervention Society (MICCAI 2019), is the first image analysis challenge to apply PAIP datasets. The goal of the challenge was to evaluate new and existing algorithms for automated detection of liver cancer in whole-slide images (WSIs). Additionally, the PAIP of this year attempted to address potential future problems of AI applicability in clinical settings. In the challenge, participants were asked to use analytical data and statistical metrics to evaluate the performance of automated algorithms in two different tasks. The participants were given the two different tasks: Task 1 involved investigating Liver Cancer Segmentation and Task 2 involved investigating Viable Tumor Burden Estimation. There was a strong correlation between high performance of teams on both tasks, in which teams that performed well on Task 1 also performed well on Task 2. After evaluation, we summarized the top 11 team's algorithms. We then gave pathological implications on the easily predicted images for cancer segmentation and the challenging images for viable tumor burden estimation. Out of the 231 participants of the PAIP challenge datasets, a total of 64 were submitted from 28 team participants. The submitted algorithms predicted the automatic segmentation on the liver cancer with WSIs to an accuracy of a score estimation of 0.78. The PAIP challenge was created in an effort to combat the lack of research that has been done to address Liver cancer using digital pathology. It remains unclear of how the applicability of AI algorithms created during the challenge can affect clinical diagnoses. However, the results of this dataset and evaluation metric provided has the potential to aid the development and benchmarking of cancer diagnosis and segmentation.

CPEM: Accurate cancer type classification based on somatic alterations using an ensemble of a random forest and a deep neural network.

With recent advances in DNA sequencing technologies, fast acquisition of large-scale genomic data has become commonplace. For cancer studies, in particular, there is an increasing need for the classification of cancer type based on somatic alterations detected from sequencing analyses. However, the ever-increasing size and complexity of the data make the classification task extremely challenging. In this study, we evaluate the contributions of various input features, such as mutation profiles, mutation rates, mutation spectra and signatures, and somatic copy number alterations that can be derived from genomic data, and further utilize them for accurate cancer type classification. We introduce a novel ensemble of machine learning classifiers, called CPEM (Cancer Predictor using an Ensemble Model), which is tested on 7,002 samples representing over 31 different cancer types collected from The Cancer Genome Atlas (TCGA) database. We first systematically examined the impact of the input features. Features known to be associated with specific cancers had relatively high importance in our initial prediction model. We further investigated various machine learning classifiers and feature selection methods to derive the ensemble-based cancer type prediction model achieving up to 84% classification accuracy in the nested 10-fold cross-validation. Finally, we narrowed down the target cancers to the six most common types and achieved up to 94% accuracy.

Frequency-splitting dynamic MRI reconstruction using multi-scale 3D convolutional sparse coding and automatic parameter selection.

In this paper, we propose a novel image reconstruction algorithm using multi-scale 3D convolutional sparse coding and a spectral decomposition technique for highly undersampled dynamic Magnetic Resonance Imaging (MRI) data. The proposed method recovers high-frequency information using a shared 3D convolution-based dictionary built progressively during the reconstruction process in an unsupervised manner, while low-frequency information is recovered using a total variation-based energy minimization method that leverages temporal coherence in dynamic MRI. Additionally, the proposed 3D dictionary is built across three different scales to more efficiently adapt to various feature sizes, and elastic net regularization is employed to promote a better approximation to the sparse input data. We also propose an automatic parameter selection technique based on a genetic algorithm to find optimal parameters for our numerical solver which is a variant of the alternating direction method of multipliers (ADMM). We demonstrate the performance of our method by comparing it with state-of-the-art methods on 15 single-coil cardiac, 7 single-coil DCE, and a multi-coil brain MRI datasets at different sampling rates (12.5%, 25% and 50%). The results show that our method significantly outperforms the other state-of-the-art methods in reconstruction quality with a comparable running time and is resilient to noise.

DXR: A Toolkit for Building Immersive Data Visualizations.

This paper presents DXR, a toolkit for building immersive data visualizations based on the Unity development platform. Over the past years, immersive data visualizations in augmented and virtual reality (AR, VR) have been emerging as a promising medium for data sense-making beyond the desktop. However, creating immersive visualizations remains challenging, and often require complex low-level programming and tedious manual encoding of data attributes to geometric and visual properties. These can hinder the iterative idea-to-prototype process, especially for developers without experience in 3D graphics, AR, and VR programming. With DXR, developers can efficiently specify visualization designs using a concise declarative visualization grammar inspired by Vega-Lite. DXR further provides a GUI for easy and quick edits and previews of visualization designs in-situ, i.e., while immersed in the virtual world. DXR also provides reusable templates and customizable graphical marks, enabling unique and engaging visualizations. We demonstrate the flexibility of DXR through several examples spanning a wide range of applications.

Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss.

Compressed sensing magnetic resonance imaging (CS-MRI) has provided theoretical foundations upon which the time-consuming MRI acquisition process can be accelerated. However, it primarily relies on iterative numerical solvers, which still hinders their adaptation in time-critical applications. In addition, recent advances in deep neural networks have shown their potential in computer vision and image processing, but their adaptation to MRI reconstruction is still in an early stage. In this paper, we propose a novel deep learning-based generative adversarial model, RefineGAN, for fast and accurate CS-MRI reconstruction. The proposed model is a variant of fully-residual convolutional autoencoder and generative adversarial networks (GANs), specifically designed for CS-MRI formulation; it employs deeper generator and discriminator networks with cyclic data consistency loss for faithful interpolation in the given under-sampled -space data. In addition, our solution leverages a chained network to further enhance the reconstruction quality. RefineGAN is fast and accurate-the reconstruction process is extremely rapid, as low as tens of milliseconds for reconstruction of a image, because it is one-way deployment on a feed-forward network, and the image quality is superior even for extremely low sampling rate (as low as 10%) due to the data-driven nature of the method. We demonstrate that RefineGAN outperforms the state-of-the-art CS-MRI methods by a large margin in terms of both running time and image quality via evaluation using several open-source MRI databases.

An Intelligent System Approach for Probabilistic Volume Rendering Using Hierarchical 3D Convolutional Sparse Coding.

In this paper, we propose a novel machine learning-based voxel classification method for highly-accurate volume rendering. Unlike conventional voxel classification methods that incorporate intensity-based features, the proposed method employs dictionary based features learned directly from the input data using hierarchical multi-scale 3D convolutional sparse coding, a novel extension of the state-of-the-art learning-based sparse feature representation method. The proposed approach automatically generates high-dimensional feature vectors in up to 75 dimensions, which are then fed into an intelligent system built on a random forest classifier for accurately classifying voxels from only a handful of selection scribbles made directly on the input data by the user. We apply the probabilistic transfer function to further customize and refine the rendered result. The proposed method is more intuitive to use and more robust to noise in comparison with conventional intensity-based classification methods. We evaluate the proposed method using several synthetic and real-world volume datasets, and demonstrate the methods usability through a user study.

Whole-brain serial-section electron microscopy in larval zebrafish.

High-resolution serial-section electron microscopy (ssEM) makes it possible to investigate the dense meshwork of axons, dendrites, and synapses that form neuronal circuits. However, the imaging scale required to comprehensively reconstruct these structures is more than ten orders of magnitude smaller than the spatial extents occupied by networks of interconnected neurons, some of which span nearly the entire brain. Difficulties in generating and handling data for large volumes at nanoscale resolution have thus restricted vertebrate studies to fragments of circuits. These efforts were recently transformed by advances in computing, sample handling, and imaging techniques, but high-resolution examination of entire brains remains a challenge. Here, we present ssEM data for the complete brain of a larval zebrafish (Danio rerio) at 5.5 days post-fertilization. Our approach utilizes multiple rounds of targeted imaging at different scales to reduce acquisition time and data management requirements. The resulting dataset can be analysed to reconstruct neuronal processes, permitting us to survey all myelinated axons (the projectome). These reconstructions enable precise investigations of neuronal morphology, which reveal remarkable bilateral symmetry in myelinated reticulospinal and lateral line afferent axons. We further set the stage for whole-brain structure-function comparisons by co-registering functional reference atlases and in vivo two-photon fluorescence microscopy data from the same specimen. All obtained images and reconstructions are provided as an open-access resource.

Vivaldi: A Domain-Specific Language for Volume Processing and Visualization on Distributed Heterogeneous Systems.

As the size of image data from microscopes and telescopes increases, the need for high-throughput processing and visualization of large volumetric data has become more pressing. At the same time, many-core processors and GPU accelerators are commonplace, making high-performance distributed heterogeneous computing systems affordable. However, effectively utilizing GPU clusters is difficult for novice programmers, and even experienced programmers often fail to fully leverage the computing power of new parallel architectures due to their steep learning curve and programming complexity. In this paper, we propose Vivaldi, a new domain-specific language for volume processing and visualization on distributed heterogeneous computing systems. Vivaldi's Python-like grammar and parallel processing abstractions provide flexible programming tools for non-experts to easily write high-performance parallel computing code. Vivaldi provides commonly used functions and numerical operators for customized visualization and high-throughput image processing applications. We demonstrate the performance and usability of Vivaldi on several examples ranging from volume rendering to image segmentation.

Exploring the connectome: petascale volume visualization of microscopy data streams.

Recent advances in high-resolution microscopy let neuroscientists acquire neural-tissue volume data of extremely large sizes. However, the tremendous resolution and the high complexity of neural structures present big challenges to storage, processing, and visualization at interactive rates. A proposed system provides interactive exploration of petascale (petavoxel) volumes resulting from high-throughput electron microscopy data streams. The system can concurrently handle multiple volumes and can support the simultaneous visualization of high-resolution voxel segmentation data. Its visualization-driven design restricts most computations to a small subset of the data. It employs a multiresolution virtual-memory architecture for better scalability than previous approaches and for handling incomplete data. Researchers have employed it for a 1-teravoxel mouse cortex volume, of which several hundred axons and dendrites as well as synapses have been segmented and labeled.

Interactive Volume Exploration of Petascale Microscopy Data Streams Using a Visualization-Driven Virtual Memory Approach.

This paper presents the first volume visualization system that scales to petascale volumes imaged as a continuous stream of high-resolution electron microscopy images. Our architecture scales to dense, anisotropic petascale volumes because it: (1) decouples construction of the 3D multi-resolution representation required for visualization from data acquisition, and (2) decouples sample access time during ray-casting from the size of the multi-resolution hierarchy. Our system is designed around a scalable multi-resolution virtual memory architecture that handles missing data naturally, does not pre-compute any 3D multi-resolution representation such as an octree, and can accept a constant stream of 2D image tiles from the microscopes. A novelty of our system design is that it is visualization-driven: we restrict most computations to the visible volume data. Leveraging the virtual memory architecture, missing data are detected during volume ray-casting as cache misses, which are propagated backwards for on-demand out-of-core processing. 3D blocks of volume data are only constructed from 2D microscope image tiles when they have actually been accessed during ray-casting. We extensively evaluate our system design choices with respect to scalability and performance, compare to previous best-of-breed systems, and illustrate the effectiveness of our system for real microscopy data from neuroscience.