Prediction of drug-induced hepatotoxicity based on histopathological whole slide images.

Liver is an important metabolic organ in human body and is sensitive to toxic chemicals or drugs. Adverse reactions caused by drug hepatotoxicity will damage the liver and hepatotoxicity is the leading cause of removal of approved drugs from the market. Therefore, it is of great significance to identify liver toxicity as early as possible in the drug development process. In this study, we developed a predictive model for drug hepatotoxicity based on histopathological whole slide images (WSI) which are the by-product of drug experiments and have received little attention. To better represent the WSIs, we constructed a graph representation for each WSI by dividing it into small patches, taking sampled patches as nodes and calculating the correlation coefficients between node features as the edges of the graph structure. Then a WSI-level graph convolutional network (GCN) was built to effectively extract the node information of the graph and predict the toxicity. In addition, we introduced a gated attention global context vector (gaGCV) to combine the global context to make node features to contain more comprehensive information. The results validated on rat liver in vivo data from the Open TG-GATES show that the use of WSI for the prediction of toxicity is feasible and effective.

Domain adaptation based self-correction model for COVID-19 infection segmentation in CT images.

The capability of generalization to unseen domains is crucial for deep learning models when considering real-world scenarios. However, current available medical image datasets, such as those for COVID-19 CT images, have large variations of infections and domain shift problems. To address this issue, we propose a prior knowledge driven domain adaptation and a dual-domain enhanced self-correction learning scheme. Based on the novel learning scheme, a domain adaptation based self-correction model (DASC-Net) is proposed for COVID-19 infection segmentation on CT images. DASC-Net consists of a novel attention and feature domain enhanced domain adaptation model (AFD-DA) to solve the domain shifts and a self-correction learning process to refine segmentation results. The innovations in AFD-DA include an image-level activation feature extractor with attention to lung abnormalities and a multi-level discrimination module for hierarchical feature domain alignment. The proposed self-correction learning process adaptively aggregates the learned model and corresponding pseudo labels for the propagation of aligned source and target domain information to alleviate the overfitting to noises caused by pseudo labels. Extensive experiments over three publicly available COVID-19 CT datasets demonstrate that DASC-Net consistently outperforms state-of-the-art segmentation, domain shift, and coronavirus infection segmentation methods. Ablation analysis further shows the effectiveness of the major components in our model. The DASC-Net enriches the theory of domain adaptation and self-correction learning in medical imaging and can be generalized to multi-site COVID-19 infection segmentation on CT images for clinical deployment.

Robust Multi-Frame Adaptive Optics Image Restoration Algorithm Using Maximum Likelihood Estimation with Poisson Statistics.

An adaptive optics (AO) system provides real-time compensation for atmospheric turbulence. However, an AO image is usually of poor contrast because of the nature of the imaging process, meaning that the image contains information coming from both out-of-focus and in-focus planes of the object, which also brings about a loss in quality. In this paper, we present a robust multi-frame adaptive optics image restoration algorithm via maximum likelihood estimation. Our proposed algorithm uses a maximum likelihood method with image regularization as the basic principle, and constructs the joint log likelihood function for multi-frame AO images based on a Poisson distribution model. To begin with, a frame selection method based on image variance is applied to the observed multi-frame AO images to select images with better quality to improve the convergence of a blind deconvolution algorithm. Then, by combining the imaging conditions and the AO system properties, a point spread function estimation model is built. Finally, we develop our iterative solutions for AO image restoration addressing the joint deconvolution issue. We conduct a number of experiments to evaluate the performances of our proposed algorithm. Experimental results show that our algorithm produces accurate AO image restoration results and outperforms the current state-of-the-art blind deconvolution methods.

A robust method for high-precision quantification of the complex three-dimensional vasculatures acquired by X-ray microtomography.

The quantification of micro-vasculatures is important for the analysis of angiogenesis on which the detection of tumor growth or hepatic fibrosis depends. Synchrotron-based X-ray computed micro-tomography (SR-µCT) allows rapid acquisition of micro-vasculature images at micrometer-scale spatial resolution. Through skeletonization, the statistical features of the micro-vasculature can be extracted from the skeleton of the micro-vasculatures. Thinning is a widely used algorithm to produce the vascular skeleton in medical research. Existing three-dimensional thinning methods normally emphasize the preservation of topological structure rather than geometrical features in generating the skeleton of a volumetric object. This results in three problems and limits the accuracy of the quantitative results related to the geometrical structure of the vasculature. The problems include the excessively shortened length of elongated objects, eliminated branches of blood vessel tree structure, and numerous noisy spurious branches. The inaccuracy of the skeleton directly introduces errors in the quantitative analysis, especially on the parameters concerning the vascular length and the counts of vessel segments and branching points. In this paper, a robust method using a consolidated end-point constraint for thinning, which generates geometry-preserving skeletons in addition to maintaining the topology of the vasculature, is presented. The improved skeleton can be used to produce more accurate quantitative results. Experimental results from high-resolution SR-µCT images show that the end-point constraint produced by the proposed method can significantly improve the accuracy of the skeleton obtained using the existing ITK three-dimensional thinning filter. The produced skeleton has laid the groundwork for accurate quantification of the angiogenesis. This is critical for the early detection of tumors and assessing anti-angiogenesis treatments.

Semi-automated infrared simulation on real urban scenes based on multi-view images.

The development of modern infrared applications require simulating thermal representations for targets of interest. However, generating geometric models for simulation has been a laborious, time-consuming work, which greatly limits the practical applications in real-world. In order to reduce the man-in-the-loop requirements, we devise a method that directly and semi-automatically simulates infrared signatures of real urban scenes. From raw meshes generated by multi-view stereo, we automatically produce a simplified watertight model through piecewise-planar 3D reconstruction. Model surface is subdivided into quality mesh elements to attach material attributes. For each element, heat balance equation is solved so as to render the whole scene by synthesizing the radiance distribution in infrared waveband. The credibility and effectiveness of our method are confirmed by comparing simulation results to the measured data in real-world. Our experiments on various types of buildings and large scale scene show that the proposed pipeline simulates meaningful infrared scenes while being robust and scalable.

Clustered nuclei splitting via curvature information and gray-scale distance transform.

Clusters or clumps of cells or nuclei are frequently observed in two dimensional images of thick tissue sections. Correct and accurate segmentation of overlapping cells and nuclei is important for many biological and biomedical applications. Many existing algorithms split clumps through the binarization of the input images; therefore, the intensity information of the original image is lost during this process. In this paper, we present a curvature information, gray scale distance transform, and shortest path splitting line-based algorithm which can make full use of the concavity and image intensity information to find out markers, each of which represents an individual object, and detect accurate splitting lines between objects using shortest path and junction adjustment. The proposed algorithm is tested on both synthetic and real nuclei images. Experiment results show that the performance of the proposed method is better than that of marker-controlled watershed method and ellipse fitting method.

Towards automated quantitative vasculature understanding via ultra high-resolution imagery.

This chapter presents an approach to processing ultra high-resolution, large-size biomedical imaging data for the purposes of detecting and quantifying vasculature and microvasculature . Capturing early signs of any changes in vasculature may have significant values for early-diagnosis and treatment assessment due to the well understood observation that vascular changes precede cancerous growth and metastasis metastasis . With the advent of key enabling technologies for extremely high-resolution imaging, such as synchrotron radiation synchrotron radiation based computed tomography (CT) computed tomography , the required levels of detail have become accessible. However, these technologies also present challenges in data analysis. This chapter aims to offer some insights as to how these changes might be best dealt with. We argue that the necessary steps in quantitative understanding of vasculatures include targeted data enhancement enhancement , information reduction aimed at characterizing the linear structure of vessels vessels , and quantitatively describing the vessel hierarchy. We present results on cerebral and liver vasculatures of a mouse captured at the Shanghai Synchrotron Radiation Facility (SSRF). These results were achieved with a processing pipeline comprising of our empirically selected component for each of the above steps. Towards the end, we discuss how alternative and additional components may be incorporated for improved speed and robustness.

Cloud based toolbox for image analysis, processing and reconstruction tasks.

This chapter describes a novel way of carrying out image analysis, reconstruction and processing tasks using cloud based service provided on the Australian National eResearch Collaboration Tools and Resources (NeCTAR) infrastructure. The toolbox allows users free access to a wide range of useful blocks of functionalities (imaging functions) that can be connected together in workflows allowing creation of even more complex algorithms that can be re-run on different data sets, shared with others or additionally adjusted. The functions given are in the area of cellular imaging, advanced X-ray image analysis, computed tomography and 3D medical imaging and visualisation. The service is currently available on the website www.cloudimaging.net.au .

Prediction of anti-cancer drug synergy based on cross-matching network and cancer molecular subtypes.

At present, anti-cancer drug synergy therapy is one of the most important methods to overcome drug resistance and reduce drug toxicity in cancer treatment. High-throughput screening through deep learning can effectively improve the efficiency of discovering synergistic drugs. Nowadays, most of the existing deep learning algorithms for anti-cancer drug synergy prediction use deep neural networks and can only implicitly perform feature interaction. This study proposes a deep learning algorithm, named MolCross, which combines implicit feature interaction with explicit features to improve the accuracy of prediction of the anti-cancer drug synergy score. MolCross uses a deep autoencoder to extract features from high-dimensional input, uses the drug-specific subnetworks and cross-network to perform implicit feature interaction and explicit feature interaction respectively, and finally uses a synergy prediction network to combine the two feature interaction methods to obtain the final prediction results. We adopted a five-fold cross validation and compared MolCross with other four anti-cancer drug synergy prediction models. The results show that MolCross has better prediction performance than other models. MolCross also has good performance in terms of cross-cell line and cross-tissue type. Existing studies have demonstrated that cancer molecular subtypes have different sensitivities to targeted therapy. In this study, the features of cancer molecular subtype were introduced in the model using an embedding layer in MolCross to explore the effect of cancer molecular subtype on anti-cancer drug synergy. We also found that the cancer molecular subtype is one of the main factors affecting the synergy between drugs.

Image Intensity Variation Information for Interest Point Detection.

Interest point detection methods are gaining more attention and are widely applied in computer vision tasks such as image retrieval and 3D reconstruction. However, there still exist two main problems to be solved: (1) from the perspective of mathematical representations, the differences among edges, corners, and blobs have not been convincingly explained and the relationships among the amplitude response, scale factor, and filtering orientation for interest points have not been thoroughly explained; (2) the existing design mechanism for interest point detection does not show how to accurately obtain intensity variation information on corners and blobs. In this paper, the first- and second-order Gaussian directional derivative representations of a step edge, four common genres of corners, an anisotropic-type blob, and an isotropic-type blob are analyzed and derived. Multiple interest point characteristics are discovered. The characteristics for interest points that we obtained help us describe the differences among edges, corners, and blobs, explain why the existing interest point detection methods with multiple scales cannot properly obtain interest points from images, and present novel corner and blob detection methods. Extensive experiments demonstrate the superiority of our proposed methods in terms of detection performance, robustness to affine transformations, noise, image matching, and 3D reconstruction.

Image Feature Information Extraction for Interest Point Detection: A Comprehensive Review.

Interest point detection is one of the most fundamental and critical problems in computer vision and image processing. In this paper, we carry out a comprehensive review on image feature information (IFI) extraction techniques for interest point detection. To systematically introduce how the existing interest point detection methods extract IFI from an input image, we propose a taxonomy of the IFI extraction techniques for interest point detection. According to this taxonomy, we discuss different types of IFI extraction techniques for interest point detection. Furthermore, we identify the main unresolved issues related to the existing IFI extraction techniques for interest point detection and any interest point detection methods that have not been discussed before. The existing popular datasets and evaluation standards are provided and the performances for fifteen state-of-the-art approaches are evaluated and discussed. Moreover, future research directions on IFI extraction techniques for interest point detection are elaborated.

Super-resolution phase retrieval network for single-pattern structured light 3D imaging.

Structured light 3D imaging is often used for obtaining accurate 3D information via phase retrieval. Single-pattern structured light 3D imaging is much faster than multi-pattern versions. Current phase retrieval methods for single-pattern structured light 3D imaging are however not accurate enough. Besides, the projector resolution in a structured light 3D imaging system is expensive to improve due to hardware costs. To address the issues of low accuracy and low resolution of single-pattern structured light 3D imaging, this work proposes a super-resolution phase retrieval network (SRPRNet). Specifically, a phase-shifting module is proposed to extract multi-scale features with different phase shifts, and a refinement and super-resolution module is proposed to obtain refined and super-resolution phase components. After phase demodulation and unwrapping, high-resolution absolute phase is obtained. A sine shifting loss and a cosine shifting loss are also introduced to form the regularization term of the loss function. As far as can be ascertained, the proposed SRPRNet is the first network for super-resolution phase retrieval by using a single pattern, and it can also be used for standard-resolution phase retrieval. Experimental results on three datasets show that SRPRNet achieves state-of-the-art performance on 1×, 2×, and 4× super-resolution phase retrieval tasks.

Image local structure information learning for fine-grained visual classification.

Learning discriminative visual patterns from image local salient regions is widely used for fine-grained visual classification (FGVC) tasks such as plant or animal species classification. A large number of complex networks have been designed for learning discriminative feature representations. In this paper, we propose a novel local structure information (LSI) learning method for FGVC. Firstly, we indicate that the existing FGVC methods have not properly considered how to extract LSI from an input image for FGVC. Then an LSI extraction technique is introduced which has the ability to properly depict the properties of different local structure features in images. Secondly, a novel LSI learning module is proposed to be added into a given backbone network for enhancing the ability of the network to find salient regions. Thirdly, extensive experiments show that our proposed method achieves better performance on six image datasets. Particularly, the proposed method performs far better on datasets with a limited number of images.

Identifying weak linear features with the "coalescing shortest path image transform".

The detection of line-like features in images finds many applications in microanalysis. Actin fibers, microtubules, neurites, pilis, DNA, and other biological structures all come up as tenuous curved lines in microscopy images. A reliable tracing method that preserves the integrity and details of these structures is particularly important for quantitative analyses. We have developed a new image transform called the "Coalescing Shortest Path Image Transform" with very encouraging properties. Our scheme efficiently combines information from an extensive collection of shortest paths in the image to delineate even very weak linear features.

Corner Detection Using Second-Order Generalized Gaussian Directional Derivative Representations.

Corner detection is a critical component of many image analysis and image understanding tasks, such as object recognition and image matching. Our research indicates that existing corner detection algorithms cannot properly depict the difference between edges and corners and this results in wrong corner detections. In this paper, the capability of second-order generalized (isotropic and anisotropic) Gaussian directional derivative filters to suppress Gaussian noise is evaluated. The second-order generalized Gaussian directional derivative representations of step edge, L-type corner, Y- or T-type corner, X-type corner, and star-type corner are investigated and obtained. A number of properties for edges and corners are discovered which enable us to propose a new image corner detection method. Finally, the criteria on detection accuracy and average repeatability under affine image transformation, JPEG compression, and noise degradation, and the criteria on region repeatability are used to evaluate the proposed detector against nine state-of-the-art methods. The experimental results show that our proposed detector outperforms all the other tested detectors.

Attentive Feature Refinement Network for Single Rainy Image Restoration.

Despite the fact that great progress has been made on single image deraining tasks, it is still challenging for existing models to produce satisfactory results directly, and it often requires a single or multiple refinement stages to gradually improve the quality. However, in this paper, we demonstrate that existing image-level refinement with a stage-independent learning design is problematic with the side effect of over/under-deraining. To resolve this issue, we for the first time propose the mechanism of learning to carry out refinement on the unsatisfactory features, and propose a novel attentive feature refinement (AFR) module. Specifically, AFR is designed as a two-branched network for simultaneous rain-distribution-aware attention map learning and attention guided hierarchy-preserving feature refinement. Guided by task-specific attention, coarse features are progressively refined to better model the diversified rainy effects. By using a separable convolution as the basic component, our AFR module introduces little computation overhead and can be readily integrated into most rainy-to-clean image translation networks for achieving better deraining results. By incorporating a series of AFR modules into a general encoder-decoder network, AFR-Net is constructed for deraining and it achieves new state-of-the-art results on both synthetic and real images. Furthermore, by using AFR-Net as a teacher model, we explore the use of knowledge distillation to successfully learn a student model that is also able to achieve state-of-the-art results but with a much faster inference speed (i.e., it only takes 0.08 second to process a 512×512 rainy image). Code and pre-trained models are available at 〈 https://github.com/RobinCSIRO/AFR-Net 〉 .

Robust Tensor Decomposition for Image Representation Based on Generalized Correntropy.

Traditional tensor decomposition methods, e.g., two dimensional principal component analysis and two dimensional singular value decomposition, that minimize mean square errors, are sensitive to outliers. To overcome this problem, in this paper we propose a new robust tensor decomposition method using generalized correntropy criterion (Corr-Tensor). A Lagrange multiplier method is used to effectively optimize the generalized correntropy objective function in an iterative manner. The Corr-Tensor can effectively improve the robustness of tensor decomposition with the existence of outliers without introducing any extra computational cost. Experimental results demonstrated that the proposed method significantly reduces the reconstruction error on face reconstruction and improves the accuracies on handwritten digit recognition and facial image clustering.

Robust Blood Cell Image Segmentation Method Based on Neural Ordinary Differential Equations.

For the analysis of medical images, one of the most basic methods is to diagnose diseases by examining blood smears through a microscope to check the morphology, number, and ratio of red blood cells and white blood cells. Therefore, accurate segmentation of blood cell images is essential for cell counting and identification. The aim of this paper is to perform blood smear image segmentation by combining neural ordinary differential equations (NODEs) with U-Net networks to improve the accuracy of image segmentation. In order to study the effect of ODE-solve on the speed and accuracy of the network, the ODE-block module was added to the nine convolutional layers in the U-Net network. Firstly, blood cell images are preprocessed to enhance the contrast between the regions to be segmented; secondly, the same dataset was used for the training set and testing set to test segmentation results. According to the experimental results, we select the location where the ordinary differential equation block (ODE-block) module is added, select the appropriate error tolerance, and balance the calculation time and the segmentation accuracy, in order to exert the best performance; finally, the error tolerance of the ODE-block is adjusted to increase the network depth, and the training NODEs-UNet network model is used for cell image segmentation. Using our proposed network model to segment blood cell images in the testing set, it can achieve 95.3% pixel accuracy and 90.61% mean intersection over union. By comparing the U-Net and ResNet networks, the pixel accuracy of our network model is increased by 0.88% and 0.46%, respectively, and the mean intersection over union is increased by 2.18% and 1.13%, respectively. Our proposed network model improves the accuracy of blood cell image segmentation and reduces the computational cost of the network.

Cascaded Attention Guidance Network for Single Rainy Image Restoration.

Restoring a rainy image with raindrops or rainstreaks of varying scales, directions, and densities is an extremely challenging task. Recent approaches attempt to leverage the rain distribution (e.g., location) as prior to generate satisfactory results. However, concatenation of a single distribution map with the rainy image or with intermediate feature maps is too simplistic to fully exploit the advantages of such priors. To further explore this valuable information, an advanced cascaded attention guidance network, dubbed as CAG-Net, is formulated and designed as a three-stage model. In the first stage, a multitask learning network is constructed for producing the attention map and coarse de-raining results simultaneously. Subsequently, the coarse results and the rain distribution map are concatenated and fed to the second stage for results refinement. In this stage, the attention map generation network from the first stage is used to formulate a novel semantic consistency loss for better detail recovery. In the third stage, a novel pyramidal "whereand- how" learning mechanism is formulated. At each pyramid level, a two-branch network is designed to take the features from previous stages as inputs to generate better attention-guidance features and de-raining features, which are then combined via a gating scheme to produce the final de-raining results. Moreover, the uncertainty maps are also generated in this stage for more accurate pixel-wise loss calculation. Extensive experiments are carried out for removing raindrops or rainstreaks from both synthetic and real rainy images, and CAG-Net is demonstrated to produce significantly better results than state-of-the-art models. Code will be publicly available after paper acceptance.