Virtual blebbistatin: A robust and rapid software approach to motion artifact removal in optical mapping of cardiomyocytes.

Fluorescent reporters of cardiac electrophysiology provide valuable information on heart cell and tissue function. However, motion artifacts caused by cardiac muscle contraction interfere with accurate measurement of fluorescence signals. Although drugs such as blebbistatin can be applied to stop cardiac tissue from contracting by uncoupling calcium-contraction, their usage prevents the study of excitation-contraction coupling and, as we show, impacts cellular structure. We therefore developed a robust method to remove motion computationally from images of contracting cardiac muscle and to map fluorescent reporters of cardiac electrophysiological activity onto images of undeformed tissue. When validated on cardiomyocytes derived from human induced pluripotent stem cells (iPSCs), in both monolayers and engineered tissues, the method enabled efficient and robust reduction of motion artifact. As with pharmacologic approaches using blebbistatin for motion removal, our algorithm improved the accuracy of optical mapping, as demonstrated by spatial maps of calcium transient decay. However, unlike pharmacologic motion removal, our computational approach allowed direct analysis of calcium-contraction coupling. Results revealed calcium-contraction coupling to be more uniform across cells within engineered tissues than across cells in monolayer culture. The algorithm shows promise as a robust and accurate tool for optical mapping studies of excitation-contraction coupling in heart tissue.

Oligoclonal Band Straightening Based on Optimized Hierarchical Warping for Multiple Sclerosis Diagnosis.

The detection of immunoglobulin G (IgG) oligoclonal bands (OCB) in cerebrospinal fluid (CSF) by isoelectric focusing (IEF) is a valuable tool for the diagnosis of multiple sclerosis. Over the last decade, the results of our clinical research have suggested that tears are a non-invasive alternative to CSF. However, since tear samples have a lower IgG concentration than CSF, a sensitive OCB detection is therefore required. We are developing the first automatic tool for IEF analysis, with a view to speeding up the current visual inspection method, removing user variability, reducing misinterpretation, and facilitating OCB quantification and follow-up studies. The removal of band distortion is a key image enhancement step in increasing the reliability of automatic OCB detection. Here, we describe a novel, fully automatic band-straightening algorithm. The algorithm is based on a correlation directional warping function, estimated using an energy minimization procedure. The approach was optimized via an innovative coupling of a hierarchy of image resolutions to a hierarchy of transformation, in which band misalignment is corrected at successively finer scales. The algorithm's performance was assessed in terms of the bands' standard deviation before and after straightening, using a synthetic dataset and a set of 200 lanes of CSF, tear, serum and control samples on which experts had manually delineated the bands. The number of distorted bands was divided by almost 16 for the synthetic lanes and by 7 for the test dataset of real lanes. This method can be applied effectively to different sample types. It can realign minimal contrast bands and is robust for non-uniform deformations.

A Foreground-Aware Framework for Local Face Attribute Transfer.

In the context of social media, large amounts of headshot photos are taken everyday. Unfortunately, in addition to laborious editing and modification, creating a visually compelling photographic masterpiece for sharing requires advanced professional skills, which are difficult for ordinary Internet users. Though there are many algorithms automatically and globally transferring the style from one image to another, they fail to respect the semantics of the scene and are unable to allow users to merely transfer the attributes of one or two face organs in the foreground region leaving the background region unchanged. To overcome this problem, we developed a novel framework for semantically meaningful local face attribute transfer, which can flexibly transfer the local attribute of a face organ from the reference image to a semantically equivalent organ in the input image, while preserving the background. Our method involves warping the reference photo to match the shape, pose, location, and expression of the input image. The fusion of the warped reference image and input image is then taken as the initialized image for a neural style transfer algorithm. Our method achieves better performance in terms of inception score (3.81) and Fréchet inception distance (80.31), which is about 10% higher than those of competitors, indicating that our framework is capable of producing high-quality and photorealistic attribute transfer results. Both theoretical findings and experimental results are provided to demonstrate the efficacy of the proposed framework, reveal its superiority over other state-of-the-art alternatives.

Blind First-Order Perspective Distortion Correction Using Parallel Convolutional Neural Networks.

In this work, we present a network architecture with parallel convolutional neural networks (CNN) for removing perspective distortion in images. While other works generate corrected images through the use of generative adversarial networks or encoder-decoder networks, we propose a method wherein three CNNs are trained in parallel, to predict a certain element pair in the 3×3 transformation matrix, M^. The corrected image is produced by transforming the distorted input image using M^-1. The networks are trained from our generated distorted image dataset using KITTI images. Experimental results show promise in this approach, as our method is capable of correcting perspective distortions on images and outperforms other state-of-the-art methods. Our method also recovers the intended scale and proportion of the image, which is not observed in other works.

A Unified Framework for Street-View Panorama Stitching.

In this paper, we propose a unified framework to generate a pleasant and high-quality street-view panorama by stitching multiple panoramic images captured from the cameras mounted on the mobile platform. Our proposed framework is comprised of four major steps: image warping, color correction, optimal seam line detection and image blending. Since the input images are captured without a precisely common projection center from the scenes with the depth differences with respect to the cameras to different extents, such images cannot be precisely aligned in geometry. Therefore, an efficient image warping method based on the dense optical flow field is proposed to greatly suppress the influence of large geometric misalignment at first. Then, to lessen the influence of photometric inconsistencies caused by the illumination variations and different exposure settings, we propose an efficient color correction algorithm via matching extreme points of histograms to greatly decrease color differences between warped images. After that, the optimal seam lines between adjacent input images are detected via the graph cut energy minimization framework. At last, the Laplacian pyramid blending algorithm is applied to further eliminate the stitching artifacts along the optimal seam lines. Experimental results on a large set of challenging street-view panoramic images captured form the real world illustrate that the proposed system is capable of creating high-quality panoramas.

Perceptual Contrastive Generative Adversarial Network based on image warping for unsupervised image-to-image translation.

This paper proposes an unsupervised image-to-image (UI2I) translation model, called Perceptual Contrastive Generative Adversarial Network (PCGAN), which can mitigate the distortion problem to enhance performance of the traditional UI2I methods. The PCGAN is designed with a two-stage UI2I model. In the first stage of the PCGAN, it leverages a novel image warping to transform shapes of objects in input (source) images. In the second stage of the PCGAN, the residual prediction is devised in refinements of the outputs of the first stage of the PCGAN. To promote performance of the image warping, a loss function, called Perceptual Patch-Wise InfoNCE, is developed in the PCGAN to effectively memorize the visual correspondences between warped images and refined images. Experimental results on quantitative evaluation and visualization comparison for UI2I benchmarks show that the PCGAN is superior to other existing methods considered here.

Facial functional networks during resting state revealed by thermal infrared imaging.

In recent decades, an increasing number of studies on psychophysiology and, in general, on clinical medicine has employed the technique of facial thermal infrared imaging (IRI), which allows to obtain information about the emotional and physical states of the subjects in a completely non-invasive and contactless fashion. Several regions of interest (ROIs) have been reported in literature as salient areas for the psychophysiological characterization of a subject (i.e. nose tip and glabella ROIs). There is however a lack of studies focusing on the functional correlation among these ROIs and about the physiological basis of the relation existing between thermal IRI and vital signals, such as the electrodermal activity, i.e. the galvanic skin response (GSR). The present study offers a new methodology able to assess the functional connection between salient seed ROIs of thermal IRI and all the pixel of the face. The same approach was also applied considering as seed signal the GSR and its phasic and tonic components. Seed correlation analysis on 63 healthy volunteers demonstrated the presence of a common pathway regulating the facial thermal functionality and the electrodermal activity. The procedure was also tested on a pathological case study, finding a completely different pattern compared to the healthy cases. The method represents a promising tool in neurology, physiology and applied neurosciences.

Deep learning-based 2D/3D registration of an atlas to biplanar X-ray images.

PURPOSE: The registration of a 3D atlas image to 2D radiographs enables 3D pre-operative planning without the need to acquire costly and high-dose CT-scans. Recently, many deep-learning-based 2D/3D registration methods have been proposed which tackle the problem as a reconstruction by regressing the 3D image immediately from the radiographs, rather than registering an atlas image. Consequently, they are less constrained against unfeasible reconstructions and have no possibility to warp auxiliary data. Finally, they are, by construction, limited to orthogonal projections. METHODS: We propose a novel end-to-end trainable 2D/3D registration network that regresses a dense deformation field that warps an atlas image such that the forward projection of the warped atlas matches the input 2D radiographs. We effectively take the projection matrix into account in the regression problem by integrating a projective and inverse projective spatial transform layer into the network. RESULTS: Comprehensive experiments conducted on simulated DRRs from patient CT images demonstrate the efficacy of the network. Our network yields an average Dice score of 0.94 and an average symmetric surface distance of 0.84 mm on our test dataset. It has experimentally been determined that projection geometries with 80[Formula: see text] to 100[Formula: see text] projection angle difference result in the highest accuracy. CONCLUSION: Our network is able to accurately reconstruct patient-specific CT-images from a pair of near-orthogonal calibrated radiographs by regressing a deformation field that warps an atlas image or any other auxiliary data. Our method is not constrained to orthogonal projections, increasing its applicability in medical practices. It remains a future task to extend the network for uncalibrated radiographs.

Adjoint image warping using multivariate splines with application to four-dimensional computed tomography.

PURPOSE: Adjoint image warping is an important tool to solve image reconstruction problems that warp the unknown image in the forward model. This includes four-dimensional computed tomography (4D-CT) models in which images are compared against recorded projection images of various time frames using image warping as a model of the motion. The inversion of these models requires the adjoint of image warping, which up to now has been substituted by approximations. We introduce an efficient implementation of the exact adjoints of multivariate spline based image warping, and compare it against previously used alternatives. METHODS: Using symbolic computer algebra, we computed a list of 64 polynomials that allow us to compute a matrix representation of trivariate cubic image warping. By combining an on-the-fly computation of this matrix with a parallelized implementation of columnwise matrix multiplication, we obtained an efficient, low memory implementation of the adjoint action of 3D cubic image warping. We used this operator in the solution of a previously proposed 4D-CT reconstruction model in which the image of a single subscan was compared against projection data of multiple subscans by warping and then projecting the image. We compared the properties of our exact adjoint with those of approximate adjoints by warping along inverted motion. RESULTS: Our method requires halve the memory to store motion between subscans, compared to methods that need to compute and store an approximate inverse of the motion. It also avoids the computation time to invert the motion and the tunable parameter of the number of iterations used to perform this inversion. Yet, a similar and often better reconstruction quality was obtained in comparison with these more expensive methods, especially when the motion is large. When compared against a simpler method that is similar to ours in computational demands, our method achieves a higher reconstruction quality in general. CONCLUSIONS: Our implementation of the exact adjoint of cubic image warping improves efficiency and provides accurate reconstructions.

A semi-automated brain atlas-based analysis pipeline for c-Fos immunohistochemical data.

BACKGROUND: The use of immunohistochemistry to quantify neural markers in various brain regions is a staple of neuroscience research. Numerous programs exist to automate quantification, but manual assignment of regions of interest (ROIs) within individual brain sections remains time consuming and can introduce interobserver variability. NEW METHOD: We have developed a novel open source FIJI-based immunohistochemical data analysis pipeline, Atlas-Based Analysis (ABA). ABA uses landmark-based image warping to adjust the experimental image to closely align with a published rat brain atlas. c-Fos positive cells are then quantified within predetermined ROI coordinates derived from the brain atlas. Image warping adjusts for natural variation in brain sections to ensure reliable alignment of ROIs for data analysis. This pipeline can be adapted for new atlases, landmarks, ROIs, and quantification measurements. RESULTS: ABA permits rapid quantification of immunoreactivity in multiple ROIs and produces results with high levels of interobserver consistency. COMPARISON WITH EXISTING METHODS: Compared to manual ROI designation, ABA reduces total analysis time by ∼70%. With correct use of landmarks for image warping, ABA produces similar results to manually drawn ROIs, results in no interobserver variability, and maintains c-Fos+ pixel dimensions. CONCLUSIONS: ABA reduces time to obtain reliable results when performing automated immunoreactivity quantification and allows multiple users to analyze data without compromising the reliability of data obtained.

A new data augmentation method based on local image warping for medical image segmentation.

PURPOSE: The segmentation accuracy of medical images was improved by increasing the number of training samples using a local image warping technique. The performance of the proposed method was evaluated in the segmentation of breast masses, prostate and brain tumors, and lung nodules. METHODS: We propose a simple data augmentation method which is called stochastic evolution (SE). Specifically, the idea of SE stems from our thinking about the deterioration of the diseased tissue and the healing process. In order to simulate this natural process, we implement it according to the local distortion algorithm in image warping. In other words, the irregular deterioration and healing processes of the diseased tissue is simulated according to the direction of the local distortion, thereby producing a natural sample that is indistinguishable by humans. RESULTS: The proposed method is evaluated on four segmentation tasks of breast masses, prostate, brain tumors, and lung nodules. Comparing the experimental results of four segmentation methods based on the UNet segmentation architecture without adding any expanded data during training, the accuracy and the Hausdorff distance obtained in our approach remain almost the same as other methods. However, the dice similarity coefficient (DSC) and sensitivity (SEN) have both improved to some extent. Among them, DSC is increased by 5.2%, 2.8%, 1.0%, and 3.2%, respectively; SEN is increased by 6.9%, 4.3%, 1.2%, and 4.5%, respectively. CONCLUSIONS: Experimental results show that the proposed SE data augmentation method could improve the segmentation accuracy of breast masses, prostate, brain tumors, and lung nodules. The method also shows the robustness with different image datasets and imaging modalities.

Biomechanical modeling and computer simulation of the brain during neurosurgery.

Computational biomechanics of the brain for neurosurgery is an emerging area of research recently gaining in importance and practical applications. This review paper presents the contributions of the Intelligent Systems for Medicine Laboratory and its collaborators to this field, discussing the modeling approaches adopted and the methods developed for obtaining the numerical solutions. We adopt a physics-based modeling approach and describe the brain deformation in mechanical terms (such as displacements, strains, and stresses), which can be computed using a biomechanical model, by solving a continuum mechanics problem. We present our modeling approaches related to geometry creation, boundary conditions, loading, and material properties. From the point of view of solution methods, we advocate the use of fully nonlinear modeling approaches, capable of capturing very large deformations and nonlinear material behavior. We discuss finite element and meshless domain discretization, the use of the total Lagrangian formulation of continuum mechanics, and explicit time integration for solving both time-accurate and steady-state problems. We present the methods developed for handling contacts and for warping 3D medical images using the results of our simulations. We present two examples to showcase these methods: brain shift estimation for image registration and brain deformation computation for neuronavigation in epilepsy treatment.

An efficient intensity-based ready-to-use X-ray image stitcher.

BACKGROUND: The limited field of view of the X-ray image intensifier makes it difficult to cover a large target area with a single X-ray image. X-ray image stitching techniques have been proposed to produce a panoramic X-ray image. METHODS: This paper presents an efficient intensity-based X-ray image stitcher, which does not rely on accurate C-arm motion control or auxiliary devices and hence is ready to use in clinic. The stitcher consumes sequentially captured X-ray images with overlap areas and automatically produces a panoramic image. The gradient information for optimization of image alignment is obtained using a back-propagation scheme so that it is convenient to adopt various image warping models. RESULTS: The proposed stitcher has the following advantages over existing methods: (1) no additional hardware modification or auxiliary markers are needed; (2) it is robust against feature-based approaches; (3) arbitrary warping models and shapes of the region of interest are supported; (4) seamless stitching is achieved using multi-band blending. Experiments have been performed to confirm the effectiveness of the proposed method. CONCLUSION: The proposed X-ray image stitcher is efficient, accurate and ready to use in clinic.