HoloCamera: Advanced Volumetric Capture for Cinematic-Quality VR Applications.

High-precision virtual environments are increasingly important for various education, simulation, training, performance, and entertainment applications. We present HoloCamera, an innovative volumetric capture instrument to rapidly acquire, process, and create cinematic-quality virtual avatars and scenarios. The HoloCamera consists of a custom-designed free-standing structure with 300 high-resolution RGB cameras mounted with uniform spacing spanning the four sides and the ceiling of a room-sized studio. The light field acquired from these cameras is streamed through a distributed array of GPUs that interleave the processing and transmission of 4K resolution images. The distributed compute infrastructure that powers these RGB cameras consists of 50 Jetson AGX Xavier boards, with each processing unit dedicated to driving and processing imagery from six cameras. A high-speed Gigabit Ethernet network fabric seamlessly interconnects all computing boards. In this systems paper, we provide an in-depth description of the steps involved and lessons learned in constructing such a cutting-edge volumetric capture facility that can be generalized to other such facilities. We delve into the techniques employed to achieve precise frame synchronization and spatial calibration of cameras, careful determination of angled camera mounts, image processing from the camera sensors, and the need for a resilient and robust network infrastructure. To advance the field of volumetric capture, we are releasing a high-fidelity static light-field dataset, which will serve as a benchmark for further research and applications of cinematic-quality volumetric light fields.

Learning brain dynamics for decoding and predicting individual differences.

Insights from functional Magnetic Resonance Imaging (fMRI), as well as recordings of large numbers of neurons, reveal that many cognitive, emotional, and motor functions depend on the multivariate interactions of brain signals. To decode brain dynamics, we propose an architecture based on recurrent neural networks to uncover distributed spatiotemporal signatures. We demonstrate the potential of the approach using human fMRI data during movie-watching data and a continuous experimental paradigm. The model was able to learn spatiotemporal patterns that supported 15-way movie-clip classification (∼90%) at the level of brain regions, and binary classification of experimental conditions (∼60%) at the level of voxels. The model was also able to learn individual differences in measures of fluid intelligence and verbal IQ at levels comparable to that of existing techniques. We propose a dimensionality reduction approach that uncovers low-dimensional trajectories and captures essential informational (i.e., classification related) properties of brain dynamics. Finally, saliency maps and lesion analysis were employed to characterize brain-region/voxel importance, and uncovered how dynamic but consistent changes in fMRI activation influenced decoding performance. When applied at the level of voxels, our framework implements a dynamic version of multivariate pattern analysis. Our approach provides a framework for visualizing, analyzing, and discovering dynamic spatially distributed brain representations during naturalistic conditions.

3D-Kernel Foveated Rendering for Light Fields.

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this article, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47×-7.28× speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

Comparing functional connectivity matrices: A geometry-aware approach applied to participant identification.

Understanding the correlation structure associated with multiple brain measurements informs about potential "functional groupings" and network organization. The correlation structure can be conveniently captured in a matrix format that summarizes the relationships among a set of brain measurements involving two regions, for example. Such functional connectivity matrix is an important component of many types of investigation focusing on network-level properties of the brain, including clustering brain states, characterizing dynamic functional states, performing participant identification (so-called "fingerprinting") understanding how tasks reconfigure brain networks, and inter-subject correlation analysis. In these investigations, some notion of proximity or similarity of functional connectivity matrices is employed, such as their Euclidean distance or Pearson correlation (by correlating the matrix entries). Here, we propose the use of a geodesic distance metric that reflects the underlying non-Euclidean geometry of functional correlation matrices. The approach is evaluated in the context of participant identification (fingerprinting): given a participant's functional connectivity matrix based on resting-state or task data, how effectively can the participant be identified? Using geodesic distance, identification accuracy was over 95% on resting-state data, and exceeded the Pearson correlation approach by 20%. For whole-cortex regions, accuracy improved on a range of tasks by between 2% and as much as 20%. We also investigated identification using pairs of subnetworks (say, dorsal attention plus default mode), and particular combinations improved accuracy over whole-cortex participant identification by over 10%. The geodesic distance also outperformed Pearson correlation when the former employed a fourth of the data as the latter. Finally, we suggest that low-dimensional distance visualizations based on the geodesic approach help uncover the geometry of task functional connectivity in relation to that during resting-state. We propose that the use of the geodesic distance is an effective way to compare the correlation structure of the brain across a broad range of studies.

Towards a Physiological Scale of Vocal Fold Agent-Based Models of Surgical Injury and Repair: Sensitivity Analysis, Calibration and Verification.

Agent based models (ABM) were developed to numerically simulate the biological response to surgical vocal fold injury and repair at the physiological level. This study aimed to improve the representation of existing ABM through a combination of empirical and computational experiments. Empirical data of vocal fold cell populations including neutrophils, macrophages and fibroblasts were obtained using flow cytometry up to four weeks following surgical injury. Random Forests were used as a sensitivity analysis method to identify model parameters that were most influential to ABM outputs. Statistical Parameter Optimization Tool for Python was used to calibrate those parameter values to match the ABM-simulation data with the corresponding empirical data from Day 1 to Day 5 following surgery. Model performance was evaluated by verifying if the empirical data fell within the 95% confidence intervals of ABM outputs of cell quantities at Day 7, Week 2 and Week 4. For Day 7, all empirical data were within the ABM output ranges. The trends of ABM-simulated cell populations were also qualitatively comparable to those of the empirical data beyond Day 7. Exact values, however, fell outside of the 95% statistical confidence intervals. Parameters related to fibroblast proliferation were indicative to the ABM-simulation of fibroblast dynamics in final stages of wound healing.

Dynamic Functional Network Analysis in Mild Traumatic Brain Injury.

Mild traumatic brain injury (mTBI) is one of the most common neurological disorders for which a subset of patients develops persistent postconcussive symptoms. Previous studies discovered abnormalities and disruptions in the brain functional networks of mTBI patients principally using static functional connectivity measures which assume that neural communication across the brain is static during resting state conditions. In this study, we examine the differences in dynamic neural communication between mTBI and control participants through the application of a combination of dynamic functional analysis and graph theoretic algorithms. Resting state functional magnetic resonance imaging data was obtained on 47 mTBI patients at the acute stage of injury and 30 demographically matched healthy control participants. Results show unique alterations in both the static and dynamic functional connectivity at the acute stage in mTBI patients who suffer persistent symptoms (≥6 months after injury). In addition, mTBI patients with postconcussion syndrome demonstrated a unique allocation of time in various brain states compared to both control participants and mTBI patients with favorable outcomes. These findings suggest that global damage to the overall communication across the brain in the acute stage may contribute to chronic mTBI symptoms. Dynamic functional analysis is a powerful tool that provides insights into the brain states and the innovative analysis methodology utilized may hold the potential to delineate patients predisposed to poor outcomes upon early presentation following injury.

Brain dynamics and temporal trajectories during task and naturalistic processing.

Human functional Magnetic Resonance Imaging (fMRI) data are acquired while participants engage in diverse perceptual, motor, cognitive, and emotional tasks. Although data are acquired temporally, they are most often treated in a quasi-static manner. Yet, a fuller understanding of the mechanisms that support mental functions necessitates the characterization of dynamic properties. Here, we describe an approach employing a class of recurrent neural networks called reservoir computing, and show the feasibility and potential of using it for the analysis of temporal properties of brain data. We show that reservoirs can be used effectively both for condition classification and for characterizing lower-dimensional "trajectories" of temporal data. Classification accuracy was approximately 90% for short clips of "social interactions" and around 70% for clips extracted from movie segments. Data representations with 12 or fewer dimensions (from an original space with over 300) attained classification accuracy within 5% of the full data. We hypothesize that such low-dimensional trajectories may provide "signatures" that can be associated with tasks and/or mental states. The approach was applied across participants (that is, training in one set of participants, and testing in a separate group), showing that representations generalized well to unseen participants. Taken together, we believe the present approach provides a promising framework to characterize dynamic fMRI information during both tasks and naturalistic conditions.

High-Performance Agent-Based Modeling Applied to Vocal Fold Inflammation and Repair.

Fast and accurate computational biology models offer the prospect of accelerating the development of personalized medicine. A tool capable of estimating treatment success can help prevent unnecessary and costly treatments and potential harmful side effects. A novel high-performance Agent-Based Model (ABM) was adopted to simulate and visualize multi-scale complex biological processes arising in vocal fold inflammation and repair. The computational scheme was designed to organize the 3D ABM sub-tasks to fully utilize the resources available on current heterogeneous platforms consisting of multi-core CPUs and many-core GPUs. Subtasks are further parallelized and convolution-based diffusion is used to enhance the performance of the ABM simulation. The scheme was implemented using a client-server protocol allowing the results of each iteration to be analyzed and visualized on the server (i.e., in-situ) while the simulation is running on the same server. The resulting simulation and visualization software enables users to interact with and steer the course of the simulation in real-time as needed. This high-resolution 3D ABM framework was used for a case study of surgical vocal fold injury and repair. The new framework is capable of completing the simulation, visualization and remote result delivery in under 7 s per iteration, where each iteration of the simulation represents 30 min in the real world. The case study model was simulated at the physiological scale of a human vocal fold. This simulation tracks 17 million biological cells as well as a total of 1.7 billion signaling chemical and structural protein data points. The visualization component processes and renders all simulated biological cells and 154 million signaling chemical data points. The proposed high-performance 3D ABM was verified through comparisons with empirical vocal fold data. Representative trends of biomarker predictions in surgically injured vocal folds were observed.

LEICA: Laplacian eigenmaps for group ICA decomposition of fMRI data.

Independent component analysis (ICA) is a data-driven method that has been increasingly used for analyzing functional Magnetic Resonance Imaging (fMRI) data. However, generalizing ICA to multi-subject studies is non-trivial due to the high-dimensionality of the data, the complexity of the underlying neuronal processes, the presence of various noise sources, and inter-subject variability. Current group ICA based approaches typically use several forms of the Principal Component Analysis (PCA) method to extend ICA for generating group inferences. However, linear dimensionality reduction techniques have serious limitations including the fact that the underlying BOLD signal is a complex function of several nonlinear processes. In this paper, we propose an effective non-linear ICA-based model for extracting group-level spatial maps from multi-subject fMRI datasets. We use a non-linear dimensionality reduction algorithm based on Laplacian eigenmaps to identify a manifold subspace common to the group, such that this mapping preserves the correlation among voxels' time series as much as possible. These eigenmaps are modeled as linear mixtures of a set of group-level spatial features, which are then extracted using ICA. The resulting algorithm is called LEICA (Laplacian Eigenmaps for group ICA decomposition). We introduce a number of methods to evaluate LEICA using 100-subject resting state and 100-subject working memory task fMRI datasets from the Human Connectome Project (HCP). The test results show that the extracted spatial maps from LEICA are meaningful functional networks similar to those produced by some of the best known methods. Importantly, relative to state-of-the-art methods, our algorithm compares favorably in terms of the functional cohesiveness of the spatial maps generated, as well as in terms of the reproducibility of the results.

In Situ Visualization for 3D Agent-Based Vocal Fold Inflammation and Repair Simulation.

A fast and insightful visualization is essential in modeling biological system behaviors and understanding underlying inter-cellular mechanisms. High fidelity models produce billions of data points per time step, making in situ visualization techniques extremely desirable as they mitigate I/O bottlenecks and provide computational steering capability. In this work, we present a novel high-performance scheme to couple in situ visualization with the simulation of the vocal fold inflammation and repair using little to no extra cost in execution time or computing resources. The visualization component is first optimized with an adaptive sampling scheme to accelerate the rendering process while maintaining the precision of the displayed visual results. Our software employs VirtualGL to perform visualization in situ. The scheme overlaps visualization and simulation, resulting in the optimal utilization of computing resources. This results in an in situ system biology simulation suite capable of remote simulation of 17 million biological cells and 1.2 billion chemical data points, remote visualization of the results, and delivery of visualized frames with aggregated statistics to remote clients in real-time.

Real-Time Agent-Based Modeling Simulation with in-situ Visualization of Complex Biological Systems: A Case Study on Vocal Fold Inflammation and Healing.

We present an efficient and scalable scheme for implementing agent-based modeling (ABM) simulation with In Situ visualization of large complex systems on heterogeneous computing platforms. The scheme is designed to make optimal use of the resources available on a heterogeneous platform consisting of a multicore CPU and a GPU, resulting in minimal to no resource idle time. Furthermore, the scheme was implemented under a client-server paradigm that enables remote users to visualize and analyze simulation data as it is being generated at each time step of the model. Performance of a simulation case study of vocal fold inflammation and wound healing with 3.8 million agents shows 35× and 7× speedup in execution time over single-core and multi-core CPU respectively. Each iteration of the model took less than 200 ms to simulate, visualize and send the results to the client. This enables users to monitor the simulation in real-time and modify its course as needed.

Connectivity-Based Brain Parcellation: A Connectivity-Based Atlas for Schizophrenia Research.

Defining brain structures of interest is an important preliminary step in brain-connectivity analysis. Researchers interested in connectivity patterns among brain structures typically employ manually delineated volumes of interest, or regions in a readily available atlas, to limit the scope of connectivity analysis to relevant regions. However, most structural brain atlases, and manually delineated volumes of interest, do not take voxel-wise connectivity patterns into consideration, and therefore may not be ideal for anatomic connectivity analysis. We herein propose a method to parcellate the brain into regions of interest based on connectivity. We formulate connectivity-based parcellation as a graph-cut problem, which we solve approximately using a novel multi-class Hopfield network algorithm. We demonstrate the application of this approach using diffusion tensor imaging data from an ongoing study of schizophrenia. Compared to a standard anatomic atlas, the connectivity-based atlas supports better classification performance when distinguishing schizophrenic from normal subjects. Comparing connectivity patterns averaged across the normal and schizophrenic subjects, we note significant systematic differences between the two atlases.

Quantitative analysis of F-actin redistribution in astrocytoma cells treated with candidate pharmaceuticals.

Actin fibers (F-actin) control the shape and internal organization of cells, and generate force. It has been long appreciated that these functions are tightly coupled, and in some cases drive cell behavior and cell fate. The distribution and dynamics of F-actin is different in cancer versus normal cells and in response to small molecules, including actin-targeting natural products and anticancer drugs. Therefore, quantifying actin structural changes from high resolution fluorescence micrographs is necessary for further understanding actin cytoskeleton dynamics and phenotypic consequences of drug interactions on cells. We applied an artificial neural network algorithm, which used image intensity and anisotropy measurements, to quantitatively classify F-actin subcellular features into actin along the edges of cells, actin at the protrusions of cells, internal fibers and punctate signals. The algorithm measured significant increase in F-actin at cell edges with concomitant decrease in internal punctate actin in astrocytoma cells lacking functional neurofibromin and p53 when treated with three structurally-distinct anticancer small molecules: OSW1, Schweinfurthin A (SA) and a synthetic marine compound 23'-dehydroxycephalostatin 1. Distinctly different changes were measured in cells treated with the actin inhibitor cytochalasin B. These measurements support published reports that SA acts on F-actin in NF1(-/-) neurofibromin deficient cancer cells through changes in Rho signaling. Quantitative pattern analysis of cells has wide applications for understanding mechanisms of small molecules, because many anti-cancer drugs directly or indirectly target cytoskeletal proteins. Furthermore, quantitative information about the actin cytoskeleton may make it possible to further understand cell fate decisions using mathematically testable models.

Quantifying the astrocytoma cell response to candidate pharmaceutical from F-ACTIN image analysis.

The distribution, directionality and motility of the actin fibers control cell shape, affect cell function and are different in cancer versus normal cells. Quantification of actin structural changes is important for further understanding differences between cell types and for elucidation of the effects and dynamics of drug interactions. We have developed an image analysis framework for quantifying F-actin organization patterns in confocal microscope images in response to different candidate pharmaceutical treatments. The main problem solved was to determine which quantitative features to compute from the images that both capture the visually-observed F-actin patterns and correlate with predicted biological outcomes. The resultant numerical features were effective to quantitatively profile the changes in the spatial distribution of F-actin and facilitate the comparison of different pharmaceuticals. The validation for the segmentation was done through visual inspection and correlation to expected biological outcomes. This is the first study quantifying different structural formations of the same protein in intact cells. Preliminary results show uniquely significant increases in cortical F-actin to stress fiber ratio for increasing doses of OSW-1 and Schweinfurthin A(SA) and a less marked increase for cephalostatin 1 derivative (ceph). This increase was not observed for the actin inhibitors: cytochalasin B (cytoB) and Y-27632 (Y). Ongoing studies are further validating the algorithms, elucidating the underlying molecular pathways and will utilize the algorithms for understanding the kinetics of the F-actin changes. Since many anti-cancer drugs target the cytoskeleton, we believe that the quantitative image analysis method reported here will have broad applications to understanding the mechanisms of action of candidate pharmaceuticals.

Interactive high-resolution isosurface ray casting on multicore processors.

We present a new method for the interactive rendering of isosurfaces using ray casting on multi-core processors. This method consists of a combination of an object-order traversal that coarsely identifies possible candidate 3D data blocks for each small set of contiguous pixels, and an isosurface ray casting strategy tailored for the resulting limited-size lists of candidate 3D data blocks. While static screen partitioning is widely used in the literature, our scheme performs dynamic allocation of groups of ray casting tasks to ensure almost equal loads among the different threads running on multi-cores while maintaining spatial locality. We also make careful use of memory management environment commonly present in multi-core processors. We test our system on a two-processor Clovertown platform, each consisting of a Quad-Core 1.86-GHz Intel Xeon Processor, for a number of widely different benchmarks. The detailed experimental results show that our system is efficient and scalable, and achieves high cache performance and excellent load balancing, resulting in an overall performance that is superior to any of the previous algorithms. In fact, we achieve an interactive isosurface rendering on a 1024(2) screen for all the datasets tested up to the maximum size of the main memory of our platform.

Isosurface extraction and spatial filtering using Persistent OcTree (POT).

We propose a novel Persistent OcTree (POT) indexing structure for accelerating isosurface extraction and spatial filtering from volumetric data. This data structure efficiently handles a wide range of visualization problems such as the generation of view-dependent isosurfaces, ray tracing, and isocontour slicing for high dimensional data. POT can be viewed as a hybrid data structure between the interval tree and the Branch-On-Need Octree (BONO) in the sense that it achieves the asymptotic bound of the interval tree for identifying the active cells corresponding to an isosurface and is more efficient than BONO for handling spatial queries. We encode a compact octree for each isovalue. Each such octree contains only the corresponding active cells, in such a way that the combined structure has linear space. The inherent hierarchical structure associated with the active cells enables very fast filtering of the active cells based on spatial constraints. We demonstrate the effectiveness of our approach by performing view-dependent isosurfacing on a wide variety of volumetric data sets and 4D isocontour slicing on the time-varying Richtmyer-Meshkov instability dataset.