Enhancing Adaptive Physics Refinement Simulations Through the Addition of Realistic Red Blood Cell Counts.

Simulations of cancer cell transport require accurately modeling mm-scale and longer trajectories through a circulatory system containing trillions of deformable red blood cells, whose intercellular interactions require submicron fidelity. Using a hybrid CPU-GPU approach, we extend the advanced physics refinement (APR) method to couple a finely-resolved region of explicitly-modeled red blood cells to a coarsely-resolved bulk fluid domain. We further develop algorithms that: capture the dynamics at the interface of differing viscosities, maintain hematocrit within the cell-filled volume, and move the finely-resolved region and encapsulated cells while tracking an individual cancer cell. Comparison to a fully-resolved fluid-structure interaction model is presented for verification. Finally, we use the advanced APR method to simulate cancer cell transport over a mm-scale distance while maintaining a local region of RBCs, using a fraction of the computational power required to run a fully-resolved model.

The role of adhesive receptor patterns on cell transport in complex microvessels.

Cell transport is governed by the interaction of fluid dynamic forces and biochemical factors such as adhesion receptor expression and concentration. Although the effect of endothelial receptor density is well understood, it is not clear how the spacing and local spatial distribution of receptors affect cell adhesion in three-dimensional microvessels. To elucidate the effect of vessel shape on cell trajectory and the arrangement of endothelial receptors on cell adhesion, we employed a three-dimensional deformable cell model that incorporates microscale interactions between the cell and the endothelium. Computational cellular adhesion models are systematically altered to assess the influence of receptor spacing. We demonstrate that the patterns of receptors on the vessel walls are a key factor guiding cell movement. In straight microvessels, we show a relationship between cell velocity and the spatial distribution of adhesive endothelial receptors, with larger receptor patches producing lower translational velocities. The joint effect of the complex vessel topology seen in microvessel shapes such as curved and bifurcated vessels when compared to straight tubes is explored with results which showed the spatial distribution of receptors affecting cell trajectory. Our findings here represent demonstration of the previously undescribed relationship between receptor pattern and geometry that guides cellular movement in complex microenvironments.

Distributed Acceleration of Adhesive Dynamics Simulations.

Cell adhesion plays a critical role in processes ranging from leukocyte migration to cancer cell transport during metastasis. Adhesive cell interactions can occur over large distances in microvessel networks with cells traveling over distances much greater than the length scale of their own diameter. Therefore, biologically relevant investigations necessitate efficient modeling of large field-of-view domains, but current models are limited by simulating such geometries at the sub-micron scale required to model adhesive interactions which greatly increases the computational requirements for even small domain sizes. In this study we introduce a hybrid scheme reliant on both on-node and distributed parallelism to accelerate a fully deformable adhesive dynamics cell model. This scheme leads to performant system usage of modern supercomputers which use a many-core per-node architecture. On-node acceleration is augmented by a combination of spatial data structures and algorithmic changes to lessen the need for atomic operations. This deformable adhesive cell model accelerated with hybrid parallelization allows us to bridge the gap between high-resolution cell models which can capture the sub-micron adhesive interactions between the cell and its microenvironment, and large-scale fluid-structure interaction (FSI) models which can track cells over considerable distances. By integrating the sub-micron simulation environment into a distributed FSI simulation we enable the study of previously unfeasible research questions involving numerous adhesive cells in microvessel networks such as cancer cell transport through the microcirculation.

High Performance Adaptive Physics Refinement to Enable Large-Scale Tracking of Cancer Cell Trajectory.

The ability to track simulated cancer cells through the circulatory system, important for developing a mechanistic understanding of metastatic spread, pushes the limits of today's supercomputers by requiring the simulation of large fluid volumes at cellular-scale resolution. To overcome this challenge, we introduce a new adaptive physics refinement (APR) method that captures cellular-scale interaction across large domains and leverages a hybrid CPU-GPU approach to maximize performance. Through algorithmic advances that integrate multi-physics and multi-resolution models, we establish a finely resolved window with explicitly modeled cells coupled to a coarsely resolved bulk fluid domain. In this work we present multiple validations of the APR framework by comparing against fully resolved fluid-structure interaction methods and employ techniques, such as latency hiding and maximizing memory bandwidth, to effectively utilize heterogeneous node architectures. Collectively, these computational developments and performance optimizations provide a robust and scalable framework to enable system-level simulations of cancer cell transport.

Computational models of cancer cell transport through the microcirculation.

The transport of cancerous cells through the microcirculation during metastatic spread encompasses several interdependent steps that are not fully understood. Computational models which resolve the cellular-scale dynamics of complex microcirculatory flows offer considerable potential to yield needed insights into the spread of cancer as a result of the level of detail that can be captured. In recent years, in silico methods have been developed that can accurately and efficiently model the circulatory flows of cancer and other biological cells. These computational methods are capable of resolving detailed fluid flow fields which transport cells through tortuous physiological geometries, as well as the deformation and interactions between cells, cell-to-endothelium interactions, and tumor cell aggregates, all of which play important roles in metastatic spread. Such models can provide a powerful complement to experimental works, and a promising approach to recapitulating the endogenous setting while maintaining control over parameters such as shear rate, cell deformability, and the strength of adhesive binding to better understand tumor cell transport. In this review, we present an overview of computational models that have been developed for modeling cancer cells in the microcirculation, including insights they have provided into cell transport phenomena.

Investigating the Interaction Between Circulating Tumor Cells and Local Hydrodynamics via Experiment and Simulations.

INTRODUCTION: The biological and mechanical properties of circulating tumor cells (CTCs) in combination with the hemodynamics affect the preference of metastatic sites in the vasculature. Despite the extensive literature on the effects of biological properties on cell adhesion, the effects of hydrodynamic forces on primary attachment remains an active area of research. Using simulations in conjunction with experimentation, we provide new insight into the interplay of CTCs dynamics and local hydrodynamics. METHODS: A flow experiment of CTC attachment was performed within a bioprinted, double branching endothelialized vessel. Simulations of fluid flow and CTC transport in the reconstructed and idealized bifurcated vessel were respectively performed by HARVEY, our in-house massively parallel computational fluid dynamics solver. HARVEY is based on the lattice Boltzmann and finite element methods to model the fluid and cells dynamics. The immersed boundary method is employed for resolving the fluid-structure interaction. RESULTS: CTC attachment was quantified experimentally at all regions of the complex vessel. The results demonstrate a clear preference for CTCs to attach at the branch points. To elucidate the effect of the vessel topology on the location of attachment, a fluid-only simulation was performed assessing the differences in the hydrodynamics along the vessel. CTC transport in idealized bifurcated vessels was subsequently studied to examine the effects of cell deformability on the local hydrodynamics patterns and, thus, the preference of attachment sites. CONCLUSIONS: The current work provides evidence on the correlation of the hydrodynamics forces arising from the vessel topology and CTC properties on the attachment regions.

Computational Framework to Evaluate the Hydrodynamics of Cell Scaffold Geometries.

The fluid dynamics of microporous materials are important to many biomedical processes such as cell deposition in scaffold materials, tissue engineering, and bioreactors. Microporous scaffolds are frequently composed of suspensions of beads that have varying topology which, in turn, informs their hydrodynamic properties. Previous work has shown that shear stress distributions can affect the response of cells in microporous environments. Using computational fluid dynamics, we characterize localized differences in fluid flow attributes such wall shear stress and velocity to better understand the fluid dynamics underpinning microporous device function. We evaluated whether bead packings with similar void fractions had different fluid dynamics as characterized by the distribution of velocity magnitudes and wall shear stress and found that there are differences despite the similarities in void fraction. We show that another metric, the average distance to the nearest wall, can provide an additional variable to measure the porosity and susceptibility of microporous materials to high shear stress. By increasing our understanding of the impact of bead size on cell scaffold fluid dynamics we aim to increase the ability to predict important attributes such as loading efficiency in these devices.

Multi-GPU Immersed Boundary Method Hemodynamics Simulations.

Large-scale simulations of blood flow that resolve the 3D deformation of each comprising cell are increasingly popular owing to algorithmic developments in conjunction with advances in compute capability. Among different approaches for modeling cell-resolved hemodynamics, fluid structure interaction (FSI) algorithms based on the immersed boundary method are frequently employed for coupling separate solvers for the background fluid and the cells within one framework. GPUs can accelerate these simulations; however, both current pre-exascale and future exascale CPU-GPU heterogeneous systems face communication challenges critical to performance and scalability. We describe, to our knowledge, the largest distributed GPU-accelerated FSI simulations of high hematocrit cell-resolved flows with over 17 million red blood cells. We compare scaling on a fat node system with six GPUs per node and on a system with a single GPU per node. Through comparison between the CPU- and GPU-based implementations, we identify the costs of data movement in multiscale multi-grid FSI simulations on heterogeneous systems and show it to be the greatest performance bottleneck on the GPU.