Propagation pattern for moment representation of the lattice Boltzmann method.

A propagation pattern for the moment representation of the regularized lattice Boltzmann method (LBM) in three dimensions is presented. Using effectively lossless compression, the simulation state is stored as a set of moments of the lattice Boltzmann distribution function, instead of the distribution function itself. An efficient cache-aware propagation pattern for this moment representation has the effect of substantially reducing both the storage and memory bandwidth required for LBM simulations. This paper extends recent work with the moment representation by expanding the performance analysis on central processing unit (CPU) architectures, considering how boundary conditions are implemented, and demonstrating the effectiveness of the moment representation on a graphics processing unit (GPU) architecture.

Exascale applications: skin in the game.

As noted in Wikipedia, skin in the game refers to having 'incurred risk by being involved in achieving a goal', where 'skin is a synecdoche for the person involved, and game is the metaphor for actions on the field of play under discussion'. For exascale applications under development in the US Department of Energy Exascale Computing Project, nothing could be more apt, with the skin being exascale applications and the game being delivering comprehensive science-based computational applications that effectively exploit exascale high-performance computing technologies to provide breakthrough modelling and simulation and data science solutions. These solutions will yield high-confidence insights and answers to the most critical problems and challenges for the USA in scientific discovery, national security, energy assurance, economic competitiveness and advanced healthcare. This article is part of a discussion meeting issue 'Numerical algorithms for high-performance computational science'.

Establishing the longitudinal hemodynamic mapping framework for wearable-driven coronary digital twins.

Understanding the evolving nature of coronary hemodynamics is crucial for early disease detection and monitoring progression. We require digital twins that mimic a patient's circulatory system by integrating continuous physiological data and computing hemodynamic patterns over months. Current models match clinical flow measurements but are limited to single heartbeats. To this end, we introduced the longitudinal hemodynamic mapping framework (LHMF), designed to tackle critical challenges: (1) computational intractability of explicit methods; (2) boundary conditions reflecting varying activity states; and (3) accessible computing resources for clinical translation. We show negligible error (0.0002-0.004%) between LHMF and explicit data of 750 heartbeats. We deployed LHMF across traditional and cloud-based platforms, demonstrating high-throughput simulations on heterogeneous systems. Additionally, we established LHMFC, where hemodynamically similar heartbeats are clustered to avoid redundant simulations, accurately reconstructing longitudinal hemodynamic maps (LHMs). This study captured 3D hemodynamics over 4.5 million heartbeats, paving the way for cardiovascular digital twins.

Cloud Computing to Enable Wearable-Driven Longitudinal Hemodynamic Maps.

Tracking hemodynamic responses to treatment and stimuli over long periods remains a grand challenge. Moving from established single-heartbeat technology to longitudinal profiles would require continuous data describing how the patient's state evolves, new methods to extend the temporal domain over which flow is sampled, and high-throughput computing resources. While personalized digital twins can accurately measure 3D hemodynamics over several heartbeats, state-of-the-art methods would require hundreds of years of wallclock time on leadership scale systems to simulate one day of activity. To address these challenges, we propose a cloud-based, parallel-in-time framework leveraging continuous data from wearable devices to capture the first 3D patient-specific, longitudinal hemodynamic maps. We demonstrate the validity of our method by establishing ground truth data for 750 beats and comparing the results. Our cloud-based framework is based on an initial fixed set of simulations to enable the wearable-informed creation of personalized longitudinal hemodynamic maps.

Enhancing Adaptive Physics Refinement Simulations Through the Addition of Realistic Red Blood Cell Counts.

Simulations of cancer cell transport require accurately modeling mm-scale and longer trajectories through a circulatory system containing trillions of deformable red blood cells, whose intercellular interactions require submicron fidelity. Using a hybrid CPU-GPU approach, we extend the advanced physics refinement (APR) method to couple a finely-resolved region of explicitly-modeled red blood cells to a coarsely-resolved bulk fluid domain. We further develop algorithms that: capture the dynamics at the interface of differing viscosities, maintain hematocrit within the cell-filled volume, and move the finely-resolved region and encapsulated cells while tracking an individual cancer cell. Comparison to a fully-resolved fluid-structure interaction model is presented for verification. Finally, we use the advanced APR method to simulate cancer cell transport over a mm-scale distance while maintaining a local region of RBCs, using a fraction of the computational power required to run a fully-resolved model.

Plane-wave pseudopotential implementation of explicit integrators for time-dependent Kohn-Sham equations in large-scale simulations.

Explicit integrators for real-time propagation of time-dependent Kohn-Sham equations are compared regarding their suitability for performing large-scale simulations. Four algorithms are implemented and assessed for both stability and accuracy within a plane-wave pseudopotential framework, employing the adiabatic approximation to the exchange-correlation functional. Simulation results for a single sodium atom and a sodium atom embedded in bulk magnesium oxide are discussed. While the first-order Euler scheme and the second-order finite-difference scheme are unstable, the fourth-order Runge-Kutta scheme is found to be conditionally stable and accurate within this framework. Excellent parallel scalability of the algorithm up to more than a thousand processors is demonstrated for a system containing hundreds of electrons, evidencing the suitability for large-scale simulations based on real-time propagation of time-dependent Kohn-Sham equations.

High Performance Adaptive Physics Refinement to Enable Large-Scale Tracking of Cancer Cell Trajectory.

The ability to track simulated cancer cells through the circulatory system, important for developing a mechanistic understanding of metastatic spread, pushes the limits of today's supercomputers by requiring the simulation of large fluid volumes at cellular-scale resolution. To overcome this challenge, we introduce a new adaptive physics refinement (APR) method that captures cellular-scale interaction across large domains and leverages a hybrid CPU-GPU approach to maximize performance. Through algorithmic advances that integrate multi-physics and multi-resolution models, we establish a finely resolved window with explicitly modeled cells coupled to a coarsely resolved bulk fluid domain. In this work we present multiple validations of the APR framework by comparing against fully resolved fluid-structure interaction methods and employ techniques, such as latency hiding and maximizing memory bandwidth, to effectively utilize heterogeneous node architectures. Collectively, these computational developments and performance optimizations provide a robust and scalable framework to enable system-level simulations of cancer cell transport.

Multi-GPU Immersed Boundary Method Hemodynamics Simulations.

Large-scale simulations of blood flow that resolve the 3D deformation of each comprising cell are increasingly popular owing to algorithmic developments in conjunction with advances in compute capability. Among different approaches for modeling cell-resolved hemodynamics, fluid structure interaction (FSI) algorithms based on the immersed boundary method are frequently employed for coupling separate solvers for the background fluid and the cells within one framework. GPUs can accelerate these simulations; however, both current pre-exascale and future exascale CPU-GPU heterogeneous systems face communication challenges critical to performance and scalability. We describe, to our knowledge, the largest distributed GPU-accelerated FSI simulations of high hematocrit cell-resolved flows with over 17 million red blood cells. We compare scaling on a fat node system with six GPUs per node and on a system with a single GPU per node. Through comparison between the CPU- and GPU-based implementations, we identify the costs of data movement in multiscale multi-grid FSI simulations on heterogeneous systems and show it to be the greatest performance bottleneck on the GPU.

Accelerating massively parallel hemodynamic models of coarctation of the aorta using neural networks.

Comorbidities such as anemia or hypertension and physiological factors related to exertion can influence a patient's hemodynamics and increase the severity of many cardiovascular diseases. Observing and quantifying associations between these factors and hemodynamics can be difficult due to the multitude of co-existing conditions and blood flow parameters in real patient data. Machine learning-driven, physics-based simulations provide a means to understand how potentially correlated conditions may affect a particular patient. Here, we use a combination of machine learning and massively parallel computing to predict the effects of physiological factors on hemodynamics in patients with coarctation of the aorta. We first validated blood flow simulations against in vitro measurements in 3D-printed phantoms representing the patient's vasculature. We then investigated the effects of varying the degree of stenosis, blood flow rate, and viscosity on two diagnostic metrics - pressure gradient across the stenosis (ΔP) and wall shear stress (WSS) - by performing the largest simulation study to date of coarctation of the aorta (over 70 million compute hours). Using machine learning models trained on data from the simulations and validated on two independent datasets, we developed a framework to identify the minimal training set required to build a predictive model on a per-patient basis. We then used this model to accurately predict ΔP (mean absolute error within 1.18 mmHg) and WSS (mean absolute error within 0.99 Pa) for patients with this disease.

Computing the ankle-brachial index with parallel computational fluid dynamics.

The ankle-brachial index (ABI), a ratio of arterial blood pressure in the ankles and upper arms, is used to diagnose and monitor circulatory conditions such as coarctation of the aorta and peripheral artery disease. Computational simulations of the ABI can potentially determine the parameters that produce an ABI indicative of ischemia or other abnormalities in blood flow. However, 0- and 1-D computational methods are limited in describing a 3-D patient-derived geometry. Thus, we present a massively parallel framework for computational fluid dynamics (CFD) simulations in the full arterial system. Using the lattice Boltzmann method to solve the Navier-Stokes equations, we employ highly parallelized and scalable methods to generate the simulation domain and efficiently distribute the computational load among processors. For the first time, we compute an ABI with 3-D CFD. In this proof-of-concept study, we investigate the dependence of ABI on the presence of stenoses, or narrowed regions of the arteries, by directly modifying the arterial geometry. As a result, our framework enables the computation a hemodynamic factor characterizing flow at the scale of the full arterial system, in a manner that is extensible to patient-specific imaging data and holds potential for treatment planning.

Numerical simulation of a compound capsule in a constricted microchannel.

Simulations of the passage of eukaryotic cells through a constricted channel aid in studying the properties of cancer cells and their transport in the bloodstream. Compound capsules, which explicitly model the outer cell membrane and nuclear lamina, have the potential to improve computational model fidelity. However, general simulations of compound capsules transiting a constricted microchannel have not been conducted and the influence of the compound capsule model on computational performance is not well known. In this study, we extend a parallel hemodynamics application to simulate the fluid-structure interaction between compound capsules and fluid. With this framework, we compare the deformation of simple and compound capsules in constricted microchannels, and explore how deformation depends on the capillary number and on the volume fraction of the inner membrane. The computational framework's parallel performance in this setting is evaluated and future development lessons are discussed.

Massively parallel simulations of hemodynamics in the primary large arteries of the human vasculature.

We present a computational model of three-dimensional and unsteady hemodynamics within the primary large arteries in the human on 1,572,864 cores of the IBM Blue Gene/Q. Models of large regions of the circulatory system are needed to study the impact of local factors on global hemodynamics and to inform next generation drug delivery mechanisms. The HARVEY code successfully addresses key challenges that can hinder effective solution of image-based hemodynamics on contemporary supercomputers, such as limited memory capacity and bandwidth, flexible load balancing, and scalability. This work is the first demonstration of large fluid dynamics simulations of the aortofemoral region of the circulatory system at resolutions as small as 10 µm.

Lithium ion solvation and diffusion in bulk organic electrolytes from first-principles and classical reactive molecular dynamics.

Lithium-ion battery performance is strongly influenced by the ionic conductivity of the electrolyte, which depends on the speed at which Li ions migrate across the cell and relates to their solvation structure. The choice of solvent can greatly impact both the solvation and diffusivity of Li ions. In this work, we used first-principles molecular dynamics to examine the solvation and diffusion of Li ions in the bulk organic solvents ethylene carbonate (EC), ethyl methyl carbonate (EMC), and a mixture of EC and EMC. We found that Li ions are solvated by either carbonyl or ether oxygen atoms of the solvents and sometimes by the PF6(-) anion. Li(+) prefers a tetrahedrally coordinated first solvation shell regardless of which species are involved, with the specific preferred solvation structure dependent on the organic solvent. In addition, we calculated Li diffusion coefficients in each electrolyte, finding slightly larger diffusivities in the linear carbonate EMC compared to the cyclic carbonate EC. The magnitude of the diffusion coefficient correlates with the strength of Li(+) solvation. Corresponding analysis for the PF6(-) anion shows greater diffusivity associated with a weakly bound, poorly defined first solvation shell. These results can be used to aid in the design of new electrolytes to improve Li-ion battery performance.

Towards real-time simulation of cardiac electrophysiology in a human heart at high resolution.

We have developed the capability to rapidly simulate cardiac electrophysiological phenomena in a human heart discretised at a resolution comparable with the length of a cardiac myocyte. Previous scientific investigation has generally invoked simplified geometries or coarse-resolution hearts, with simulation duration limited to 10s of heartbeats. Using state-of-the-art high-performance computing techniques coupled with one of the most powerful computers available (the 20 PFlop/s IBM BlueGene/Q at Lawrence Livermore National Laboratory), high-resolution simulation of the human heart can now be carried out over 1200 times faster compared with published results in the field. We demonstrate the utility of this capability by simulating, for the first time, the formation of transmural re-entrant waves in a 3D human heart. Such wave patterns are thought to underlie Torsades de Pointes, an arrhythmia that indicates a high risk of sudden cardiac death. Our new simulation capability has the potential to impact a multitude of applications in medicine, pharmaceuticals and implantable devices.

Influence of synthesis conditions on the structural and optical properties of passivated silicon nanoclusters.

First-principles molecular dynamics and quantum Monte Carlo techniques are employed to gain insight into the effect of preparation conditions on the structural and optical properties of silicon nanoparticles. Our results demonstrate that (i) kinetically limited nanostructures form different core structures than bulk-derived crystalline clusters, (ii) the type of core structure that forms depends on how the cluster is passivated during synthesis, and (iii) good agreement with measured optical gaps can be obtained for nanoparticles with core structures different from those derived from the bulk.

Towards an assessment of the accuracy of density functional theory for first principles simulations of water.

A series of Car-Parrinello (CP) molecular dynamics simulations of water are presented, aimed at assessing the accuracy of density functional theory in describing the structural and dynamical properties of water at ambient conditions. We found negligible differences in structural properties obtained using the Perdew-Burke-Ernzerhof or the Becke-Lee-Yang-Parr exchange and correlation energy functionals; we also found that size effects, although not fully negligible when using 32 molecule cells, are rather small. In addition, we identified a wide range of values of the fictitious electronic mass (micro) entering the CP Lagrangian for which the electronic ground state is accurately described, yielding trajectories and average properties that are independent of the value chosen. However, care must be exercised not to carry out simulations outside this range, where structural properties may artificially depend on micro. In the case of an accurate description of the electronic ground state, and in the absence of proton quantum effects, we obtained an oxygen-oxygen correlation function that is overstructured compared to experiment, and a diffusion coefficient which is approximately ten times smaller.

Optical properties of passivated silicon nanoclusters: the role of synthesis.

The effect of preparation conditions on the structural and optical properties of silicon nanoparticles is investigated. Nanoscale reconstructions, unique to curved nanosurfaces, are presented for silicon nanocrystals and shown to have lower energy and larger optical gaps than bulk-derived structures. We find that high-temperature synthesis processes can produce metastable noncrystalline nanostructures with different core structures than bulk-derived crystalline clusters. The type of core structure that forms from a given synthesis process may depend on the passivation mechanism and time scale. The effect of oxygen on the optical of different types of silicon structures is calculated. In contrast to the behavior of bulklike nanostructures, for noncrystalline and reconstructed crystalline structures surface oxygen atoms do not decrease the gap. In some cases, the presence of oxygen atoms at the nanocluster surface can significantly increase the optical absorption gap, due to decreased angular distortion of the silicon bonds. The relationship between strain and the optical gap in silicon nanoclusters is discussed.