Generative Design of Inorganic Compounds Using Deep Diffusion Language Models.

Due to the vast chemical space, discovering materials with a specific function is challenging. Chemical formulas are obligated to conform to a set of exacting criteria, such as charge neutrality, balanced electronegativity, synthesizability, and mechanical stability. In response to this formidable task, we introduce a deep-learning-based generative model for material composition and structure design by learning and exploiting explicit and implicit chemical knowledge. Our pipeline first uses deep diffusion language models as the generator of compositions and then applies a template-based crystal structure prediction algorithm to predict their corresponding structures, which is then followed by structure relaxation using a universal graph neural network-based potential. Density functional theory (DFT) calculations of the formation energies and energy-above-the-hull analysis are used to validate new structures generated through our pipeline. Based on the DFT calculation results, six new materials, including Ti2HfO5, TaNbP, YMoN2, TaReO4, HfTiO2, and HfMnO2, with formation energy less than zero have been found. Remarkably, among these, four materials, namely, Ti2HfO5, TaNbP, YMoN2, and TaReO4, exhibit an e-above-hull energy of less than 0.3 eV. These findings have proved the effectiveness of our approach.

Global Mapping of Structures and Properties of Crystal Materials.

Understanding materials' composition-structure-function relationships is of critical importance for the design and discovery of novel functional materials. While most such studies focus on individual materials, we conducted a global mapping study of all known materials deposited in the Materials Project database to investigate their distributions in the space of a set of seven compositional, structural, physical, and neural latent descriptors. These two-dimensional materials maps along with their density maps allow us to illustrate the distribution of the patterns and clusters of different shapes, which indicates the propensity of these materials and the tinkering history of existing materials. We then overlap the material properties such as composition prototypes and piezoelectric properties over the background material maps to study the relationships of how material compositions and structures affect their physical properties. We also use these maps to study the spatial distributions of properties of known inorganic materials, in particular those of local vicinities in structural space such as structural density and functional diversity. These maps provide a uniquely comprehensive overview of materials and space and thus reveal previously undescribed fundamental properties. Our methodology can be easily extended by other researchers to generate their own global material maps with different background maps and overlap properties for both distribution understanding and cluster-based new material discovery. The source code for feature generation and generated maps is available at https://github.com/usccolumbia/matglobalmapping.

TCSP: a Template-Based Crystal Structure Prediction Algorithm for Materials Discovery.

Fast and accurate crystal structure prediction (CSP) algorithms and web servers are highly desirable for the exploration and discovery of new materials out of the infinite chemical design space. However, currently, the computationally expensive first-principles calculation-based CSP algorithms are applicable to relatively small systems and are out of reach of most materials researchers. Several teams have used an element substitution approach for generating or predicting new structures, but usually in an ad hoc way. Here we develop a template-based crystal structure prediction (TCSP) algorithm and its companion web server, which makes this tool accessible to all materials researchers. Our algorithm uses elemental/chemical similarity and oxidation states to guide the selection of template structures and then rank them based on the substitution compatibility and can return multiple predictions with ranking scores in a few minutes. A benchmark study on the 98290 formulas of the Materials Project database using leave-one-out evaluation shows that our algorithm can achieve high accuracy (for 13145 target structures, TCSP predicted their structures with root-mean-square deviation < 0.1) for a large portion of the formulas. We have also used TCSP to discover new materials of the Ga-B-N system, showing its potential for high-throughput materials discovery. Our user-friendly web app TCSP can be accessed freely at www.materialsatlas.org/crystalstructure on our MaterialsAtlas.org web app platform.

Physics-Guided Dual Self-Supervised Learning for Structure-Based Material Property Prediction.

Deep learning models have been widely used for high-performance material property prediction. However, training such models usually requires a large amount of labeled data, which are usually unavailable. Self-supervised learning (SSL) methods have been proposed to address this data scarcity issue. Herein, we present DSSL, a physics-guided dual SSL framework, for graph neural network-based material property prediction, which combines node masking-based generative SSL with atomic coordinate perturbation-based contrastive SSL strategies to capture local and global information about input crystals. Moreover, we achieve physics-guided pretraining by using the macroproperty (e.g., elasticity)-related microproperty prediction of atomic stiffness as an additional pretext task. We pretrain our DSSL model on the Materials Project database and fine-tune it with 10 material property data sets. The experimental results demonstrate that teaching neural networks some physics using the SSL strategy can afford ≤26.89% performance improvement compared to that of the baseline models.

Crystal Composition Transformer: Self-Learning Neural Language Model for Generative and Tinkering Design of Materials.

Self-supervised neural language models have recently achieved unprecedented success from natural language processing to learning the languages of biological sequences and organic molecules. These models have demonstrated superior performance in the generation, structure classification, and functional predictions for proteins and molecules with learned representations. However, most of the masking-based pre-trained language models are not designed for generative design, and their black-box nature makes it difficult to interpret their design logic. Here a Blank-filling Language Model for Materials (BLMM) Crystal Transformer is proposed, a neural network-based probabilistic generative model for generative and tinkering design of inorganic materials. The model is built on the blank-filling language model for text generation and has demonstrated unique advantages in learning the "materials grammars" together with high-quality generation, interpretability, and data efficiency. It can generate chemically valid materials compositions with as high as 89.7% charge neutrality and 84.8% balanced electronegativity, which are more than four and eight times higher compared to a pseudo-random sampling baseline. The probabilistic generation process of BLMM allows it to recommend materials tinkering operations based on learned materials chemistry, which makes it useful for materials doping. The model is applied to discover a set of new materials as validated using the Density Functional Theory (DFT) calculations. This work thus brings the unsupervised transformer language models based generative artificial intelligence to inorganic materials. A user-friendly web app for tinkering materials design has been developed and can be accessed freely at www.materialsatlas.org/blmtinker.

Simultaneously-Collected Multimodal Lying Pose Dataset: Enabling In-Bed Human Pose Monitoring.

Computer vision field has achieved great success in interpreting semantic meanings from images, yet its algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. Among these tasks is in-bed human pose monitoring with significant value in many healthcare applications. In-bed pose monitoring in natural settings involves pose estimation in complete darkness or full occlusion. The lack of publicly available in-bed pose datasets hinders the applicability of many successful human pose estimation algorithms for this task. In this paper, we introduce our Simultaneously-collected multimodal Lying Pose (SLP) dataset, which includes in-bed pose images from 109 participants captured using multiple imaging modalities including RGB, long wave infrared (LWIR), depth, and pressure map. We also present a physical hyper parameter tuning strategy for ground truth pose label generation under adverse vision conditions. The SLP design is compatible with the mainstream human pose datasets; therefore, the state-of-the-art 2D pose estimation models can be trained effectively with the SLP data with promising performance as high as 95% at PCKh@0.5 on a single modality. The pose estimation performance of these models can be further improved by including additional modalities through the proposed collaborative scheme.

Probabilistic generative transformer language models for generative design of molecules.

Self-supervised neural language models have recently found wide applications in the generative design of organic molecules and protein sequences as well as representation learning for downstream structure classification and functional prediction. However, most of the existing deep learning models for molecule design usually require a big dataset and have a black-box architecture, which makes it difficult to interpret their design logic. Here we propose the Generative Molecular Transformer (GMTransformer), a probabilistic neural network model for generative design of molecules. Our model is built on the blank filling language model originally developed for text processing, which has demonstrated unique advantages in learning the "molecules grammars" with high-quality generation, interpretability, and data efficiency. Benchmarked on the MOSES datasets, our models achieve high novelty and Scaf compared to other baselines. The probabilistic generation steps have the potential in tinkering with molecule design due to their capability of recommending how to modify existing molecules with explanation, guided by the learned implicit molecule chemistry. The source code and datasets can be accessed freely at https://github.com/usccolumbia/GMTransformer.

Composition Based Oxidation State Prediction of Materials Using Deep Learning Language Models.

Oxidation states (OS) are the charges on atoms due to electrons gained or lost upon applying an ionic approximation to their bonds. As a fundamental property, OS has been widely used in charge-neutrality verification, crystal structure determination, and reaction estimation. Currently, only heuristic rules exist for guessing the oxidation states of a given compound with many exceptions. Recent work has developed machine learning models based on heuristic structural features for predicting the oxidation states of metal ions. However, composition-based oxidation state prediction still remains elusive so far, which has significant implications for the discovery of new materials for which the structures have not been determined. This work proposes a novel deep learning-based BERT transformer language model BERTOS for predicting the oxidation states for all elements of inorganic compounds given only their chemical composition. This model achieves 96.82% accuracy for all-element oxidation states prediction benchmarked on the cleaned ICSD dataset and achieves 97.61% accuracy for oxide materials. It is also demonstrated how it can be used to conduct large-scale screening of hypothetical material compositions for materials discovery.

DeepXRD, a Deep Learning Model for Predicting XRD spectrum from Material Composition.

One of the long-standing problems in materials science is how to predict a material's structure and then its properties given only its composition. Experimental characterization of crystal structures has been widely used for structure determination, which is, however, too expensive for high-throughput screening. At the same time, directly predicting crystal structures from compositions remains a challenging unsolved problem. Herein we propose a deep learning algorithm for predicting the XRD spectrum given only the composition of a material, which can then be used to infer key structural features for downstream structural analysis such as crystal system or space group classification or crystal lattice parameter determination or materials property prediction. Benchmark studies on two data sets show that our DeepXRD algorithm can achieve good performance for XRD prediction as evaluated over our test sets. It can thus be used in high-throughput screening in the huge materials composition space for materials discovery.

Scalable deeper graph neural networks for high-performance materials property prediction.

Machine-learning-based materials property prediction models have emerged as a promising approach for new materials discovery, among which the graph neural networks (GNNs) have shown the best performance due to their capability to learn high-level features from crystal structures. However, existing GNN models suffer from their lack of scalability, high hyperparameter tuning complexity, and constrained performance due to over-smoothing. We propose a scalable global graph attention neural network model DeeperGATGNN with differentiable group normalization (DGN) and skip connections for high-performance materials property prediction. Our systematic benchmark studies show that our model achieves the state-of-the-art prediction results on five out of six datasets, outperforming five existing GNN models by up to 10%. Our model is also the most scalable one in terms of graph convolution layers, which allows us to train very deep networks (e.g., >30 layers) without significant performance degradation. Our implementation is available at https://github.com/usccolumbia/deeperGATGNN.