Explainability and human intervention in autonomous scanning probe microscopy.

The broad adoption of machine learning (ML)-based autonomous experiments (AEs) in material characterization and synthesis requires strategies development for understanding and intervention in the experimental workflow. Here, we introduce and realize a post-experimental analysis strategy for deep kernel learning-based autonomous scanning probe microscopy. This approach yields real-time and post-experimental indicators for the progression of an active learning process interacting with an experimental system. We further illustrate how this approach can be applied to human-in-the-loop AEs, where human operators make high-level decisions at high latencies setting the policies for AEs, and the ML algorithm performs low-level, fast decisions. The proposed approach is universal and can be extended to other techniques and applications such as combinatorial library analysis.

Ferroelectricity in hafnia controlled via surface electrochemical state.

Ferroelectricity in binary oxides including hafnia and zirconia has riveted the attention of the scientific community due to the highly unconventional physical mechanisms and the potential for the integration of these materials into semiconductor workflows. Over the last decade, it has been argued that behaviours such as wake-up phenomena and an extreme sensitivity to electrode and processing conditions suggest that ferroelectricity in these materials is strongly influenced by other factors, including electrochemical boundary conditions and strain. Here we argue that the properties of these materials emerge due to the interplay between the bulk competition between ferroelectric and structural instabilities, similar to that in classical antiferroelectrics, coupled with non-local screening mediated by the finite density of states at surfaces and internal interfaces. Via the decoupling of electrochemical and electrostatic controls, realized via environmental and ultra-high vacuum piezoresponse force microscopy, we show that these materials demonstrate a rich spectrum of ferroic behaviours including partial-pressure-induced and temperature-induced transitions between ferroelectric and antiferroelectric behaviours. These behaviours are consistent with an antiferroionic model and suggest strategies for hafnia-based device optimization.

Disentangling Electronic Transport and Hysteresis at Individual Grain Boundaries in Hybrid Perovskites via Automated Scanning Probe Microscopy.

Underlying the rapidly increasing photovoltaic efficiency and stability of metal halide perovskites (MHPs) is the advancement in the understanding of the microstructure of polycrystalline MHP thin film. Over the past decade, intense efforts have been aimed at understanding the effect of microstructures on MHP properties, including chemical heterogeneity, strain disorder, phase impurity, etc. It has been found that grain and grain boundary (GB) are tightly related to lots of microscale and nanoscale behavior in MHP thin films. Atomic force microscopy (AFM) is widely used to observe grain and boundary structures in topography and subsequently to study the correlative surface potential and conductivity of these structures. For now, most AFM measurements have been performed in imaging mode to study the static behavior; in contrast, AFM spectroscopy mode allows us to investigate the dynamic behavior of materials, e.g., conductivity under sweeping voltage. However, a major limitation of AFM spectroscopy measurements is that they require manual operation by human operators, and as such only limited data can be obtained, hindering systematic investigations of these microstructures. In this work, we designed a workflow combining the conductive AFM measurement with a machine learning (ML) algorithm to systematically investigate grain boundaries in MHPs. The trained ML model can extract GBs locations from the topography image, and the workflow drives the AFM probe to each GB location to perform a current-voltage (IV) curve automatically. Then, we are able to have IV curves at all GB locations, allowing us to systematically understand the property of GBs. Using this method, we discovered that the GB junction points are less conductive, potentially more photoactive, and can play critical roles in MHP stability, while most previous works only focused on the difference between GB and grains.

Autonomous scanning probe microscopy with hypothesis learning: Exploring the physics of domain switching in ferroelectric materials.

Using hypothesis-learning-driven automated scanning probe microscopy (SPM), we explore the bias-induced transformations that underpin the functionality of broad classes of devices and materials from batteries and memristors to ferroelectrics and antiferroelectrics. Optimization and design of these materials require probing the mechanisms of these transformations on the nanometer scale as a function of a broad range of control parameters, leading to experimentally intractable scenarios. Meanwhile, often these behaviors are understood within potentially competing theoretical hypotheses. Here, we develop a hypothesis list covering possible limiting scenarios for domain growth in ferroelectric materials, including thermodynamic, domain-wall pinning, and screening limited. The hypothesis-driven SPM autonomously identifies the mechanisms of bias-induced domain switching, and the results indicate that domain growth is ruled by kinetic control. We note that the hypothesis learning can be broadly used in other automated experiment settings.

Exploring the Relationship of Microstructure and Conductivity in Metal Halide Perovskites via Active Learning-Driven Automated Scanning Probe Microscopy.

Electronic transport and hysteresis in metal halide perovskites (MHPs) are key to the applications in photovoltaics, light emitting devices, and light and chemical sensors. These phenomena are strongly affected by the materials microstructure including grain boundaries, ferroic domain walls, and secondary phase inclusions. Here, we demonstrate an active machine learning framework for "driving" an automated scanning probe microscope (SPM) to discover the microstructures responsible for specific aspects of transport behavior in MHPs. In our setup, the microscope can discover the microstructural elements that maximize the onset of conduction, hysteresis, or any other characteristic that can be derived from a set of current-voltage spectra. This approach opens new opportunities for exploring the origins of materials functionality in complex materials by SPM and can be integrated with other characterization techniques either before (prior knowledge) or after (identification of locations of interest for detail studies) functional probing.

Learning and Predicting Photonic Responses of Plasmonic Nanoparticle Assemblies via Dual Variational Autoencoders.

The application of machine learning is demonstrated for rapid and accurate extraction of plasmonic particles cluster geometries from hyperspectral image data via a dual variational autoencoder (dual-VAE). In this approach, the information is shared between the latent spaces of two VAEs acting on the particle shape data and spectral data, respectively, but enforcing a common encoding on the shape-spectra pairs. It is shown that this approach can establish the relationship between the geometric characteristics of nanoparticles and their far-field photonic responses, demonstrating that hyperspectral darkfield microscopy can be used to accurately predict the geometry (number of particles, arrangement) of a multiparticle assemblies below the diffraction limit in an automated fashion with high fidelity (for monomers (0.96), dimers (0.86), and trimers (0.58). This approach of building structure-property relationships via shared encoding is universal and should have applications to a broader range of materials science and physics problems in imaging of both molecular and nanomaterial systems.

Roadmap on Deep Learning for Microscopy.

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

Physics Discovery in Nanoplasmonic Systems via Autonomous Experiments in Scanning Transmission Electron Microscopy.

Physics-driven discovery in an autonomous experiment has emerged as a dream application of machine learning in physical sciences. Here, this work develops and experimentally implements a deep kernel learning (DKL) workflow combining the correlative prediction of the target functional response and its uncertainty from the structure, and physics-based selection of acquisition function, which autonomously guides the navigation of the image space. Compared to classical Bayesian optimization (BO) methods, this approach allows to capture the complex spatial features present in the images of realistic materials, and dynamically learn structure-property relationships. In combination with the flexible scalarizer function that allows to ascribe the degree of physical interest to predicted spectra, this enables physical discovery in automated experiment. Here, this approach is illustrated for nanoplasmonic studies of nanoparticles and experimentally implemented in a truly autonomous fashion for bulk- and edge plasmon discovery in MnPS3 , a lesser-known beam-sensitive layered 2D material. This approach is universal, can be directly used as-is with any specimen, and is expected to be applicable to any probe-based microscopic techniques including other STEM modalities, scanning probe microscopies, chemical, and optical imaging.

Hierarchical Materials from High Information Content Macromolecular Building Blocks: Construction, Dynamic Interventions, and Prediction.

Hierarchical materials that exhibit order over multiple length scales are ubiquitous in nature. Because hierarchy gives rise to unique properties and functions, many have sought inspiration from nature when designing and fabricating hierarchical matter. More and more, however, nature's own high-information content building blocks, proteins, peptides, and peptidomimetics, are being coopted to build hierarchy because the information that determines structure, function, and interfacial interactions can be readily encoded in these versatile macromolecules. Here, we take stock of recent progress in the rational design and characterization of hierarchical materials produced from high-information content blocks with a focus on stimuli-responsive and "smart" architectures. We also review advances in the use of computational simulations and data-driven predictions to shed light on how the side chain chemistry and conformational flexibility of macromolecular blocks drive the emergence of order and the acquisition of hierarchy and also on how ionic, solvent, and surface effects influence the outcomes of assembly. Continued progress in the above areas will ultimately usher in an era where an understanding of designed interactions, surface effects, and solution conditions can be harnessed to achieve predictive materials synthesis across scale and drive emergent phenomena in the self-assembly and reconfiguration of high-information content building blocks.

Automated Experiments of Local Non-Linear Behavior in Ferroelectric Materials.

An automated experiment in multimodal imaging to probe structural, chemical, and functional behaviors in complex materials and elucidate the dominant physical mechanisms that control device function is developed and implemented. Here, the emergence of non-linear electromechanical responses in piezoresponse force microscopy (PFM) is explored. Non-linear responses in PFM can originate from multiple mechanisms, including intrinsic material responses often controlled by domain structure, surface topography that affects the mechanical phenomena at the tip-surface junction, and the presence of surface contaminants. Using an automated experiment to probe the origins of non-linear behavior in ferroelectric lead titanate (PTO) and ferroelectric Al0.93 B0.07 N films, it is found that PTO shows asymmetric nonlinear behavior across a/c domain walls and a broadened high nonlinear response region around c/c domain walls. In contrast, for Al0.93 B0.07 N, well-poled regions show high linear piezoelectric responses, when paired with low non-linear responses regions that are multidomain show low linear responses and high nonlinear responses. It is shown that formulating dissimilar exploration strategies in deep kernel learning as alternative hypotheses allows for establishing the preponderant physical mechanisms behind the non-linear behaviors, suggesting that automated experiments can potentially discern between competing physical mechanisms. This technique can also be extended to electron, probe, and chemical imaging.

Probing Electron Beam Induced Transformations on a Single-Defect Level via Automated Scanning Transmission Electron Microscopy.

A robust approach for real-time analysis of the scanning transmission electron microscopy (STEM) data streams, based on ensemble learning and iterative training (ELIT) of deep convolutional neural networks, is implemented on an operational microscope, enabling the exploration of the dynamics of specific atomic configurations under electron beam irradiation via an automated experiment in STEM. Combined with beam control, this approach allows studying beam effects on selected atomic groups and chemical bonds in a fully automated mode. Here, we demonstrate atomically precise engineering of single vacancy lines in transition metal dichalcogenides and the creation and identification of topological defects in graphene. The ELIT-based approach facilitates direct on-the-fly analysis of the STEM data and engenders real-time feedback schemes for probing electron beam chemistry, atomic manipulation, and atom by atom assembly.

Decoding the Mechanisms of Phase Transitions from In Situ Microscopy Observations.

Analysis of the temperature- and stimulus-dependent imaging data toward elucidation of the physical transformations is an ubiquitous problem in multiple fields. Here, temperature-induced phase transition in BaTiO3 is explored using the machine learning analysis of domain morphologies visualized via variable-temperature scanning transmission electron microscopy (STEM) imaging data. This approach is based on the multivariate statistical analysis of the time or temperature dependence of the statistical descriptors of the system, derived in turn from the categorical classification of observed domain structures or projection on the continuous parameter space of the feature extraction-dimensionality reduction transform. The proposed workflow offers a powerful tool for the exploration of the dynamic data based on the statistics of image representation as a function of the external control variable to visualize the transformation pathways during phase transitions and chemical reactions. This can include the mesoscopic STEM data as demonstrated here, but also optical, chemical imaging, etc., data. It can further be extended to the higher dimensional spaces, for example, analysis of the combinatorial libraries of materials compositions.