Learning and Predicting Photonic Responses of Plasmonic Nanoparticle Assemblies via Dual Variational Autoencoders.

The application of machine learning is demonstrated for rapid and accurate extraction of plasmonic particles cluster geometries from hyperspectral image data via a dual variational autoencoder (dual-VAE). In this approach, the information is shared between the latent spaces of two VAEs acting on the particle shape data and spectral data, respectively, but enforcing a common encoding on the shape-spectra pairs. It is shown that this approach can establish the relationship between the geometric characteristics of nanoparticles and their far-field photonic responses, demonstrating that hyperspectral darkfield microscopy can be used to accurately predict the geometry (number of particles, arrangement) of a multiparticle assemblies below the diffraction limit in an automated fashion with high fidelity (for monomers (0.96), dimers (0.86), and trimers (0.58). This approach of building structure-property relationships via shared encoding is universal and should have applications to a broader range of materials science and physics problems in imaging of both molecular and nanomaterial systems.

Automated Experiments of Local Non-Linear Behavior in Ferroelectric Materials.

An automated experiment in multimodal imaging to probe structural, chemical, and functional behaviors in complex materials and elucidate the dominant physical mechanisms that control device function is developed and implemented. Here, the emergence of non-linear electromechanical responses in piezoresponse force microscopy (PFM) is explored. Non-linear responses in PFM can originate from multiple mechanisms, including intrinsic material responses often controlled by domain structure, surface topography that affects the mechanical phenomena at the tip-surface junction, and the presence of surface contaminants. Using an automated experiment to probe the origins of non-linear behavior in ferroelectric lead titanate (PTO) and ferroelectric Al0.93 B0.07 N films, it is found that PTO shows asymmetric nonlinear behavior across a/c domain walls and a broadened high nonlinear response region around c/c domain walls. In contrast, for Al0.93 B0.07 N, well-poled regions show high linear piezoelectric responses, when paired with low non-linear responses regions that are multidomain show low linear responses and high nonlinear responses. It is shown that formulating dissimilar exploration strategies in deep kernel learning as alternative hypotheses allows for establishing the preponderant physical mechanisms behind the non-linear behaviors, suggesting that automated experiments can potentially discern between competing physical mechanisms. This technique can also be extended to electron, probe, and chemical imaging.

Investigating Carboxysome Morphology Dynamics with a Rotationally Invariant Variational Autoencoder.

Carboxysomes are a class of bacterial microcompartments that form proteinaceous organelles within the cytoplasm of cyanobacteria and play a central role in photosynthetic metabolism by defining a cellular microenvironment permissive to CO2 fixation. Critical aspects of the assembly of the carboxysomes remain relatively unknown, especially with regard to the dynamics of this microcompartment. Progress in understanding carboxysome dynamics is impeded in part because analysis of the subtle changes in carboxysome morphology with microscopy remains a low-throughput and subjective process. Here we use deep learning techniques, specifically a Rotationally Invariant Variational Autoencoder (rVAE), to analyze fluorescence microscopy images of cyanobacteria bearing a carboxysome reporter and quantitatively evaluate how carboxysome shell remodelling impacts subtle trends in the morphology of the microcompartment over time. Toward this goal, we use a recently developed tool to control endogenous protein levels, including carboxysomal components, in the model cyanobacterium Synechococcous elongatus PCC 7942. By utilization of this system, proteins that compose the carboxysome can be tuned in real time as a method to examine carboxysome dynamics. We find that rVAEs are able to assist in the quantitative evaluation of changes in carboxysome numbers, shape, and size over time. We propose that rVAEs may be a useful tool to accelerate the analysis of carboxysome assembly and dynamics in response to genetic or environmental perturbation and may be more generally useful to probe regulatory processes involving a broader array of bacterial microcompartments.

Quantifying the Dynamics of Protein Self-Organization Using Deep Learning Analysis of Atomic Force Microscopy Data.

The dynamics of protein self-assembly on the inorganic surface and the resultant geometric patterns are visualized using high-speed atomic force microscopy. The time dynamics of the classical macroscopic descriptors such as 2D fast Fourier transforms, correlation, and pair distribution functions are explored using the unsupervised linear unmixing, demonstrating the presence of static ordered and dynamic disordered phases and establishing their time dynamics. The deep learning (DL)-based workflow is developed to analyze detailed particle dynamics and explore the evolution of local geometries. Finally, we use a combination of DL feature extraction and mixture modeling to define particle neighborhoods free of physics constraints, allowing for a separation of possible classes of particle behavior and identification of the associated transitions. Overall, this work establishes the workflow for the analysis of the self-organization processes in complex systems from observational data and provides insight into the fundamental mechanisms.

High-Throughput Study of Antisolvents on the Stability of Multicomponent Metal Halide Perovskites through Robotics-Based Synthesis and Machine Learning Approaches.

Antisolvent crystallization methods are frequently used to fabricate high-quality metal halide perovskite (MHP) thin films, to produce sizable single crystals, and to synthesize nanoparticles at room temperature. However, a systematic exploration of the effect of specific antisolvents on the intrinsic stability of multicomponent MHPs has yet to be demonstrated. Here, we develop a high-throughput experimental workflow that incorporates chemical robotic synthesis, automated characterization, and machine learning techniques to explore how the choice of antisolvent affects the intrinsic stability of binary MHP systems in ambient conditions over time. Different combinations of the end-members, MAPbI3, MAPbBr3, FAPbI3, FAPbBr3, CsPbI3, and CsPbBr3 (MA, methylammonium; FA+, formamidinium), are used to synthesize 15 combinatorial libraries, each with 96 unique combinations. In total, roughly 1100 different compositions are synthesized. Each library is fabricated twice by using two different antisolvents: toluene and chloroform. Once synthesized, photoluminescence spectroscopy is automatically performed every 5 min for approximately 6 h. Nonnegative matrix factorization (NMF) is then utilized to map the time- and compositional-dependent optoelectronic properties. Through the utilization of this workflow for each library, we demonstrate that the selection of antisolvent is critical to the intrinsic stability of MHPs in ambient conditions. We explore possible dynamical processes, such as halide segregation, responsible for either the stability or eventual degradation as caused by the choice of antisolvent. Overall, this high-throughput study demonstrates the vital role that antisolvents play in the synthesis of high-quality multicomponent MHP systems.

Predictability of Localized Plasmonic Responses in Nanoparticle Assemblies.

Design of nanoscale structures with desired optical properties is a key task for nanophotonics. Here, the correlative relationship between local nanoparticle geometries and their plasmonic responses is established using encoder-decoder neural networks. In the im2spec network, the relationship between local particle geometries and local spectra is established via encoding the observed geometries to a small number of latent variables and subsequently decoding into plasmonic spectra; in the spec2im network, the relationship is reversed. Surprisingly, these reduced descriptions allow high-veracity predictions of local responses based on geometries for fixed compositions and surface chemical states. Analysis of the latent space distributions and the corresponding decoded and closest (in latent space) encoded images yields insight into the generative mechanisms of plasmonic interactions in the nanoparticle arrays. Ultimately, this approach creates a path toward determining configurations that yield the spectrum closest to the desired one, paving the way for stochastic design of nanoplasmonic structures.

Alignment of Au nanorods along de novo designed protein nanofibers studied with automated image analysis.

In this study, we focus on exploring the directional assembly of anisotropic Au nanorods along de novo designed 1D protein nanofiber templates. Using machine learning and automated image processing, we analyze scanning electron microscopy (SEM) images to study how the attachment density and alignment fidelity are influenced by variables such as the aspect ratio of the Au nanorods, and the salt concentration of the solution. We find that the Au nanorods prefer to align parallel to the protein nanofibers. This preference decreases with increasing salt concentration, but is only weakly sensitive to the nanorod aspect ratio. While the overall specific Au nanorod attachment density to the protein fibers increases with increasing solution ionic strength, this increase is dominated primarily by non-specific binding to the substrate background, and we find that greater specific attachment (nanorods attached to the nanofiber template as compared to the substrates) occurs at the lower studied salt concentrations, with the maximum ratio of specific to non-specific binding occurring when the protein fiber solutions are prepared in 75 mM NaCl concentration.

Exploring electron beam induced atomic assembly via reinforcement learning in a molecular dynamics environment.

Atom-by-atom assembly of functional materials and devices is perceived as one of the ultimate targets of nanotechnology. Recently it has been shown that the beam of a scanning transmission electron microscope can be used for targeted manipulation of individual atoms. However, the process is highly dynamic in nature rendering control difficult. One possible solution is to instead train artificial agents to perform the atomic manipulation in an automated manner without need for human intervention. As a first step to realizing this goal, we explore how artificial agents can be trained for atomic manipulation in a simplified molecular dynamics environment of graphene with Si dopants, using reinforcement learning. We find that it is possible to engineer the reward function of the agent in such a way as to encourage formation of local clusters of dopants under different constraints. This study shows the potential for reinforcement learning in nanoscale fabrication, and crucially, that the dynamics learned by agents encode specific elements of important physics that can be learned.

Disentangling ferroelectric domain wall geometries and pathways in dynamic piezoresponse force microscopy via unsupervised machine learning.

Domain switching pathways in ferroelectric materials visualized by dynamic piezoresponse force microscopy (PFM) are explored via variational autoencoder, which simplifies the elements of the observed domain structure, crucially allowing for rotational invariance, thereby reducing the variability of local polarization distributions to a small number of latent variables. For small sampling window sizes the latent space is degenerate, and variability is observed only in the direction of a single latent variable that can be identified with the presence of domain wall. For larger window sizes, the latent space is 2D, and the disentangled latent variables can be generally interpreted as the degree of switching and complexity of domain structure. Applied to multiple consecutive PFM images acquired while monitoring domain switching, the polarization switching mechanism can thus be visualized in the latent space, providing insight into domain evolution mechanisms and their correlation with the microstructure.

Fast Scanning Probe Microscopy via Machine Learning: Non-Rectangular Scans with Compressed Sensing and Gaussian Process Optimization.

Fast scanning probe microscopy enabled via machine learning allows for a broad range of nanoscale, temporally resolved physics to be uncovered. However, such examples for functional imaging are few in number. Here, using piezoresponse force microscopy (PFM) as a model application, a factor of 5.8 reduction in data collection using a combination of sparse spiral scanning with compressive sensing and Gaussian process regression reconstruction is demonstrated. It is found that even extremely sparse spiral scans offer strong reconstructions with less than 6% error for Gaussian process regression reconstructions. Further, the error associated with each reconstructive technique per reconstruction iteration is analyzed, finding the error is similar past ≈15 iterations, while at initial iterations Gaussian process regression outperforms compressive sensing. This study highlights the capabilities of reconstruction techniques when applied to sparse data, particularly sparse spiral PFM scans, with broad applications in scanning probe and electron microscopies.

Phases and Interfaces from Real Space Atomically Resolved Data: Physics-Based Deep Data Image Analysis.

Advances in electron and scanning probe microscopies have led to a wealth of atomically resolved structural and electronic data, often with ∼1-10 pm precision. However, knowledge generation from such data requires the development of a physics-based robust framework to link the observed structures to macroscopic chemical and physical descriptors, including single phase regions, order parameter fields, interfaces, and structural and topological defects. Here, we develop an approach based on a synergy of sliding window Fourier transform to capture the local analog of traditional structure factors combined with blind linear unmixing of the resultant 4D data set. This deep data analysis is ideally matched to the underlying physics of the problem and allows reconstruction of the a priori unknown structure factors of individual components and their spatial localization. We demonstrate the principles of this approach using a synthetic data set and further apply it for extracting chemical and physically relevant information from electron and scanning tunneling microscopy data. This method promises to dramatically speed up crystallographic analysis in atomically resolved data, paving the road toward automatic local structure-property determinations in crystalline and quasi-ordered systems, as well as systems with competing structural and electronic order parameters.