Machine learning in microscopy - insights, opportunities and challenges.

Machine learning (ML) is transforming the field of image processing and analysis, from automation of laborious tasks to open-ended exploration of visual patterns. This has striking implications for image-driven life science research, particularly microscopy. In this Review, we focus on the opportunities and challenges associated with applying ML-based pipelines for microscopy datasets from a user point of view. We investigate the significance of different data characteristics - quantity, transferability and content - and how this determines which ML model(s) to use, as well as their output(s). Within the context of cell biological questions and applications, we further discuss ML utility range, namely data curation, exploration, prediction and explanation, and what they entail and translate to in the context of microscopy. Finally, we explore the challenges, common artefacts and risks associated with ML in microscopy. Building on insights from other fields, we propose how these pitfalls might be mitigated for in microscopy.

A framework for evaluating the performance of SMLM cluster analysis algorithms.

Single-molecule localization microscopy (SMLM) generates data in the form of coordinates of localized fluorophores. Cluster analysis is an attractive route for extracting biologically meaningful information from such data and has been widely applied. Despite a range of cluster analysis algorithms, there exists no consensus framework for the evaluation of their performance. Here, we use a systematic approach based on two metrics to score the success of clustering algorithms in simulated conditions mimicking experimental data. We demonstrate the framework using seven diverse analysis algorithms: DBSCAN, ToMATo, KDE, FOCAL, CAML, ClusterViSu and SR-Tesseler. Given that the best performer depended on the underlying distribution of localizations, we demonstrate an analysis pipeline based on statistical similarity measures that enables the selection of the most appropriate algorithm, and the optimized analysis parameters for real SMLM data. We propose that these standard simulated conditions, metrics and analysis pipeline become the basis for future analysis algorithm development and evaluation.

Event-driven acquisition for content-enriched microscopy.

A common goal of fluorescence microscopy is to collect data on specific biological events. Yet, the event-specific content that can be collected from a sample is limited, especially for rare or stochastic processes. This is due in part to photobleaching and phototoxicity, which constrain imaging speed and duration. We developed an event-driven acquisition framework, in which neural-network-based recognition of specific biological events triggers real-time control in an instant structured illumination microscope. Our setup adapts acquisitions on-the-fly by switching between a slow imaging rate while detecting the onset of events, and a fast imaging rate during their progression. Thus, we capture mitochondrial and bacterial divisions at imaging rates that match their dynamic timescales, while extending overall imaging durations. Because event-driven acquisition allows the microscope to respond specifically to complex biological events, it acquires data enriched in relevant content.

Correction of multiple-blinking artifacts in photoactivated localization microscopy.

Photoactivated localization microscopy (PALM) produces an array of localization coordinates by means of photoactivatable fluorescent proteins. However, observations are subject to fluorophore multiple blinking and each protein is included in the dataset an unknown number of times at different positions, due to localization error. This causes artificial clustering to be observed in the data. We present a 'model-based correction' (MBC) workflow using calibration-free estimation of blinking dynamics and model-based clustering to produce a corrected set of localization coordinates representing the true underlying fluorophore locations with enhanced localization precision, outperforming the state of the art. The corrected data can be reliably tested for spatial randomness or analyzed by other clustering approaches, and descriptors such as the absolute number of fluorophores per cluster are now quantifiable, which we validate with simulated data and experimental data with known ground truth. Using MBC, we confirm that the adapter protein, the linker for activation of T cells, is clustered at the T cell immunological synapse.

Differential nanoscale organisation of LFA-1 modulates T-cell migration.

Effector T-cells rely on integrins to drive adhesion and migration to facilitate their immune function. The heterodimeric transmembrane integrin LFA-1 (αLß2 integrin) regulates adhesion and migration of effector T-cells through linkage of the extracellular matrix with the intracellular actin treadmill machinery. Here, we quantified the velocity and direction of F-actin flow in migrating T-cells alongside single-molecule localisation of transmembrane and intracellular LFA-1. Results showed that actin retrograde flow positively correlated and immobile actin negatively correlated with T-cell velocity. Plasma membrane-localised LFA-1 forms unique nano-clustering patterns in the leading edge, compared to the mid-focal zone, of migrating T-cells. Deleting the cytosolic phosphatase PTPN22, loss-of-function mutations of which have been linked to autoimmune disease, increased T-cell velocity, and leading-edge co-clustering of pY397 FAK, pY416 Src family kinases and LFA-1. These data suggest that differential nanoclustering patterns of LFA-1 in migrating T-cells may instruct intracellular signalling. Our data presents a paradigm where T-cells modulate the nanoscale organisation of adhesion and signalling molecules to fine tune their migration speed, with implications for the regulation of immune and inflammatory responses.This article has an associated First Person interview with the first author of the paper.

Bayesian cluster identification in single-molecule localization microscopy data.

Single-molecule localization-based super-resolution microscopy techniques such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM) produce pointillist data sets of molecular coordinates. Although many algorithms exist for the identification and localization of molecules from raw image data, methods for analyzing the resulting point patterns for properties such as clustering have remained relatively under-studied. Here we present a model-based Bayesian approach to evaluate molecular cluster assignment proposals, generated in this study by analysis based on Ripley's K function. The method takes full account of the individual localization precisions calculated for each emitter. We validate the approach using simulated data, as well as experimental data on the clustering behavior of CD3ζ, a subunit of the CD3 T cell receptor complex, in resting and activated primary human T cells.

Quantification of fibrous spatial point patterns from single-molecule localization microscopy (SMLM) data.

MOTIVATION: Unlike conventional microscopy which produces pixelated images, SMLM produces data in the form of a list of localization coordinates-a spatial point pattern (SPP). Often, such SPPs are analyzed using cluster analysis algorithms to quantify molecular clustering within, for example, the plasma membrane. While SMLM cluster analysis is now well developed, techniques for analyzing fibrous structures remain poorly explored. RESULTS: Here, we demonstrate a statistical methodology, based on Ripley's K-function to quantitatively assess fibrous structures in 2D SMLM datasets. Using simulated data, we present the underlying theory to describe fiber spatial arrangements and show how these descriptions can be quantitatively derived from pointillist datasets. We also demonstrate the techniques on experimental data acquired using the image reconstruction by integrating exchangeable single-molecule localization (IRIS) approach to SMLM, in the context of the fibrous actin meshwork at the T cell immunological synapse, whose structure has been shown to be important for T cell activation. AVAILABILITY AND IMPLEMENTATION: Freely available on the web at https://github.com/RubyPeters/Angular-Ripleys-K . Implemented in MatLab. CONTACT: dylan.owen@kcl.ac.uk. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

The nanoscale organization of signaling domains at the plasma membrane.

In this chapter, we present an overview of the role of the nanoscale organization of signaling domains in regulating key cellular processes. In particular, we illustrate the importance of protein and lipid nanodomains as triggers and mediators of cell signaling. As particular examples, we summarize the state of the art of understanding the role of nanodomains in the mounting of an immune response, cellular adhesion, intercellular communication, and cell proliferation. Thus, this chapter underlines the essential role the nanoscale organization of key signaling proteins and lipid domains. We will also see how nanodomains play an important role in the lifecycle of many pathogens relevant to human disease and therefore illustrate how these structures may become future therapeutic targets.

Cluster analysis for localisation-based data sets: dos and don'ts when quantifying protein aggregates.

Many proteins display a non-random distribution on the cell surface. From dimers to nanoscale clusters to large, micron-scale aggregations, these distributions regulate protein-protein interactions and signalling. Although these distributions show organisation on length-scales below the resolution limit of conventional optical microscopy, single molecule localisation microscopy (SMLM) can map molecule locations with nanometre precision. The data from SMLM is not a conventional pixelated image and instead takes the form of a point-pattern-a list of the x, y coordinates of the localised molecules. To extract the biological insights that researchers require cluster analysis is often performed on these data sets, quantifying such parameters as the size of clusters, the percentage of monomers and so on. Here, we provide some guidance on how SMLM clustering should best be performed.

Fluorescent d-Amino Acids for Super-resolution Microscopy of the Bacterial Cell Wall.

Fluorescent d-amino acids (FDAAs) have previously been developed to enable in situ highlighting of locations of bacterial cell wall growth. Most bacterial cells lie at the edge of the diffraction limit of visible light; thus, resolving the precise details of peptidoglycan (PG) biosynthesis requires super-resolution microscopy after probe incorporation. Single molecule localization microscopy (SMLM) has stringent requirements on the fluorophore photophysical properties and therefore has remained challenging in this context. Here, we report the synthesis and characterization of new FDAAs compatible with one-step labeling and SMLM imaging. We demonstrate the incorporation of our probes and their utility for visualizing PG at the nanoscale in Gram-negative, Gram-positive, and mycobacteria species. This improved FDAA toolkit will endow researchers with a nanoscale perspective on the spatial distribution of PG biosynthesis for a broad range of bacterial species.

Single-molecule localization microscopy.

Single-molecule localization microscopy (SMLM) describes a family of powerful imaging techniques that dramatically improve spatial resolution over standard, diffraction-limited microscopy techniques and can image biological structures at the molecular scale. In SMLM, individual fluorescent molecules are computationally localized from diffraction-limited image sequences and the localizations are used to generate a super-resolution image or a time course of super-resolution images, or to define molecular trajectories. In this Primer, we introduce the basic principles of SMLM techniques before describing the main experimental considerations when performing SMLM, including fluorescent labelling, sample preparation, hardware requirements and image acquisition in fixed and live cells. We then explain how low-resolution image sequences are computationally processed to reconstruct super-resolution images and/or extract quantitative information, and highlight a selection of biological discoveries enabled by SMLM and closely related methods. We discuss some of the main limitations and potential artefacts of SMLM, as well as ways to alleviate them. Finally, we present an outlook on advanced techniques and promising new developments in the fast-evolving field of SMLM. We hope that this Primer will be a useful reference for both newcomers and practitioners of SMLM.

An agent-based model of molecular aggregation at the cell membrane.

Molecular clustering at the plasma membrane has long been identified as a key process and is associated with regulating signalling pathways across cell types. Recent advances in microscopy, in particular the rise of super-resolution, have allowed the experimental observation of nanoscale molecular clusters in the plasma membrane. However, modelling approaches capable of recapitulating these observations are in their infancy, partly because of the extremely complex array of biophysical factors which influence molecular distributions and dynamics in the plasma membrane. We propose here a highly abstracted approach: an agent-based model dedicated to the study of molecular aggregation at the plasma membrane. We show that when molecules are modelled as though they can act (diffuse) in a manner which is influenced by their molecular neighbourhood, many of the distributions observed in cells can be recapitulated, even though such sensing and response is not possible for real membrane molecules. As such, agent-based offers a unique platform which may lead to a new understanding of how molecular clustering in extremely complex molecular environments can be abstracted, simulated and interpreted using simple rules.

Multi-color Molecular Visualization of Signaling Proteins Reveals How C-Terminal Src Kinase Nanoclusters Regulate T Cell Receptor Activation.

Elucidating the mechanisms that controlled T cell activation requires visualization of the spatial organization of multiple proteins on the submicron scale. Here, we use stoichiometrically accurate, multiplexed, single-molecule super-resolution microscopy (DNA-PAINT) to image the nanoscale spatial architecture of the primary inhibitor of the T cell signaling pathway, Csk, and two binding partners implicated in its membrane association, PAG and TRAF3. Combined with a newly developed co-clustering analysis framework, we find that Csk forms nanoscale clusters proximal to the plasma membrane that are lost post-stimulation and are re-recruited at later time points. Unexpectedly, these clusters do not co-localize with PAG at the membrane but instead provide a ready pool of monomers to downregulate signaling. By generating CRISPR-Cas9 knockout T cells, our data also identify that a major risk factor for autoimmune diseases, the protein tyrosine phosphatase non-receptor type 22 (PTPN22) locus, is essential for Csk nanocluster re-recruitment and for maintenance of the synaptic PAG population.

Machine learning for cluster analysis of localization microscopy data.

Quantifying the extent to which points are clustered in single-molecule localization microscopy data is vital to understanding the spatial relationships between molecules in the underlying sample. Many existing computational approaches are limited in their ability to process large-scale data sets, to deal effectively with sample heterogeneity, or require subjective user-defined analysis parameters. Here, we develop a supervised machine-learning approach to cluster analysis which is fast and accurate. Trained on a variety of simulated clustered data, the neural network can classify millions of points from a typical single-molecule localization microscopy data set, with the potential to include additional classifiers to describe different subtypes of clusters. The output can be further refined for the measurement of cluster area, shape, and point-density. We demonstrate this approach on simulated data and experimental data of the kinase Csk and the adaptor PAG in primary human T cell immunological synapses.

Strategies for increasing the throughput of super-resolution microscopies.

Fluorescence microscopy methods have been developed to circumvent the diffraction limit by exploiting nonlinearities in the interactions between light and fluorophores. Initially, these methods were up to orders of magnitude slower than standard microscopies. In recent years, a wide array of technological advances have increased the throughput of super-resolution microscopies, through parallelization, smart scanning or data processing, and sample expansion. Here, we review recent innovations for increased throughput, some applications that have benefitted from them, and how they could be improved in the future.

Quantitative fibre analysis of single-molecule localization microscopy data.

Single molecule localization microscopy (SMLM) methods produce data in the form of a spatial point pattern (SPP) of all localized emitters. Whilst numerous tools exist to quantify molecular clustering in SPP data, the analysis of fibrous structures has remained understudied. Taking the SMLM localization coordinates as input, we present an algorithm capable of tracing fibrous structures in data generated by SMLM. Based upon a density parameter tracing routine, the algorithm outputs several fibre descriptors, such as number of fibres, length of fibres, area of enclosed regions and locations and angles of fibre branch points. The method is validated in a variety of simulated conditions and experimental data acquired using the image reconstruction by integrating exchangeable single-molecule localization (IRIS) technique. For this, the nanoscale architecture of F-actin at the T cell immunological synapse in both untreated and pharmacologically treated cells, designed to perturb actin structure, was analysed.

3D Bayesian cluster analysis of super-resolution data reveals LAT recruitment to the T cell synapse.

Single-molecule localisation microscopy (SMLM) allows the localisation of fluorophores with a precision of 10-30 nm, revealing the cell's nanoscale architecture at the molecular level. Recently, SMLM has been extended to 3D, providing a unique insight into cellular machinery. Although cluster analysis techniques have been developed for 2D SMLM data sets, few have been applied to 3D. This lack of quantification tools can be explained by the relative novelty of imaging techniques such as interferometric photo-activated localisation microscopy (iPALM). Also, existing methods that could be extended to 3D SMLM are usually subject to user defined analysis parameters, which remains a major drawback. Here, we present a new open source cluster analysis method for 3D SMLM data, free of user definable parameters, relying on a model-based Bayesian approach which takes full account of the individual localisation precisions in all three dimensions. The accuracy and reliability of the method is validated using simulated data sets. This tool is then deployed on novel experimental data as a proof of concept, illustrating the recruitment of LAT to the T-cell immunological synapse in data acquired by iPALM providing ~10 nm isotropic resolution.

A Bayesian cluster analysis method for single-molecule localization microscopy data.

Cell function is regulated by the spatiotemporal organization of the signaling machinery, and a key facet of this is molecular clustering. Here, we present a protocol for the analysis of clustering in data generated by 2D single-molecule localization microscopy (SMLM)-for example, photoactivated localization microscopy (PALM) or stochastic optical reconstruction microscopy (STORM). Three features of such data can cause standard cluster analysis approaches to be ineffective: (i) the data take the form of a list of points rather than a pixel array; (ii) there is a non-negligible unclustered background density of points that must be accounted for; and (iii) each localization has an associated uncertainty in regard to its position. These issues are overcome using a Bayesian, model-based approach. Many possible cluster configurations are proposed and scored against a generative model, which assumes Gaussian clusters overlaid on a completely spatially random (CSR) background, before every point is scrambled by its localization precision. We present the process of generating simulated and experimental data that are suitable to our algorithm, the analysis itself, and the extraction and interpretation of key cluster descriptors such as the number of clusters, cluster radii and the number of localizations per cluster. Variations in these descriptors can be interpreted as arising from changes in the organization of the cellular nanoarchitecture. The protocol requires no specific programming ability, and the processing time for one data set, typically containing 30 regions of interest, is ∼18 h; user input takes ∼1 h.

Superresolution imaging of the cytoplasmic phosphatase PTPN22 links integrin-mediated T cell adhesion with autoimmunity.

Integrins are heterodimeric transmembrane proteins that play a fundamental role in the migration of leukocytes to sites of infection or injury. We found that protein tyrosine phosphatase nonreceptor type 22 (PTPN22) inhibits signaling by the integrin lymphocyte function-associated antigen-1 (LFA-1) in effector T cells. PTPN22 colocalized with its substrates at the leading edge of cells migrating on surfaces coated with the LFA-1 ligand intercellular adhesion molecule-1 (ICAM-1). Knockout or knockdown of PTPN22 or expression of the autoimmune disease-associated PTPN22-R620W variant resulted in the enhanced phosphorylation of signaling molecules downstream of integrins. Superresolution imaging revealed that PTPN22-R620 (wild-type PTPN22) was present as large clusters in unstimulated T cells and that these disaggregated upon stimulation of LFA-1, enabling increased association of PTPN22 with its binding partners at the leading edge. The failure of PTPN22-R620W molecules to be retained at the leading edge led to increased LFA-1 clustering and integrin-mediated cell adhesion. Our data define a previously uncharacterized mechanism for fine-tuning integrin signaling in T cells, as well as a paradigm of autoimmunity in humans in which disease susceptibility is underpinned by inherited phosphatase mutations that perturb integrin function.

Topographic prominence as a method for cluster identification in single-molecule localisation data.

Single-molecule localisation based super-resolution fluorescence imaging produces maps of the coordinates of fluorescent molecules in a region of interest. Cluster analysis algorithms provide information concerning the clustering characteristics of these molecules, often through the generation of cluster heat maps based on local molecular density. The goal of this study was to generate a new cluster analysis method based on a topographic approach. In particular, a topographic map of the level of clustering across a region is generated based on Getis' variant of Ripley's K-function. By using the relative heights (topographic prominence, TP) of the peaks in the map, cluster characteristics can be identified more accurately than by using previously demonstrated height thresholds. Analogous to geological TP, the concepts of wet and dry TP and topographic isolation are introduced to generate binary maps. The algorithm is validated using simulated and experimental data and found to significantly outperform previous cluster identification methods. Illustration of the topographic prominence based cluster analysis algorithm.