Bragg Spot Finder (BSF): a new machine-learning-aided approach to deal with spot finding for rapidly filtering diffraction pattern images.

Macromolecular crystallography contributes significantly to understanding diseases and, more importantly, how to treat them by providing atomic resolution 3D structures of proteins. This is achieved by collecting X-ray diffraction images of protein crystals from important biological pathways. Spotfinders are used to detect the presence of crystals with usable data, and the spots from such crystals are the primary data used to solve the relevant structures. Having fast and accurate spot finding is essential, but recent advances in synchrotron beamlines used to generate X-ray diffraction images have brought us to the limits of what the best existing spotfinders can do. This bottleneck must be removed so spotfinder software can keep pace with the X-ray beamline hardware improvements and be able to see the weak or diffuse spots required to solve the most challenging problems encountered when working with diffraction images. In this paper, we first present Bragg Spot Detection (BSD), a large benchmark Bragg spot image dataset that contains 304 images with more than 66 000 spots. We then discuss the open source extensible U-Net-based spotfinder Bragg Spot Finder (BSF), with image pre-processing, a U-Net segmentation backbone, and post-processing that includes artifact removal and watershed segmentation. Finally, we perform experiments on the BSD benchmark and obtain results that are (in terms of accuracy) comparable to or better than those obtained with two popular spotfinder software packages (Dozor and DIALS), demonstrating that this is an appropriate framework to support future extensions and improvements.

Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments.

Structural biology experiments benefit significantly from state-of-the-art synchrotron data collection. One can acquire macromolecular crystallography (MX) diffraction data on large-area photon-counting pixel-array detectors at framing rates exceeding 1000 frames per second, using 200 Gbps network connectivity, or higher when available. In extreme cases this represents a raw data throughput of about 25 GB s-1, which is nearly impossible to deliver at reasonable cost without compression. Our field has used lossless compression for decades to make such data collection manageable. Many MX beamlines are now fitted with DECTRIS Eiger detectors, all of which are delivered with optimized compression algorithms by default, and they perform well with current framing rates and typical diffraction data. However, better lossless compression algorithms have been developed and are now available to the research community. Here one of the latest and most promising lossless compression algorithms is investigated on a variety of diffraction data like those routinely acquired at state-of-the-art MX beamlines.

Approximating lattice similarity.

A method is proposed for choosing unit cells for a group of crystals so that they all appear as nearly similar as possible to a selected cell. Related unit cells with varying cell parameters or indexed with different lattice centering can be accommodated.

An invertible seven-dimensional Dirichlet cell characterization of lattices.

Characterization of crystallographic lattices is an important tool in structure solution, crystallographic database searches and clustering of diffraction images in serial crystallography. Characterization of lattices by Niggli-reduced cells (based on the three shortest non-coplanar lattice vectors) or by Delaunay-reduced cells (based on four non-coplanar vectors summing to zero and all meeting at obtuse or right angles) is commonly performed. The Niggli cell derives from Minkowski reduction. The Delaunay cell derives from Selling reduction. All are related to the Wigner-Seitz (or Dirichlet, or Voronoi) cell of the lattice, which consists of the points at least as close to a chosen lattice point as they are to any other lattice point. The three non-coplanar lattice vectors chosen are here called the Niggli-reduced cell edges. Starting from a Niggli-reduced cell, the Dirichlet cell is characterized by the planes determined by 13 lattice half-edges: the midpoints of the three Niggli cell edges, the six Niggli cell face-diagonals and the four body-diagonals, but seven of the lengths are sufficient: three edge lengths, the three shorter of each pair of face-diagonal lengths, and the shortest body-diagonal length. These seven are sufficient to recover the Niggli-reduced cell.

AMX - the highly automated macromolecular crystallography (17-ID-1) beamline at the NSLS-II.

The highly automated macromolecular crystallography beamline AMX/17-ID-1 is an undulator-based high-intensity (>5 × 1012 photons s-1), micro-focus (7 µm × 5 µm), low-divergence (1 mrad × 0.35 mrad) energy-tunable (5-18 keV) beamline at the NSLS-II, Brookhaven National Laboratory, Upton, NY, USA. It is one of the three life science beamlines constructed by the NIH under the ABBIX project and it shares sector 17-ID with the FMX beamline, the frontier micro-focus macromolecular crystallography beamline. AMX saw first light in March 2016 and started general user operation in February 2017. At AMX, emphasis has been placed on high throughput, high capacity, and automation to enable data collection from the most challenging projects using an intense micro-focus beam. Here, the current state and capabilities of the beamline are reported, and the different macromolecular crystallography experiments that are routinely performed at AMX/17-ID-1 as well as some plans for the near future are presented.

Generating random unit cells.

Methods of generating random unit-cell data for testing software are discussed. Working within the space S 6 appears to be the most expeditious.

Serial crystallography with multi-stage merging of thousands of images.

KAMO and BLEND provide particularly effective tools to automatically manage the merging of large numbers of data sets from serial crystallography. The requirement for manual intervention in the process can be reduced by extending BLEND to support additional clustering options such as the use of more accurate cell distance metrics and the use of reflection-intensity correlation coefficients to infer `distances' among sets of reflections. This increases the sensitivity to differences in unit-cell parameters and allows clustering to assemble nearly complete data sets on the basis of intensity or amplitude differences. If the data sets are already sufficiently complete to permit it, one applies KAMO once and clusters the data using intensities only. When starting from incomplete data sets, one applies KAMO twice, first using unit-cell parameters. In this step, either the simple cell vector distance of the original BLEND or the more sensitive NCDist is used. This step tends to find clusters of sufficient size such that, when merged, each cluster is sufficiently complete to allow reflection intensities or amplitudes to be compared. One then uses KAMO again using the correlation between reflections with a common hkl to merge clusters in a way that is sensitive to structural differences that may not have perturbed the unit-cell parameters sufficiently to make meaningful clusters. Many groups have developed effective clustering algorithms that use a measurable physical parameter from each diffraction still or wedge to cluster the data into categories which then can be merged, one hopes, to yield the electron density from a single protein form. Since these physical parameters are often largely independent of one another, it should be possible to greatly improve the efficacy of data-clustering software by using a multi-stage partitioning strategy. Here, one possible approach to multi-stage data clustering is demonstrated. The strategy is to use unit-cell clustering until the merged data are sufficiently complete and then to use intensity-based clustering. Using this strategy, it is demonstrated that it is possible to accurately cluster data sets from crystals that have subtle differences.

A simple technique to classify diffraction data from dynamic proteins according to individual polymorphs.

One often observes small but measurable differences in the diffraction data measured from different crystals of a single protein. These differences might reflect structural differences in the protein and may reveal the natural dynamism of the molecule in solution. Partitioning these mixed-state data into single-state clusters is a critical step that could extract information about the dynamic behavior of proteins from hundreds or thousands of single-crystal data sets. Mixed-state data can be obtained deliberately (through intentional perturbation) or inadvertently (while attempting to measure highly redundant single-crystal data). To the extent that different states adopt different molecular structures, one expects to observe differences in the crystals; each of the polystates will create a polymorph of the crystals. After mixed-state diffraction data have been measured, deliberately or inadvertently, the challenge is to sort the data into clusters that may represent relevant biological polystates. Here, this problem is addressed using a simple multi-factor clustering approach that classifies each data set using independent observables, thereby assigning each data set to the correct location in conformational space. This procedure is illustrated using two independent observables, unit-cell parameters and intensities, to cluster mixed-state data from chymotrypsinogen (ChTg) crystals. It is observed that the data populate an arc of the reaction trajectory as ChTg is converted into chymotrypsin.

Robotic sample changers for macromolecular X-ray crystallography and biological small-angle X-ray scattering at the National Synchrotron Light Source II. Corrigendum.

A correction in the paper by Lazo et al. [(2021). J. Synchrotron Rad. 28, 1649-1661] is made.

Robotic sample changers for macromolecular X-ray crystallography and biological small-angle X-ray scattering at the National Synchrotron Light Source II.

Here we present two robotic sample changers integrated into the experimental stations for the macromolecular crystallography (MX) beamlines AMX and FMX, and the biological small-angle scattering (bioSAXS) beamline LiX. They enable fully automated unattended data collection and remote access to the beamlines. The system designs incorporate high-throughput, versatility, high-capacity, resource sharing and robustness. All systems are centered around a six-axis industrial robotic arm coupled with a force torque sensor and in-house end effectors (grippers). They have the same software architecture and the facility standard EPICS-based BEAST alarm system. The MX system is compatible with SPINE bases and Unipucks. It comprises a liquid nitrogen dewar holding 384 samples (24 Unipucks) and a stay-cold gripper, and utilizes machine vision software to track the sample during operations and to calculate the final mount position on the goniometer. The bioSAXS system has an in-house engineered sample storage unit that can hold up to 360 samples (20 sample holders) which keeps samples at a user-set temperature (277 K to 300 K). The MX systems were deployed in early 2017 and the bioSAXS system in early 2019.

FMX - the Frontier Microfocusing Macromolecular Crystallography Beamline at the National Synchrotron Light Source II.

Two new macromolecular crystallography (MX) beamlines at the National Synchrotron Light Source II, FMX and AMX, opened for general user operation in February 2017 [Schneider et al. (2013). J. Phys. Conf. Ser. 425, 012003; Fuchs et al. (2014). J. Phys. Conf. Ser. 493, 012021; Fuchs et al. (2016). AIP Conf. Proc. SRI2015, 1741, 030006]. FMX, the micro-focusing Frontier MX beamline in sector 17-ID-2 at NSLS-II, covers a 5-30 keV photon energy range and delivers a flux of 4.0 × 1012 photons s-1 at 1 Šinto a 1 µm × 1.5 µm to 10 µm × 10 µm (V × H) variable focus, expected to reach 5 × 1012 photons s-1 at final storage-ring current. This flux density surpasses most MX beamlines by nearly two orders of magnitude. The high brightness and microbeam capability of FMX are focused on solving difficult crystallographic challenges. The beamline's flexible design supports a wide range of structure determination methods - serial crystallography on micrometre-sized crystals, raster optimization of diffraction from inhomogeneous crystals, high-resolution data collection from large-unit-cell crystals, room-temperature data collection for crystals that are difficult to freeze and for studying conformational dynamics, and fully automated data collection for sample-screening and ligand-binding studies. FMX's high dose rate reduces data collection times for applications like serial crystallography to minutes rather than hours. With associated sample lifetimes as short as a few milliseconds, new rapid sample-delivery methods have been implemented, such as an ultra-high-speed high-precision piezo scanner goniometer [Gao et al. (2018). J. Synchrotron Rad. 25, 1362-1370], new microcrystal-optimized micromesh well sample holders [Guo et al. (2018). IUCrJ, 5, 238-246] and highly viscous media injectors [Weierstall et al. (2014). Nat. Commun. 5, 3309]. The new beamline pushes the frontier of synchrotron crystallography and enables users to determine structures from difficult-to-crystallize targets like membrane proteins, using previously intractable crystals of a few micrometres in size, and to obtain quality structures from irregular larger crystals.

Gold Standard for macromolecular crystallography diffraction data.

Macromolecular crystallography (MX) is the dominant means of determining the three-dimensional structures of biological macromolecules. Over the last few decades, most MX data have been collected at synchrotron beamlines using a large number of different detectors produced by various manufacturers and taking advantage of various protocols and goniometries. These data came in their own formats: sometimes proprietary, sometimes open. The associated metadata rarely reached the degree of completeness required for data management according to Findability, Accessibility, Interoperability and Reusability (FAIR) principles. Efforts to reuse old data by other investigators or even by the original investigators some time later were often frustrated. In the culmination of an effort dating back more than two decades, a large portion of the research community concerned with high data-rate macromolecular crystallography (HDRMX) has now agreed to an updated specification of data and metadata for diffraction images produced at synchrotron light sources and X-ray free-electron lasers (XFELs). This 'Gold Standard' will facilitate the processing of data sets independent of the facility at which they were collected and enable data archiving according to FAIR principles, with a particular focus on interoperability and reusability. This agreed standard builds on the NeXus/HDF5 NXmx application definition and the International Union of Crystallo-graphy (IUCr) imgCIF/CBF dictionary, and it is compatible with major data-processing programs and pipelines. Just as with the IUCr CBF/imgCIF standard from which it arose and to which it is tied, the NeXus/HDF5 NXmx Gold Standard application definition is intended to be applicable to all detectors used for crystallography, and all hardware and software developers in the field are encouraged to adopt and contribute to the standard.

Best practices for high data-rate macromolecular crystallography (HDRMX).

In macromolecular crystallography, higher flux, smaller beams, and faster detectors open the door to experiments with very large numbers of very small samples that can reveal polymorphs and dynamics but require re-engineering of approaches to the clustering of images both at synchrotrons and XFELs (X-ray free electron lasers). The need for the management of orders of magnitude more images and limitations of file systems favor a transition from simple one-file-per-image systems such as CBF to image container systems such as HDF5. This further increases the load on computers and networks and requires a re-examination of the presentation of metadata. In this paper, we discuss three important components of this problem-improved approaches to the clustering of images to better support experiments on polymorphs and dynamics, recent and upcoming changes in metadata for Eiger images, and software to rapidly validate images in the revised Eiger format.

Converting three-space matrices to equivalent six-space matrices for Delone scalars in S6.

The transformations from the primitive cells of the centered Bravais lattices to the corresponding centered cells have conventionally been listed as three-by-three matrices that transform three-space lattice vectors. Using those three-by-three matrices when working in the six-dimensional space of lattices represented as Selling scalars as used in Delone (Delaunay) reduction, one could transform to the three-space representation, apply the three-by-three matrices and then back-transform to the six-space representation, but it is much simpler to have the equivalent six-by-six matrices and apply them directly. The general form of the transformation from the three-space matrix to the corresponding matrix operating on Selling scalars (expressed in space S6) is derived, and the particular S6matrices for the centered Delone types are listed. (Note: in his later publications, Boris Delaunay used the Russian version of his surname, Delone.).

A space for lattice representation and clustering.

Algorithms for quantifying the differences between two lattices are used for Bravais lattice determination, database lookup for unit cells to select candidates for molecular replacement, and recently for clustering to group together images from serial crystallography. It is particularly desirable for the differences between lattices to be computed as a perturbation-stable metric, i.e. as distances that satisfy the triangle inequality, so that standard tree-based nearest-neighbor algorithms can be used, and for which small changes in the lattices involved produce small changes in the distances computed. A perturbation-stable metric space related to the reduction algorithm of Selling and to the Bravais lattice determination methods of Delone is described. Two ways of representing the space, as six-dimensional real vectors or equivalently as three-dimensional complex vectors, are presented and applications of these metrics are discussed. (Note: in his later publications, Boris Delaunay used the Russian version of his surname, Delone.).

Selling reduction versus Niggli reduction for crystallographic lattices.

The unit-cell reduction described by Selling and used by Delone (whose early publications were under the spelling Delaunay) is explained in a simple form. The transformations needed to implement the reduction are listed. The simplicity of this reduction contrasts with the complexity of Niggli reduction.

NearTree, a data structure and a software toolkit for the nearest-neighbor problem.

Many problems in crystallography and other fields can be treated as nearest-neighbor problems. The neartree data structure provides a flexible way to organize and retrieve metric data. In some cases, it can provide near-optimal performance. NearTree is a software tool that constructs neartrees and provides a number of different query tools.

Automated protein motif generation in the structure-based protein function prediction tool ProMOL.

ProMOL, a plugin for the PyMOL molecular graphics system, is a structure-based protein function prediction tool. ProMOL includes a set of routines for building motif templates that are used for screening query structures for enzyme active sites. Previously, each motif template was generated manually and required supervision in the optimization of parameters for sensitivity and selectivity. We developed an algorithm and workflow for the automation of motif building and testing routines in ProMOL. The algorithm uses a set of empirically derived parameters for optimization and requires little user intervention. The automated motif generation algorithm was first tested in a performance comparison with a set of manually generated motifs based on identical active sites from the same 112 PDB entries. The two sets of motifs were equally effective in identifying alignments with homologs and in rejecting alignments with unrelated structures. A second set of 296 active site motifs were generated automatically, based on Catalytic Site Atlas entries with literature citations, as an expansion of the library of existing manually generated motif templates. The new motif templates exhibited comparable performance to the existing ones in terms of hit rates against native structures, homologs with the same EC and Pfam designations, and randomly selected unrelated structures with a different EC designation at the first EC digit, as well as in terms of RMSD values obtained from local structural alignments of motifs and query structures. This research is supported by NIH grant GM078077.

The NeXus data format.

NeXus is an effort by an international group of scientists to define a common data exchange and archival format for neutron, X-ray and muon experiments. NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files, in addition to a dictionary of well defined domain-specific field names. The NeXus data format has two purposes. First, it defines a format that can serve as a container for all relevant data associated with a beamline. This is a very important use case. Second, it defines standards in the form of application definitions for the exchange of data between applications. NeXus provides structures for raw experimental data as well as for processed data.

Development of nanosecond time-resolved infrared detection at the LEAF pulse radiolysis facility.

When coupled with transient absorption spectroscopy, pulse radiolysis, which utilizes high-energy electron pulses from an accelerator, is a powerful tool for investigating the kinetics and thermodynamics of a wide range of radiation-induced redox and electron transfer processes. The majority of these investigations detect transient species in the UV, visible, or near-IR spectral regions. Unfortunately, the often-broad and featureless absorption bands in these regions can make the definitive identification of intermediates difficult. Time-resolved vibrational spectroscopy would offer much improved structural characterization, but has received only limited application in pulse radiolysis. In this paper, we describe in detail the development of a unique nanosecond time-resolved infrared (TRIR) detection capability for condensed-phase pulse radiolysis on a new beam line at the LEAF facility of Brookhaven National Laboratory. The system makes use of a suite of high-power, continuous wave external-cavity quantum cascade lasers as the IR probe source, with coverage from 2330 to 1051 cm(-1). The response time of the TRIR detection setup is ∼40 ns, with a typical sensitivity of ∼100 µOD after 4-8 signal averages using a dual-beam probe/reference normalization detection scheme. This new detection method has enabled mechanistic investigations of a range of radiation-induced chemical processes, some of which are highlighted here.