Evaluating language models for mathematics through interactions.

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs; this is insufficient for making an informed decision about which LLMs are best to use in an interactive setting, and how that varies by setting. Static assessment therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analyzing MathConverse, we derive a taxonomy of human query behaviors and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, among other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by experienced mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, may constitute better assistants. Humans should inspect LLM output carefully given their current shortcomings and potential for surprising fallibility.

Weak Intermolecular CH···N Hydrogen Bonding: Determination of 13CH-15N Hydrogen-Bond Mediated J Couplings by Solid-State NMR Spectroscopy and First-Principles Calculations.

Weak hydrogen bonds are increasingly hypothesized to play key roles in a wide range of chemistry from catalysis to gelation to polymer structure. Here, 15N/13C spin-echo magic-angle spinning (MAS) solid-state nuclear magnetic resonance (NMR) experiments are applied to "view" intermolecular CH···N hydrogen bonding in two selectively labeled organic compounds, 4-[15N] cyano-4'-[13C2] ethynylbiphenyl (1) and [15N3,13C6]-2,4,6-triethynyl-1,3,5-triazine (2). The synthesis of 2-15N3,13C6 is reported here for the first time via a multistep procedure, where the key element is the reaction of [15N3]-2,4,6-trichloro-1,3,5-triazine (5) with [13C2]-[(trimethylsilyl)ethynyl]zinc chloride (8) to afford its immediate precursor [15N3,13C6]-2,4,6-tris[(trimethylsilyl)ethynyl]-1,3,5-triazine (9). Experimentally determined hydrogen-bond-mediated 2hJCN couplings (4.7 ± 0.4 Hz (1) and 4.1 ± 0.3 Hz (2)) are compared with density functional theory (DFT) gauge-including projector augmented wave (GIPAW) calculations, whereby species-independent coupling values 2hKCN (29.0 × 1019 kg m-2 s-2 A-2 (1) and 27.9 × 1019 kg m-2 s-2 A-2 (2)) quantitatively demonstrate the J couplings for these "weak" CH···N hydrogen bonds to be of a similar magnitude to those for conventionally observed NH···O hydrogen-bonding interactions in uracil (2hKNO: 28.1 and 36.8 × 1019 kg m-2 s-2 A-2). Moreover, the GIPAW calculations show a clear correlation between increasing 2hJCN (and 3hJCN) coupling and reducing C(H)···N and H···N hydrogen-bonding distances, with the Fermi contact term accounting for at least 98% of the isotropic 2hJCN coupling.

An NMR crystallography investigation of furosemide.

This paper presents an NMR crystallography study of three polymorphs of furosemide. Experimental magic-angle spinning (MAS) solid-state NMR spectra are reported for form I of furosemide, and these are assigned using density-functional theory (DFT)-based gauge-including projector augmented wave (GIPAW) calculations. Focusing on the three known polymorphs, we examine the changes to the NMR parameters due to crystal packing effects. We use a recently developed formalism to visualise which regions are responsible for the chemical shielding of particular sites and hence understand the variation in NMR parameters between the three polymorphs.

Reducing the computational cost of NMR crystallography of organic powders at natural isotopic abundance with the help of 13 C-13 C dipolar couplings.

Structure determination of functional organic compounds remains a formidable challenge when the sample exists as a powder. Nuclear magnetic resonance crystallography approaches based on the comparison of experimental and Density Functional Theory (DFT)-computed 1 H chemical shifts have already demonstrated great potential for structure determination of organic powders, but limitations still persist. In this study, we discuss the possibility of using 13 C-13 C dipolar couplings quantified on powdered theophylline at natural isotopic abundance with the help of dynamic nuclear polarization, to realize a DFT-free, rapid screening of a pool of structures predicted by ab initio random structure search. We show that although 13 C-13 C dipolar couplings can identify structures possessing long range structural motifs and unit cell parameters close to those of the true structure, it must be complemented with other data to recover information about the presence and the chemical nature of the supramolecular interactions.

Ab initio random structure searching of organic molecular solids: assessment and validation against experimental data.

This paper explores the capability of using the DFT-D ab initio random structure searching (AIRSS) method to generate crystal structures of organic molecular materials, focusing on a system (m-aminobenzoic acid; m-ABA) that is known from experimental studies to exhibit abundant polymorphism. Within the structural constraints selected for the AIRSS calculations (specifically, centrosymmetric structures with Z = 4 for zwitterionic m-ABA molecules), the method is shown to successfully generate the two known polymorphs of m-ABA (form III and form IV) that have these structural features. We highlight various issues that are encountered in comparing crystal structures generated by AIRSS to experimental powder X-ray diffraction (XRD) data and solid-state magic-angle spinning (MAS) NMR data, demonstrating successful fitting for some of the lowest energy structures from the AIRSS calculations against experimental low-temperature powder XRD data for known polymorphs of m-ABA, and showing that comparison of computed and experimental solid-state NMR parameters allows different hydrogen-bonding motifs to be discriminated.

Visualising crystal packing interactions in solid-state NMR: Concepts and applications.

In this article, we introduce and apply a methodology, based on density functional theory and the gauge-including projector augmented wave approach, to explore the effects of packing interactions on solid-state nuclear magnetic resonance (NMR) parameters. A visual map derived from a so-termed "magnetic shielding contribution field" can be made of the contributions to the magnetic shielding of a specific site-partitioning the chemical shift to specific interactions. The relation to the established approaches of examining the molecule to crystal change in the chemical shift and the nuclear independent chemical shift is established. The results are applied to a large sample of 71 molecular crystals and three further specific examples from supermolecular chemistry and pharmaceuticals. This approach extends the NMR crystallography toolkit and provides insight into the development of both cluster based approaches to the predictions of chemical shifts and for empirical predictions of chemical shifts in solids.

Automated detection of feeding strikes by larval fish using continuous high-speed digital video: a novel method to extract quantitative data from fast, sparse kinematic events.

Using videography to extract quantitative data on animal movement and kinematics constitutes a major tool in biomechanics and behavioral ecology. Advanced recording technologies now enable acquisition of long video sequences encompassing sparse and unpredictable events. Although such events may be ecologically important, analysis of sparse data can be extremely time-consuming and potentially biased; data quality is often strongly dependent on the training level of the observer and subject to contamination by observer-dependent biases. These constraints often limit our ability to study animal performance and fitness. Using long videos of foraging fish larvae, we provide a framework for the automated detection of prey acquisition strikes, a behavior that is infrequent yet critical for larval survival. We compared the performance of four video descriptors and their combinations against manually identified feeding events. For our data, the best single descriptor provided a classification accuracy of 77-95% and detection accuracy of 88-98%, depending on fish species and size. Using a combination of descriptors improved the accuracy of classification by ∼2%, but did not improve detection accuracy. Our results indicate that the effort required by an expert to manually label videos can be greatly reduced to examining only the potential feeding detections in order to filter false detections. Thus, using automated descriptors reduces the amount of manual work needed to identify events of interest from weeks to hours, enabling the assembly of an unbiased large dataset of ecologically relevant behaviors.

Visualization and processing of computed solid-state NMR parameters: MagresView and MagresPython.

We introduce two open source tools to aid the processing and visualisation of ab-initio computed solid-state NMR parameters. The Magres file format for computed NMR parameters (as implemented in CASTEP v8.0 and QuantumEspresso v5.0.0) is implemented. MagresView is built upon the widely used Jmol crystal viewer, and provides an intuitive environment to display computed NMR parameters. It can provide simple pictorial representation of one- and two-dimensional NMR spectra as well as output a selected spin-system for exact simulations with dedicated spin-dynamics software. MagresPython provides a simple scripting environment to manipulate large numbers of computed NMR parameters to search for structural correlations.

Hydrodynamic Constraints of Suction Feeding in Low Reynolds Numbers, and the Critical Period of Larval Fishes.

Larval fishes suffer prodigious mortality rates, eliminating 99% of the cohort within a few days after their first feeding. Hjort (1914) famously attributed this "critical period" of low survival to larval inability to obtain sufficient food. We discuss recent experimental and modeling work, suggesting that the viscous hydrodynamic regime have marked effects on the mechanism of suction feeding in larval fish. As larvae grow, the size of the gape and associated volume of the mouth increase. At the same time, larvae swim faster and can generate faster suction flows, thus transiting to a hydrodynamic regime of higher Reynolds numbers. This hydrodynamic regime further leads to changes in the spatio-temporal patterns of flow in front of the mouth, and an increasing ability in larger larvae to exert suction forces on the prey. Simultaneously, the increase in swimming speed and the distance from which the prey is attacked result in higher rates of encountering prey by larger (older) larvae. In contrast, during the first few days after feeding commence the lower rates of encounter and success in feeding translate to low feeding rates. We conclude that young larvae experience "hydrodynamic starvation," in which low Reynolds numbers mechanically limit their feeding performance even under high densities of prey.