REFORMS: Consensus-based Recommendations for Machine-learning-based Science.

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability.

Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.

Integrating explanation and prediction in computational social science.

Computational social science is more than just large repositories of digital data and the computational methods needed to construct and analyse them. It also represents a convergence of different fields with different ways of thinking about and doing science. The goal of this Perspective is to provide some clarity around how these approaches differ from one another and to propose how they might be productively integrated. Towards this end we make two contributions. The first is a schema for thinking about research activities along two dimensions-the extent to which work is explanatory, focusing on identifying and estimating causal effects, and the degree of consideration given to testing predictions of outcomes-and how these two priorities can complement, rather than compete with, one another. Our second contribution is to advocate that computational social scientists devote more attention to combining prediction and explanation, which we call integrative modelling, and to outline some practical suggestions for realizing this goal.

Prediction and explanation in social systems.

Historically, social scientists have sought out explanations of human and social phenomena that provide interpretable causal mechanisms, while often ignoring their predictive accuracy. We argue that the increasingly computational nature of social science is beginning to reverse this traditional bias against prediction; however, it has also highlighted three important issues that require resolution. First, current practices for evaluating predictions must be better standardized. Second, theoretical limits to predictive accuracy in complex social systems must be better characterized, thereby setting expectations for what can be predicted or explained. Third, predictive accuracy and interpretability must be recognized as complements, not substitutes, when evaluating explanations. Resolving these three issues will lead to better, more replicable, and more useful social science.

Online and Social Media Data As an Imperfect Continuous Panel Survey.

There is a large body of research on utilizing online activity as a survey of political opinion to predict real world election outcomes. There is considerably less work, however, on using this data to understand topic-specific interest and opinion amongst the general population and specific demographic subgroups, as currently measured by relatively expensive surveys. Here we investigate this possibility by studying a full census of all Twitter activity during the 2012 election cycle along with the comprehensive search history of a large panel of Internet users during the same period, highlighting the challenges in interpreting online and social media activity as the results of a survey. As noted in existing work, the online population is a non-representative sample of the offline world (e.g., the U.S. voting population). We extend this work to show how demographic skew and user participation is non-stationary and difficult to predict over time. In addition, the nature of user contributions varies substantially around important events. Furthermore, we note subtle problems in mapping what people are sharing or consuming online to specific sentiment or opinion measures around a particular topic. We provide a framework, built around considering this data as an imperfect continuous panel survey, for addressing these issues so that meaningful insight about public interest and opinion can be reliably extracted from online and social media data.

Graphical models for inferring single molecule dynamics.

BACKGROUND: The recent explosion of experimental techniques in single molecule biophysics has generated a variety of novel time series data requiring equally novel computational tools for analysis and inference. This article describes in general terms how graphical modeling may be used to learn from biophysical time series data using the variational Bayesian expectation maximization algorithm (VBEM). The discussion is illustrated by the example of single-molecule fluorescence resonance energy transfer (smFRET) versus time data, where the smFRET time series is modeled as a hidden Markov model (HMM) with Gaussian observables. A detailed description of smFRET is provided as well. RESULTS: The VBEM algorithm returns the model's evidence and an approximating posterior parameter distribution given the data. The former provides a metric for model selection via maximum evidence (ME), and the latter a description of the model's parameters learned from the data. ME/VBEM provide several advantages over the more commonly used approach of maximum likelihood (ML) optimized by the expectation maximization (EM) algorithm, the most important being a natural form of model selection and a well-posed (non-divergent) optimization problem. CONCLUSIONS: The results demonstrate the utility of graphical modeling for inference of dynamic processes in single molecule biophysics.

Predicting consumer behavior with Web search.

Recent work has demonstrated that Web search volume can "predict the present," meaning that it can be used to accurately track outcomes such as unemployment levels, auto and home sales, and disease prevalence in near real time. Here we show that what consumers are searching for online can also predict their collective future behavior days or even weeks in advance. Specifically we use search query volume to forecast the opening weekend box-office revenue for feature films, first-month sales of video games, and the rank of songs on the Billboard Hot 100 chart, finding in all cases that search counts are highly predictive of future outcomes. We also find that search counts generally boost the performance of baseline models fit on other publicly available data, where the boost varies from modest to dramatic, depending on the application in question. Finally, we reexamine previous work on tracking flu trends and show that, perhaps surprisingly, the utility of search data relative to a simple autoregressive model is modest. We conclude that in the absence of other data sources, or where small improvements in predictive performance are material, search queries provide a useful guide to the near future.

Learning rates and states from biophysical time series: a Bayesian approach to model selection and single-molecule FRET data.

Time series data provided by single-molecule Förster resonance energy transfer (smFRET) experiments offer the opportunity to infer not only model parameters describing molecular complexes, e.g., rate constants, but also information about the model itself, e.g., the number of conformational states. Resolving whether such states exist or how many of them exist requires a careful approach to the problem of model selection, here meaning discrimination among models with differing numbers of states. The most straightforward approach to model selection generalizes the common idea of maximum likelihood--selecting the most likely parameter values--to maximum evidence: selecting the most likely model. In either case, such an inference presents a tremendous computational challenge, which we here address by exploiting an approximation technique termed variational Bayesian expectation maximization. We demonstrate how this technique can be applied to temporal data such as smFRET time series; show superior statistical consistency relative to the maximum likelihood approach; compare its performance on smFRET data generated from experiments on the ribosome; and illustrate how model selection in such probabilistic or generative modeling can facilitate analysis of closely related temporal data currently prevalent in biophysics. Source code used in this analysis, including a graphical user interface, is available open source via http://vbFRET.sourceforge.net.

Allosteric collaboration between elongation factor G and the ribosomal L1 stalk directs tRNA movements during translation.

Determining the mechanism by which tRNAs rapidly and precisely transit through the ribosomal A, P, and E sites during translation remains a major goal in the study of protein synthesis. Here, we report the real-time dynamics of the L1 stalk, a structural element of the large ribosomal subunit that is implicated in directing tRNA movements during translation. Within pretranslocation ribosomal complexes, the L1 stalk exists in a dynamic equilibrium between open and closed conformations. Binding of elongation factor G (EF-G) shifts this equilibrium toward the closed conformation through one of at least two distinct kinetic mechanisms, where the identity of the P-site tRNA dictates the kinetic route that is taken. Within posttranslocation complexes, L1 stalk dynamics are dependent on the presence and identity of the E-site tRNA. Collectively, our data demonstrate that EF-G and the L1 stalk allosterically collaborate to direct tRNA translocation from the P to the E sites, and suggest a model for the release of E-site tRNA.

Quantification of cell edge velocities and traction forces reveals distinct motility modules during cell spreading.

Actin-based cell motility and force generation are central to immune response, tissue development, and cancer metastasis, and understanding actin cytoskeleton regulation is a major goal of cell biologists. Cell spreading is a commonly used model system for motility experiments -- spreading fibroblasts exhibit stereotypic, spatially-isotropic edge dynamics during a reproducible sequence of functional phases: 1) During early spreading, cells form initial contacts with the surface. 2) The middle spreading phase exhibits rapidly increasing attachment area. 3) Late spreading is characterized by periodic contractions and stable adhesions formation. While differences in cytoskeletal regulation between phases are known, a global analysis of the spatial and temporal coordination of motility and force generation is missing. Implementing improved algorithms for analyzing edge dynamics over the entire cell periphery, we observed that a single domain of homogeneous cytoskeletal dynamics dominated each of the three phases of spreading. These domains exhibited a unique combination of biophysical and biochemical parameters -- a motility module. Biophysical characterization of the motility modules revealed that the early phase was dominated by periodic, rapid membrane blebbing; the middle phase exhibited continuous protrusion with very low traction force generation; and the late phase was characterized by global periodic contractions and high force generation. Biochemically, each motility module exhibited a different distribution of the actin-related protein VASP, while inhibition of actin polymerization revealed different dependencies on barbed-end polymerization. In addition, our whole-cell analysis revealed that many cells exhibited heterogeneous combinations of motility modules in neighboring regions of the cell edge. Together, these observations support a model of motility in which regions of the cell edge exhibit one of a limited number of motility modules that, together, determine the overall motility function. Our data and algorithms are publicly available to encourage further exploration.

Bayesian approach to network modularity.

We present an efficient, principled, and interpretable technique for inferring module assignments and for identifying the optimal number of modules in a given network. We show how several existing methods for finding modules can be described as variant, special, or limiting cases of our work, and how the method overcomes the resolution limit problem, accurately recovering the true number of modules. Our approach is based on Bayesian methods for model selection which have been used with success for almost a century, implemented using a variational technique developed only in the past decade. We apply the technique to synthetic and real networks and outline how the method naturally allows selection among competing models.

Opposing effects of PKCtheta and WASp on symmetry breaking and relocation of the immunological synapse.

The immunological synapse (IS) is a junction between the T cell and antigen-presenting cell and is composed of supramolecular activation clusters (SMACs). No studies have been published on naive T cell IS dynamics. Here, we find that IS formation during antigen recognition comprises cycles of stable IS formation and autonomous naive T cell migration. The migration phase is driven by PKCtheta, which is localized to the F-actin-dependent peripheral (p)SMAC. PKCtheta(-/-) T cells formed hyperstable IS in vitro and in vivo and, like WT cells, displayed fast oscillations in the distal SMAC, but they showed reduced slow oscillations in pSMAC integrity. IS reformation is driven by the Wiscott Aldrich Syndrome protein (WASp). WASp(-/-) T cells displayed normal IS formation but were unable to reform IS after migration unless PKCtheta was inhibited. Thus, opposing effects of PKCtheta and WASp control IS stability through pSMAC symmetry breaking and reformation.

Lateral membrane waves constitute a universal dynamic pattern of motile cells.

We have monitored active movements of the cell circumference on specifically coated substrates for a variety of cells including mouse embryonic fibroblasts and T cells, as well as wing disk cells from fruit flies. Despite having different functions and being from multiple phyla, these cell types share a common spatiotemporal pattern in their normal membrane velocity; we show that protrusion and retraction events are organized in lateral waves along the cell membrane. These wave patterns indicate both spatial and temporal long-range periodic correlations of the actomyosin gel.

Nonmuscle myosin IIA-dependent force inhibits cell spreading and drives F-actin flow.

Nonmuscle myosin IIA (NMM-IIA) is involved in the formation of focal adhesions and neurite retraction. However, the role of NMM-IIA in these functions remains largely unknown. Using RNA interference as a tool to decrease NMM-IIA expression, we have found that NMM-IIA is the major myosin involved in traction force generation and retrograde F-actin flow in mouse embryonic fibroblast cells. Quantitative analyses revealed that approximately 60% of traction force on fibronectin-coated surfaces is contributed by NMM-IIA and approximately 30% by NMM-IIB. The retrograde F-actin flow decreased dramatically in NMM-IIA-depleted cells, but seemed unaffected by NMM-IIB deletion. In addition, we found that depletion of NMM-IIA caused cells to spread at a higher rate and to a greater area on fibronectin substrates during the early spreading period, whereas deletion of NMM-IIB appeared to have no effect on spreading. The distribution of NMM-IIA was concentrated on the dorsal surface and approached the ventral surface in the periphery, whereas NMM-IIB was primarily concentrated around the nucleus and to a lesser extent at the ventral surface in cell periphery. Our results suggest that NMM-IIA is involved in generating a coherent cytoplasmic contractile force from one side of the cell to the other through the cross-linking and the contraction of dorsal actin filaments.

The small GTPase R-Ras regulates organization of actin and drives membrane protrusions through the activity of PLCepsilon.

R-Ras, an atypical member of the Ras subfamily of small GTPases, enhances integrin-mediated adhesion and signaling through a poorly understood mechanism. Dynamic analysis of cell spreading by total internal reflection fluorescence (TIRF) microscopy demonstrated that active R-Ras lengthened the duration of initial membrane protrusion, and promoted the formation of a ruffling lamellipod, rich in branched actin structures and devoid of filopodia. By contrast, dominant-negative R-Ras enhanced filopodia formation. Moreover, RNA interference (RNAi) approaches demonstrated that endogenous R-Ras contributed to cell spreading. These observations suggest that R-Ras regulates membrane protrusions through organization of the actin cytoskeleton. Our results suggest that phospholipase Cepsilon (PLCepsilon) is a novel R-Ras effector mediating the effects of R-Ras on the actin cytoskeleton and membrane protrusion, because R-Ras was co-precipitated with PLCepsilon and increased its activity. Knockdown of PLCepsilon with siRNA reduced the formation of the ruffling lamellipod in R-Ras cells. Consistent with this pathway, inhibitors of PLC activity, or chelating intracellular Ca2+ abolished the ability of R-Ras to promote membrane protrusions and spreading. Overall, these data suggest that R-Ras signaling regulates the organization of the actin cytoskeleton to sustain membrane protrusion through the activity of PLCepsilon.