Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model.

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote "statistical significance" and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

On the Effectiveness of Social Norms Intervention in College Drinking: The Roles of Identity Verification and Peer Influence.

BACKGROUND: The application of social norms theory in the study of college drinking centers on the ideas that incorrect perceptions of drinking norms encourage problematic drinking behavior and that correcting misperceptions can mitigate problems. The design and execution of social norms interventions can be improved with a deeper understanding of causal mechanisms connecting misperception to drinking behavior. METHODS: We develop an agent-based computational simulation that uses identity control theory and peer influence (PI) to model interactions that affect drinking. Using data from the College Alcohol Survey and Social Norms Marketing Research Project, we inform model parameters for agent drinking identities and perceptions. We simulate social norms campaigns that reach progressively larger fractions of the student population, and we consider the strength of the campaign in terms of changing student perception and resulting behavior. RESULTS: We observe a general reduction in heavy episodic drinking (HED) as students are affected by the intervention. As campaigns reached larger fractions of students, the reduction rate diminishes, in some cases actually making a slight reverse. The way in which students "take the message to heart" can have a significant impact as well: The psychological factors involved in identity control and PI have both positive and negative effects on HED rates. With whom agents associate at drinking events also impacts drinking behavior and intervention effectiveness. CONCLUSIONS: Simulations suggest that reducing misperception can reduce HED. When agents adhere strongly to identity verification and when misperceptions affect identity appraisals, social norms campaigns can bring about large reductions. PI, self-monitoring, and socializing with like-drinking peers appear to moderate the effect.

Parameter Estimation for Gene Regulatory Networks from Microarray Data: Cold Shock Response in Saccharomyces cerevisiae.

We investigated the dynamics of a gene regulatory network controlling the cold shock response in budding yeast, Saccharomyces cerevisiae. The medium-scale network, derived from published genome-wide location data, consists of 21 transcription factors that regulate one another through 31 directed edges. The expression levels of the individual transcription factors were modeled using mass balance ordinary differential equations with a sigmoidal production function. Each equation includes a production rate, a degradation rate, weights that denote the magnitude and type of influence of the connected transcription factors (activation or repression), and a threshold of expression. The inverse problem of determining model parameters from observed data is our primary interest. We fit the differential equation model to published microarray data using a penalized nonlinear least squares approach. Model predictions fit the experimental data well, within the 95% confidence interval. Tests of the model using randomized initial guesses and model-generated data also lend confidence to the fit. The results have revealed activation and repression relationships between the transcription factors. Sensitivity analysis indicates that the model is most sensitive to changes in the production rate parameters, weights, and thresholds of Yap1, Rox1, and Yap6, which form a densely connected core in the network. The modeling results newly suggest that Rap1, Fhl1, Msn4, Rph1, and Hsf1 play an important role in regulating the early response to cold shock in yeast. Our results demonstrate that estimation for a large number of parameters can be successfully performed for nonlinear dynamic gene regulatory networks using sparse, noisy microarray data.

Forecasting the effect of the Amethyst initiative on college drinking.

BACKGROUND: A number of college presidents have endorsed the Amethyst Initiative, a call to consider lowering the minimum legal drinking age (MLDA). Our objective is to forecast the effect of the Amethyst Initiative on college drinking. METHODS: A system model of college drinking simulates MLDA changes through (i) a decrease in heavy episodic drinking (HED) because of the lower likelihood of students drinking in unsupervised settings where they model irresponsible drinking (misperception), and (ii) an increase in overall drinking among currently underage students because of increased social availability of alcohol (wetness). RESULTS: For the proportion of HEDs on campus, effects of large decreases in misperception of responsible drinking behavior were more than offset by modest increases in wetness. CONCLUSIONS: For the effect of lowering the MLDA, it appears that increases in social availability of alcohol have a stronger impact on drinking behavior than decreases in misperceptions.

Heavy episodic drinking on college campuses: does changing the legal drinking age make a difference?

OBJECTIVE: This article extends the compartmental model previously developed by Scribner et al. in the context of college drinking to a mathematical model of the consequences of lowering the legal drinking age. METHOD: Using data available from 32 U.S. campuses, the analyses separate underage and legal age drinking groups into an eight-compartment model with different alcohol availability (wetness) for the underage and legal age groups. The model evaluates the likelihood that underage students will incorrectly perceive normative drinking levels to be higher than they actually are (i.e., misperception) and adjust their drinking accordingly by varying the interaction between underage students in social and heavy episodic drinking compartments. RESULTS: The results evaluate the total heavy episodic drinker population and its dependence on the difference in misperception, as well as its dependence on underage wetness, legal age wetness, and drinking age. CONCLUSIONS: Results suggest that an unrealistically extreme combination of high wetness and low enforcement would be needed for the policies related to lowering the drinking age to be effective.

A systems approach to college drinking: development of a deterministic model for testing alcohol control policies.

OBJECTIVE: The misuse and abuse of alcohol among college students remain persistent problems. Using a systems approach to understand the dynamics of student drinking behavior and thus forecasting the impact of campus policy to address the problem represents a novel approach. Toward this end, the successful development of a predictive mathematical model of college drinking would represent a significant advance for prevention efforts. METHOD: A deterministic, compartmental model of college drinking was developed, incorporating three processes: (1) individual factors, (2) social interactions, and (3) social norms. The model quantifies these processes in terms of the movement of students between drinking compartments characterized by five styles of college drinking: abstainers, light drinkers, moderate drinkers, problem drinkers, and heavy episodic drinkers. Predictions from the model were first compared with actual campus-level data and then used to predict the effects of several simulated interventions to address heavy episodic drinking. RESULTS: First, the model provides a reasonable fit of actual drinking styles of students attending Social Norms Marketing Research Project campuses varying by "wetness" and by drinking styles of matriculating students. Second, the model predicts that a combination of simulated interventions targeting heavy episodic drinkers at a moderately "dry" campus would extinguish heavy episodic drinkers, replacing them with light and moderate drinkers. Instituting the same combination of simulated interventions at a moderately "wet" campus would result in only a moderate reduction in heavy episodic drinkers (i.e., 50% to 35%). CONCLUSIONS: A simple, five-state compartmental model adequately predicted the actual drinking patterns of students from a variety of campuses surveyed in the Social Norms Marketing Research Project study. The model predicted the impact on drinking patterns of several simulated interventions to address heavy episodic drinking on various types of campuses.

Ecosystem Modeling of College Drinking: Parameter Estimation and Comparing Models to Data.

Recently we developed a model composed of five impulsive differential equations that describes the changes in drinking patterns (that persist at epidemic level) amongst college students. Many of the model parameters cannot be measured directly from data; thus, an inverse problem approach, which chooses the set of parameters that results in the "best" model to data fit, is crucial for using this model as a predictive tool. The purpose of this paper is to present the procedure and results of an unconventional approach to parameter estimation that we developed after more common approaches were unsuccessful for our specific problem. The results show that our model provides a good fit to survey data for 32 campuses. Using these parameter estimates, we examined the effect of two hypothetical intervention policies: 1) reducing environmental wetness, and 2) penalizing students who are caught drinking. The results suggest that reducing campus wetness may be a very effective way of reducing heavy episodic (binge) drinking on a college campus, while a policy that penalizes students who drink is not nearly as effective.

Statistical considerations and techniques for understanding physiological data, modeling, and treatments.

Comparing models with data always forces us to deal with uncertainty. This uncertainty may take many different forms and involve multiple scales of resolution in the model and in the experiment. In this paper, we discuss issues surrounding the development of deterministic dynamic models of mean behavior and the associated statistical models of the difference between model and experiment. We touch on a variety of topics, including basic exploratory data analysis, confidence bounds and model reduction hypothesis tests. Tools ranging from nonlinear regression to time series to Bayesian decision theory are presented.