Registration and primary outcome reporting in behavioral health trials.

BACKGROUND: Registration of research studies is designed to lock investigators into a data collection and analysis plan before a study starts and thereby limit their ability to engage in flexible data analysis and selective outcome reporting. Studies of registered clinical trials show that one- to two-thirds are registered after the study has started and that non-adherence to important design and analytic features, such as reporting data pertaining to all primary outcomes, remains high. Less is known about the effects of registration on research transparency and integrity outside of clinical trials. To address this gap in knowledge, the current study examined the effects of registration on the reporting of research findings in a sample of behavioral health trials published in BMC Public Health. METHODS: Registered trials published in the BMC Public Health section "Health Behavior, Health Promotion and Society" between 2011 and 2015 were included in the study. For each trial, we reviewed associated online submissions from 13 different registration sites. For those determined to have been prospectively registered, we used the trial registry, MEDLINE (Pubmed), PsychINFO, Web of Science and e-mails to investigators to identify subsequent publications from the study that reported results pertaining to primary outcomes. The two investigators then independently reviewed the outcome publication(s) and compared the primary outcomes reported in these to the registered primary outcomes. RESULTS: The final analytic sample comprised 136 locatable, registered trials with an identifiable start date. Sixty-eight of the 136 were prospectively registered. Among these prospectively registered trials, only 16 published manuscripts reported outcomes and methods that were concordant with their registrations. CONCLUSIONS: Retrospective submission of protocols for publication and retrospective registration remain common in public health research, and adherence to prespecified outcomes is rare. In its current form, registration of behavioral and health promotion trials is likely to have minimal effect on preventing selective outcome reporting in publications, and the pervasiveness of vague and incomplete registry entries means that registries will have limited utility in terms of facilitating replication studies.

Availability of Research Data in High-Impact Addiction Journals with Data Sharing Policies.

Although data sharing is one of the primary measures proposed to improve the integrity and quality of published research, studies show it remains the exception not the rule. The current study examines the availability of data in papers reporting the results of analyses of empirical data from original research in high-impact addiction journals. Thirteen high-impact journals with data sharing policies were selected from those included in the substance abuse category of the 2018 Clarivate Analytics' Journal Citation Report. The first 10 full or short original research reports that included empirical data in the most recent complete issue of each journal were electronically searched and reviewed for reference to where their data can be obtained and for a formal data sharing statement. Only eight of the 130 papers contained a data sharing statement in their text or supplementary online materials, and just one contained a direct link to the data analyzed. Data sharing was rare in the 13 high-impact addiction journals reviewed. The nature of the data reported in addiction journals might partly explain this. Currently, data sharing is not a procedure likely to improve the quality and integrity of published addiction research.

A Systems Approach to Understanding and Improving Research Integrity.

Concern about the integrity of empirical research has arisen in recent years in the light of studies showing the vast majority of publications in academic journals report positive results, many of these results are false and cannot be replicated, and many positive results are the product of data dredging and the application of flexible data analysis practices coupled with selective reporting. While a number of potential solutions have been proposed, the effects of these are poorly understood and empirical evaluation of each would take many years. We propose that methods from the systems sciences be used to assess the effects, both positive and negative, of proposed solutions to the problem of declining research integrity such as study registration, Registered Reports, and open access to methods and data. In order to illustrate the potential application of systems science methods to the study of research integrity, we describe three broad types of models: one built on the characteristics of specific academic disciplines; one a diffusion of research norms model conceptualizing researchers as susceptible, "infected" and recovered; and one conceptualizing publications as a product produced by an industry comprised of academics who respond to incentives and disincentives.

Can We Trust Positive Findings of Intervention Research? The Role of Conflict of Interest.

In recent years, there has been increased attention to the issue of conflict of interest within prevention research. The aims of this paper are to discuss these developments and to relate them to discussions of conflict of interest in the broader scientific literature. Although there has been concern expressed about the extent to which conflicts of interest can be defined and measured, empirical research suggests that financial conflicts can be easily identified and assessed in meta-analyses focused on their effects on research quality. Research evidence also shows that conflict of interest is associated with use of flexible data analysis practices and the reporting of chance positive findings, both within prevention research and related disciplines such as public health and psychology. However, the overwhelming majority of published studies report positive results, and there are a number of other influences within academia (such as pressure to publish) that account for this and for the use of flexible data analysis practices. Accordingly, introducing measures to improve research quality in general, rather than just focusing on problems specific to research in which there is a clearly identifiable conflict of interest, may prove more effective and less controversial. Most such efforts focus on introducing greater transparency into research design, practice, and reporting. These both curtail employment of flexible data analysis practices and make their use transparent to investigators seeking to assess their effects on research quality. Also, requiring detailed disclosures of conflicts be reported by all investigators (not just senior authors) would improve current disclosure practices.

Is Project Towards No Drug Abuse (Project TND) an evidence-based drug and violence prevention program? A review and reappraisal of the evaluation studies.

This paper critically reviews the published evidence pertaining to Project Towards No Drug Abuse (Project TND). Publications from seven evaluation studies of Project TND are reviewed, and the results from these are discussed as related to the following outcomes: main effects on the use of cigarettes, alcohol and marijuana; main effects on the use of "hard drugs," defined in the evaluations as cocaine, hallucinogens, stimulants, inhalants, ecstasy and other drugs (e.g., depressants, PCP, steroids and heroin); subgroup and interaction analyses of drug use; and violence-related behaviors. Very few main effects have been found for cigarette, alcohol and marijuana use in the Project TND evaluations. While studies do report main effects for hard drug use, these findings are subject to numerous threats to validity and may be attributable to the data analyses employed. Similarly, while isolated subgroup and interaction effects were found for alcohol use among baseline nonusers and some violence-related behaviors in the early Project TND evaluations, these findings have not been replicated in more recent studies and may result from multiple comparisons between study conditions. In conclusion, there is little evidence to support the assertion that Project TND is an effective drug or violence prevention program. The broader implications of these findings for prevention science are discussed and suggestions are made as to how the quality of research in the field might be improved.

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model.

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote "statistical significance" and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

Mapping the spread of methamphetamine abuse in California from 1995 to 2008.

OBJECTIVES: From 1983 to 2008, the incidence of methamphetamine abuse and dependence (MA) presenting at hospitals in California increased 13-fold. We assessed whether this growth could be characterized as a drug epidemic. METHODS: We geocoded MA discharges to residential zip codes from 1995 through 2008. We related discharges to population and environmental characteristics using Bayesian Poisson conditional autoregressive models, correcting for small area effects and spatial misalignment and enabling an assessment of contagion between areas. RESULTS: MA incidence increased exponentially in 3 phases interrupted by implementation of laws limiting access to methamphetamine precursors. MA growth from 1999 through 2008 was 17% per year. MA was greatest in areas with larger White or Hispanic low-income populations, small household sizes, and good connections to highway systems. Spatial misalignment was a source of bias in estimated effects. Spatial autocorrelation was substantial, accounting for approximately 80% of error variance in the model. CONCLUSIONS: From 1995 through 2008, MA exhibited signs of growth and spatial spread characteristic of drug epidemics, spreading most rapidly through low-income White and Hispanic populations living outside dense urban areas.

Evaluating the effects of the introduction of off-sale alcohol outlets on violent crime.

AIMS: To examine the effects on violence of a policy change that ended prohibition of off-sale alcohol outlets in Lubbock, Texas. METHODS: Time-series analysis of violent crime data from police records comparing the periods before and after the policy change. RESULTS: The effect of the policy change on both total violent crime and aggregated assault was small and did not approach statistical significance. CONCLUSIONS: Increased availability of alcohol through off-sale premises may not influence the type of violence reported to the police in Lubbock, Texas.

PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews.

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

Ranking of addiction journals in eight widely used impact metrics.

Background and aims: Journal metrics assess impact upon the research literature, and are now used to assess individual researchers in hiring and promotion decisions. This study compared the ranking of addiction journals according to eight widely used metrics; assessed the correlations between journal rankings; and assessed changes over time in metric scores. Methods: Data pertaining to the 2020 scores on eight metrics for 43 journals were obtained and the top 20 ranking in each compared and the correlations between rankings assessed. The Impact Factor was employed to assess changes over time. Results: Ignoring the two categorization systems used by some metrics, 31 journals appeared in at least one metric top 20 and 11 in all eight. The top rank in each was occupied by one of three journals. Three-quarters of the correlations between rankings were above 6.0. The number of journals with an Impact Factor rose from 23 in 1997 to 38 in 2020, and the journals added tended to focus on addictions other than alcohol and drugs or have a specific focus. Conclusions and discussion: The results indicate a concentration of journals at the top of the metrics and moderate to strong agreement between them, but almost three-quarters of journals appeared in at least one metric. The longitudinal data reflect both a broadening and specialization of the addiction field. The study limitations include exclusion of some journals and metrics.

The Campbell Collaboration's systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards.

BACKGROUND: Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration's mandatory methodological standards. ISSUES: In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review. CONCLUSIONS: Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

The Making of Evidence-based Practice: The Case of Project ALERT.

Evidence-based practice has been enthusiastically embraced within the field of drug prevention during the past decade. Project ALERT, a school-based universal prevention program, is among the most widely advocated evidence-based interventions. We examined the results of three large-scale evaluations of Project ALERT, and concluded that assessment of data from the total samples shows that the program has little effect on drug use. Despite this, Project ALERT is included on evidence-based drug prevention lists because the criteria for inclusion are extremely weak. We discuss the implications of this for drug prevention evaluation research and the creation of evidence-based practice lists.

Impact of Relative Residence Times in Highly Distinct Environments on the Distribution of Heavy Drinkers.

Alcohol consumption is a function of social dynamics, environmental contexts, individuals' preferences and family history. Empirical surveys have focused primarily on identification of risk factors for high-level drinking but have done little to clarify the underlying mechanisms at work. Also, there have been few attempts to apply nonlinear dynamics to the study of these mechanisms and processes at the population level. A simple framework where drinking is modeled as a socially contagious process in low- and high-risk connected environments is introduced. Individuals are classified as light, moderate (assumed mobile), and heavy drinkers. Moderate drinkers provide the link between both environments, that is, they are assumed to be the only individuals drinking in both settings. The focus here is on the effect of moderate drinkers, measured by the proportion of their time spent in "low-" versus "high-" risk drinking environments, on the distribution of drinkers.A simple model within our contact framework predicts that if the relative residence times of moderate drinkers is distributed randomly between low- and high-risk environments then the proportion of heavy drinkers is likely to be higher than expected. However, the full story even in a highly simplified setting is not so simple because "strong" local social mixing tends to increase high-risk drinking on its own. High levels of social interaction between light and moderate drinkers in low-risk environments can diminish the importance of the distribution of relative drinking times on the prevalence of heavy drinking.

High impact nutrition and dietetics journals' use of publication procedures to increase research transparency.

BACKGROUND: The rigor and integrity of the published research in nutrition studies has come into serious question in recent years. Concerns focus on the use of flexible data analysis practices and selective reporting and the failure of peer review journals to identify and correct these practices. In response, it has been proposed that journals employ editorial procedures designed to improve the transparency of published research. OBJECTIVE: The present study examines the adoption of editorial procedures designed to improve the reporting of empirical studies in the field of nutrition and dietetics research. DESIGN: The instructions for authors of 43 journals included in Quartiles 1 and 2 of the Clarivate Analytics' 2018 Journal Citation Report category Nutrition and Dietetics were reviewed. For journals that published original research, conflict of interest disclosure, recommendation of reporting guidelines, registration of clinical trials, registration of other types of studies, encouraging data sharing, and use of the Registered Reports were assessed. For journals that only published reviews, all of the procedures except clinical trial registration were assessed. RESULTS: Thirty-three journals published original research and 10 published only reviews. Conflict of interest disclosure was required by all 33 original research journals. Use of guidelines, trial registration and encouragement of data sharing were mentioned by 30, 27 and 25 journals, respectively. Registration of other studies was required by eight and none offered Registered Reports as a publication option at the time of the review. All 10 review journals required conflict of interest disclosure, four recommended data sharing and three the use of guidelines. None mentioned the other two procedures. CONCLUSIONS: While nutrition journals have adopted a number of procedures designed to improve the reporting of research findings, their limited effects likely result from the mechanisms through which they influence analytic flexibility and selective reporting and the extent to which they are properly implemented and enforced by journals.

The social construction of "evidence-based'' drug prevention programs: a reanalysis of data from the Drug Abuse Resistance Education (DARE) program.

This study explores the possibility that any drug prevention program might be considered ;;evidence-based'' given the use of data analysis procedures that optimize the chance of producing statistically significant results by reanalyzing data from a Drug Abuse Resistance Education (DARE) program evaluation. The analysis produced a number of statistically significant differences between the DARE and control conditions on alcohol and marijuana use measures. Many of these differences occurred at cutoff points on the assessment scales for which post hoc meaningful labels were created. Our results are compared to those from evaluations of programs that appear on evidence-based drug prevention lists.

Use of publication procedures to improve research integrity by addiction journals.

BACKGROUND AND AIMS: The credibility crisis evident in many academic disciplines has led peer-reviewed journals to implement procedures to reduce use of flexible data analysis practices and selective reporting of results. This exploratory study examined the adoption of six of these procedures by addiction journals. METHODS: Thirty-eight high-impact addiction journals were identified using the 2018 Clarivate Analytics Journal Citation Report for 2017 ranks. The online instructions for authors were reviewed for references to six publication procedures: conflict of interest disclosure, reporting guidelines, clinical trial registration, registration of other study designs, data-sharing and registered reports. The webpages of the Center for Open Science and Consolidated Standards of Reporting Trials (CONSORT) were also reviewed for data pertaining to registered reports and reporting guidelines, respectively. RESULTS: The range of procedures adopted by the addiction journals was 0-5, with a mean of 2.66. Conflict-of-interest disclosure was required by all but one journal. Encouraging data-sharing was the next most commonly required procedure. Fewer than half the journals recommended specific reporting guidelines or required registration of clinical trials, and only four required procedures to pre-specify hypotheses and analytical methods. CONCLUSIONS: While many addiction journals have adopted publication procedures to improve research integrity, these can be limited by their voluntary nature and monitoring difficulties. More stringent requirements that lock researchers into specific hypotheses and analyses have not been widely adopted.

Violent crime redistribution in a city following a substantial increase in the number of off-sale alcohol outlets: A Bayesian analysis.

INTRODUCTION AND AIMS: This study examined whether the introduction of a large number of off-premise alcohol outlets into a city over a brief period of time could affect rates of violent crime. DESIGN AND METHODS: The study analysed annual counts of violent crime across 172 US Census block groups in Lubbock, Texas from 2006 through 2011. Spatial Poisson models related annual violent crime counts within each block group to off-premise and on-premise alcohol outlets active during this time period as well as neighbourhood socio-demographic characteristics. The effects of alcohol outlets were assessed both within block groups and across adjacent block groups. RESULTS: On-premise outlets had a small, significant positive association with violence within a given block group. A similar well-supported local effect for off-premise outlets was not found. However, the spatially lagged effect for off-sale premises was well-supported, indicating that greater densities of these outlets were related to greater rates of violent crime in adjacent areas. DISCUSSION AND CONCLUSIONS: While these analyses confirmed a previous time-series analysis in finding no city-wide effect of the increase in off-premise outlets, they do suggest that such outlets in a local area may be related to violence in nearby geographic areas. They indicate the importance of examining neighbourhood-specific effects of alcohol outlets on violence in addition to the city-wide effects. They also present further evidence supporting the need to examine the differential effects of on-sale and off-sale premises.