Heterogeneity in effect size estimates.

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.

Variability in the analysis of a single neuroimaging dataset by many teams.

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs.

Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity-variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity-estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs-indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis.

Examining the generalizability of research findings from archival data.

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Meta-GWAS Accuracy and Power (MetaGAP) Calculator Shows that Hiding Heritability Is Partially Due to Imperfect Genetic Correlations across Studies.

Large-scale genome-wide association results are typically obtained from a fixed-effects meta-analysis of GWAS summary statistics from multiple studies spanning different regions and/or time periods. This approach averages the estimated effects of genetic variants across studies. In case genetic effects are heterogeneous across studies, the statistical power of a GWAS and the predictive accuracy of polygenic scores are attenuated, contributing to the so-called 'missing heritability'. Here, we describe the online Meta-GWAS Accuracy and Power (MetaGAP) calculator (available at www.devlaming.eu) which quantifies this attenuation based on a novel multi-study framework. By means of simulation studies, we show that under a wide range of genetic architectures, the statistical power and predictive accuracy provided by this calculator are accurate. We compare the predictions from the MetaGAP calculator with actual results obtained in the GWAS literature. Specifically, we use genomic-relatedness-matrix restricted maximum likelihood to estimate the SNP heritability and cross-study genetic correlation of height, BMI, years of education, and self-rated health in three large samples. These estimates are used as input parameters for the MetaGAP calculator. Results from the calculator suggest that cross-study heterogeneity has led to attenuation of statistical power and predictive accuracy in recent large-scale GWAS efforts on these traits (e.g., for years of education, we estimate a relative loss of 51-62% in the number of genome-wide significant loci and a relative loss in polygenic score R2 of 36-38%). Hence, cross-study heterogeneity contributes to the missing heritability.

Using prediction markets to estimate the reproducibility of scientific research.

Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.

Personality Polygenes, Positive Affect, and Life Satisfaction.

Approximately half of the variation in wellbeing measures overlaps with variation in personality traits. Studies of non-human primate pedigrees and human twins suggest that this is due to common genetic influences. We tested whether personality polygenic scores for the NEO Five-Factor Inventory (NEO-FFI) domains and for item response theory (IRT) derived extraversion and neuroticism scores predict variance in wellbeing measures. Polygenic scores were based on published genome-wide association (GWA) results in over 17,000 individuals for the NEO-FFI and in over 63,000 for the IRT extraversion and neuroticism traits. The NEO-FFI polygenic scores were used to predict life satisfaction in 7 cohorts, positive affect in 12 cohorts, and general wellbeing in 1 cohort (maximal N = 46,508). Meta-analysis of these results showed no significant association between NEO-FFI personality polygenic scores and the wellbeing measures. IRT extraversion and neuroticism polygenic scores were used to predict life satisfaction and positive affect in almost 37,000 individuals from UK Biobank. Significant positive associations (effect sizes <0.05%) were observed between the extraversion polygenic score and wellbeing measures, and a negative association was observed between the polygenic neuroticism score and life satisfaction. Furthermore, using GWA data, genetic correlations of -0.49 and -0.55 were estimated between neuroticism with life satisfaction and positive affect, respectively. The moderate genetic correlation between neuroticism and wellbeing is in line with twin research showing that genetic influences on wellbeing are also shared with other independent personality domains.

Molecular genetics and subjective well-being.

Subjective well-being (SWB) is a major topic of research across the social sciences. Twin and family studies have found that genetic factors may account for as much as 30-40% of the variance in SWB. Here, we study genetic contributions to SWB in a pooled sample of ≈ 11,500 unrelated, comprehensively-genotyped Swedish and Dutch individuals. We apply a recently developed method to estimate "common narrow heritability": the fraction of variance in SWB that can be explained by the cumulative additive effects of genetic polymorphisms that are common in the population. Our estimates are 5-10% for single-question survey measures of SWB, and 12-18% after correction for measurement error in the SWB measures. Our results suggest guarded optimism about the prospects of using genetic data in SWB research because, although the common narrow heritability is not large, the polymorphisms that contribute to it could feasibly be discovered with a sufficiently large sample of individuals.

The genetic architecture of economic and political preferences.

Preferences are fundamental building blocks in all models of economic and political behavior. We study a new sample of comprehensively genotyped subjects with data on economic and political preferences and educational attainment. We use dense single nucleotide polymorphism (SNP) data to estimate the proportion of variation in these traits explained by common SNPs and to conduct genome-wide association study (GWAS) and prediction analyses. The pattern of results is consistent with findings for other complex traits. First, the estimated fraction of phenotypic variation that could, in principle, be explained by dense SNP arrays is around one-half of the narrow heritability estimated using twin and family samples. The molecular-genetic-based heritability estimates, therefore, partially corroborate evidence of significant heritability from behavior genetic studies. Second, our analyses suggest that these traits have a polygenic architecture, with the heritable variation explained by many genes with small effects. Our results suggest that most published genetic association studies with economic and political traits are dramatically underpowered, which implies a high false discovery rate. These results convey a cautionary message for whether, how, and how soon molecular genetic data can contribute to, and potentially transform, research in social science. We propose some constructive responses to the inferential challenges posed by the small explanatory power of individual SNPs.

Replicability and robustness of genome-wide-association studies for behavioral traits.

A recent genome-wide-association study of educational attainment identified three single-nucleotide polymorphisms (SNPs) whose associations, despite their small effect sizes (each R (2) ≈ 0.02%), reached genome-wide significance (p < 5 × 10(-8)) in a large discovery sample and were replicated in an independent sample (p < .05). The study also reported associations between educational attainment and indices of SNPs called "polygenic scores." In three studies, we evaluated the robustness of these findings. Study 1 showed that the associations with all three SNPs were replicated in another large (N = 34,428) independent sample. We also found that the scores remained predictive (R (2) ≈ 2%) in regressions with stringent controls for stratification (Study 2) and in new within-family analyses (Study 3). Our results show that large and therefore well-powered genome-wide-association studies can identify replicable genetic associations with behavioral traits. The small effect sizes of individual SNPs are likely to be a major contributing factor explaining the striking contrast between our results and the disappointing replication record of most candidate-gene studies.

Genetic influences on political ideologies: twin analyses of 19 measures of political ideologies from five democracies and genome-wide findings from three populations.

Almost 40 years ago, evidence from large studies of adult twins and their relatives suggested that between 30 and 60% of the variance in social and political attitudes could be explained by genetic influences. However, these findings have not been widely accepted or incorporated into the dominant paradigms that explain the etiology of political ideology. This has been attributed in part to measurement and sample limitations, as well the relative absence of molecular genetic studies. Here we present results from original analyses of a combined sample of over 12,000 twins pairs, ascertained from nine different studies conducted in five democracies, sampled over the course of four decades. We provide evidence that genetic factors play a role in the formation of political ideology, regardless of how ideology is measured, the era, or the population sampled. The only exception is a question that explicitly uses the phrase "Left-Right". We then present results from one of the first genome-wide association studies on political ideology using data from three samples: a 1990 Australian sample involving 6,894 individuals from 3,516 families; a 2008 Australian sample of 1,160 related individuals from 635 families and a 2010 Swedish sample involving 3,334 individuals from 2,607 families. No polymorphisms reached genome-wide significance in the meta-analysis. The combined evidence suggests that political ideology constitutes a fundamental aspect of one's genetically informed psychological disposition, but as Fisher proposed long ago, genetic influences on complex traits will be composed of thousands of markers of very small effects and it will require extremely large samples to have enough power in order to identify specific polymorphisms related to complex social traits.

Limbic justice--amygdala involvement in immediate rejection in the Ultimatum Game.

Imaging studies have revealed a putative neural account of emotional bias in decision making. However, it has been difficult in previous studies to identify the causal role of the different sub-regions involved in decision making. The Ultimatum Game (UG) is a game to study the punishment of norm-violating behavior. In a previous influential paper on UG it was suggested that frontal insular cortex has a pivotal role in the rejection response. This view has not been reconciled with a vast literature that attributes a crucial role in emotional decision making to a subcortical structure (i.e., amygdala). In this study we propose an anatomy-informed model that may join these views. We also present a design that detects the functional anatomical response to unfair proposals in a subcortical network that mediates rapid reactive responses. We used a functional MRI paradigm to study the early components of decision making and challenged our paradigm with the introduction of a pharmacological intervention to perturb the elicited behavioral and neural response. Benzodiazepine treatment decreased the rejection rate (from 37.6% to 19.0%) concomitantly with a diminished amygdala response to unfair proposals, and this in spite of an unchanged feeling of unfairness and unchanged insular response. In the control group, rejection was directly linked to an increase in amygdala activity. These results allow a functional anatomical detection of the early neural components of rejection associated with the initial reactive emotional response. Thus, the act of immediate rejection seems to be mediated by the limbic system and is not solely driven by cortical processes, as previously suggested. Our results also prompt an ethical discussion as we demonstrated that a commonly used drug influences core functions in the human brain that underlie individual autonomy and economic decision making.

Swedish experience-based value sets for EQ-5D health states.

PURPOSE: To estimate Swedish experience-based value sets for EQ-5D health states using general population health survey data. METHODS: Approximately 45,000 individuals valued their current health status by means of time trade off (TTO) and visual analogue scale (VAS) methods and answered the EQ-5D questionnaire, making it possible to model the association between the experience-based TTO and VAS values and the EQ-5D dimensions and severity levels. The association between TTO and VAS values and the different severity levels of respondents' answers on a self-rated health (SRH) question was assessed. RESULTS: Almost all dimensions (except usual activity) and severity levels had less impact on TTO valuations compared with the UK study based on hypothetical values. Anxiety/depression had the greatest impact on both TTO and VAS values. TTO and VAS values were consistently related to SRH. The inclusion of age, sex, education and socioeconomic group affected the main effect coefficients and the explanatory power modestly. CONCLUSIONS: A value set for EQ-5D health states based on Swedish valuations has been lacking. Several authors have recently advocated the normative standpoint of using experience-based values. Guidelines of economic evaluation for reimbursement decisions in Sweden recommend the use of experience-based values for QALY calculations. Our results that anxiety/depression had the greatest impact on both TTO and VAS values underline the importance of mental health for individuals' overall HRQoL. Using population surveys is in line with recent thinking on valuing health states and could reduce some of the focusing effects potentially appearing in hypothetical valuation studies.

Subjective evidence evaluation survey for many-analysts studies.

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

The Swedish Twin Registry: establishment of a biobank and other recent developments.

The Swedish Twin Registry (STR) today contains more than 194,000 twins and more than 75,000 pairs have zygosity determined by an intra-pair similarity algorithm, DNA, or by being of opposite sex. Of these, approximately 20,000, 25,000, and 30,000 pairs are monozygotic, same-sex dizygotic, and opposite-sex dizygotic pairs, respectively. Since its establishment in the late 1950s, the STR has been an important epidemiological resource for the study of genetic and environmental influences on a multitude of traits, behaviors, and diseases. Following large investments in the collection of biological specimens in the past 10 years we have now established a Swedish twin biobank with DNA from 45,000 twins and blood serum from 15,000 twins, which effectively has also transformed the registry into a powerful resource for molecular studies. We here describe the main projects within which the new collections of both biological samples as well as phenotypic measures have been collected. Coverage by year of birth, zygosity determination, ethnic heterogeneity, and influences of in vitro fertilization are also described.

[Same data - different conclusions: Analytical flexibility in medical research can be considerable]. / Olika slutsatser från samma data.

Analysis of research data entails many choices. As a result, a space of different analytical strategies is open to researchers. Different justifiable analyses may not give similar results. The method of multiple analysts is a way to study the analytical flexibility and behaviour of researchers under naturalistic conditions, as part of the field known as metascience. Analytical flexibility and risks of bias can be counteracted by open data sharing, pre-registration of analysis plans, and registration of clinical trials in trial registers. These measures are particularly important for retrospective studies where analytical flexibility can be greatest, although pre-registration is less useful in this context. Synthetic datasets can be an alternative to pre-registration when used to decide what analyses should be conducted on real datasets by independent parties. All these strategies help build trustworthiness in scientific reports, and improve the reliability of research findings.

Genome-wide analysis identifies genetic effects on reproductive success and ongoing natural selection at the FADS locus.

Identifying genetic determinants of reproductive success may highlight mechanisms underlying fertility and identify alleles under present-day selection. Using data in 785,604 individuals of European ancestry, we identified 43 genomic loci associated with either number of children ever born (NEB) or childlessness. These loci span diverse aspects of reproductive biology, including puberty timing, age at first birth, sex hormone regulation, endometriosis and age at menopause. Missense variants in ARHGAP27 were associated with higher NEB but shorter reproductive lifespan, suggesting a trade-off at this locus between reproductive ageing and intensity. Other genes implicated by coding variants include PIK3IP1, ZFP82 and LRP4, and our results suggest a new role for the melanocortin 1 receptor (MC1R) in reproductive biology. As NEB is one component of evolutionary fitness, our identified associations indicate loci under present-day natural selection. Integration with data from historical selection scans highlighted an allele in the FADS1/2 gene locus that has been under selection for thousands of years and remains so today. Collectively, our findings demonstrate that a broad range of biological mechanisms contribute to reproductive success.

Does parametric fMRI analysis with SPM yield valid results? An empirical study of 1484 rest datasets.

The validity of parametric functional magnetic resonance imaging (fMRI) analysis has only been reported for simulated data. Recent advances in computer science and data sharing make it possible to analyze large amounts of real fMRI data. In this study, 1484 rest datasets have been analyzed in SPM8, to estimate true familywise error rates. For a familywise significance threshold of 5%, significant activity was found in 1%-70% of the 1484 rest datasets, depending on repetition time, paradigm and parameter settings. This means that parametric significance thresholds in SPM both can be conservative or very liberal. The main reason for the high familywise error rates seems to be that the global AR(1) auto correlation correction in SPM fails to model the spectra of the residuals, especially for short repetition times. The findings that are reported in this study cannot be generalized to parametric fMRI analysis in general, other software packages may give different results. By using the computational power of the graphics processing unit (GPU), the 1484 rest datasets were also analyzed with a random permutation test. Significant activity was then found in 1%-19% of the datasets. These findings speak to the need for a better model of temporal correlations in fMRI timeseries.