Variability in the analysis of a single neuroimaging dataset by many teams.

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Substitution of social sustainability concerns under the Covid-19 pandemic.

Think tanks and political leaders have raised concerns about the implications that the Covid-19 response and reconstruction might have on other social objectives that were setting the international agenda before the Covid-19 pandemic. We present evidence for eight consecutive weeks during April-May 2020 for Austria, testing the extent to which Covid-19 concerns substitute other social concerns such as the climate crisis or the protection of vulnerable sectors of the society. We measure behavior in a simple donation task where participants receive €3 that they can distribute between themselves and a list of charitable organizations, which vary between treatments. We consider initially a list of eight charities, including a broad set of social concerns. Results show that introducing the WHO Covid-19 Solidarity Response Fund significantly reduces the sum of donations to the original eight charities. This derives from two effects: First, introducing the Covid-19 Solidarity Response Fund does not significantly change aggregate donations. Second, results point to a high support to the WHO Covid-19 Fund. Overall, our results indicate that donations to diverse social concerns are partially substituted by donations to the Covid-19 fund; yet, this substitution does not fully replace all other social concerns. Results are robust to a 10-fold increase in endowment, with decisions made over €30.

Forecasting the publication and citation outcomes of COVID-19 preprints.

Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.

The risk elicitation puzzle revisited: Across-methods (in)consistency?

With the rise of experimental research in the social sciences, numerous methods to elicit and classify people's risk attitudes in the laboratory have evolved. However, evidence suggests that attitudes towards risk may vary considerably when measured with different methods. Based on a within-subject experimental design using four widespread risk preference elicitation tasks, we find that the different methods indeed give rise to considerably varying estimates of individual and aggregate level risk preferences. Conducting simulation exercises to obtain benchmarks for subjects' behavior, we find that the observed heterogeneity in risk preference estimates across methods is qualitatively similar to the heterogeneity arising from independent random draws from the choice distributions observed in the experiment. Our study, however, provides evidence that subjects are surprisingly well aware of the variation in the riskiness of their choices. We argue that this calls into question the common interpretation of variation in revealed risk preferences as being inconsistent.

Long Term Effects of the COVID-19 Pandemic on Social Concerns.

While some local, temporary past crises have boosted overall charitable donations, there have been concerns about potential substitution effects that the Covid-19 pandemic might have on other social objectives, such as tackling climate change and reducing inequality. We present results from a donation experiment (n = 1, 762), with data collected between April 2020 and January 2021. We combine data from (i) an online donation experiment, (ii) an extended questionnaire including perceptions, actions, and motives on the Covid-19 pandemic, the climate crisis, and poverty, as well as charitable behavior and (iii) epidemiological data. The experimental results show that donations to diverse social concerns are partially substituted by donations to the Covid-19 fund; yet, this substitution does not fully replace all other social concerns. Over time we observe no systematic trend in charitable donations. In regards to the determinants of individual donations, we observe that women donate more, people taking actions against Covid-19 and against poverty donate more, while those fearing risks from poverty donate less. In addition, we observe that the population under consideration is sensitive to the needs of others, enhancing total donations for higher Covid-19 incidence. For donations to each charity, we find that trusting a given charitable organization is the strongest explanatory factor of donations. JEL: L3, D64, Q54, I3, D9.

Consensus-based guidance for conducting and reporting multi-analyst studies.

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme.

The Defense Advanced Research Projects Agency (DARPA) programme 'Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.

fMRI data of mixed gambles from the Neuroimaging Analysis Replication and Prediction Study.

There is an ongoing debate about the replicability of neuroimaging research. It was suggested that one of the main reasons for the high rate of false positive results is the many degrees of freedom researchers have during data analysis. In the Neuroimaging Analysis Replication and Prediction Study (NARPS), we aim to provide the first scientific evidence on the variability of results across analysis teams in neuroscience. We collected fMRI data from 108 participants during two versions of the mixed gambles task, which is often used to study decision-making under risk. For each participant, the dataset includes an anatomical (T1 weighted) scan and fMRI as well as behavioral data from four runs of the task. The dataset is shared through OpenNeuro and is formatted according to the Brain Imaging Data Structure (BIDS) standard. Data pre-processed with fMRIprep and quality control reports are also publicly shared. This dataset can be used to study decision-making under risk and to test replicability and interpretability of previous results in the field.

Ethnical discrimination in Europe: Field evidence from the finance industry.

The integration of ethnical minorities has been a hotly discussed topic in the political, societal, and economic debate. Persistent discrimination of ethnical minorities can hinder successful integration. Given that unequal access to investment and financing opportunities can cause social and economic disparities due to inferior economic prospects, we conducted a field experiment on ethnical discrimination in the finance sector with 1,218 banks in seven European countries. We contacted banks via e-mail, either with domestic or Arabic sounding names, asking for contact details only. We find pronounced discrimination in terms of a substantially lower response rate to e-mails from Arabic senders. Remarkably, the observed discrimination effect is robust for loan- and investment-related requests, across rural and urban locations of banks, and across countries.

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015.

Being able to replicate scientific findings is crucial for scientific progress1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

Evaluating replicability of laboratory experiments in economics.

The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that were made publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.