RESUMO
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
RESUMO
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Assuntos
Análise de Dados , Ciência de Dados/métodos , Ciência de Dados/normas , Conjuntos de Dados como Assunto , Neuroimagem Funcional , Imageamento por Ressonância Magnética , Pesquisadores/organização & administração , Encéfalo/diagnóstico por imagem , Encéfalo/fisiologia , Conjuntos de Dados como Assunto/estatística & dados numéricos , Feminino , Humanos , Modelos Logísticos , Masculino , Metanálise como Assunto , Modelos Neurológicos , Reprodutibilidade dos Testes , Pesquisadores/normas , SoftwareRESUMO
Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity-variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity-estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs-indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis.
RESUMO
We investigate how the experience of extreme events, such as the COVID-19 market crash, influence risk-taking behavior. To isolate changes in risk-taking from other factors, we ran controlled experiments with finance professionals in December 2019 and March 2020. We observe that their investments in the experiment were 12 percent lower in March 2020 than in December 2019, although their price expectations had not changed, and although they considered the experimental asset less risky during the crash than before. This lower perceived risk is likely due to adaptive normalization, as volatility during the shock is compared to volatility experienced in real markets (which was low in December 2019, but very high in March 2020). Lower investments during the crash can be supported by higher risk aversion, not by changes in beliefs.
Assuntos
Comportamento Cooperativo , Tomada de Decisões , Ego , Teoria dos Jogos , Doações , Comportamento Impulsivo , Intuição , Modelos Psicológicos , Feminino , Humanos , MasculinoRESUMO
Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
RESUMO
Incentivized experiments in which individuals receive monetary rewards according to the outcomes of their decisions are regarded as the gold standard for preference elicitation in experimental economics. These task-related real payments are considered necessary to reveal subjects' "true preferences." Using a systematic, large-sample approach with three subject pools of private investors, professional investors, and students, we test the effect of task-related monetary incentives on risk preferences in four standard experimental tasks. We find no significant differences in behavior between and within subjects in the incentivized and non-incentivized regimes. We discuss implications for academic research and forions in the field. Supplementary Information: The online version contains supplementary material available at 10.1007/s11166-022-09377-w.
RESUMO
As the introduction of financial transaction taxes is increasingly discussed by political leaders we explore possible consequences such taxes could have on markets. Here we examine how "stylized facts", namely fat tails and volatility clustering, are affected by different tax regimes in laboratory experiments. We find that leptokurtosis of price returns is highest and clustered volatility is weakest in unilaterally taxed markets (where tax havens exist). Instead, tails are slimmest and volatility clustering is strongest in tax havens. When an encompassing financial transaction tax is levied, stylized facts hardly change compared to a scenario with no tax on all markets.
RESUMO
Background: COVID-19 booster vaccine uptake rates are behind the rate of primary vaccination in many countries. Governments and non-governmental institutions rely on a range of interventions aiming to increase booster uptake. Yet, little is known how experts and the general public evaluate these interventions. Methods: We applied a novel crowdsourcing approach to provide rapid insights on the most promising interventions to promote uptake of COVID-19 booster vaccines. In the first phase (December 2021), international experts (n = 78 from 17 countries) proposed 46 unique interventions. To reduce noise and potential bias, in the second phase (January 2022), experts (n = 307 from 34 countries) and representative general population samples from the UK (n = 299) and the US (n = 300) rated the proposed interventions on several evaluation criteria, including effectiveness and acceptability, on a 5-point Likert-type scale. Findings: Sanctions were evaluated as potentially most effective but least accepted. Evaluations by expert and general population samples were considerably aligned. Interventions that received the most positive evaluations regarding both effectiveness and acceptability across evaluation groups were: a day off work after getting vaccinated, financial incentives, tax benefits, promotional campaigns, and mobile vaccination teams. Interpretation: The results provide useful insights to help governmental and non-governmental institutions in their decisions about which interventions to implement. Additionally, the applied crowdsourcing method may be used in future studies to retrieve rapid insights on the comparative evaluation of (health) policies. Funding: This study received funding from the Austrian Science Fund (SFB F63) and the University of Vienna.
RESUMO
TRADING IN FX MARKETS IS DOMINATED BY TWO MICROSTRUCTURES: exchanges with market makers and OTC-markets without market makers. Using laboratory experiments we test whether the impact of a Tobin tax is different in these two market microstructures. We find that (i) in markets without market makers an unilaterally imposed Tobin tax (i.e. a tax haven exists) increases volatility. (ii) In contrast, in markets with market makers we observe a decrease in volatility in unilaterally taxed markets. (iii) An encompassing Tobin tax has no impact on volatility in either setting. Efficiency does not vary significantly across tax regimes.
RESUMO
Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
Assuntos
Consenso , Análise de Dados , Conjuntos de Dados como Assunto , PesquisaRESUMO
There is a heated debate on whether markets erode social responsibility and moral behavior. However, it is a challenging task to identify and measure moral behavior in markets. Based on a theoretical model, we examine in an experiment the relation between trading volume, prices and moral behavior by setting up markets that either impose a negative externality on third parties or not. We find that moral behavior reveals itself in lower trading volume in markets with a negative externality, while prices mostly depend on the market structure. We further investigate individual characteristics that explain trading behavior in markets with negative externalities.
RESUMO
We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
Assuntos
Laboratórios , Pesquisa , Ciências Sociais , Algoritmos , Modelos Estatísticos , Curva ROC , Análise de Regressão , Reprodutibilidade dos TestesRESUMO
There is an ongoing debate about the replicability of neuroimaging research. It was suggested that one of the main reasons for the high rate of false positive results is the many degrees of freedom researchers have during data analysis. In the Neuroimaging Analysis Replication and Prediction Study (NARPS), we aim to provide the first scientific evidence on the variability of results across analysis teams in neuroscience. We collected fMRI data from 108 participants during two versions of the mixed gambles task, which is often used to study decision-making under risk. For each participant, the dataset includes an anatomical (T1 weighted) scan and fMRI as well as behavioral data from four runs of the task. The dataset is shared through OpenNeuro and is formatted according to the Brain Imaging Data Structure (BIDS) standard. Data pre-processed with fMRIprep and quality control reports are also publicly shared. This dataset can be used to study decision-making under risk and to test replicability and interpretability of previous results in the field.
Assuntos
Encéfalo/diagnóstico por imagem , Neuroimagem , Encéfalo/fisiologia , Mapeamento Encefálico , Humanos , Processamento de Imagem Assistida por Computador/métodos , Imageamento por Ressonância Magnética/métodos , Valor Preditivo dos TestesRESUMO
Reciprocation of monetary gifts is well-understood in economics. In contrast, there is little research on reciprocal behavior following immaterial gifts like compliments. We narrow this gap and investigate how employees reciprocate after receiving immaterial gifts and material gifts over time. We purchase (1) ice cream from fast food restaurants, and (2) durum doner, a common lunch snack, from independent vendors. Prior to the food's preparation, we either compliment or tip the salesperson. We find that salespersons reciprocate compliments with higher product weight than in a control treatment. Importantly, this reciprocal behavior following immaterial gifts grows over repeated transactions. Tips, in contrast, have a stronger level effect which does not change over time.
RESUMO
The integration of ethnical minorities has been a hotly discussed topic in the political, societal, and economic debate. Persistent discrimination of ethnical minorities can hinder successful integration. Given that unequal access to investment and financing opportunities can cause social and economic disparities due to inferior economic prospects, we conducted a field experiment on ethnical discrimination in the finance sector with 1,218 banks in seven European countries. We contacted banks via e-mail, either with domestic or Arabic sounding names, asking for contact details only. We find pronounced discrimination in terms of a substantially lower response rate to e-mails from Arabic senders. Remarkably, the observed discrimination effect is robust for loan- and investment-related requests, across rural and urban locations of banks, and across countries.
Assuntos
Economia , Etnicidade , Indústrias , Preconceito , Europa (Continente) , HumanosRESUMO
Being able to replicate scientific findings is crucial for scientific progress1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
Assuntos
Reprodutibilidade dos Testes , Pesquisa/estatística & dados numéricos , Ciências Sociais/estatística & dados numéricos , Teorema de Bayes , Humanos , Publicações Periódicas como Assunto/estatística & dados numéricos , Tamanho da Amostra , Ciências Sociais/métodosRESUMO
The effects of a financial transaction tax (FTT) are scientifically disputed, as seemingly small details of its implementation may matter a lot. In this article, we provide experimental evidence on the different effects of an FTT, depending on whether it is implemented as a tax on markets, on residents, or a combination of both. We find that a tax on markets has negative effects on volatility and trading volume, whereas a tax on residents shows none of these undesired effects. Additionally, we observe that individual risk attitude is not related to traders' reaction to the different forms of an FTT.
RESUMO
We experimentally compare fast and slow decisions in a series of experiments on financial risk taking in three countries involving over 1700 subjects. To manipulate fast and slow decisions, subjects were randomly allocated to responding within 7 seconds (time pressure) or waiting for at least 7 or 20 seconds (time delay) before responding. To control for different effects of time pressure and time delay on measurement noise, we estimate separate parameters for noise and risk preferences within a random utility framework. We find that time pressure increases risk aversion for gains and risk taking for losses compared to time delay, implying that time pressure increases the reflection effect of Prospect Theory. The results for gains are weaker and less robust than the results for losses. We find no significant difference between time pressure and time delay for loss aversion (tested in only one of the experiments). Time delay also leads to less measurement noise than time pressure and unconstrained decisions, and appears to be an effective way of decreasing noise in experiments.
RESUMO
Do individuals intuitively favor certain moral actions over others? This study explores the role of intuitive thinking-induced by time pressure and cognitive load-in moral judgment and behavior. We conduct experiments in three different countries (Sweden, Austria, and the United States) involving over 1,400 subjects. All subjects responded to four trolley type dilemmas and four dictator games involving different charitable causes. Decisions were made under time pressure/time delay or while experiencing cognitive load or control. Overall we find converging evidence that intuitive states do not influence moral decisions. Neither time-pressure nor cognitive load had any effect on moral judgments or altruistic behavior. Thus we find no supporting evidence for the claim that intuitive moral judgments and dictator game giving differ from more reflectively taken decisions. Across all samples and decision tasks men were more likely to make utilitarian moral judgments and act selfishly compared to women, providing further evidence that there are robust gender differences in moral decision-making. However, there were no significant interactions between gender and the treatment manipulations of intuitive versus reflective decision-making.