Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Data from a pre-publication independent replication initiative examining ten moral judgement effects.

We present the data from a crowdsourced project seeking to replicate findings in independent laboratories before (rather than after) they are published. In this Pre-Publication Independent Replication (PPIR) initiative, 25 research groups attempted to replicate 10 moral judgment effects from a single laboratory's research pipeline of unpublished findings. The 10 effects were investigated using online/lab surveys containing psychological manipulations (vignettes) followed by questionnaires. Results revealed a mix of reliable, unreliable, and culturally moderated findings. Unlike any previous replication project, this dataset includes the data from not only the replications but also from the original studies, creating a unique corpus that researchers can use to better understand reproducibility and irreproducibility in science.

Diverse According to Whom? Racial group membership and concerns about discrimination shape diversity judgments.

People often treat diversity as an objective feature of situations that everyone perceives similarly. The current research shows, however, that disagreement often exists over whether a group is diverse. We argue that diversity judgments diverge because they are social perceptions that reflect, in part, individuals' motivations and experiences, including concerns about how a group would treat them. Therefore, whether a group includes in-group members should affect how diverse a group appears because the inclusion or apparent exclusion of in-group members signals whether perceivers can expect to be accepted and treated fairly. Supporting our claims, three experiments demonstrate that racial minority group members perceive more diversity when groups included racial in-group members rather than members of other racial minority groups. Moreover, important differences exist between Asian Americans and African Americans, which underscore the need for more research to explore uniqueness rather than commonalities across racial minority groups.

Limits on legitimacy: moral and religious convictions as constraints on deference to authority.

Various versions of legitimacy theory predict that a duty and obligation to obey legitimate authorities generally trumps people's personal moral and religious values. However, most research has assumed rather than measured the degree to which people have a moral or religious stake in the situations studied. This study tested compliance with and reactions to legitimate authorities in the context of a natural experiment that tracked public opinion before and after the U.S. Supreme Court ruled in a case that challenged states' rights to legalize physician-assisted suicide. Results indicated that citizens' degree of moral conviction about the issue of physician-assisted suicide predicted post-ruling perceptions of outcome fairness, decision acceptance, and changes in perceptions of the Court's legitimacy from pre- to post-ruling. Other results revealed that the effects of religious conviction independently predicted outcome fairness and decision acceptance but not perceptions of post-ruling legitimacy.

Moral conviction: another contributor to attitude strength or something more?

Attitudes held with strong moral conviction (moral mandates) were predicted to have different interpersonal consequences than strong but nonmoral attitudes. After controlling for indices of attitude strength, the authors explored the unique effect of moral conviction on the degree that people preferred greater social (Studies 1 and 2) and physical (Study 3) distance from attitudinally dissimilar others and the effects of moral conviction on group interaction and decision making in attitudinally homogeneous versus heterogeneous groups (Study 4). Results supported the moral mandate hypothesis: Stronger moral conviction led to (a) greater preferred social and physical distance from attitudinally dissimilar others, (b) intolerance of attitudinally dissimilar others in both intimate (e.g., friend) and distant relationships (e.g., owner of a store one frequents), (c) lower levels of good will and cooperativeness in attitudinally heterogeneous groups, and (d) a greater inability to generate procedural solutions to resolve disagreements.

Political tolerance and coming to psychological closure following the September 11, 2001, terrorist attacks: an integrative approach.

This study tested hypotheses generated from an integrative model of political tolerance that derived hypotheses from a number of different social psychological theories (e.g., appraisal tendency theory, intergroup emotion theory, and value protection models) to explain political tolerance following the September 11, 2001, terrorist attacks. A national field study (N = 550) found that immediate post attack anger and fear had different implications for political tolerance 4 months later. The effects of anger on political tolerance were mediated through moral outrage and outgroup derogation, whereas the effects of fear on political tolerance were mediated through personal threat, ingroup enhancement, and value affirmation. Value affirmation led to increased political tolerance, whereas moral outrage, outgroup derogation, ingroup enhancement, and personal threat led to decreased political tolerance. Value affirmation, moral outrage, and outgroup derogation also facilitated post-9/11 psychological closure and increased psychological closure led to greater political tolerance.