Exploring a formal approach to selecting studies for replication: A feasibility study in social neuroscience.

Replication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility - a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van 't Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed.

The process of replication target selection in psychology: what to consider?

Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of individuals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.

Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.

Robust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The Language of Social Touch Is Intuitive and Quantifiable.

Touch is a powerful communication tool, but we have a limited understanding of the role played by particular physical features of interpersonal touch communication. In this study, adults living in Sweden performed a task in which messages (attention, love, happiness, calming, sadness, and gratitude) were conveyed by a sender touching the forearm of a receiver, who interpreted the messages. Two experiments (N = 32, N = 20) showed that within close relationships, receivers could identify the intuitive touch expressions of the senders, and we characterized the physical features of the touches associated with successful communication. Facial expressions measured with electromyography varied by message but were uncorrelated with communication performance. We developed standardized touch expressions and quantified the physical features with 3D hand tracking. In two further experiments (N = 20, N = 16), these standardized expressions were conveyed by trained senders and were readily understood by strangers unacquainted with the senders. Thus, the possibility emerges of a standardized, intuitively understood language of social touch.

Why Hypothesis Testers Should Spend Less Time Testing Hypotheses.

For almost half a century, Paul Meehl educated psychologists about how the mindless use of null-hypothesis significance tests made research on theories in the social sciences basically uninterpretable. In response to the replication crisis, reforms in psychology have focused on formalizing procedures for testing hypotheses. These reforms were necessary and influential. However, as an unexpected consequence, psychological scientists have begun to realize that they may not be ready to test hypotheses. Forcing researchers to prematurely test hypotheses before they have established a sound "derivation chain" between test and theory is counterproductive. Instead, various nonconfirmatory research activities should be used to obtain the inputs necessary to make hypothesis tests informative. Before testing hypotheses, researchers should spend more time forming concepts, developing valid measures, establishing the causal relationships between concepts and the functional form of those relationships, and identifying boundary conditions and auxiliary assumptions. Providing these inputs should be recognized and incentivized as a crucial goal in itself. In this article, we discuss how shifting the focus to nonconfirmatory research can tie together many loose ends of psychology's reform movement and help us to develop strong, testable theories, as Paul Meehl urged.

Improving Inferences About Null Effects With Bayes Factors and Equivalence Tests.

Researchers often conclude an effect is absent when a null-hypothesis significance test yields a nonsignificant p value. However, it is neither logically nor statistically correct to conclude an effect is absent when a hypothesis test is not significant. We present two methods to evaluate the presence or absence of effects: Equivalence testing (based on frequentist statistics) and Bayes factors (based on Bayesian statistics). In four examples from the gerontology literature, we illustrate different ways to specify alternative models that can be used to reject the presence of a meaningful or predicted effect in hypothesis tests. We provide detailed explanations of how to calculate, report, and interpret Bayes factors and equivalence tests. We also discuss how to design informative studies that can provide support for a null model or for the absence of a meaningful effect. The conceptual differences between Bayes factors and equivalence tests are discussed, and we also note when and why they might lead to similar or different inferences in practice. It is important that researchers are able to falsify predictions or can quantify the support for predicted null effects. Bayes factors and equivalence tests provide useful statistical tools to improve inferences about null effects.

Evidence for magnitude representations of social hierarchies: Size and distance effects.

Social status is often metaphorically construed in terms of spatial relations such as height, size, and numerosity. This has led to the idea that social status might partially be represented by an analogue magnitude system, responsible for processing the magnitude of various physical and abstract dimensions. Accordingly, processing of social status should obey Weber's law. We conducted three studies to investigate whether social status comparisons would indicate behavioral outcomes derived from Weber's law: the distance effect and the size effect. Dependent variable was the latency of status comparisons for a variety of both learned and familiar hierarchies. As predicted and in line with previous findings, we observed a clear distance effect. However, the effect of size variation differed from the size effect hypothesized a priori, and an unexpected interaction between the two effects was observed. In conclusion, we provide a robust confirmation of previous observations of the distance effect in social status comparisons, but the shape of the size effect requires new theorizing.

Morphine reduced perceived anger from neutral and implicit emotional expressions.

The µ-opioid system modulates responses to pain and psychosocial stress and mediates non-social and social reward. In humans, the µ-opioid agonist morphine can increase overt attention to the eye-region and visual exploration of faces with neutral expressions. However, little is known about how the human µ-opioid system influences sensitivity to and appraisal of subtle and explicit cues of social threats and reward. Here, we examined the effects of selective µ-opioid stimulation on perception of anger and happiness in faces with explicit, neutral or implicit emotion expressions. Sixty-three healthy adults (32 females) attended two sessions where they received either placebo or 10 mg per oral morphine in randomised order under double-blind conditions. Based on the known µ-opioid reduction of pain and discomfort, as well as reports suggesting that the non-specific partial agonist buprenorphine or the non-specific antagonist naltrexone affect appraisal of social emotional stimuli, we hypothesised that morphine would reduce threat sensitivity and enhance perception of happy facial expressions. While overall perception of others' happiness was unaffected by morphine treatment, morphine reduced perception of anger in stimuli with neutral and implicit expressions without affecting perception of explicit anger. This effect was statistically unrelated to gender, subjective drug effects, mood and autism trait measures. The finding that a low dose of µ-agonist reduced the propensity to perceive anger in photos with subtle facial expressions is consistent with the notion that µ-opioids mediate social confidence and reduce sensitivity to threat cues.

The costs and benefits of replication studies.

The debate about whether replication studies should become mainstream is essentially driven by disagreements about their costs and benefits and the best ways to allocate limited resources. Determining when replications are worthwhile requires quantifying their expected utility. We argue that a formalized framework for such evaluations can be useful for both individual decision-making and collective discussions about replication.