Evaluations are inherently comparative, but are compared to what?

Understanding how objective quantities are translated into subjective evaluations has long been of interest to social scientists, medical professionals, and policymakers with an interest in how people process and act on quantitative information. The theory of decision by sampling proposes a comparative procedure: Values seem larger or smaller based on how they rank in a comparison set, the decision sample. But what values are included in this decision sample? We identify and test four mechanistic accounts, each suggesting that how previously encountered attribute values are processed determines whether they linger in the sample to guide the subjective interpretation, and thus the influence, of newly encountered values. Testing our ideas through studies of loss aversion, delay discounting, and vaccine hesitancy, we find strongest support for one account: Quantities need to be subjectively evaluated-rather than merely encountered-for them to enter the decision sample, alter the subjective interpretation of other values, and then guide decision making. Discussion focuses on how the present findings inform understanding of the nature of the decision sample and identify new research directions for the longstanding question of how comparison standards influence decision making. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Once and Again: Repeated Viewing Affects Judgments of Spontaneity and Preparation.

Reality is fleeting, and any moment can only be experienced once. Rewatching a video, however, allows people to repeatedly observe the exact same moment. We propose that people may fail to fully distinguish between merely observing behavior again (through replay) from that behavior being performed again in the exact same way. Using an assortment of stimuli that included auditions, commercials, and potential trial evidence, we demonstrated through nine experiments (N = 10,412 adults in the United States) that rewatching makes a recorded behavior appear more rehearsed and less spontaneous, as if the actors were simply precisely repeating their actions. These findings contribute to an emerging literature showing that incidental video features, like perspective or slow motion, can meaningfully change evaluations. Replay may inadvertently shape judgments in both mundane and consequential contexts. To understand how a video will influence its viewer, one will need to consider not only its content, but also how often it is viewed.

Prosocial spending encourages happiness: A replication of the only experiment reported in Dunn, Aknin, and Norton (2008).

Spending money on one's self, whether to solve a problem, fulfill a need, or increase enjoyment, often heightens one's sense of happiness. It is therefore both surprising and important that people can be even happier after spending money on someone else. We conducted a close replication of a key experiment from Dunn, Aknin, and Norton (2008) to verify and expand upon their findings. Participants were given money and randomly assigned to either spend it on themselves or on someone else. Although the original study (N = 46) found that the latter group was happier, when we used the same analysis in our replication (N = 133), we did not observe a significant difference. However, we report an additional analysis, focused on a more direct measure of happiness, that does show a significant effect in the direction of the original. Follow-up analyses shed new insights into people's predictions about their own and others' happiness and their actual happiness when spending money for themselves or others.

Empirical audit and review and an assessment of evidentiary value in research on the psychological consequences of scarcity.

Empirical audit and review is an approach to assessing the evidentiary value of a research area. It involves identifying a topic and selecting a cross-section of studies for replication. We apply the method to research on the psychological consequences of scarcity. Starting with the papers citing a seminal publication in the field, we conducted replications of 20 studies that evaluate the role of scarcity priming in pain sensitivity, resource allocation, materialism, and many other domains. There was considerable variability in the replicability, with some strong successes and other undeniable failures. Empirical audit and review does not attempt to assign an overall replication rate for a heterogeneous field, but rather facilitates researchers seeking to incorporate strength of evidence as they refine theories and plan new investigations in the research area. This method allows for an integration of qualitative and quantitative approaches to review and enables the growth of a cumulative science.

Publisher Correction: Specification curve analysis.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Specification curve analysis.

Empirical results hinge on analytical decisions that are defensible, arbitrary and motivated. These decisions probably introduce bias (towards the narrative put forward by the authors), and they certainly involve variability not reflected by standard errors. To address this source of noise and bias, we introduce specification curve analysis, which consists of three steps: (1) identifying the set of theoretically justified, statistically valid and non-redundant specifications; (2) displaying the results graphically, allowing readers to identify consequential specifications decisions; and (3) conducting joint inference across all specifications. We illustrate the use of this technique by applying it to three findings from two different papers, one investigating discrimination based on distinctively Black names, the other investigating the effect of assigning female versus male names to hurricanes. Specification curve analysis reveals that one finding is robust, one is weak and one is not robust at all.

Overestimating the valuations and preferences of others.

People often make judgments about their own and others' valuations and preferences. Across 12 studies (N = 17,594), we find a robust bias in these judgments such that people overestimate the valuations and preferences of others. This overestimation arises because, when making predictions about others, people rely on their intuitive core representation of the experience (e.g., is the experience generally positive?) in lieu of a more complex representation that might also include countervailing aspects (e.g., is any of the experience negative?). We first demonstrate that the overestimation bias is pervasive for a wide range of positive (Studies 1-5) and negative experiences (Study 6). Furthermore, the bias is not merely an artifact of how preferences are measured (Study 7). Consistent with judgments based on core representations, the bias significantly reduces when the core representation is uniformly positive (Studies 8A-8B). Such judgments lead to a paradox in how people see others trade off between valuation and utility (Studies 9A-9B). Specifically, relative to themselves, people believe that an identically paying other will get more enjoyment from the same experience, but paradoxically, that an identically enjoying other will pay more for the same experience. Finally, consistent with a core representation explanation, explicitly prompting people to consider the entire distribution of others' preferences significantly reduced or eliminated the bias (Study 10). These findings suggest that social judgments of others' preferences are not only largely biased, but they also ignore how others make trade-offs between evaluative metrics. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

99% impossible: A valid, or falsifiable, internal meta-analysis.

Several researchers have relied on, or advocated for, internal meta-analysis, which involves statistically aggregating multiple studies in a paper to assess their overall evidential value. Advocates of internal meta-analysis argue that it provides an efficient approach to increasing statistical power and solving the file-drawer problem. Here we show that the validity of internal meta-analysis rests on the assumption that no studies or analyses were selectively reported. That is, the technique is only valid if (a) all conducted studies were included (i.e., an empty file drawer), and (b) for each included study, exactly one analysis was attempted (i.e., there was no p-hacking). We show that even very small doses of selective reporting invalidate internal meta-analysis. For example, the kind of minimal p-hacking that increases the false-positive rate of 1 study to just 8% increases the false-positive rate of a 10-study internal meta-analysis to 83%. If selective reporting is approximately zero, but not exactly zero, then internal meta-analysis is invalid. To be valid, (a) an internal meta-analysis would need to contain exclusively studies that were properly preregistered, (b) those preregistrations would have to be followed in all essential aspects, and (c) the decision of whether to include a given study in an internal meta-analysis would have to be made before any of those studies are run. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

P-curve won't do your laundry, but it will distinguish replicable from non-replicable findings in observational research: Comment on Bruns & Ioannidis (2016).

p-curve, the distribution of significant p-values, can be analyzed to assess if the findings have evidential value, whether p-hacking and file-drawering can be ruled out as the sole explanations for them. Bruns and Ioannidis (2016) have proposed p-curve cannot examine evidential value with observational data. Their discussion confuses false-positive findings with confounded ones, failing to distinguish correlation from causation. We demonstrate this important distinction by showing that a confounded but real, hence replicable association, gun ownership and number of sexual partners, leads to a right-skewed p-curve, while a false-positive one, respondent ID number and trust in the supreme court, leads to a flat p-curve. P-curve can distinguish between replicable and non-replicable findings. The observational nature of the data is not consequential.

False-Positive Citations.

We describe why we wrote "False-Positive Psychology," analyze how it has been cited, and explain why the integrity of experimental psychology hinges on the full disclosure of methods, the sharing of materials and data, and, especially, the preregistration of analyses.

Registered Replication Report: Dijksterhuis and van Knippenberg (1998).

Dijksterhuis and van Knippenberg (1998) reported that participants primed with a category associated with intelligence ("professor") subsequently performed 13% better on a trivia test than participants primed with a category associated with a lack of intelligence ("soccer hooligans"). In two unpublished replications of this study designed to verify the appropriate testing procedures, Dijksterhuis, van Knippenberg, and Holland observed a smaller difference between conditions (2%-3%) as well as a gender difference: Men showed the effect (9.3% and 7.6%), but women did not (0.3% and -0.3%). The procedure used in those replications served as the basis for this multilab Registered Replication Report. A total of 40 laboratories collected data for this project, and 23 of these laboratories met all inclusion criteria. Here we report the meta-analytic results for those 23 direct replications (total N = 4,493), which tested whether performance on a 30-item general-knowledge trivia task differed between these two priming conditions (results of supplementary analyses of the data from all 40 labs, N = 6,454, are also reported). We observed no overall difference in trivia performance between participants primed with the "professor" category and those primed with the "hooligan" category (0.14%) and no moderation by gender.

Psychology's Renaissance.

In 2010-2012, a few largely coincidental events led experimental psychologists to realize that their approach to collecting, analyzing, and reporting data made it too easy to publish false-positive findings. This sparked a period of methodological reflection that we review here and call Psychology's Renaissance. We begin by describing how psychologists' concerns with publication bias shifted from worrying about file-drawered studies to worrying about p-hacked analyses. We then review the methodological changes that psychologists have proposed and, in some cases, embraced. In describing how the renaissance has unfolded, we attempt to describe different points of view fairly but not neutrally, so as to identify the most promising paths forward. In so doing, we champion disclosure and preregistration, express skepticism about most statistical solutions to publication bias, take positions on the analysis and interpretation of replication failures, and contend that meta-analytical thinking increases the prevalence of false positives. Our general thesis is that the scientific practices of experimental psychologists have improved dramatically.

Rejecting a bad option feels like choosing a good one.

Across 4,151 participants, the authors demonstrate a novel framing effect, attribute matching, whereby matching a salient attribute of a decision frame with that of a decision's options facilitates decision-making. This attribute matching is shown to increase decision confidence and, ultimately, consensus estimates by increasing feelings of metacognitive ease. In Study 1, participants choosing the more attractive of two faces or rejecting the less attractive face reported greater confidence in and perceived consensus around their decision. Using positive and negative words, Study 2 showed that the attribute's extremity moderates the size of the effect. Study 3 found decision ease mediates these changes in confidence and consensus estimates. Consistent with a misattribution account, when participants were warned about this external source of ease in Study 4, the effect disappeared. Study 5 extended attribute matching beyond valence to objective judgments. The authors conclude by discussing related psychological constructs as well as downstream consequences. (PsycINFO Database Record

Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01 s) than high (.04 s) p values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (a) p-curvers selecting the wrong p values, (b) fake data, (c) honest errors, and (d) ambitiously p-hacked (beyond p < .05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.

From "Where" to "What": Distributed Representations of Brand Associations in the Human Brain.

Considerable attention has been given to the notion that there exists a set of human-like characteristics associated with brands, referred to as brand personality. Here we combine newly available machine learning techniques with functional neuroimaging data to characterize the set of processes that give rise to these associations. We show that brand personality traits can be captured by the weighted activity across a widely distributed set of brain regions previously implicated in reasoning, imagery, and affective processing. That is, as opposed to being constructed via reflective processes, brand personality traits appear to exist a priori inside the minds of consumers, such that we were able to predict what brand a person is thinking about based solely on the relationship between brand personality associations and brain activity. These findings represent an important advance in the application of neuroscientific methods to consumer research, moving from work focused on cataloguing brain regions associated with marketing stimuli to testing and refining mental constructs central to theories of consumer behavior.

Paying more when paying for others.

Social behavior is heavily influenced by the perception of the behaviors of others. We considered how perceptions (and misperceptions) of kindness can increase generosity in economic transactions. We investigated how these perceptions can alter behavior in a novel real-life situation that pitted kindness against selfishness. That situation, consumer elective pricing, is defined by an economic transaction allowing people to purchase goods or services for any price (including zero). Field and lab experiments compared how people behave in 2 financially identical circumstances: pay-what-you-want (in which people are ostensibly paying for themselves) and pay-it-forward (in which people are ostensibly paying on behalf of someone else). In 4 field experiments, people paid more under pay-it-forward than pay-what-you-want (Studies 1-4). Four subsequent lab studies assessed whether the salience of others explains the increased payments (Study 5), whether ability to justify lowered payments (Study 6), and whether the manipulation was operating through changing the perceptions of others (Studies 7 and 8). When people rely on ambiguous perceptions, pay-it-forward leads to overestimating the kindness of others and a corresponding increase in personal payment. When those perceptions are replaced with explicit descriptive norms (i.e., others' payment amounts), that effect is eliminated. Finally, subsequent studies confirmed that the effects were not driven by participant confusion (Studies 9A and 9B) and not limited by the specificity of the referent other in the pay-it-forward framing (Study 9C).

P-curve: a key to the file-drawer.

Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that "work," readers must ask, "Are these effects true, or do they merely reflect selective reporting?" We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves-containing more low (.01s) than high (.04s) significant p values--only right-skewed p--curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.

p-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results.

Journals tend to publish only statistically significant evidence, creating a scientific record that markedly overstates the size of effects. We provide a new tool that corrects for this bias without requiring access to nonsignificant results. It capitalizes on the fact that the distribution of significant p values, p-curve, is a function of the true underlying effect. Researchers armed only with sample sizes and test results of the published findings can correct for publication bias. We validate the technique with simulations and by reanalyzing data from the Many-Labs Replication project. We demonstrate that p-curve can arrive at conclusions opposite that of existing tools by reanalyzing the meta-analysis of the "choice overload" literature.