A Validity-Based Framework for Understanding Replication in Psychology.

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication "failure." Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

Holding replication studies to mainstream standards of evidence.

Replications can make theoretical contributions, but are unlikely to do so if their findings are open to multiple interpretations (especially violations of psychometric invariance). Thus, just as studies demonstrating novel effects are often expected to empirically evaluate competing explanations, replications should be held to similar standards. Unfortunately, this is rarely done, thereby undermining the value of replication research.

Accumulating evidence across studies: Consistent methods protect against false findings produced by p-hacking.

Much empirical science involves evaluating alternative explanations for the obtained data. For example, given certain assumptions underlying a statistical test, a "significant" result generally refers to implausibility of a null (zero) effect in the population producing the obtained study data. However, methodological work on various versions of p-hacking (i.e., using different analysis strategies until a "significant" result is produced) questions whether significant p-values might often reflect false findings. Indeed, initial simulations of single studies showed that the potential for finding "significant" but false findings might be much higher than the nominal .05 value when various analysis flexibilities are undertaken. In many settings, however, research articles report multiple studies using consistent methods across the studies, where those consistent methods would constrain the flexibilities used to produce high false-finding rates for simulations of single studies. Thus, we conducted simulations of study sets. These simulations show that consistent methods across studies (i.e., consistent in terms of which measures are analyzed, which conditions are included, and whether and how covariates are included) dramatically reduce the potential for flexible research practices (p-hacking) to produce consistent sets of significant results across studies. For p-hacking to produce even modest probabilities of a consistent set of studies would require (a) a large amount of selectivity in study reporting and (b) severe (and quite intentional) versions of p-hacking. With no more than modest selective reporting and with consistent methods across studies, p-hacking does not provide a plausible explanation for consistent empirical results across studies, especially as the size of the reported study set increases. In addition, the simulations show that p-hacking can produce high rates of false findings for single studies with very large samples. In contrast, a series of methodologically-consistent studies (even with much smaller samples) is much less vulnerable to the forms of p-hacking examined in the simulations.

Falling Vocal Intonation Signals Speaker Confidence and Conditionally Boosts Persuasion.

People are often advised to project confidence with their bodies and voices to convince others. Prior research has focused on the high and low thinking processes through which vocal confidence signals (e.g., fast speed, falling intonation, low pitch) can influence attitude change. In contrast, this research examines how the vocal confidence of speakers operates under more moderate elaboration levels, revealing that falling intonation only benefits persuasion under certain circumstances. In three experiments, we show that falling (vs. rising) vocal intonation at the ends of sentences can signal speaker confidence. Under moderate elaboration conditions, falling (vs. rising) vocal intonation increased message processing, bolstering the benefit of strong over weak messages, increasing the proportion of message-relevant thoughts, and increasing thought-attitude correspondence. In sum, the present work examined an unstudied role of vocal confidence in guiding persuasion, revealing new processes by which vocal signals increase or fail to increase persuasion.

Pitch as a Recipient, Channel, and Context Factor Affecting Thought Reliance and Persuasion.

Three experiments tested how low versus high pitch generated from sources beyond a message communicator can affect reliance on thoughts and influence recipients' attitudes. First, participants wrote positive or negative thoughts about an exam proposal (Experiments 1, 2) or their academic abilities (Experiment 3). Then, pitch from the message recipient (Experiment 1), channel (Experiment 2), or context (Experiment 3) was manipulated to be high or low. Experiment 1 showed that when participants vocally expressed their thoughts using low (vs. high) pitch, thoughts had a greater effect on attitudes toward exams. Experiment 2 revealed low (vs. high) pitch sounds from the keyboard participants used to write their thoughts produced the same effect on thought usage. Experiment 3 demonstrated that thoughts influenced attitudes more when listed while background music was low (vs. high) Pitch can influence attitudes through a meta-cognitive thought reliance process whether emerging from the recipient, channel, or context.

Evaluating Research in Personality and Social Psychology: Considerations of Statistical Power and Concerns About False Findings.

Traditionally, statistical power was viewed as relevant to research planning but not evaluation of completed research. However, following discussions of high false finding rates (FFRs) associated with low statistical power, the assumed level of statistical power has become a key criterion for research acceptability. Yet, the links between power and false findings are not as straightforward as described. Assumptions underlying FFR calculations do not reflect research realities in personality and social psychology. Even granting the assumptions, the FFR calculations identify important limitations to any general influences of statistical power. Limits for statistical power in inflating false findings can also be illustrated through the use of FFR calculations to (a) update beliefs about the null or alternative hypothesis and (b) assess the relative support for the null versus alternative hypothesis when evaluating a set of studies. Taken together, statistical power should be de-emphasized in comparison to current uses in research evaluation.

I'll scratch your back if you give me a compliment: Exploring psychological mechanisms underlying compliments' effects on compliance.

Although compliments can be an effective compliance tactic, little is known about the reasons for their effectiveness. Two studies tested three potential mechanisms underlying the use of compliments as a compliance tactic: reciprocity, positive mood, and liking. In both studies, participants were either primed with the reciprocity norm or not, then received either complimentary or neutral feedback from a stranger. Participants were later faced with a request from the stranger. Mood, liking for the requestor, and compliance were measured. As predicted, compliments increased compliance in both studies. Neither study found evidence for positive mood nor liking as a mediator of the compliment effect. However, reciprocity priming was found to moderate the compliment effect in both studies, suggesting that compliments are effective, at least in part, because they invoke the reciprocity norm.

Extraversion as a Moderator of the Efficacy of Self-Esteem Maintenance Strategies.

Four experiments explored how extraversion's connection with self-esteem may depend on specific self-enhancement strategies. Participants' self-esteem threatening feedback indicating that they had performed poorly on a vocabulary or emotional intelligence test. In Experiment 1, participants (N = 80) were randomly assigned to either a control condition (no self-enhancement) or a downward social comparison condition. The procedures for Experiments 2 (N = 470) and 3 (N = 514) were similar, adding a self-serving attribution condition (Experiments 2 and 3) and Basking-in-Reflected-Glory (BIRG) condition (Experiment 3). Across the experiments, extraversion was more related to self-esteem under downward social comparison versus other conditions. BIRGing produced higher self-esteem in Experiment 3 across extraversion levels. Experiment 4 (N = 355) focused on downward social comparison versus control, and provided evidence that an increased perception of being similar to the comparison targets may partially explain extraversion's self-esteem link. Theoretical implications concerning both extraversion and self-enhancement are discussed.

Paralinguistic Features Communicated through Voice can Affect Appraisals of Confidence and Evaluative Judgments.

This article unpacks the basic mechanisms by which paralinguistic features communicated through the voice can affect evaluative judgments and persuasion. Special emphasis is placed on exploring the rapidly emerging literature on vocal features linked to appraisals of confidence (e.g., vocal pitch, intonation, speech rate, loudness, etc.), and their subsequent impact on information processing and meta-cognitive processes of attitude change. The main goal of this review is to advance understanding of the different psychological processes by which paralinguistic markers of confidence can affect attitude change, specifying the conditions under which they are more likely to operate. In sum, we highlight the importance of considering basic mechanisms of attitude change to predict when and why appraisals of paralinguistic markers of confidence can lead to more or less persuasion.

Neuroticism and the architecture of the self: exploring neuroticism as a moderator of the impact of ideal self-discrepancies on emotion.

Two studies examined the moderating role of neuroticism in discrepancy-emotion relations. In Study 1, neuroticism, self-discrepancies, and depression were measured. Multiple regression analyses revealed a significant interaction between neuroticism and ideal self-discrepancies such that the magnitude of ideal self-discrepancies was a stronger predictor of depression for people high in neuroticism than people low in neuroticism. Study 2 used an experimental paradigm to test the same hypothesis. Participants were randomly assigned to an ideal self-discrepancy salience condition or a control condition in which ideal self-discrepancies were not made salient. A significant interaction between self-discrepancy condition and neuroticism emerged such that the ideal self-discrepancy condition produced higher dejection-related affect relative to the control condition for people high in neuroticism compared to people low in neuroticism.

Perceived Knowledge Moderates the Relation Between Subjective Ambivalence and the "Impact" of Attitudes: An Attitude Strength Perspective.

Previous work has reliably demonstrated that when people experience more subjective ambivalence about an attitude object, their attitudes have less impact on strength-related outcomes such as attitude-related thinking, judging, or behaving. However, previous research has not considered whether the amount of perceived knowledge a person has about the topic might moderate these effects. Across eight studies on different topics using a variety of outcome measures, the current research demonstrates that perceived knowledge can moderate the relation between ambivalence and the impact of attitudes on related thinking, judging, and behaving. Although the typical Attitude × Ambivalence effect emerged when participants had relatively high perceived knowledge, this interaction did not emerge when participants were lower in perceived knowledge. This work provides a more nuanced view of the effects of subjective ambivalence on attitude impact and highlights the importance of understanding the combined impact of attitude strength antecedents.

Altering category-level beliefs: the impact of level of representation at belief formation and belief disconfirmation.

This research program investigates whether representational level of information underlying initial beliefs (individual vs. category) and disconfirming information (individual vs. category) influence the magnitude of belief and attitude change regarding categories of objects. In 3 experiments, 2 key effects emerged. A main effect of type of disconfirming information indicated that category-level information produced more belief and attitude change than did individual-level information. Also, a significant interaction between type of information at formation and disconfirmation indicated a relative matching effect, with category-level disconfirmation producing substantially more belief and attitude change than individual-level disconfirmation when initial beliefs were based on category-level information but only slightly greater change when initial beliefs were based on individual-level information.

A painful reminder: the role of level and salience of attitude importance in cognitive dissonance.

In his seminal book, L. Festinger (1957) emphasized the role of attitude importance in cognitive dissonance. This study (N = 308) explored whether people's use of dissonance reduction strategies differs as a function of level of attitude importance and whether the personal importance of an attitude is salient. Results showed that level and salience of attitude importance interacted to affect high-choice (HC) participants' tendency to use attitude change and trivialization to reduce dissonance. When HC participants were not reminded of the personal importance of their attitude (i.e., it was not salient), they changed their attitudes equally irrespective of attitude importance, but engaged in greater trivialization with increasing levels of importance. In contrast, when attitude importance was salient, HC participants changed their attitudes less with increasing attitude importance and showed no evidence of trivializing under any level of importance.

Speech Rate, Intonation, and Pitch: Investigating the Bias and Cue Effects of Vocal Confidence on Persuasion.

Three experiments were designed to investigate the effects and psychological mechanisms of three vocal qualities on persuasion. Experiment 1 (N = 394) employed a 2 (elaboration: high vs. low) × 2 (vocal speed: fast vs. slow) × 2 (vocal intonation: falling vs. rising) between-participants factorial design. As predicted, vocal speed and vocal intonation influenced global perceptions of speaker confidence. Under high-elaboration, vocal confidence biased thought-favorability, which influenced attitudes. Under low-elaboration, vocal confidence directly influenced attitudes as a peripheral cue. Experiments 2 (N = 412) and 3 (N = 397) conceptually replicated the bias and cue effects in Experiment 1, using a 2 (elaboration: high vs. low) × 2 (vocal pitch: raised vs. lowered) between-participants factorial design. Vocal pitch influenced perceptions of speaker confidence as predicted. These studies demonstrate that changes in three vocal properties influence global perceptions of speaker confidence, influencing attitudes via different mediating processes moderated by amount of thought. Evaluation of alternative mediators in Experiments 2 and 3 failed to support these alternatives to global perceptions of speaker confidence.

Affective and cognitive meta-bases of attitudes: Unique effects on information interest and persuasion.

The authors investigated the predictive utility of people's subjective assessments of whether their evaluations are affect- or cognition driven (i.e., meta-cognitive bases) as separate from whether people's attitudes are actually affect- or cognition based (i.e., structural bases). Study 1 demonstrated that meta-bases uniquely predict interest in affective versus cognitive information above and beyond structural bases and other related variables (i.e., need for cognition and need for affect). In Study 2, meta-bases were shown to account for unique variance in attitude change as a function of appeal type. Finally, Study 3 showed that as people became more deliberative in their judgments, meta-bases increased in predictive utility, and structural bases decreased in predictive utility. These findings support the existence of meta-bases of attitudes and demonstrate that meta-bases are distinguishable from structural bases in their predictive utility.

Attitudinal ambivalence and message-based persuasion: motivated processing of proattitudinal information and avoidance of counterattitudinal information.

Attitudinal ambivalence has been found to increase processing of attitude-relevant information. In this research, the authors suggest that ambivalence can also create the opposite effect: avoidance of thinking about persuasive messages. If processing is intended to reduce experienced ambivalence, then ambivalent people should increase processing of information perceived as proattitudinal (agreeable) and able to decrease ambivalence. However, ambivalence should also lead people to avoid processing of counterattitudinal (disagreeable) information that threatens to increase ambivalence. Three studies provide evidence consistent with this proposal. When participants were relatively ambivalent, they processed messages to a greater extent when the messages were proattitudinal rather than counterattitudinal. However, when participants were relatively unambivalent, they processed messages more when the messages were counterattitudinal rather than proattitudinal. In addition, ambivalent participants perceived proattitudinal messages as more likely than counterattitudinal messages to reduce ambivalence, and these perceptions accounted for message position effects on amount of processing.

Chronic pain assessment: a seven-factor model.

BACKGROUND: There are many measures assessing related dimensions of the chronic pain experience (eg, pain severity, pain coping, depression, activity level), but the relationships among them have not been systematically established. OBJECTIVE: The present study set out to determine the core dimensions requiring assessment in individuals with chronic pain. METHODS: Individuals with chronic pain (n=126) completed the Beck Anxiety Inventory, Beck Depression Inventory, Beck Hopelessness Scale, Chronic Pain Coping Index, Multidimensional Pain Inventory, Pain Catastrophizing Scale, McGill Pain Questionnaire--Short Form, Pain Disability Index and the Tampa Scale of Kinesiophobia. RESULTS: Before an exploratory factor analysis (EFA) of the nine chronic pain measures, EFAs were conducted on each of the individual measures, and the derived factors (subscales) from each measure were submitted together for a single EFA. A seven-factor model best fit the data, representing the core factors of pain and disability, pain description, affective distress, support, positive coping strategies, negative coping strategies and activity. CONCLUSIONS: Seven meaningful dimensions of the pain experience were reliably and systematically extracted. Implications and future directions for this work are discussed.

The role of information-processing capacity and goals in attitude-congruent selective exposure effects.

Two experiments explored the role of information-processing capacity and strategies in regulating attitude-congruent selective exposure. In Experiment 1, participants were placed under time pressure and randomly assigned to conditions in which either an attitude-expressive or no-information processing goal was made salient. Analyses revealed an attitude-congruent selective exposure effect and indicated that this effect was stronger when an attitude-expressive goal was made salient than when no goal was made salient. In Experiment 2, information-processing goals and time pressure were factorially manipulated. Analyses revealed an attitude-congruent selective exposure effect and indicated that this effect was especially strong when time pressure was high and an attitude-expressive goal was made salient. In both experiments, bias at exposure was found to predict bias at later stages of information processing (attention and memory). Supplementary analyses and data confirmed that the attitude-expressive goal manipulation activated its intended motivational processing strategy.

Understanding knowledge effects on attitude-behavior consistency: the role of relevance, complexity, and amount of knowledge.

The role of properties of attitude-relevant knowledge in attitude- behavior consistency was explored in 3 experiments. In Experiment 1, attitudes based on behaviorally relevant knowledge predicted behavior better than attitudes based on low-relevance knowledge, especially when people had time to deliberate. Relevance, complexity, and amount of knowledge were investigated in Experiment 2. It was found that complexity increased attitude- behavior consistency when knowledge was of low-behavioral relevance. Under high-behavioral relevance, attitudes predicted behavior well regardless of complexity. Amount of knowledge had no effect on attitude- behavior consistency. In Experiment 3, the findings of Experiment 2 were replicated, and the complexity effect was extended to behaviors of ambiguous relevance. Together, these experiments support an attitude inference perspective, which holds that under high deliberation conditions, people consider the behavioral relevance and dimensional complexity of knowledge underlying their attitudes before deciding to act on them.

The Personal Acquaintance Measure: a tool for appraising one's acquaintance with any person.

The authors developed and evaluated the psychometric properties of the 18-item Personal Acquaintance Measure (PAM) and investigated how the PAM relates to self- other agreement in personality ratings. Results support that 6 factors represent the PAM (Duration, Frequency of Interaction, Knowledge of Goals, Physical Intimacy, Self-Disclosure, Social Network Familiarity), which showed evidence of internal consistency, test-retest reliability over 3 weeks, sensitivity to known group differences, discriminant validity from socially desirable responding, and convergent validity with other relationship inventories. Results also show that the PAM positively predicted self-other agreement. The authors discuss the implications of these findings for the PAM and research in person perception, although this measure may also be used in other research areas.