Integrating explanation and prediction in computational social science.

Computational social science is more than just large repositories of digital data and the computational methods needed to construct and analyse them. It also represents a convergence of different fields with different ways of thinking about and doing science. The goal of this Perspective is to provide some clarity around how these approaches differ from one another and to propose how they might be productively integrated. Towards this end we make two contributions. The first is a schema for thinking about research activities along two dimensions-the extent to which work is explanatory, focusing on identifying and estimating causal effects, and the degree of consideration given to testing predictions of outcomes-and how these two priorities can complement, rather than compete with, one another. Our second contribution is to advocate that computational social scientists devote more attention to combining prediction and explanation, which we call integrative modelling, and to outline some practical suggestions for realizing this goal.

Reporting all results efficiently: A RARE proposal to open up the file drawer.

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Replicability, Robustness, and Reproducibility in Psychological Science.

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.

The crisis from above: Gatekeepers need better standards.

Improvements to the validity of psychological science depend upon more than the actions of individual researchers. Editors, journals, and publishers wield considerable power in shaping the incentives that have ushered in the generalizability crisis. These gatekeepers must raise their standards to ensure authors' claims are supported by evidence. Unless gatekeepers change, changes made by individual scientists will not be sustainable.

Do People Know What They're Like in the Moment?

Knowing yourself requires knowing not only what you are like in general (trait self-knowledge) but also how your personality fluctuates from moment to moment (state self-knowledge). We examined this latter form of self-knowledge. Participants (248 people; 2,938 observations) wore the Electronically Activated Recorder (EAR), an unobtrusive audio recorder, and completed experience-sampling self-reports of their personality states four times each day for 1 week. We estimated state self-knowledge by comparing self-reported personality states with consensual observer ratings of personality states coded from the EAR files, which formed the criterion for what participants were "actually" like in the moment. People had self-insight into their momentary extraversion, conscientiousness, and likely neuroticism, suggesting that people can accurately detect fluctuations in some aspects of their personality. However, the evidence for self-insight was weaker for agreeableness. This apparent self-ignorance may be partly responsible for interpersonal problems and for blind spots in trait self-knowledge.

"Eavesdropping on Happiness" Revisited: A Pooled, Multisample Replication of the Association Between Life Satisfaction and Observed Daily Conversation Quantity and Quality.

In the present study, we aimed to replicate and extend findings by Mehl, Vazire, Holleran, and Clark (2010) that individuals with higher well-being tend to spend less time alone and more time interacting with others (e.g., greater conversation quantity) and engage in less small talk and more substantive conversations (e.g., greater conversation quality). To test the robustness of these effects in a larger and more diverse sample, we used Bayesian integrative data analysis to pool data on subjective life satisfaction and observed daily conversations from three heterogeneous adult samples, in addition to the original sample ( N = 486). We found moderate associations between life satisfaction and amount of alone time, conversation time, and substantive conversations, but no reliable association with small talk. Personality did not substantially moderate these associations. The failure to replicate the original small-talk effect is theoretically and practically important, as it has garnered considerable scientific and lay interest.

Scientific progress is like doing a puzzle, not building a wall.

We contest the "building a wall" analogy of scientific progress. We argue that this analogy unfairly privileges original research (which is perceived as laying bricks and, therefore, constructive) over replication research (which is perceived as testing and removing bricks and, therefore, destructive). We propose an alternative analogy for scientific progress: solving a jigsaw puzzle.

Improving the dependability of research in personality and social psychology: recommendations for research and educational practice.

In this article, the Society for Personality and Social Psychology (SPSP) Task Force on Publication and Research Practices offers a brief statistical primer and recommendations for improving the dependability of research. Recommendations for research practice include (a) describing and addressing the choice of N (sample size) and consequent issues of statistical power, (b) reporting effect sizes and 95% confidence intervals (CIs), (c) avoiding "questionable research practices" that can inflate the probability of Type I error, (d) making available research materials necessary to replicate reported results, (e) adhering to SPSP's data sharing policy, (f) encouraging publication of high-quality replication studies, and (g) maintaining flexibility and openness to alternative standards and methods. Recommendations for educational practice include (a) encouraging a culture of "getting it right," (b) teaching and encouraging transparency of data reporting, (c) improving methodological instruction, and (d) modeling sound science and supporting junior researchers who seek to "get it right."

Self-other knowledge asymmetries in personality pathology.

OBJECTIVE: Self-reports of personality provide valid information about personality disorders (PDs). However, informant reports provide information about PDs that self-reports alone do not provide. The current article examines whether and when one perspective is more valid than the other in identifying PDs. METHOD: Using a representative sample of adults 55 to 65 years of age (N = 991; 45% males), we compared the validity of self- and informant (e.g., spouse, family, or friend) reports of the Five-Factor Model traits in predicting PD scores (i.e., composite of interviewer, self-, and informant reports of PDs). RESULTS: Self-reports (particularly of Neuroticism) were more valid than informant reports for most internalizing PDs (i.e., PDs defined by high Neuroticism). Informant reports (particularly of Agreeableness and Conscientiousness) were more valid than self-reports for externalizing and/or antagonistic PDs (i.e., PDs defined by low Agreeableness and Conscientiousness). Neither report was consistently more valid for thought disorder PDs (i.e., PDs defined by low Extraversion). However, informant reports (particularly of Agreeableness) were more valid than self-reports for PDs that were both internalizing and externalizing (i.e., PDs defined by high Neuroticism and low Agreeableness). CONCLUSIONS: The intrapersonal and interpersonal manifestations of PDs differ, and these differences influence who knows more about pathology.

Reckoning With Our Crisis: An Agenda for the Field of Social and Personality Psychology.

The replication crisis and credibility revolution in the 2010s brought a wave of doubts about the credibility of social and personality psychology. We argue that as a field, we must reckon with the concerns brought to light during this critical decade. How the field responds to this crisis will reveal our commitment to self-correction. If we do not take the steps necessary to address our problems and simply declare the crisis to be over or the problems to be fixed without evidence, we risk further undermining our credibility. To fully reckon with this crisis, we must empirically assess the state of the field to take stock of how credible our science actually is and whether it is improving. We propose an agenda for metascientific research, and we review approaches to empirically evaluate and track where we are as a field (e.g., analyzing the published literature, surveying researchers). We describe one such project (Surveying the Past and Present State of Published Studies in Social and Personality Psychology) underway in our research group. Empirical evidence about the state of our field is necessary if we are to take self-correction seriously and if we hope to avert future crises.

What limitations are reported in short articles in social and personality psychology?

Every research project has limitations. The limitations that authors acknowledge in their articles offer a glimpse into some of the concerns that occupy a field's attention. We examine the types of limitations authors discuss in their published articles by categorizing them according to the four validities framework and investigate whether the field's attention to each of the four validities has shifted from 2010 to 2020. We selected one journal in social and personality psychology (Social Psychological and Personality Science; SPPS), the subfield most in the crosshairs of psychology's replication crisis. We sampled 440 articles (with half of those articles containing a subsection explicitly addressing limitations), and we identified and categorized 831 limitations across the 440 articles. Articles with limitations sections reported more limitations than those without (avg. 2.6 vs. 1.2 limitations per article). Threats to external validity were the most common type of reported limitation (est. 52% of articles), and threats to statistical conclusion validity were the least common (est. 17% of articles). Authors reported slightly more limitations over time. Despite the extensive attention paid to statistical conclusion validity in the scientific discourse throughout psychology's credibility revolution, our results suggest that concerns about statistics-related issues were not reflected in social and personality psychologists' reported limitations. The high prevalence of limitations concerning external validity might suggest it is time that we improve our practices in this area, rather than apologizing for these limitations after the fact. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The transparency of quantitative empirical legal research published in highly ranked law journals (2018-2020): an observational study.

Background: Scientists are increasingly concerned with making their work easy to verify and build upon. Associated practices include sharing data, materials, and analytic scripts, and preregistering protocols. This shift towards increased transparency and rigor has been referred to as a "credibility revolution." The credibility of empirical legal research has been questioned in the past due to its distinctive peer review system and because the legal background of its researchers means that many often are not trained in study design or statistics. Still, there has been no systematic study of transparency and credibility-related characteristics of published empirical legal research. Methods: To fill this gap and provide an estimate of current practices that can be tracked as the field evolves, we assessed 300 empirical articles from highly ranked law journals including both faculty-edited journals and student-edited journals. Results: We found high levels of article accessibility (86%, 95% CI = [82%, 90%]), especially among student-edited journals (100%). Few articles stated that a study's data are available (19%, 95% CI = [15%, 23%]). Statements of preregistration (3%, 95% CI = [1%, 5%]) and availability of analytic scripts (6%, 95% CI = [4%, 9%]) were very uncommon. (i.e., they collected new data using the study's reported methods, but found results inconsistent or not as strong as the original). Conclusion: We suggest that empirical legal researchers and the journals that publish their work cultivate norms and practices to encourage research credibility. Our estimates may be revisited to track the field's progress in the coming years.

Personality evaluated: What do people most like and dislike about themselves and their friends?

What do people think their best and worst personality traits are? Do their friends agree? Across three samples, 463 college students ("targets") and their friends freely described two traits they most liked and two traits they most disliked about the target. Coders categorized these open-ended trait descriptors into high or low poles of six trait domains (extraversion, agreeableness, conscientiousness, emotional stability, openness, and honesty-humility) and judged whether targets and friends reported the same specific best and worst traits. Best traits almost exclusively reflected high levels of the major trait domains (especially high agreeableness and extraversion). In contrast, although worst traits typically reflected low levels of these traits (especially low emotional stability), they sometimes also revealed the downsides of having high levels of these traits (e.g., high extraversion: "loud"; high agreeableness: "people-pleaser"). Overall, targets and friends mentioned similar kinds of best traits; however, targets emphasized low emotional stability worst traits more than friends did, whereas friends emphasized low prosociality worst traits more than targets did. Targets and friends also showed a moderate amount of self-other agreement on what the targets' best and worst traits were. These results (a) shed light on the traits that people consider to be most important in themselves and their friends, (b) suggest that the desirability of some traits may be in the eye of the beholder, (c) reveal the mixed blessings of different traits, and, ultimately, (d) provide a nuanced perspective on what it means for a trait to be "good" or "bad." (PsycInfo Database Record (c) 2022 APA, all rights reserved).

What do participants think of our research practices? An examination of behavioural psychology participants' preferences.

What research practices should be considered acceptable? Historically, scientists have set the standards for what constitutes acceptable research practices. However, there is value in considering non-scientists' perspectives, including research participants'. 1873 participants from MTurk and university subject pools were surveyed after their participation in one of eight minimal-risk studies. We asked participants how they would feel if (mostly) common research practices were applied to their data: p-hacking/cherry-picking results, selective reporting of studies, Hypothesizing After Results are Known (HARKing), committing fraud, conducting direct replications, sharing data, sharing methods, and open access publishing. An overwhelming majority of psychology research participants think questionable research practices (e.g. p-hacking, HARKing) are unacceptable (68.3-81.3%), and were supportive of practices to increase transparency and replicability (71.4-80.1%). A surprising number of participants expressed positive or neutral views toward scientific fraud (18.7%), raising concerns about data quality. We grapple with this concern and interpret our results in light of the limitations of our study. Despite the ambiguity in our results, we argue that there is evidence (from our study and others') that researchers may be violating participants' expectations and should be transparent with participants about how their data will be used.

Aspiring to greater intellectual humility in science.

The replication crisis in the social, behavioural and life sciences has spurred a reform movement aimed at increasing the credibility of scientific studies. Many of these credibility-enhancing reforms focus, appropriately, on specific research and publication practices. A less often mentioned aspect of credibility is the need for intellectual humility or being transparent about and owning the limitations of our work. Although intellectual humility is presented as a widely accepted scientific norm, we argue that current research practice does not incentivize intellectual humility. We provide a set of recommendations on how to increase intellectual humility in research articles and highlight the central role peer reviewers can play in incentivizing authors to foreground the flaws and uncertainty in their work, thus enabling full and transparent evaluation of the validity of research.

Eavesdropping on Missing Data: What Are University Students Doing When They Miss Experience Sampling Reports?

Participants in experience sampling method (ESM) studies are "beeped" several times per day to report on their momentary experiences-but participants do not always answer the beep. Knowing whether there are systematic predictors of missing a report is critical for understanding the extent to which missing data threatens the validity of inferences from ESM studies. Here, 228 university students completed up to four ESM reports per day while wearing the Electronically Activated Recorder (EAR)-an unobtrusive audio recording device-for a week. These audio recordings provided an alternative source of information about what participants were doing when they missed or completed reports (3,678 observations). We predicted missing ESM reports from 46 variables coded from the EAR recordings, and found very little evidence that missing an ESM report was correlated with constructs typically of interest to ESM researchers. These findings provide reassuring evidence for the validity of ESM research among relatively healthy university student samples.