An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability.

Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.

A 680,000-person megastudy of nudges to encourage vaccination in pharmacies.

Encouraging vaccination is a pressing policy problem. To assess whether text-based reminders can encourage pharmacy vaccination and what kinds of messages work best, we conducted a megastudy. We randomly assigned 689,693 Walmart pharmacy patients to receive one of 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination or to a business-as-usual control condition that received no messages. We found that the reminder texts that we tested increased pharmacy vaccination rates by an average of 2.0 percentage points, or 6.8%, over a 3-mo follow-up period. The most-effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top-performing intervention included two texts delivered 3 d apart and communicated to patients that a vaccine was "waiting for you." Neither experts nor lay people anticipated that this would be the best-performing treatment, underscoring the value of simultaneously testing many different nudges in a highly powered megastudy.

Comparison of Clinical Decision Support Tools to Improve Pediatric Lipid Screening.

OBJECTIVE: To test whether different clinical decision support tools increase clinician orders and patient completions relative to standard practice and each other. STUDY DESIGN: A pragmatic, patient-randomized clinical trial in the electronic health record was conducted between October 2019 and April 2020 at Geisinger Health System in Pennsylvania, with 4 arms: care gap-a passive listing recommending screening; alert-a panel promoting and enabling lipid screen orders; both; and a standard practice-no guideline-based notification-control arm. Data were analyzed for 13 346 9- to 11-year-old patients seen within Geisinger primary care, cardiology, urgent care, or nutrition clinics, or who had an endocrinology visit. Principal outcomes were lipid screening orders by clinicians and completions by patients within 1 week of orders. RESULTS: Active (care gap and/or alert) vs control arm patients were significantly more likely (P < .05) to have lipid screening tests ordered and completed, with ORs ranging from 1.67 (95% CI 1.28-2.19) to 5.73 (95% CI 4.46-7.36) for orders and 1.54 (95% CI 1.04-2.27) to 2.90 (95% CI 2.02-4.15) for completions. Alerts, with or without care gaps listed, outperformed care gaps alone on orders, with odds ratios ranging from 2.92 (95% CI 2.32-3.66) to 3.43 (95% CI 2.73-4.29). CONCLUSIONS: Electronic alerts can increase lipid screening orders and completions, suggesting clinical decision support can improve guideline-concordant screening. The study also highlights electronic record-based patient randomization as a way to determine relative effectiveness of support tools. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT04118348.

A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor's appointment.

Many Americans fail to get life-saving vaccines each year, and the availability of a vaccine for COVID-19 makes the challenge of encouraging vaccination more urgent than ever. We present a large field experiment (N = 47,306) testing 19 nudges delivered to patients via text message and designed to boost adoption of the influenza vaccine. Our findings suggest that text messages sent prior to a primary care visit can boost vaccination rates by an average of 5%. Overall, interventions performed better when they were 1) framed as reminders to get flu shots that were already reserved for the patient and 2) congruent with the sort of communications patients expected to receive from their healthcare provider (i.e., not surprising, casual, or interactive). The best-performing intervention in our study reminded patients twice to get their flu shot at their upcoming doctor's appointment and indicated it was reserved for them. This successful script could be used as a template for campaigns to encourage the adoption of life-saving vaccines, including against COVID-19.

Objecting to experiments even while approving of the policies or treatments they compare.

We resolve a controversy over two competing hypotheses about why people object to randomized experiments: 1) People unsurprisingly object to experiments only when they object to a policy or treatment the experiment contains, or 2) people can paradoxically object to experiments even when they approve of implementing either condition for everyone. Using multiple measures of preference and test criteria in five preregistered within-subjects studies with 1,955 participants, we find that people often disapprove of experiments involving randomization despite approving of the policies or treatments to be tested.

Objecting to experiments that compare two unobjectionable policies or treatments.

Randomized experiments have enormous potential to improve human welfare in many domains, including healthcare, education, finance, and public policy. However, such "A/B tests" are often criticized on ethical grounds even as similar, untested interventions are implemented without objection. We find robust evidence across 16 studies of 5,873 participants from three diverse populations spanning nine domains-from healthcare to autonomous vehicle design to poverty reduction-that people frequently rate A/B tests designed to establish the comparative effectiveness of two policies or treatments as inappropriate even when universally implementing either A or B, untested, is seen as appropriate. This "A/B effect" is as strong among those with higher educational attainment and science literacy and among relevant professionals. It persists even when there is no reason to prefer A to B and even when recipients are treated unequally and randomly in all conditions (A, B, and A/B). Several remaining explanations for the effect-a belief that consent is required to impose a policy on half of a population but not on the entire population; an aversion to controlled but not to uncontrolled experiments; and a proxy form of the illusion of knowledge (according to which randomized evaluations are unnecessary because experts already do or should know "what works")-appear to contribute to the effect, but none dominates or fully accounts for it. We conclude that rigorously evaluating policies or treatments via pragmatic randomized trials may provoke greater objection than simply implementing those same policies or treatments untested.

Using game-like animations of geometric shapes to simulate social interactions: An evaluation of group score differences.

This study introduces a novel, game-like method for measuring social intelligence: the Social Shapes Test. Unlike other existing video or game-based tests, the Shapes Test uses animations of abstract shapes to represent social interactions. We explore demographic differences in Shapes Test scores compared to a written situational judgment test. Gender and race/ethnicity only had meaningful effects on written SJT scores while no effects were found for Shapes Test scores. This pattern of results remained after controlling for general mental ability and English language exposure. We also found metric invariance between demographic groups for both tests. Our results demonstrate the potential for using animated shape tasks as an alternative to written SJTs when designing future game-based assessments.

Attributing social meaning to animated shapes: A new experimental study of apparent behavior.

In 1944, Heider and Simmel reported that observers could perceive simple animated geometric shapes as characters with emotions, intentions, and other social attributes. This work has been cited over 3000 times and has had wide and ongoing influence on the study of social cognition and social intelligence. However, many researchers in this area have continued to use the original Heider and Simmel black-and-white video. We asked whether the original findings could be reproduced 75 years later by creating 32 new colored animated shape videos designed to depict various social plots and testing whether they can evoke similar spontaneous social attributions. Participants (N = 66) viewed our videos and were asked to write narratives which we coded for indicia of different types of social attributions. Consistent with Heider and Simmel, we found that participants spontaneously attributed social meaning to the videos. We observed that responses to our videos were also similar to responses to the original video reported by Klin (2000), despite being only 13-23 s and portraying a broader range of social plots. Participants varied in how many social attributions they made in response, and the videos varied in how much they elicited such responses. Our set of animated shape videos is freely available online for all researchers to use and forms the basis of a multiple-choice assessment of social intelligence (Brown et al., 2019).

The causal influence of brain size on human intelligence: Evidence from within-family phenotypic associations and GWAS modeling.

There exists a moderate correlation between MRI-measured brain size and the general factor of IQ performance (g), but the question of whether the association reflects a theoretically important causal relationship or spurious confounding remains somewhat open. Previous small studies (n < 100) looking for the persistence of this correlation within families failed to find a tendency for the sibling with the larger brain to obtain a higher test score. We studied the within-family relationship between brain volume and intelligence in the much larger sample provided by the Human Connectome Project (n = 1,022) and found a highly significant correlation (disattenuated ρ = 0.18, p < .001). We replicated this result in the Minnesota Center for Twin and Family Research (n = 2,698), finding a highly significant within-family correlation between head circumference and intelligence (disattenuated ρ = 0.19, p < .001). We also employed novel methods of causal inference relying on summary statistics from genome-wide association studies (GWAS) of head size (n ≈ 10,000) and measures of cognition (257,000 < n < 767,000). Using bivariate LD Score regression, we found a genetic correlation between intracranial volume (ICV) and years of education (EduYears) of 0.41 (p < .001). Using the Latent Causal Variable method, we found a genetic causality proportion of 0.72 (p < .001); thus the genetic correlation arises from an asymmetric pattern, extending to sub-significant loci, of genetic variants associated with ICV also being associated with EduYears but many genetic variants associated with EduYears not being associated with ICV. This is the pattern of genetic results expected from a causal effect of brain size on intelligence. These findings give reason to take up the hypothesis that the dramatic increase in brain volume over the course of human evolution has been the result of natural selection favoring general intelligence.

Common genetic variants associated with cognitive performance identified using the proxy-phenotype method.

We identify common genetic variants associated with cognitive performance using a two-stage approach, which we call the proxy-phenotype method. First, we conduct a genome-wide association study of educational attainment in a large sample (n = 106,736), which produces a set of 69 education-associated SNPs. Second, using independent samples (n = 24,189), we measure the association of these education-associated SNPs with cognitive performance. Three SNPs (rs1487441, rs7923609, and rs2721173) are significantly associated with cognitive performance after correction for multiple hypothesis testing. In an independent sample of older Americans (n = 8,652), we also show that a polygenic score derived from the education-associated SNPs is associated with memory and absence of dementia. Convergent evidence from a set of bioinformatics analyses implicates four specific genes (KNCMA1, NRXN1, POU2F3, and SCRT). All of these genes are associated with a particular neurotransmitter pathway involved in synaptic plasticity, the main cellular mechanism for learning and memory.

Genomic data can illuminate the architecture and evolution of cognitive abilities.

Does general intelligence exist across species, and has it been a target of natural selection? These questions can be addressed with genomic data, which can rule out artifacts by demonstrating that distinct cognitive abilities are genetically correlated and thus share a biological substrate. This work has begun with data from humans and can be extended to other species; it should focus not only on general intelligence but also specific capacities like language and spatial ability.

The genetic architecture of economic and political preferences.

Preferences are fundamental building blocks in all models of economic and political behavior. We study a new sample of comprehensively genotyped subjects with data on economic and political preferences and educational attainment. We use dense single nucleotide polymorphism (SNP) data to estimate the proportion of variation in these traits explained by common SNPs and to conduct genome-wide association study (GWAS) and prediction analyses. The pattern of results is consistent with findings for other complex traits. First, the estimated fraction of phenotypic variation that could, in principle, be explained by dense SNP arrays is around one-half of the narrow heritability estimated using twin and family samples. The molecular-genetic-based heritability estimates, therefore, partially corroborate evidence of significant heritability from behavior genetic studies. Second, our analyses suggest that these traits have a polygenic architecture, with the heritable variation explained by many genes with small effects. Our results suggest that most published genetic association studies with economic and political traits are dramatically underpowered, which implies a high false discovery rate. These results convey a cautionary message for whether, how, and how soon molecular genetic data can contribute to, and potentially transform, research in social science. We propose some constructive responses to the inferential challenges posed by the small explanatory power of individual SNPs.

Replicability and robustness of genome-wide-association studies for behavioral traits.

A recent genome-wide-association study of educational attainment identified three single-nucleotide polymorphisms (SNPs) whose associations, despite their small effect sizes (each R (2) ≈ 0.02%), reached genome-wide significance (p < 5 × 10(-8)) in a large discovery sample and were replicated in an independent sample (p < .05). The study also reported associations between educational attainment and indices of SNPs called "polygenic scores." In three studies, we evaluated the robustness of these findings. Study 1 showed that the associations with all three SNPs were replicated in another large (N = 34,428) independent sample. We also found that the scores remained predictive (R (2) ≈ 2%) in regressions with stringent controls for stratification (Study 2) and in new within-family analyses (Study 3). Our results show that large and therefore well-powered genome-wide-association studies can identify replicable genetic associations with behavioral traits. The small effect sizes of individual SNPs are likely to be a major contributing factor explaining the striking contrast between our results and the disappointing replication record of most candidate-gene studies.

Development and multi-site external validation of a generalizable risk prediction model for bipolar disorder.

Bipolar disorder is a leading contributor to disability, premature mortality, and suicide. Early identification of risk for bipolar disorder using generalizable predictive models trained on diverse cohorts around the United States could improve targeted assessment of high risk individuals, reduce misdiagnosis, and improve the allocation of limited mental health resources. This observational case-control study intended to develop and validate generalizable predictive models of bipolar disorder as part of the multisite, multinational PsycheMERGE Network across diverse and large biobanks with linked electronic health records (EHRs) from three academic medical centers: in the Northeast (Massachusetts General Brigham), the Mid-Atlantic (Geisinger) and the Mid-South (Vanderbilt University Medical Center). Predictive models were developed and valid with multiple algorithms at each study site: random forests, gradient boosting machines, penalized regression, including stacked ensemble learning algorithms combining them. Predictors were limited to widely available EHR-based features agnostic to a common data model including demographics, diagnostic codes, and medications. The main study outcome was bipolar disorder diagnosis as defined by the International Cohort Collection for Bipolar Disorder, 2015. In total, the study included records for 3,529,569 patients including 12,533 cases (0.3%) of bipolar disorder. After internal and external validation, algorithms demonstrated optimal performance in their respective development sites. The stacked ensemble achieved the best combination of overall discrimination (AUC = 0.82-0.87) and calibration performance with positive predictive values above 5% in the highest risk quantiles at all three study sites. In conclusion, generalizable predictive models of risk for bipolar disorder can be feasibly developed across diverse sites to enable precision medicine. Comparison of a range of machine learning methods indicated that an ensemble approach provides the best performance overall but required local retraining. These models will be disseminated via the PsycheMERGE Network website.

Whole-exome sequencing in UK Biobank reveals rare genetic architecture for depression.

Nearly two hundred common-variant depression risk loci have been identified by genome-wide association studies (GWAS). However, the impact of rare coding variants on depression remains poorly understood. Here, we present whole-exome sequencing analyses of depression with seven different definitions based on survey, questionnaire, and electronic health records in 320,356 UK Biobank participants. We showed that the burden of rare damaging coding variants in loss-of-function intolerant genes is significantly associated with risk of depression with various definitions. We compared the rare and common genetic architecture across depression definitions by genetic correlation and showed different genetic relationships between definitions across common and rare variants. In addition, we demonstrated that the effects of rare damaging coding variant burden and polygenic risk score on depression risk are additive. The gene set burden analyses revealed overlapping rare genetic variant components with developmental disorder, autism, and schizophrenia. Our study provides insights into the contribution of rare coding variants, separately and in conjunction with common variants, on depression with various definitions and their genetic relationships with neurodevelopmental disorders.

General cognitive ability and the psychological refractory period: individual differences in the mind's bottleneck.

Identifying the precise locus of general cognitive ability (g) in the flow of information between perception and action is an important goal of differential psychology. To localize the negative correlation between g and reaction time to a specific processing stage, we administered a speeded number-comparison task to two groups differing in average g. The participants had to respond to two stimuli in each trial, which produced the well-known slowing of the second reaction time known as the psychological refractory period. The difference in the second reaction time favoring the high-g group doubled as the stimulus onsets became very close together. This finding affirms that the faster reaction times of higher-g individuals reflect an advantage exclusively in the serial bottleneck of central processing and not in the parallel peripheral stages.

Why it is hard to find genes associated with social science traits: theoretical and empirical considerations.

OBJECTIVES: We explain why traits of interest to behavioral scientists may have a genetic architecture featuring hundreds or thousands of loci with tiny individual effects rather than a few with large effects and why such an architecture makes it difficult to find robust associations between traits and genes. METHODS: We conducted a genome-wide association study at 2 sites, Harvard University and Union College, measuring more than 100 physical and behavioral traits with a sample size typical of candidate gene studies. We evaluated predictions that alleles with large effect sizes would be rare and most traits of interest to social science are likely characterized by a lack of strong directional selection. We also carried out a theoretical analysis of the genetic architecture of traits based on R.A. Fisher's geometric model of natural selection and empirical analyses of the effects of selection bias and phenotype measurement stability on the results of genetic association studies. RESULTS: Although we replicated several known genetic associations with physical traits, we found only 2 associations with behavioral traits that met the nominal genome-wide significance threshold, indicating that physical and behavioral traits are mainly affected by numerous genes with small effects. CONCLUSIONS: The challenge for social science genomics is the likelihood that genes are connected to behavioral variation by lengthy, nonlinear, interactive causal chains, and unraveling these chains requires allying with personal genomics to take advantage of the potential for large sample sizes as well as continuing with traditional epidemiological studies.

Human face recognition ability is specific and highly heritable.

Compared with notable successes in the genetics of basic sensory transduction, progress on the genetics of higher level perception and cognition has been limited. We propose that investigating specific cognitive abilities with well-defined neural substrates, such as face recognition, may yield additional insights. In a twin study of face recognition, we found that the correlation of scores between monozygotic twins (0.70) was more than double the dizygotic twin correlation (0.29), evidence for a high genetic contribution to face recognition ability. Low correlations between face recognition scores and visual and verbal recognition scores indicate that both face recognition ability itself and its genetic basis are largely attributable to face-specific mechanisms. The present results therefore identify an unusual phenomenon: a highly specific cognitive ability that is highly heritable. Our results establish a clear genetic basis for face recognition, opening this intensively studied and socially advantageous cognitive trait to genetic investigation.