Megastudies improve the impact of applied behavioural science.

Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens' decisions and outcomes1. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals2. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy-a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research3-6. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.

Comparing the reliability and predictive power of child, teacher, and guardian reports of noncognitive skills.

Children's noncognitive or socioemotional skills (e.g., persistence and self-control) are typically measured using surveys in which either children rate their own skills or adults rate the skills of children. For many purposes-including program evaluation and monitoring school systems-ratings are often collected from multiple perspectives about a single child (e.g., from both the child and an adult). Collecting data from multiple perspectives is costly, and there is limited evidence on the benefits of this approach. Using a longitudinal survey, this study compares children's noncognitive skills as reported by themselves, their guardians, and their teachers. Although reports from all three types of respondents are correlated with each other, teacher reports have the highest internal consistency and are the most predictive of children's later cognitive outcomes and behavior in school. The teacher reports add predictive power beyond baseline measures of Intelligence Quotient (IQ) for most outcomes in schools. Measures collected from children and guardians add minimal predictive power beyond the teacher reports.

Sensitivity of self-reported noncognitive skills to survey administration conditions.

Noncognitive skills (e.g., persistence and self-control) are typically measured using self-reported questionnaires in which respondents rate their own skills. In many applications-including program evaluation and school accountability systems-such reports are assumed to measure only the skill of interest. However, self-reports might also capture other dimensions aside from the skill, such as aspects of a respondent's situation, which could include incentives and the conditions in which they complete the questionnaire. To explore this possibility, this study conducted 2 experiments to estimate the extent to which survey administration conditions can affect student responses on noncognitive skill questionnaires. The first experiment tested whether providing information about the importance of noncognitive skills to students directly affects their responses, and the second experiment tested whether incentives tied to performance on another task indirectly affect responses. Both experiments suggest that self-reports of noncognitive skills are sensitive to survey conditions. The effects of the conditions are relatively large compared with those found in the program evaluation literature, ranging from 0.05 to 0.11 SDs. These findings suggest that the effects of interventions or other social policies on self-reported noncognitive skills should be interpreted with caution.

Hard evidence on soft skills.

This paper summarizes recent evidence on what achievement tests measure; how achievement tests relate to other measures of "cognitive ability" like IQ and grades; the important skills that achievement tests miss or mismeasure, and how much these skills matter in life. Achievement tests miss, or perhaps more accurately, do not adequately capture, soft skills-personality traits, goals, motivations, and preferences that are valued in the labor market, in school, and in many other domains. The larger message of this paper is that soft skills predict success in life, that they causally produce that success, and that programs that enhance soft skills have an important place in an effective portfolio of public policies.

Large studies reveal how reference bias limits policy applications of self-report measures.

There is growing policy interest in identifying contexts that cultivate self-regulation. Doing so often entails comparing groups of individuals (e.g., from different schools). We show that self-report questionnaires-the most prevalent modality for assessing self-regulation-are prone to reference bias, defined as systematic error arising from differences in the implicit standards by which individuals evaluate behavior. In three studies, adolescents (N = 229,685) whose peers performed better academically rated themselves lower in self-regulation and held higher standards for self-regulation. This effect was not observed for task measures of self-regulation and led to paradoxical predictions of college persistence 6 years later. These findings suggest that standards for self-regulation vary by social group, limiting the policy applications of self-report questionnaires.

AIDS and declining support for dependent elderly people in Africa: retrospective analysis using demographic and health surveys.

OBJECTIVES: To determine the relation between the HIV/AIDS epidemic and support for dependent elderly people in Africa. DESIGN: Retrospective analysis using data from Demographic and Health Surveys. SETTING: 22 African countries between 1991 and 2006. PARTICIPANTS: 123,176 individuals over the age of 60. MAIN OUTCOME MEASURES: We investigated how three measures of the living arrangements of older people have been affected by the HIV/AIDS epidemic: the number of older individuals living alone (that is, the number of unattended elderly people); the number of older individuals living with only dependent children under the age of 10 (that is, in missing generation households); and the number of adults age 18-59 (that is, prime age adults) per household where an older person lives. RESULTS: An increase in annual AIDS mortality of one death per 1000 people was associated with a 1.5% increase in the proportion of older individuals living alone (95% CI 1.2% to 1.9%) and a 0.4% increase in the number of older individuals living in missing generation households (95% CI 0.3% to 0.6%). Increases in AIDS mortality were also associated with fewer prime age adults in households with at least one older person and at least one prime age adult (P<0.001). These findings suggest that in our study countries, which encompass 70% of the sub-Saharan population, the HIV/AIDS epidemic could be responsible for 582,200-917,000 older individuals living alone without prime age adults and 141,000-323,100 older individuals being the sole caregivers for young children. CONCLUSIONS: Africa's HIV/AIDS epidemic might be responsible for a large number of older people losing their support and having to care for young children. This population has previously been under-recognised. Efforts to reduce HIV/AIDS deaths could have large "spillover" benefits for elderly people in Africa.