Using Behavioral Insights to Market a Workplace Safety Program: Evidence From a Multi-Armed Experiment.

This paper describes how a multi-armed randomized experiment was used to test multiple variants of a behaviorally informed marketing strategy. In particular, we tested whether specific behavioral messages could be used to increase demand for a safety consultation service offered by the U.S. Occupational Safety and Health Administration. Our experiment used a partial factorial design with 19 study arms and a very large research sample-97,182 establishments-to test the impact of various message, formats, and delivery modes compared with an existing (not behaviorally informed) informational brochure and a no-marketing counterfactual. A secondary research goal was to predict the impact of the most successful marketing strategy (i.e., combination of message, format, and mode) so that OSHA would know what to anticipate if that strategy were implemented at scale. We used two related (but distinct) methods to address these two goals. Both begin with a common mixed (i.e., fixed and random effects) ANOVA model. We addressed the first research goal primarily from the fixed effects; we addressed the second research goal by calculating best linear unbiased predictions (BLUPs) from the full mixed model, where the BLUP involves "shrinkage" as in empirical Bayes (EB) approaches. Marketing via brochures was effective overall, nearly doubling the rate of requests for services. However, the behaviorally informed materials performed no better than OSHA's existing informational brochure. This study also highlights the conditions under which a factorial design can be used to efficiently address questions about which of several program variants are most effective.

Meta-analysis of Federally Funded Adolescent Pregnancy Prevention Program Evaluations.

Beginning in 2010, the U.S. Department of Health and Human Services (HHS) sponsored numerous studies testing the effectiveness of Adolescent Pregnancy Prevention programs on youths' risky sexual behaviors. This article presents results from a meta-analysis of such studies completed between 2015 and 2019 and provided to us by HHS. Studies were eligible for inclusion if they used an experimental/quasi-experimental design, included a comparison condition, assigned at least 10 study participants to each condition, and measured and reported at least one outcome related to sexual behavior or consequences. Data were extracted using a standardized coding protocol and outcomes were coded as, or transformed to, log odds ratio effect sizes for analysis and then transformed to odds ratio effect sizes for presentation in the text. All outcomes were coded such that log odds ratios greater than zero (odds ratios greater than 1) indicate beneficial effects (e.g., reductions in risky sexual behavior). A total of 52 studies met the inclusion criteria. Across 7 outcome types, pooled effects were generally small, the direction of effects was mixed, and none achieved statistical significance: recent pregnancy (OR = 1.30, 95% CI [1.00, 1.68]), ever had sex (OR = 1.07, 95% CI [0.99, 1.15]), recent sexual activity (OR = 0.95, 95% CI [0.84, 1.09]), recent unprotected sexual activity (OR = 1.05, 95% CI [0.96, 1.16]), ever pregnant (OR = 1.21, 95% CI [0.51, 2.89]), number of sexual partners (OR = 1.08, 95% CI = [0.28, 4.22]), and proportion of recent sexual experiences that were unprotected (OR = 0.74, 95% CI = [0.43, 1.31]). Moderator analyses pooling across all 7 outcome types indicated minor differences in program effectiveness between program settings, with those delivered in classroom settings being the least effective (b = - 0.17, 95% CI [- 0.33, - 0.01]). These findings do not lend meaningful support for the overall effectiveness of this group of pregnancy prevention programs. However, because the meta-analysis sample only included programs recently funded by HHS, these findings may have limited generalizability to programs funded by other entities or implemented in other countries.

Do home pregnancy tests bring women to community health workers for antenatal care counselling? A randomized controlled trial in Madagascar.

Task-shifting the provision of pregnancy tests to community health workers (CHWs) in low-resource settings has the potential to reach significantly more underserved women at risk of pregnancy with essential reproductive health services. This study assessed whether an intervention to supply CHWs with home pregnancy tests brought more clients for antenatal care (ANC) counselling. We implemented a randomized controlled trial among CHWs providing reproductive health services to women in Eastern Madagascar. We used ordinary least squares regressions to estimate the effect of the intervention, with district- and month-fixed effects and CHW baseline characteristics as control variables. Our outcomes of interest included whether the intervention increased: (1) the number of women at risk of pregnancy who sought services from CHWs; (2) the number of these women who knew they were pregnant by the end of visit; and (3) the number of these women who received ANC counselling during visit. We found that providing pregnancy tests to CHWs to distribute to their clients for free significantly increased the number of women at risk of pregnancy who sought services from CHWs. At follow-up, treatment-group CHWs provided services to 6.3 clients compared with 4.2 clients among control-group CHWs, which represents a 50% relative increase from the control-group mean. A significantly higher number of these clients knew they were pregnant by the end of the visit, with a mean of 0.95 in treatment compared with 0.10 in control (Coeff. 0.86; 95% CI 0.59-1.13). A significantly higher number of these clients received antenatal counselling at the visit (Coeff. 0.4; 95% CI 0.14-0.64). Introducing free home pregnancy tests as part of community-based health services can improve pregnancy care by attracting more clients at risk of pregnancy to services at the community level, enabling more women to confirm they are pregnant and receive antenatal counselling.

Adolescent Pregnancy Prevention: Meta-Analysis of Federally Funded Program Evaluations.

BACKGROUND: Beginning in 2010, the US Department of Health and Human Services (HHS) funded more than 40 evaluations of adolescent pregnancy prevention interventions. The government's emphasis on rigor and transparency, along with a requirement that grantees collect standardized behavioral outcomes, ensured that findings could be meaningfully compared across evaluations. OBJECTIVES: We used random and mixed-effects meta-analysis to analyze the findings generated by these evaluations to learn whether program elements, program implementation features, and participant demographics were associated with effects on adolescent sexual risk behavior. SEARCH METHODS: We screened all 43 independent evaluation reports, some of which included multiple studies, funded by HHS and completed before October 1, 2016. HHS released, and our team considered, all such studies regardless of favorability or statistical significance. SELECTION CRITERIA: Of these studies, we included those that used a randomized or high-quality quasi-experimental research design. We excluded studies that did not use statistical matching or provide pretest equivalence data on a measure of sexual behavior or a close proxy. We also excluded studies that compared 2 pregnancy prevention interventions without a control group. A total of 44 studies from 39 reports, comprising 51 150 youths, met the inclusion criteria. DATA COLLECTION AND ANALYSIS: Two researchers extracted data from each study by using standard systematic reviewing and meta-analysis procedures. In addition, study authors provided individual participant data for a subset of 34 studies. We used mixed-effects meta-regressions with aggregate data to examine whether program or participant characteristics were associated with program effects on adolescent sexual risk behaviors and consequences. To examine whether individual-level participant characteristics such as age, gender, and race/ethnicity were associated with program effects, we used a 1-stage meta-regression approach combining participant-level data (48 635 youths) with aggregate data from the 10 studies for which participant-level data were not available. MAIN RESULTS: Across all 44 studies, we found small but statistically insignificant mean effects favoring the programs and little variability around those means. Only 2 program characteristics showed statistically reliable relationships with program effects. First, gender-specific (girl-only) programs yielded a statistically significant average effect size (P < .05). Second, programs with individualized service delivery were more effective than programs delivering services to youths in small groups (P < .05). We found no other statistically significant associations between program effects and program or participant characteristics, or evaluation methods. Nor was there a statistically significant difference in the mean effect sizes for programs with previous evidence of effectiveness and previously untested programs. CONCLUSIONS: Although several individual studies reported positive impacts, the average effects were small and there was minimal variation in effect sizes across studies on all of the outcomes assessed. Thus, we were unable to confidently identify which individual program characteristics were associated with effects. However, these studies examined relatively short-term effects and it is an open question whether some programs, perhaps with distinctive characteristics, will show longer-term effects as more of the adolescent participants become sexually active. Public Health Implications. The success of a small number of individualized interventions designed specifically for girls in changing behavioral outcomes suggests the need to reexamine the assumptions that underlie coed group approaches. However, given the almost total absence of similar programs targeting male adolescents, it is likely to be some time before evidence to support or reject such an approach for boys is available.

A randomized controlled trial of the impact of a family planning mHealth service on knowledge and use of contraception.

OBJECTIVES: mHealth, or the use of mobile phones for health, is a promising but largely untested method for increasing family planning knowledge in developing countries. This study estimates the effect of m4RH, an mHealth service in Kenya that provides family planning information via text message, on consumers' knowledge and use of contraception. STUDY DESIGN: We randomly assigned new consumers of the m4RH service to receive either full access or limited access to m4RH. We collected data on outcomes by sending questions directly to consumers via text message. RESULTS: Response rates to the text message surveys ranged from 51.8% to 13.5%. Despite relatively low response rates, response rates were very similar across the full-access and limited-access groups. We find that full access to m4RH increased consumers' scores on a test of contraceptive knowledge by 14% (95% confidence interval: 9.9%-18.2%) compared to a control group with limited access to m4RH. m4RH did not increase consumers' use of contraception, likelihood of discussing family planning with their partners, or likelihood of visiting a clinic to discuss family planning. CONCLUSION: Text messages may increase family planning knowledge but do not, by themselves, lead to behavior change. IMPLICATIONS: Text messages can be an effective method of increasing family planning knowledge but may be insufficient on their own to cause behavior change.

Estimates of Intraclass Correlation Coefficients and Other Design Parameters for Studies of School-Based Nutritional Interventions.

INTRODUCTION: When cluster randomized trials are used to evaluate school-based nutritional interventions such as school lunch programs, design-stage estimates of the required sample size must take into account the correlation in outcomes among individuals within each cluster (e.g., classrooms, schools, or districts). Estimates of the necessary parameters have been carefully developed for educational interventions, but for nutritional interventions the literature is thin. METHODS: Using data from two large multi-school, multi-district impact evaluations conducted in the United States, this article calculates estimates of the design parameters required for sizing school-based nutritional studies. The large size of the trials (252 and 1,327 schools) yields precise estimates of the parameters of interest. Variance components are estimated by fitting random-intercept multilevel models in Stata. RESULTS: School-level intraclass correlations are similar to those typically found for educational outcomes. In particular, school-level estimates range from less than .01 to .26 across the two studies, and district-level estimates ranged from less than .01 to .19. This suggests that cluster randomized trials of nutritional interventions may require samples with numbers of schools similar to the education studies to detect similar effect sizes.

Replicating Reducing the Risk: 12-Month Impacts of a Cluster Randomized Controlled Trial.

OBJECTIVES: To test the effectiveness of Reducing the Risk, an evidence-based sexual health curriculum designed to help prevent adolescent pregnancy and sexually transmitted infections, on youth sexual behavior and intermediate outcomes thought to lead to these behaviors. METHODS: Classes within schools in St. Louis, Missouri; Austin, Texas; and San Diego, California; were randomly assigned to receive Reducing the Risk or "business as usual." Youths completed Web-based surveys at baseline (preintervention, August 2012-January 2014) and 12 months later (August 2013-January 2015). Intent-to-treat analyses were conducted across sites; we tested for differences in impacts between sites and other subgroups. RESULTS: The program had no overall impact on sexual behaviors. However, at 1 site, program participants were significantly less likely to have engaged in recent sexual intercourse than were control group members. There were positive overall impacts on intermediate outcomes (e.g., knowledge, attitudes). CONCLUSIONS: After 12 months, Reducing the Risk was unsuccessful at changing sexual behaviors. Other results were mixed, but promising evidence (e.g., behavioral impacts at 1 site, impacts on intermediate outcomes) suggests potential for more widespread behavioral impacts over a longer term.

Replicating the Safer Sex Intervention: 9-Month Impact Findings of a Randomized Controlled Trial.

OBJECTIVES: To test the effects of the Safer Sex Intervention (SSI) on female adolescents' sexual behavior and possible antecedents of behavior such as sexual health attitudes, knowledge, motivation, intentions, and skills. METHODS: A randomized controlled trial compared SSI (n = 1196) with no intervention (n = 613) among female adolescents aged 13 to 20 years at 3 sites across the United States from 2012 to 2015. Intent-to-treat impacts were estimated at 9 months after baseline, overall, and for key subgroups. RESULTS: Compared with control participants, SSI participants were less likely to have sexual intercourse without birth control, more likely to report positive attitudes toward protection and intention to use condoms, and more confident of their ability to refuse sex. SSI did not affect sexual risk knowledge or motivation to delay childbearing. Positive impacts on sexual behavior and sexual risk were observed among key subgroups of youths who were aged 18 years or older, Hispanic, not sexually experienced at baseline, and enrolled at the Minnesota site. CONCLUSIONS: SSI produced meaningful changes in sexual behavior and sexual risk and successfully addressed some potential antecedents of sexual risk behavior.

Replicating ¡Cuídate!: 6-Month Impact Findings of a Randomized Controlled Trial.

OBJECTIVES: To test whether ¡Cuídate!, a program culturally adapted for Hispanic youths, affects sexual risk behavior. METHODS: We evaluated 3 replications of ¡Cuídate! in California, Arizona, and Massachusetts in a randomized controlled trial (registry no. NCT02540304) in which 2169 primarily Hispanic participants were randomly assigned to an intervention (n = 1326) or a control (n = 870) group. Youths were surveyed at baseline (September 2012-April 2014) and 6 months postbaseline (March 2013-October 2014). We estimated pooled and subgroup impacts using a regression framework with baseline covariates to increase statistical precision (1216 youths analyzed in the treatment group, 806 analyzed in the control group). RESULTS: We found no impacts on the study's primary outcomes of recent sexual activity or recent unprotected sexual activity. However, ¡Cuídate! improved knowledge (10%-20% increase; P < .001), attitudes (effect size = .24; P < .001), and skills (effect size = .14; P = .002). Exploratory subgroup analyses suggest potentially problematic effects for some groups. CONCLUSIONS: Findings suggest that ¡Cuídate! was effective in improving youths' knowledge and attitudes. However, after 6 months, these changes did not translate to improvements in reported sexual risk behaviors.

Providing free pregnancy test kits to community health workers increases distribution of contraceptives: results from an impact evaluation in Madagascar.

OBJECTIVES: To improve access to contraceptives in remote and rural areas, sub-Saharan African countries are allowing community health workers (CHWs) to distribute hormonal contraceptives. Before offering hormonal contraceptives, CHWs must determine pregnancy status but often lack a reliable way to do so. No studies have evaluated the impact of providing CHWs with urine pregnancy test kits. We assessed the impact of giving CHWs free pregnancy test kits on the number of new clients purchasing hormonal contraceptives from CHWs. STUDY DESIGN: We implemented a randomized experiment in Eastern Madagascar among CHWs who sell injectable and oral hormonal contraceptives. A total of 622 CHWs were stratified by region and randomly assigned at the individual level. Treatment-group CHWs were given free pregnancy tests to distribute (n analyzed=272) and control-group CHWs did not receive the tests (n analyzed=263). We estimated an ordinary least-squares regression model, with the monthly number of new hormonal contraceptive clients per CHW as our primary outcome. RESULTS: We find that providing CHWs with free pregnancy test kits increases the number of new hormonal contraceptive clients. Treatment-group CHWs provide hormonal contraceptives to 3.1 new clients per month, compared to 2.5 in the control group. This difference of 0.7 clients per month (95% confidence interval 0.13-1.18; p=.014) represents a 26% increase. CONCLUSIONS: Giving CHWs free pregnancy tests is an effective way to increase distribution of hormonal contraceptives. As pregnancy tests become increasingly affordable for health-care systems in developing countries, community-based distribution programs should consider including the tests as a low-cost addition to CHWs' services. IMPLICATIONS: No study has evaluated the impact of giving CHWs free urine pregnancy test kits for distribution to improve provision of hormonal contraceptives. Giving CHWs free pregnancy test kits significantly increases the number of clients to whom they sell hormonal contraceptives. Community-based distribution programs should consider including these tests among CHWs' services.