The Revolution Will Be Hard to Evaluate: How Co-Occurring Policy Changes Affect Research on the Health Effects of Social Policies.

Extensive empirical health research leverages variation in the timing and location of policy changes as quasi-experiments. Multiple social policies may be adopted simultaneously in the same locations, creating co-occurrence that must be addressed analytically for valid inferences. The pervasiveness and consequences of co-occurring policies have received limited attention. We analyzed a systematic sample of 13 social policy databases covering diverse domains including poverty, paid family leave, and tobacco use. We quantified policy co-occurrence in each database as the fraction of variation in each policy measure across different jurisdictions and times that could be explained by covariation with other policies. We used simulations to estimate the ratio of the variance of effect estimates under the observed policy co-occurrence to variance if policies were independent. Policy co-occurrence ranged from very high for state-level cannabis policies to low for country-level sexual minority-rights policies. For 65% of policies, greater than 90% of the place-time variation was explained by other policies. Policy co-occurrence increased the variance of effect estimates by a median of 57-fold. Co-occurring policies are common and pose a major methodological challenge to rigorously evaluating health effects of individual social policies. When uncontrolled, co-occurring policies confound one another, and when controlled, resulting positivity violations may substantially inflate the variance of estimated effects. Tools to enhance validity and precision for evaluating co-occurring policies are needed.

What to Do When Everything Happens at Once: Analytic Approaches to Estimate the Health Effects of Co-Occurring Social Policies.

Social policies have great potential to improve population health and reduce health disparities. Increasingly, those doing empirical research have sought to quantify the health effects of social policies by exploiting variation in the timing of policy changes across places. Multiple social policies are often adopted simultaneously or in close succession in the same locations, creating co-occurrence that must be handled analytically for valid inferences. Although this is a substantial methodological challenge for researchers aiming to isolate social policy effects, only in a limited number of studies have researchers systematically considered analytic solutions within a causal framework or assessed whether these solutions are being adopted. We designated 7 analytic solutions to policy co-occurrence, including efforts to disentangle individual policy effects and efforts to estimate the combined effects of co-occurring policies. We used an existing systematic review of social policies and health to evaluate how often policy co-occurrence is identified as a threat to validity and how often each analytic solution is applied in practice. Of the 55 studies, only in 17 (31%) did authors report checking for any co-occurring policies, although in 36 studies (67%), at least 1 approach was used that helps address policy co-occurrence. The most common approaches were adjusting for measures of co-occurring policies; defining the outcome on subpopulations likely to be affected by the policy of interest (but not other co-occurring policies); and selecting a less-correlated measure of policy exposure. As health research increasingly focuses on policy changes, we must systematically assess policy co-occurrence and apply analytic solutions to strengthen studies on the health effects of social policies.

A quantitative assessment of the frequency and magnitude of heterogeneous treatment effects in studies of the health effects of social policies.

Substantial heterogeneity in effects of social policies on health across subgroups may be common, but has not been systematically characterized. Using a sample of 55 contemporary studies on health effects of social policies, we recorded how often heterogeneous treatment effects (HTEs) were assessed, for what subgroups (e.g., male, female), and the subgroup-specific effect estimates expressed as Standardized Mean Differences (SMDs). For each study, outcome, and dimension (e.g., gender), we fit a random-effects meta-analysis. We characterized the magnitude of heterogeneity in policy effects using the standard deviation of the subgroup-specific effect estimates (τ). Among the 44% of studies reporting subgroup-specific estimates, policy effects were generally small (<0.1 SMDs) with mixed impacts on health (67% beneficial) and disparities (50% implied narrowing of disparities). Across study-outcome-dimensions, 54% indicated any heterogeneity in effects, and 20% had τ > 0.1 SMDs. For 26% of study-outcome-dimensions, the magnitude of τ indicated that effects of opposite signs were plausible across subgroups. Heterogeneity was more common in policy effects not specified a priori. Our findings suggest social policies commonly have heterogeneous effects on health of different populations; these HTEs may substantially impact disparities. Studies of social policies and health should routinely evaluate HTEs.

Heterogeneous treatment effects in social policy studies: An assessment of contemporary articles in the health and social sciences.

PURPOSE: Social policies are important determinants of population health but may have varying effects on subgroups of people. Evaluating heterogeneous treatment effects (HTEs) of social policies is critical to determine how social policies will affect health inequities. Methods for evaluating HTEs are not standardized. Little is known about how often and by what methods HTEs are assessed in social policy and health research. METHODS: A sample of 55 articles from 2019 on the health effects of social policies were evaluated for frequency of reporting HTEs; for what subgroupings HTEs were reported; frequency of a priori specification of intent to assess HTEs; and methods used for assessing HTEs. RESULTS: A total of 24 (44%) studies described some form of HTE assessment, including by age, gender, education, race/ethnicity, and/or geography. Among studies assessing HTEs, 63% specified HTE assessment a priori, and most (71%) used descriptive methods such as stratification; 21% used statistical tests (e.g., interaction terms in a regression); and no studies used data-driven algorithms. CONCLUSIONS: Although understanding HTEs could enhance policy and practice-based efforts to reduce inequities, it is not routine research practice. Increased evaluation of HTEs across relevant subgroups is needed.

Communicating fish consumption advisories in California: what works, what doesn't.

State agencies face many challenges in creating sport fish consumption advisories that can be readily understood by diverse populations. In this study, our objectives were to identify barriers to understanding consumption advisories and recommend more effective approaches for communicating advisory concepts. We conducted key informant interviews with demographically diverse consumers of sport fish from the Sacramento-San Joaquin Delta watershed in California to explore how intended audiences perceive consumption advisories and identify factors that influence comprehension. Some barriers to communication included the use of portion sizes that departed from commonly consumed amounts, poorly understood terminology, misleading category headings, and ineffective visual tools. Comprehension was enhanced when advisory information did not contradict existing beliefs about fish or fish consumption, and when advisories provided information about contaminant levels in specific kinds of fish. Using certain methods, such as portion sizes that reflect commonly consumed amounts, mercury meters to convey contaminant levels, three advice categories (e.g., high, medium, low), and population definitions that identify specific age ranges, improved the clarity of advisory concepts for intended audiences.

Powering population health research: Considerations for plausible and actionable effect sizes.

Evidence for Action (E4A), a signature program of the Robert Wood Johnson Foundation, funds investigator-initiated research on the impacts of social programs and policies on population health and health inequities. Across thousands of letters of intent and full proposals E4A has received since 2015, one of the most common methodological challenges faced by applicants is selecting realistic effect sizes to inform calculations of power, sample size, and minimum detectable effect (MDE). E4A prioritizes health studies that are both (1) adequately powered to detect effect sizes that may reasonably be expected for the given intervention and (2) likely to achieve intervention effects sizes that, if demonstrated, correspond to actionable evidence for population health stakeholders. However, little guidance exists to inform the selection of effect sizes for population health research proposals. We draw on examples of five rigorously evaluated population health interventions. These examples illustrate considerations for selecting realistic and actionable effect sizes as inputs to calculations of power, sample size and MDE for research proposals to study population health interventions. We show that plausible effects sizes for population health interventions may be smaller than commonly cited guidelines suggest. Effect sizes achieved with population health interventions depend on the characteristics of the intervention, the target population, and the outcomes studied. Population health impact depends on the proportion of the population receiving the intervention. When adequately powered, even studies of interventions with small effect sizes can offer valuable evidence to inform population health if such interventions can be implemented broadly. Demonstrating the effectiveness of such interventions, however, requires large sample sizes.

Community Eligibility Provision and School Meal Participation among Student Subgroups.

BACKGROUND: The National School Lunch and Breakfast Programs help to reduce food insecurity and improve nutrition. The Community Eligibility Provision (CEP) enables high-poverty schools to offer breakfast and lunch at no cost to all students. This study examines associations between CEP and participation among students eligible for free or reduced-price meals ("FRPM"), possibly eligible ("near-cutoff"), or ineligible ("full-price"). METHODS: Using data from the 2013-2015 Healthy Communities Study, we compared school breakfast and lunch participation between 842 students in K-8 at 80 CEP schools and 1463 students at 118 schools without CEP. Cross-sectional difference-in-difference (DID) models compared meal participation among near-cutoff and full-price groups to that in the FRPM group. RESULTS: Overall, FRPM students had high participation in school lunch and breakfast at both types of schools. In adjusted DID models, lunch participation among near-cutoff students was 12 points higher in CEP versus comparison schools (p < .05). Among full-price students, breakfast participation was 20 points higher and lunch participation 19 points higher in CEP than comparison schools (p < .001). CONCLUSIONS: Community Eligibility Provision improves access to school breakfast and lunch in high-poverty schools, particularly for students who are near or above the cutoff for FRPM eligibility.

Alternative causal inference methods in population health research: Evaluating tradeoffs and triangulating evidence.

Population health researchers from different fields often address similar substantive questions but rely on different study designs, reflecting their home disciplines. This is especially true in studies involving causal inference, for which semantic and substantive differences inhibit interdisciplinary dialogue and collaboration. In this paper, we group nonrandomized study designs into two categories: those that use confounder-control (such as regression adjustment or propensity score matching) and those that rely on an instrument (such as instrumental variables, regression discontinuity, or differences-in-differences approaches). Using the Shadish, Cook, and Campbell framework for evaluating threats to validity, we contrast the assumptions, strengths, and limitations of these two approaches and illustrate differences with examples from the literature on education and health. Across disciplines, all methods to test a hypothesized causal relationship involve unverifiable assumptions, and rarely is there clear justification for exclusive reliance on one method. Each method entails trade-offs between statistical power, internal validity, measurement quality, and generalizability. The choice between confounder-control and instrument-based methods should be guided by these tradeoffs and consideration of the most important limitations of previous work in the area. Our goals are to foster common understanding of the methods available for causal inference in population health research and the tradeoffs between them; to encourage researchers to objectively evaluate what can be learned from methods outside one's home discipline; and to facilitate the selection of methods that best answer the investigator's scientific questions.

Building the evidence on Making Health a Shared Value: Insights and considerations for research.

The Robert Wood Johnson Foundation (RWJF)'s Culture of Health Action Framework guides a movement to improve health and advance health equity across the nation. Action Area One of the Framework, Making Health a Shared Value, highlights the role of individual and community factors in achieving a societal commitment to health and health equity, centered around three drivers: Mindset and Expectations, Sense of Community, and Civic Engagement. To stimulate research about how Action Area One and its drivers may impact health, Evidence for Action (E4A), a signature research funding program of RWJF, developed and released a national Call for Proposals (CFP). The process of formulating the CFP and reviewing proposals surfaced important challenges for research on creating and sustaining shared values to foster and maintain a Culture of Health. In this essay, we describe these considerations and provide examples from funded projects regarding how challenges can be addressed.

Child Food Insecurity Is Associated with Energy Intake among Fourth- and Fifth-Grade Girls.

BACKGROUND: Food insecurity is associated with poor diet and obesity among adult women, but evidence among children is mixed, and few studies have examined differences between boys and girls. OBJECTIVE: This study examined the relationship between self-reported food insecurity and dietary intake among boys and girls. DESIGN: Cross-sectional survey data were used from the Children's PowerPlay! Campaign evaluation. PARTICIPANTS AND SETTING: In all, 3,547 fourth- and fifth-grade students (9 to 11 years old) from 44 San Diego-area elementary schools in 2012 completed diary-assisted 24-hour recalls and a questionnaire that included five questions from the Child Food Security Assessment. MAIN OUTCOME MEASURES: Individual dietary components (including total energy, nutrients, and sugar-sweetened beverages), Healthy Eating Index-2010 scores, and meal patterns (such as meal sizes and missed meals) were derived from 24-hour recalls. STATISTICAL ANALYSES: Multivariable linear and logistic regression models were used to estimate the relationships between food insecurity and diet characteristics. RESULTS: Girls with the highest food insecurity consumed 135 total kilocalories (P<0.005) and 60 snack kilocalories (P<0.05) more per day than girls with no food insecurity. These relationships were absent among boys. CONCLUSIONS: Food insecurity among girls in grades 4 and 5 was associated with higher energy intake. Findings support the need for further research to better understand the nature of this relationship and its implications for energy balance.

Development and implementation of a food store-based intervention to improve diet in the Republic of the Marshall Islands.

Effective approaches for the prevention and reduction of obesity and obesity-related chronic diseases are urgently needed. Food store-centered programs represent one approach that may be both effective and sustainable. The authors developed a food store-based intervention in the Marshall Islands using qualitative and quantitative formative research methods, including a store usage survey (n = 184) and in-depth interviews with large-store managers (n = 13), small-store managers (n = 7), customers (n = 10), and community leaders (n = 4). This process was followed up by development and piloting of specific intervention components and workshops to finalize materials. The final intervention combined mass media (newspaper articles, video, radio announcements) and in-store components (shelf labels, cooking demonstrations, posters, recipe cards) and had high store-owner support and participation. High levels of exposure to the intervention were achieved during the 10-week period of implementation. This model for developing food store-based interventions is applicable to other settings.