Estimating generalized propensity scores with survey and attrition weighted data.

Prior work in causal inference has shown that using survey sampling weights in the propensity score estimation stage and the outcome model stage for binary treatments can result in a more robust estimator of the effect of the binary treatment being analyzed. However, to date, extending this work to continuous treatments and exposures has not been explored nor has consideration been given for how to handle attrition weights in the propensity score model. Nonetheless, generalized propensity score (GPS) analyses are being used for estimating continuous treatment effects on outcomes when researchers have observational data, and those data sets often have survey or attrition weights that need to be accounted for in the analysis. Here, we extend prior work and show with analytic results that using survey sampling or attrition weights in the GPS estimation stage and the outcome model stage for continuous treatments can result in a more robust estimator than one that does not. Simulation study results show that, although using weights in both estimation stages is sufficient for robust estimation, it is not necessary and unbiased estimation is possible in some cases under various approaches to using weights in estimation. Analysts do not know if the conditions of our simulation studies hold, so use of weights in both estimation stages might provide insurance for reducing potential bias. We discuss the implications of our results in the context of an empirical example.

Sampling weighting strategies in causal mediation analysis.

BACKGROUND: Causal mediation analysis plays a crucial role in examining causal effects and causal mechanisms. Yet, limited work has taken into consideration the use of sampling weights in causal mediation analysis. In this study, we compared different strategies of incorporating sampling weights into causal mediation analysis. METHODS: We conducted a simulation study to assess 4 different sampling weighting strategies-1) not using sampling weights, 2) incorporating sampling weights into mediation "cross-world" weights, 3) using sampling weights when estimating the outcome model, and 4) using sampling weights in both stages. We generated 8 simulated population scenarios comprising an exposure (A), an outcome (Y), a mediator (M), and six covariates (C), all of which were binary. The data were generated so that the true model of A given C and the true model of A given M and C were both logit models. We crossed these 8 population scenarios with 4 different sampling methods to obtain 32 total simulation conditions. For each simulation condition, we assessed the performance of 4 sampling weighting strategies when calculating sample-based estimates of the total, direct, and indirect effects. We also applied the four sampling weighting strategies to a case study using data from the National Survey on Drug Use and Health (NSDUH). RESULTS: Using sampling weights in both stages (mediation weight estimation and outcome models) had the lowest bias under most simulation conditions examined. Using sampling weights in only one stage led to greater bias for multiple simulation conditions. DISCUSSION: Using sampling weights in both stages is an effective approach to reduce bias in causal mediation analyses under a variety of conditions regarding the structure of the population data and sampling methods.

A Tutorial for Propensity Score Weighting for Moderation Analysis With Categorical Variables: An Application Examining Smoking Disparities Among Sexual Minority Adults.

OBJECTIVE: To provide step-by-step guidance and STATA and R code for using propensity score (PS) weighting to estimate moderation effects with categorical variables. RESEARCH DESIGN: Tutorial illustrating the key steps for estimating and testing moderation using observational data. Steps include: (1) examining covariate overlap across treatment groups within levels of the moderator; (2) estimating the PS weights; (3) evaluating whether PS weights improved covariate balance; (4) estimating moderated treatment effects; and (5) assessing the sensitivity of findings to unobserved confounding. Our illustrative case study uses data from 41,832 adults from the 2019 National Survey on Drug Use and Health to examine if gender moderates the association between sexual minority status (eg, lesbian, gay, or bisexual [LGB] identity) and adult smoking prevalence. RESULTS: For our case study, there were no noted concerns about covariate overlap, and we were able to successfully estimate the PS weights within each level of the moderator. Moreover, balance criteria indicated that PS weights successfully achieved covariate balance for both moderator groups. PS-weighted results indicated there was significant evidence of moderation for the case study, and sensitivity analyses demonstrated that results were highly robust for one level of the moderator but not the other. CONCLUSIONS: When conducting moderation analyses, covariate imbalances across levels of the moderator can cause biased estimates. As demonstrated in this tutorial, PS weighting within each level of the moderator can improve the estimated moderation effects by minimizing bias from imbalance within the moderator subgroups.

Best practices in statistical computing.

The world is becoming increasingly complex, both in terms of the rich sources of data we have access to and the statistical and computational methods we can use on data. These factors create an ever-increasing risk for errors in code and the sensitivity of findings to data preparation and the execution of complex statistical and computing methods. The consequences of coding and data mistakes can be substantial. In this paper, we describe the key steps for implementing a code quality assurance (QA) process that researchers can follow to improve their coding practices throughout a project to assure the quality of the final data, code, analyses, and results. These steps include: (i) adherence to principles for code writing and style that follow best practices; (ii) clear written documentation that describes code, workflow, and key analytic decisions; (iii) careful version control; (iv) good data management; and (v) regular testing and review. Following these steps will greatly improve the ability of a study to assure results are accurate and reproducible. The responsibility for code QA falls not only on individual researchers but institutions, journals, and funding agencies as well.

The Right Tool for the Job: Choosing Between Covariate-balancing and Generalized Boosted Model Propensity Scores.

Estimating the causal effect of an exposure (vs. some control) on an outcome using observational data often requires addressing the fact that exposed and control groups differ on pre-exposure characteristics that may be related to the outcome (confounders). Propensity score methods have long been used as a tool for adjusting for observed confounders in order to produce more valid causal effect estimates under the strong ignorability assumption. In this article, we compare two promising propensity score estimation methods (for time-invariant binary exposures) when assessing the average treatment effect on the treated: the generalized boosted models and covariate-balancing propensity scores, with the main objective to provide analysts with some rules-of-thumb when choosing between these two methods. We compare the methods across different dimensions including the presence of extraneous variables, the complexity of the relationship between exposure or outcome and covariates, and the residual variance in outcome and exposure. We found that when noncomplex relationships exist between outcome or exposure and covariates, the covariate-balancing method outperformed the boosted method, but under complex relationships, the boosted method performed better. We lay out criteria for when one method should be expected to outperform the other with no blanket statement on whether one method is always better than the other.

Time-varying effect moderation using the structural nested mean model: estimation using inverse-weighted regression with residuals.

This article considers the problem of examining time-varying causal effect moderation using observational, longitudinal data in which treatment, candidate moderators, and possible confounders are time varying. The structural nested mean model (SNMM) is used to specify the moderated time-varying causal effects of interest in a conditional mean model for a continuous response given time-varying treatments and moderators. We present an easy-to-use estimator of the SNMM that combines an existing regression-with-residuals (RR) approach with an inverse-probability-of-treatment weighting (IPTW) strategy. The RR approach has been shown to identify the moderated time-varying causal effects if the time-varying moderators are also the sole time-varying confounders. The proposed IPTW+RR approach provides estimators of the moderated time-varying causal effects in the SNMM in the presence of an additional, auxiliary set of known and measured time-varying confounders. We use a small simulation experiment to compare IPTW+RR versus the traditional regression approach and to compare small and large sample properties of asymptotic versus bootstrap estimators of the standard errors for the IPTW+RR approach. This article clarifies the distinction between time-varying moderators and time-varying confounders. We illustrate the methodology in a case study to assess if time-varying substance use moderates treatment effects on future substance use.

A tutorial on propensity score estimation for multiple treatments using generalized boosted models.

The use of propensity scores to control for pretreatment imbalances on observed variables in non-randomized or observational studies examining the causal effects of treatments or interventions has become widespread over the past decade. For settings with two conditions of interest such as a treatment and a control, inverse probability of treatment weighted estimation with propensity scores estimated via boosted models has been shown in simulation studies to yield causal effect estimates with desirable properties. There are tools (e.g., the twang package in R) and guidance for implementing this method with two treatments. However, there is not such guidance for analyses of three or more treatments. The goals of this paper are twofold: (1) to provide step-by-step guidance for researchers who want to implement propensity score weighting for multiple treatments and (2) to propose the use of generalized boosted models (GBM) for estimation of the necessary propensity score weights. We define the causal quantities that may be of interest to studies of multiple treatments and derive weighted estimators of those quantities. We present a detailed plan for using GBM to estimate propensity scores and using those scores to estimate weights and causal effects. We also provide tools for assessing balance and overlap of pretreatment variables among treatment groups in the context of multiple treatments. A case study examining the effects of three treatment programs for adolescent substance abuse demonstrates the methods.

Subgroups analysis when treatment and moderators are time-varying.

Prevention scientists are often interested in understanding characteristics of participants that are predictive of treatment effects because these characteristics can be used to inform the types of individuals who benefit more or less from treatment or prevention programs. Often, effect moderation questions are examined using subgroups analysis or, equivalently, using covariate × treatment interactions in the context of regression analysis. This article focuses on conceptualizing and examining causal effect moderation in longitudinal settings in which both treatment and the putative moderators are time-varying. Studying effect moderation in the time-varying setting helps identify which individuals will benefit more or less from additional treatment services on the basis of both individual characteristics and their evolving outcomes, symptoms, severity, and need. Examining effect moderation in these longitudinal settings, however, is difficult because moderators of future treatment may themselves be affected by prior treatment (for example, future moderators may be mediators of prior treatment). This article introduces moderated intermediate causal effects in the time-varying setting, describes how they are part of Robins' Structural Nested Mean Model, discusses two problems with using a traditional regression approach to estimate these effects, and describes a new approach (a two-stage regression estimator) to estimate these effects. The methodology is illustrated using longitudinal data to examine the time-varying effects of receiving community-based substance abuse treatment as a function of time-varying severity (or need).

Comparative effectiveness of individual versus family-based substance use treatment on adolescent self-injurious thoughts and behaviors.

INTRODUCTION: Self-injurious thoughts and behaviors (SITB) are of increasing concern among adolescents, especially those who use substances. Some evidence suggests that existing evidence-based substance use treatments (EBTs) could impact not only their intended substance use targets but also SITB. However, which types of substance use treatments may have the greatest impact on youth SITB is not yet clear. Based on prior literature showing that family support and connection may buffer youth from SITB, we initially hypothesized that family-based EBTs would show greater improvement in SITB compared to those receiving individually focused EBTs and that the size of the effects would be small given the comparison between two active, evidence-based interventions, and base rates of SITB. METHODS: In a sample of 2893 youth in substance use treatment, we compared the effectiveness of individually and family-based EBTs in reducing SITBs. The study used entropy balancing and regression modeling to balance the groups on pre-treatment characteristics and examine change in outcomes over a one-year follow-up period. RESULTS: Both groups improved in self-injury and suicide attempts over the one-year study period, but only youth in individual treatment improved in suicidal ideation. However, the study found no significant difference between the changes over time in the two groups for any outcome. As expected, effect sizes were small and power was constrained in this study given the rarity of the outcomes, but effect sizes are similar to those observed with substance use outcomes. CONCLUSIONS: The results provide important exploratory evidence on the potential relative effectiveness of these two treatments for SITBs. This study supports prior findings that EBTs for youth substance use may help to improve SITB and suggests that different treatment formats (individual or family-based) could result in different benefits for SITB outcomes.

Smoothing across time in repeated cross-sectional data.

Repeated cross-sectional samples are common in national surveys of health like the National Health Interview Survey (NHIS). Because population health outcomes generally evolve slowly, pooling data across years can improve the precision of current-year annual estimates of disease prevalence and other health outcomes. Pooling over time is particularly valuable in health disparities research, where outcomes for small groups are often of interest and pooling data across groups would bias disparity estimates. State-space modeling and Kalman filtering are appealing choices for smoothing data across time. However, filtering can be problematic when few time points are available, as is common with annual cross-sectional data. Problems arise because filtering relies on estimated variance components, which can be biased and imprecise when estimated with small samples, especially when estimated in tandem with linear trends. We conduct a simulation study showing that even when trends and variance components are estimated poorly, smoothing with these estimates can improve the mean squared error (MSE) of estimated health states for multiple racial/ethnic groups when the variance components are estimated with the pooled sample. We consider frequentist estimators with no trends, one common trend across groups, and separate trends for every group, as well as shrinkage estimators of trends through a Bayesian model. We show that the Bayesian model offers the greatest improvement in MSE, and that Bayesian Information Criterion (BIC)-based model averaging of the frequentist estimators with different trend assumptions performs nearly as well. We present empirical examples using the NHIS data.

Nonparametric Estimation of Population Average Dose-Response Curves using Entropy Balancing Weights for Continuous Exposures.

Weighted estimators are commonly used for estimating exposure effects in observational settings to establish causal relations. These estimators have a long history of development when the exposure of interest is binary and where the weights are typically functions of an estimated propensity score. Recent developments in optimization-based estimators for constructing weights in binary exposure settings, such as those based on entropy balancing, have shown more promise in estimating treatment effects than those methods that focus on the direct estimation of the propensity score using likelihood-based methods. This paper explores recent developments of entropy balancing methods to continuous exposure settings and the estimation of population dose-response curves using nonparametric estimation combined with entropy balancing weights, focusing on factors that would be important to applied researchers in medical or health services research. The methods developed here are applied to data from a study assessing the effect of non-randomized components of an evidence-based substance use treatment program on emotional and substance use clinical outcomes.

Marijuana use and high school dropout: the influence of unobservables.

In this study, we reconsider the relationship between heavy and persistent marijuana use and high school dropout status. Using a unique prospective panel study of over 4500 7th grade students from South Dakota who are followed through high school, we developed propensity score weights to adjust for baseline differences found to exist before marijuana initiation occurs for most students (7th grade). We then used weighted logistic regression that incorporates these propensity score weights to examine the extent to which time-varying factors, including substance use, also influence the likelihood of dropping out of school. We found a positive association between marijuana use and dropping out (OR=5.6, RR=3.8), over half of which was explained by prior differences in observational characteristics and behaviors. The remaining association (OR=2.4, RR=1.7) became statistically insignificant when measures of cigarette smoking were included in the analysis. Because cigarette smoking is unlikely to seriously impair cognition, we interpret this result as evidence that the association between marijuana use and high school dropout is unlikely to be due to its adverse effects on cognition. We then explored which constructs drive this result, determining that they are time-varying parental and peer influences.

Switching cluster membership in cluster randomized control trials: Implications for design and analysis.

Randomized control trials (RCTs) often use clustered designs, where intact clusters (such as classroom, schools, or treatment centers) are randomly assigned to treatment and control conditions. Hierarchical linear models (HLMs) are used almost universally to estimate the effects in such experiments. While study designs that utilize intact clusters have many potential advantages, there is little guidance in the literature on how to respond when cluster switching induces noncompliance with the randomization protocol. In the presence of noncompliance the intent-to-treat (ITT) effect becomes the estimand of interest. When fitting the HLM, these individuals who switch clusters can be assigned to either their as-assigned cluster (the cluster they belonged to at the time of randomization) or their as-treated cluster (the cluster they belonged to at the time the outcome was collected). We show analytically and via simulation, that using the as-treated cluster in HLM will bias the estimate of the ITT effect and using the as-assigned cluster will bias the standard error estimates when heterogeneity among clusters is because of heterogeneity in treatment effects. We show that using linear regression with two-way cluster adjusted standard errors can yield unbiased ITT estimates and consistent standard errors regardless of the source of the random effects. We recommend this method replace HLM as the method of choice for testing intervention effects with cluster-randomized trials with noncompliance and cluster switching. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Creating a developmentally sensitive measure of adolescent alcohol misuse: an application of item response theory.

This study, funded by the US National Institute of Drug Abuse, evaluates the usefulness of item response theory (IRT) to create a developmental alcohol misuse scale. Data were collected during 1997-2006 from 5,828 Midwestern US students who completed annual surveys at grades 7 through 11 and 2 and 4 years after high school. Seventeen alcohol misuse items were calibrated with IRT and examined for differential item functioning (DIF) across 5 study waves. Eight items displayed DIF; in most cases, properties for items assessed 2 years after high school were different from those assessed in grades 8-11. Implications and suggestions for future research are discussed.

Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy.

BACKGROUND: Analysis of covariance (ANCOVA) is commonly used to adjust for potential confounders in observational studies of intervention effects. Measurement error in the covariates used in ANCOVA models can lead to inconsistent estimators of intervention effects. While errors-in-variables (EIV) regression can restore consistency, it requires surrogacy assumptions for the error-prone covariates that may be violated in practical settings. OBJECTIVES: The objectives of this article are (1) to derive asymptotic results for ANCOVA using EIV regression when measurement errors may not satisfy the standard surrogacy assumptions and (2) to demonstrate how these results can be used to explore the potential bias from ANCOVA models that either ignore measurement error by using ordinary least squares (OLS) regression or use EIV regression when its required assumptions do not hold. RESULTS: The article derives asymptotic results for ANCOVA with error-prone covariates that cover a variety of cases relevant to applications. It then uses the results in a case study of choosing among ANCOVA model specifications for estimating teacher effects using longitudinal data from a large urban school system. It finds evidence that estimates of teacher effects computed using EIV regression may have smaller bias than estimates computed using OLS regression when the data available for adjusting for students' prior achievement are limited.

Power of tests for a dichotomous independent variable measured with error.

OBJECTIVE: To examine the implications for statistical power of using predicted probabilities for a dichotomous independent variable, rather than the actual variable. DATA SOURCES/STUDY SETTING: An application uses 271,479 observations from the 2000 to 2002 CAHPS Medicare Fee-for-Service surveys. STUDY DESIGN AND DATA: A methodological study with simulation results and a substantive application to previously collected data. PRINCIPLE FINDINGS: Researchers often must employ key dichotomous predictors that are unobserved but for which predictions exist. We consider three approaches to such data: the classification estimator (1); the direct substitution estimator (2); the partial information maximum likelihood estimator (3, PIMLE). The efficiency of (1) (its power relative to testing with the true variable) roughly scales with the square of one less the classification error. The efficiency of (2) roughly scales with the R(2) for predicting the unobserved dichotomous variable, and is usually more powerful than (1). Approach (3) is most powerful, but for testing differences in means of 0.2-0.5 standard deviations, (2) is typically more than 95 percent as efficient as (3). CONCLUSIONS: The information loss from not observing actual values of dichotomous predictors can be quite large. Direct substitution is easy to implement and interpret and nearly as efficient as the PIMLE.

Inconsistencies in self-reported drug use by adolescents in substance abuse treatment: implications for outcome and performance measurements.

This article presents an analysis of logical inconsistencies in adolescents' reporting of recent substance use to assess the potential effect of inaccurate reporting on measures of treatment outcomes and program performance. We used data on 1,463 clients from 10 adolescent treatment programs to assess the relationship between inconsistent reports and various factors that contribute to program assignment and treatment outcomes. Our results suggest that inconsistencies do not arise at random. Instead, inconsistencies are associated with program assignment and factors widely considered to influence treatment outcomes, including age at first use, living situation, race/ethnicity, and mental distress. We also found a positive relationship between level of inconsistent reporting of drug use and self-reports of improvement over time on several well-established treatment outcome measures. Our study highlights the need for greater awareness on the potential impact of inaccuracies in the reporting of substance use on outcome and performance measurements and that for the development of methodologies to improve accuracy.

A prospective investigation of suicide ideation, attempts, and use of mental health service among adolescents in substance abuse treatment.

This study examined suicide ideation, attempts, and subsequent mental health service among a sample of 948 youth from substance abuse treatment facilities across the United States. Youth were surveyed at intake and every 3 months for a 1-year period. Thirty percent of youth reported ideating in at least one interview, and 12% reported attempting suicide; almost half of all youth reported receiving outpatient mental health treatment at least once, and close to one-third of all youth reported being on prescription drugs for an emotional or behavioral problem. Higher levels of conduct disorder symptoms were associated with both ideation and attempts, while higher levels of depressive symptoms and being female were associated with ideation only. Among all youth, older youth were less likely to receive outpatient and prescription drug treatment, and Black and Hispanic youth were less likely to receive prescription drug treatment than White youth. Among youth who reported ideating, those with conduct disorder were less likely to receive prescription drug treatment 3 months later. These findings emphasize a high prevalence of suicide risk behavior in substance abuse treatment programs and provide insight into the specialized treatment youth in substance abuse treatment at risk for suicide currently receive.

Developing Optimized Adaptive Interventions in Education.

Hedges (2018) encourages us to consider asking new scientific questions concerning the optimization of adaptive interventions in education. In this commentary, we have expanded on this (albeit briefly) by providing concrete examples of scientific questions and associated experimental designs to optimize adaptive interventions, and commenting on some of the ways such designs might challenge us to think differently. A great deal of methodological work remains to be done. For example, we have only begun to consider experimental design and analysis methods for developing "cluster-level adaptive interventions" (NeCamp, Kilbourne, & Almirall, 2017), or to extend methods for comparing the marginal mean trajectories between the adaptive interventions embedded in a SMART (Lu et al., 2016) to accommodate random effects. These methodological advances, among others, will propel educational research concerning the construction of more complex, yet meaningful, interventions that are necessary for improving student and teacher outcomes.

Interpreting treatment effects when cases are institutionalized after treatment.

Drug treatment clients are at high risk for institutionalization, i.e., spending a day or more in a controlled environment where their freedom to use drugs, commit crimes, or engage in risky behavior may be circumscribed. For example, in recent large studies of drug treatment outcomes, more than 40% of participants were institutionalized for a portion of the follow-up period. When longitudinal studies ignore institutionalization at follow-up, outcome measures and treatment effect estimates conflate treatment effects on institutionalization with effects on many of the outcomes of interest. In this paper, we develop a causal modeling framework for evaluating the four standard approaches for addressing this institutionalization confound, and illustrate the effects of each approach using a case study comparing drug use outcomes of youths who enter either residential or outpatient treatment modalities. Common methods provide biased estimates of the treatment effect except under improbable assumptions. In the case study, the effect of residential care ranged from beneficial and significant to detrimental and significant depending on the approach used to account for institutionalization. We discuss the implications of our analysis for longitudinal studies of all populations at high risk for institutionalization.