Problematic meta-analyses: Bayesian and frequentist perspectives on combining randomized controlled trials and non-randomized studies.

PURPOSE: In the literature, the propriety of the meta-analytic treatment-effect produced by combining randomized controlled trials (RCT) and non-randomized studies (NRS) is questioned, given the inherent confounding in NRS that may bias the meta-analysis. The current study compared an implicitly principled pooled Bayesian meta-analytic treatment-effect with that of frequentist pooling of RCT and NRS to determine how well each approach handled the NRS bias. MATERIALS & METHODS: Binary outcome Critical-Care meta-analyses, reflecting the importance of such outcomes in Critical-Care practice, combining RCT and NRS were identified electronically. Bayesian pooled treatment-effect and 95% credible-intervals (BCrI), posterior model probabilities indicating model plausibility and Bayes-factors (BF) were estimated using an informative heavy-tailed heterogeneity prior (half-Cauchy). Preference for pooling of RCT and NRS was indicated for Bayes-factors > 3 or < 0.333 for the converse. All pooled frequentist treatment-effects and 95% confidence intervals (FCI) were re-estimated using the popular DerSimonian-Laird (DSL) random effects model. RESULTS: Fifty meta-analyses were identified (2009-2021), reporting pooled estimates in 44; 29 were pharmaceutical-therapeutic and 21 were non-pharmaceutical therapeutic. Re-computed pooled DSL FCI excluded the null (OR or RR = 1) in 86% (43/50). In 18 meta-analyses there was an agreement between FCI and BCrI in excluding the null. In 23 meta-analyses where FCI excluded the null, BCrI embraced the null. BF supported a pooled model in 27 meta-analyses and separate models in 4. The highest density of the posterior model probabilities for 0.333 < Bayes factor < 1 was 0.8. CONCLUSIONS: In the current meta-analytic cohort, an integrated and multifaceted Bayesian approach gave support to including NRS in a pooled-estimate model. Conversely, caution should attend the reporting of naïve frequentist pooled, RCT and NRS, meta-analytic treatment effects.

Modelling of intensive care unit (ICU) length of stay as a quality measure: a problematic exercise.

BACKGROUND: Intensive care unit (ICU) length of stay (LOS) and the risk adjusted equivalent (RALOS) have been used as quality metrics. The latter measures entail either ratio or difference formulations or ICU random effects (RE), which have not been previously compared. METHODS: From calendar year 2016 data of an adult ICU registry-database (Australia & New Zealand Intensive Care Society (ANZICS) CORE), LOS predictive models were established using linear (LMM) and generalised linear (GLMM) mixed models. Model fixed effects quality-metric formulations were estimated as RALOSR for LMM (geometric mean derived from log(ICU LOS)) and GLMM (day) and observed minus expected ICU LOS (OMELOS from GLMM). Metric confidence intervals (95%CI) were estimated by bootstrapping; random effects (RE) were predicted for LMM and GLMM. Forest-plot displays of ranked quality-metric point-estimates (95%CI) were generated for ICU hospital classifications (metropolitan, private, rural/regional, and tertiary). Robust rank confidence sets (point estimate and 95%CI), both marginal (pertaining to a singular ICU) and simultaneous (pertaining to all ICU differences), were established. RESULTS: The ICU cohort was of 94,361 patients from 125 ICUs (metropolitan 16.9%, private 32.8%, rural/regional 6.4%, tertiary 43.8%). Age (mean, SD) was 61.7 (17.5) years; 58.3% were male; APACHE III severity-of-illness score 54.6 (25.7); ICU annual patient volume 1192 (702) and ICU LOS 3.2 (4.9). There was no concordance of ICU ranked model predictions, GLMM versus LMM, nor for the quality metrics used, RALOSR, OMELOS and site-specific RE for each of the ICU hospital classifications. Furthermore, there was no concordance between ICU ranking confidence sets, marginal and simultaneous for models or quality metrics. CONCLUSIONS: Inference regarding adjusted ICU LOS was dependent upon the statistical estimator and the quality index used to quantify any LOS differences across ICUs. That is, there was no "one best model"; thus, ICU "performance" is determined by model choice and any rankings thereupon should be circumspect.

Estimating causal effects for multivalued treatments: a comparison of approaches.

Interventions with multivalued treatments are common in medical and health research, such as when comparing the efficacy of competing drugs or interventions, or comparing between various doses of a particular drug. In recent years, there has been a growing interest in the development of multivalued treatment effect estimators using observational data. In this paper, we compare the performance of commonly used regression-based methods that estimate multivalued treatment effects based on the unconfoundedness assumption. These estimation methods fall into three general categories: (i) estimators based on a model for the outcome variable using conventional regression adjustment; (ii) weighted estimators based on a model for the treatment assignment; and (iii) 'doubly-robust' estimators that model both the treatment assignment and outcome variable within the same framework. We assess the performance of these models using Monte Carlo simulation and demonstrate their application with empirical data. Our results show that (i) when models estimating both the treatment and outcome are correctly specified, all adjustment methods provide similar unbiased estimates; (ii) when the outcome model is misspecified, regression adjustment performs poorly, while all the weighting methods provide unbiased estimates; (iii) when the treatment model is misspecified, methods based solely on modeling the treatment perform poorly, while regression adjustment and the doubly robust models provide unbiased estimates; and (iv) when both the treatment and outcome models are misspecified, all methods perform poorly. Given that researchers will rarely know which of the two models is misspecified, our results support the use of doubly robust estimation.

Assessing regression to the mean effects in health care initiatives.

BACKGROUND: Interventions targeting individuals classified as "high-risk" have become common-place in health care. High-risk may represent outlier values on utilization, cost, or clinical measures. Typically, such individuals are invited to participate in an intervention intended to reduce their level of risk, and after a period of time, a follow-up measurement is taken. However, individuals initially identified by their outlier values will likely have lower values on re-measurement in the absence of an intervention. This statistical phenomenon is known as "regression to the mean" (RTM) and often leads to an inaccurate conclusion that the intervention caused the effect. Concerns about RTM are rarely raised in connection with most health care interventions, and it is uncommon to find evaluators who estimate its effect. This may be due to lack of awareness, cognitive biases that may cause people to systematically misinterpret RTM effects by creating (erroneous) explanations to account for it, or by design. METHODS: In this paper, the author fully describes the RTM phenomenon, and tests the accuracy of the traditional approach in calculating RTM assuming normality, using normally distributed data from a Monte Carlo simulation and skewed data from a control group in a pre-post evaluation of a health intervention. Confidence intervals are generated around the traditional RTM calculation to provide more insight into the potential magnitude of the bias introduced by RTM. Finally, suggestions are offered for designing interventions and evaluations to mitigate the effects of RTM. RESULTS: On multivariate normal data, the calculated RTM estimates are identical to true estimates. As expected, when using skewed data the calculated method underestimated the true RTM effect. Confidence intervals provide helpful guidance on the magnitude of the RTM effect. CONCLUSION: Decision-makers should always consider RTM to be a viable explanation of the observed change in an outcome in a pre-post study, and evaluators of health care initiatives should always take the appropriate steps to estimate the magnitude of the effect and control for it when possible. Regardless of the cause, failure to address RTM may result in wasteful pursuit of ineffective interventions, both at the organizational level and at the policy level.

Advancing current approaches to disease management evaluation: capitalizing on heterogeneity to understand what works and for whom.

BACKGROUND: Evaluating large-scale disease management interventions implemented in actual health care settings is a complex undertaking for which universally accepted methods do not exist. Fundamental issues, such as a lack of control patients and limited generalizability, hamper the use of the 'gold-standard' randomized controlled trial, while methodological shortcomings restrict the value of observational designs. Advancing methods for disease management evaluation in practice is pivotal to learn more about the impact of population-wide approaches. Methods must account for the presence of heterogeneity in effects, which necessitates a more granular assessment of outcomes. METHODS: This paper introduces multilevel regression methods as valuable techniques to evaluate 'real-world' disease management approaches in a manner that produces meaningful findings for everyday practice. In a worked example, these methods are applied to retrospectively gathered routine health care data covering a cohort of 105,056 diabetes patients who receive disease management for type 2 diabetes mellitus in the Netherlands. Multivariable, multilevel regression models are fitted to identify trends in clinical outcomes and correct for differences in characteristics of patients (age, disease duration, health status, diabetes complications, smoking status) and the intervention (measurement frequency and range, length of follow-up). RESULTS: After a median one year follow-up, the Dutch disease management approach was associated with small average improvements in systolic blood pressure and low-density lipoprotein, while a slight deterioration occurred in glycated hemoglobin. Differential findings suggest that patients with poorly controlled diabetes tend to benefit most from disease management in terms of improved clinical measures. Additionally, a greater measurement frequency was associated with better outcomes, while longer length of follow-up was accompanied by less positive results. CONCLUSIONS: Despite concerted efforts to adjust for potential sources of confounding and bias, there ultimately are limits to the validity and reliability of findings from uncontrolled research based on routine intervention data. While our findings are supported by previous randomized research in other settings, the trends in outcome measures presented here may have alternative explanations. Further practice-based research, perhaps using historical data to retrospectively construct a control group, is necessary to confirm results and learn more about the impact of population-wide disease management.

Electronic connectivity between hospital pairs: impact on emergency department-related utilization.

OBJECTIVE: To use more precise measures of which hospitals are electronically connected to determine whether health information exchange (HIE) is associated with lower emergency department (ED)-related utilization. MATERIALS AND METHODS: We combined 2018 Medicare fee-for-service claims to identify beneficiaries with 2 ED encounters within 30 days, and Definitive Healthcare and AHA IT Supplement data to identify hospital participation in HIE networks (HIOs and EHR vendor networks). We determined whether the 2 encounters for the same beneficiary occurred at: the same organization, different organizations connected by HIE, or different organizations not connected by HIE. Outcomes were: (1) whether any repeat imaging occurred during the second ED visit; (2) for beneficiaries with a treat-and-release ED visit followed by a second ED visit, whether they were admitted to the hospital after the second visit; (3) for beneficiaries discharged from the hospital followed by an ED visit, whether they were admitted to the hospital. RESULTS: In adjusted mixed effects models, for all outcomes, beneficiaries returning to the same organization had significantly lower utilization compared to those going to different organizations. Comparing only those going to different organizations, HIE was not associated with lower levels of repeat imaging. HIE was associated with lower likelihood of hospital admission following a treat-and-release ED visit (1.83 percentage points [-3.44 to -0.21]) but higher likelihood of admission following hospital discharge (2.78 percentage points [0.48-5.08]). DISCUSSION: Lower utilization for beneficiaries returning to the same organization could reflect better access to information or other factors such as aligned incentives. CONCLUSION: HIE is not consistently associated with utilization outcomes reflecting more coordinated care in the ED setting.

Association of Primary Care Engagement in Value-Based Reform Programs With Health Services Outcomes: Participation and Synergies.

Importance: Policy makers envision synergistic benefits from primary care reform programs that advance infrastructure and processes in the context of a supportive payment environment. However, these programs have been operationalized and implemented separately, raising the question of whether synergies are achieved. Objective: To evaluate associations between primary care engagement in voluntary delivery system and/or payment reform programs and health services outcomes. Design Setting and Participants: This was an observational longitudinal analysis of US ambulatory primary care organizations (PCOs) with attributed Medicare fee-for-service beneficiaries (1.6-1.9 million unique beneficiaries annually) using data for 2009, 2010, and 2015 to 2017; PCOs included multispecialty practices that delivered primary care. Data analyses were performed from January 2020 to December 2021. Exposures: Annual PCO participation in or recognition by (1) the Centers for Medicare & Medicaid's meaningful use (MU) program, (2) the National Committee for Quality Assurance's Patient-Centered Medical Home (PCMH) program, and/or (3) the Medicare Shared Savings Program (MSSP), an Accountable Care Organizations program. Main Outcomes and Measures: Independent and joint associations between an additional year of participation by a PCO in each of the 3 reform programs, and 3 types of outcomes: (1) hospital utilization (all-cause admissions, ambulatory care sensitive admissions, all-cause readmissions, all-cause emergency department visits); (2) evidence-based diabetes guideline adherence (≥1 annual glycated hemoglobin test, low-density lipoprotein cholesterol test, nephropathy screening, and eye examination); and (3) Medicare spending (total, acute inpatient, and skilled nursing facility). Results: The study sample comprised 47 880 unique PCOs (size ≤10 beneficiaries, 50%; ≤1-2 clinicians, 65%) and approximately 5.61 million unique Medicare beneficiaries (mean [SD] age, 71.4 [12.7] years; 3 207 568 [57.14%] women; 4 474 541 [79.71%] non-Hispanic White individuals) across the study years (2009, 2010, 2015-2017). Of the hospital utilization measures, only ambulatory care sensitive admission was associated with improved performance, showing a statistically significant marginal effect size for joint participation in MU and MSSP (-0.0002; 95% CI, -0.0005 to 0.0000) and MSSP alone (-0.0003; 95% CI, -0.0005 to -0.0001). For diabetes adherence, joint participation in PCMH and MU was associated with 0.06 more measures met (95% CI, 0.03 to 0.10) while participation in all 3 programs was associated with 0.05 more measures met (95% CI, 0.02 to 0.09). Stand-alone PCMH and stand-alone MU participation were also associated with improved performance. Joint participation in MU and MSSP was associated with $33.89 lower spending (95% CI, -$65.79 to -$1.99) as was stand-alone MSSP participation (-$37.04; 95% CI, -$65.73 to -$8.35). Conclusions and Relevance: This longitudinal observational study found that participation by PCOs in single or multiple reform programs was associated with better performance for only a subset of health services outcomes. More consistent and larger synergies may be realized with improved alignment of program requirements and goals.

Longitudinal participation in delivery and payment reform programs among US Primary Care Organizations.

OBJECTIVE: To assess longitudinal primary care organization participation patterns in large-scale reform programs and identify organizational characteristics associated with multiprogram participation. DATA SOURCES: Secondary data analysis of national program participation data over an eight-year period (2009-2016). STUDY DESIGN: We conducted a retrospective, observational study by creating a unique set of data linkages (including Medicare and Medicaid Meaningful Use and Medicare Shared Savings Program Accountable Care Organization (MSSP ACO) participation from CMS, Patient-Centered Medical Home (PCMH) participation from the National Committee for Quality Assurance, and organizational characteristics) to measure longitudinal participation and identify what types of organizations participate in one or more of these reform programs. We used multivariate models to identify organizational characteristics that differentiate those that participate in none, one, or two-to-three programs. DATA EXTRACTION METHODS: We used Medicare claims to identify organizations that delivered primary care services (n = 56 ,287) and then linked organizations to program participation data and characteristics. PRINCIPAL FINDINGS: No program achieved more than 50% participation across the 56,287 organizations in a given year, and participation levels flattened or decreased in later years. 36% of organizations did not participate in any program over the eight-year study period; 50% participated in one; 13% in two; and 1% in all three. 14.31% of organizations participated in five or more years of Meaningful Use while 3.84% of organizations participated in five years of the MSSP ACO Program and 0.64% participated in at least five years of PCMH. Larger organizations, those with younger providers, those with more primary care providers, and those with larger Medicare patient panels were more likely to participate in more programs. CONCLUSIONS AND RELEVANCE: Primary care transformation via use of voluntary programs, each with their own participation requirements and approach to incentives, has failed to broadly engage primary care organizations. Those that have chosen to participate in multiple programs are likely those already providing high-quality care.

Using randomization tests to assess treatment effects in multiple-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple-group ITSA typically has small sample sizes, and parametric methods for multiple-group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non-parametric, distribution-free option for computing exact P values. METHOD: The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)-the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)-a commonly used parametric approach for evaluating treatment effects in ITSA studies. RESULTS: Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P < 0.01). CONCLUSIONS: In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.

Now trending: Coping with non-parallel trends in difference-in-differences analysis.

Difference-in-differences (DID) analysis is used widely to estimate the causal effects of health policies and interventions. A critical assumption in DID is "parallel trends": that pre-intervention trends in outcomes are the same between treated and comparison groups. To date, little guidance has been available to researchers who wish to use DID when the parallel trends assumption is violated. Using a Monte Carlo simulation experiment, we tested the performance of several estimators (standard DID; DID with propensity score matching; single-group interrupted time-series analysis; and multi-group interrupted time-series analysis) when the parallel trends assumption is violated. Using nationwide data from US hospitals (n = 3737) for seven data periods (four pre-interventions and three post-interventions), we used alternative estimators to evaluate the effect of a placebo intervention on common outcomes in health policy (clinical process quality and 30-day risk-standardized mortality for acute myocardial infarction, heart failure, and pneumonia). Estimator performance was assessed using mean-squared error and estimator coverage. We found that mean-squared error values were considerably lower for the DID estimator with matching than for the standard DID or interrupted time-series analysis models. The DID estimator with matching also had superior performance for estimator coverage. Our findings were robust across all outcomes evaluated.

Practice strategies to improve primary care for chronic disease patients under a pay-for-value program.

BACKGROUND: Improving primary care for patients with chronic illness is critical to advancing healthcare quality and value. Yet, little is known about what strategies are successful in helping primary care practices deliver high-quality care for this population under value-based payment models. METHODS: Double-blind interviews in 14 primary care practices in the state of Michigan, stratified based on whether they did (n = 7) or did not (n = 7) demonstrate improvement in primary care outcomes for patients with at least one reported chronic disease between 2010 and 2013. All practices participate in a statewide pay-for-performance program run by a large commercial payer. Using an implementation science framework to identify leverage points for effecting organizational change, we sought to identify, describe and compare strategies among improving and non-improving practices across three domains: (1) organizational learning opportunities, (2) approaches to motivating staff, and (3) acquisition and use of resources. RESULTS: We identified 10 strategies; 6 were "differentiating" - that is, more prevalent among improving practices. These differentiating strategies included: (1) participation in learning collaboratives, (2) accessing payer tools to monitor quality performance, (3) framing pay-for-performance as a practice transformation opportunity, (4) reinvesting earned incentive money in equitable, practice-centric improvement, (5) employing a care manager, and (6) using available technical support from local hospitals and provider organizations to support performance improvement. Implementation of these strategies varied based on organizational context and relative strengths. CONCLUSIONS: Practices that succeeded in improving care for chronic disease patients pursued a mix of strategies that helped meet immediate care delivery needs while also creating new adaptive structures and processes to better respond to changing pressures and demands. These findings help inform payers and primary care practices seeking evidence-based strategies to foster a stronger delivery system for patients with significant healthcare needs.

Medicare disease management in policy context.

Interim results of the Medicare health support (MHS) demonstration projects suggest that commercial disease management (DM) is unable to deliver short-term medical cost savings. This is not surprising given the current DM program focus on compliance with process measures that may only lead to cost savings in the long-term. A program focused on reducing near-term hospitalizations is more likely to deliver savings during the initial 3-year phase of MHS. If the early trends in MHS are indicative of the final results, CMS will face the decision of whether to abandon commercial DM in favor of other chronic care management strategies. This article supports the upcoming assessment by describing the characteristics of the current commercial DM model that limit its ability to deliver short-term medical cost savings and the changes required to overcome these limitations.

Sample size in disease management program evaluation: the challenge of demonstrating a statistically significant reduction in admissions.

Prior to implementing a disease management (DM) strategy, a needs assessment should be conducted to determine whether sufficient opportunity exists for an intervention to be successful in the given population. A central component of this assessment is a sample size analysis to determine whether the population is of sufficient size to allow the expected program effect to achieve statistical significance. This paper discusses the parameters that comprise the generic sample size formula for independent samples and their interrelationships, followed by modifications for the DM setting. In addition, a table is provided with sample size estimates for various effect sizes. Examples are described in detail along with strategies for overcoming common barriers. Ultimately, conducting these calculations up front will help set appropriate expectations about the ability to demonstrate the success of the intervention.

A matching framework to improve causal inference in interrupted time-series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time-series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. When ITSA is implemented without a comparison group, the internal validity may be quite poor. Therefore, adding a comparable control group to serve as the counterfactual is always preferred. This paper introduces a novel matching framework, ITSAMATCH, to create a comparable control group by matching directly on covariates and then use these matches in the outcomes model. METHOD: We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. We compare ITSAMATCH results to 2 commonly used matching approaches, synthetic controls (SYNTH), and regression adjustment; SYNTH reweights nontreated units to make them comparable to the treated unit, and regression adjusts covariates directly. Methods are compared by assessing covariate balance and treatment effects. RESULTS: Both ITSAMATCH and SYNTH achieved covariate balance and estimated similar treatment effects. The regression model found no treatment effect and produced inconsistent covariate adjustment. CONCLUSIONS: While the matching framework achieved results comparable to SYNTH, it has the advantage of being technically less complicated, while producing statistical estimates that are straightforward to interpret. Conversely, regression adjustment may "adjust away" a treatment effect. Given its advantages, ITSAMATCH should be considered as a primary approach for evaluating treatment effects in multiple-group time-series analysis.

Using group-based trajectory modelling to enhance causal inference in interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Several enhancements have been proposed for interrupted time series analysis (ITSA) to improve causal inference. Presently, group-based trajectory modelling (GBTM) is introduced as a complement to ITSA. GBTM assumes a certain number of discrete groups in the sample have unique trajectories of the outcome. GBTM is used herein for 2 purposes: (1) to compare outcomes across all trajectory groups via a stand-alone GBTM and (2) to identify comparable non-treated units to serve as controls in the ITSA outcome model. Examples of each are offered. METHOD: The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California to other states not exposed to smoking reduction initiatives. In the stand-alone GBTM, distinct trajectory groups are identified based on cigarette sales for the entire observation period (1970-2000). In the second approach, a GBTM is generated using only baseline period observations (1970-1988), and treatment effects (difference in post-intervention trends) are then estimated for the treatment unit versus non-treated units in the treated unit's trajectory group. RESULTS: In the stand-alone GBTM, 3 distinct trajectory groups were identified: low-decreasing, medium-decreasing, and high-decreasing (California and 26 other states were in the low-decreasing group). When using baseline data for matching, California and 19 non-treated states comprised the low group. California had a significantly larger decrease in post-intervention cigarette sales than these controls (P < 0.01). CONCLUSIONS: GBTM enhances ITSA by providing perspective for the outcome trajectory in the treated unit's group versus all others and can identify non-treated units to be used for estimating treatment effects.

Using forecast modelling to evaluate treatment effects in single-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. ITSA is commonly evaluated using methods which may produce biased results if model assumptions are violated. In this paper, treatment effects are alternatively assessed by using forecasting methods to closely fit the preintervention observations and then forecast the post-intervention trend. A treatment effect may be inferred if the actual post-intervention observations diverge from the forecasts by some specified amount. METHOD: The forecasting approach is demonstrated using the effect of California's Proposition 99 for reducing cigarette sales. Three forecast models are fit to the preintervention series-linear regression (REG), Holt-Winters (HW) non-seasonal smoothing, and autoregressive moving average (ARIMA)-and forecasts are generated into the post-intervention period. The actual observations are then compared with the forecasts to assess intervention effects. RESULTS: The preintervention data were fit best by HW, followed closely by ARIMA. REG fit the data poorly. The actual post-intervention observations were above the forecasts in HW and ARIMA, suggesting no intervention effect, but below the forecasts in the REG (suggesting a treatment effect), thereby raising doubts about any definitive conclusion of a treatment effect. CONCLUSIONS: In a single-group ITSA, treatment effects are likely to be biased if the model is misspecified. Therefore, evaluators should consider using forecast models to accurately fit the preintervention data and generate plausible counterfactual forecasts, thereby improving causal inference of treatment effects in single-group ITSA studies.

Using permutation tests to enhance causal inference in interrupted time series analysis.

RATIONALE, AIMS AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robustness check based on permutation tests to further improve causal inference. METHOD: We evaluate the effect of California's Proposition 99 for reducing cigarette sales by iteratively casting each nontreated state into the role of "treated," creating a comparable control group using the ITSAMATCH package in Stata, and then evaluating treatment effects using ITSA regression. If statistically significant "treatment effects" are estimated for pseudotreated states, then any significant changes in the outcome of the actual treatment unit (California) cannot be attributed to the intervention. We perform these analyses setting the cutpoint significance level to P > .40 for identifying balanced matches (the highest threshold possible for which controls could still be found for California) and use the difference in differences of trends as the treatment effect estimator. RESULTS: Only California attained a statistically significant treatment effect, strengthening confidence in the conclusion that Proposition 99 reduced cigarette sales. CONCLUSIONS: The proposed permutation testing framework provides an additional robustness check to either support or refute a treatment effect identified in for the true treated unit in ITSA. Given its value and ease of implementation, this framework should be considered as a standard robustness test in all multiple group interrupted time series analyses.

Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation.

RATIONALE, AIMS AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robust evaluation framework that combines the synthetic controls method (SYNTH) to generate a comparable control group and ITSA regression to assess covariate balance and estimate treatment effects. METHODS: We evaluate the effect of California's Proposition 99 for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. SYNTH is used to reweight nontreated units to make them comparable to the treated unit. These weights are then used in ITSA regression models to assess covariate balance and estimate treatment effects. RESULTS: Covariate balance was achieved for all but one covariate. While California experienced a significant decrease in the annual trend of cigarette sales after Proposition 99, there was no statistically significant treatment effect when compared to synthetic controls. CONCLUSIONS: The advantage of using this framework over regression alone is that it ensures that a comparable control group is generated. Additionally, it offers a common set of statistical measures familiar to investigators, the capability for assessing covariate balance, and enhancement of the evaluation with a comprehensive set of postestimation measures. Therefore, this robust framework should be considered as a primary approach for evaluating treatment effects in multiple group time series analysis.

Estimating causal effects for survival (time-to-event) outcomes by combining classification tree analysis and propensity score weighting.

RATIONALE, AIMS AND OBJECTIVES: A common approach to assessing treatment effects in nonrandomized studies with time-to-event outcomes is to estimate propensity scores and compute weights using logistic regression, test for covariate balance, and then estimate treatment effects using Cox regression. A machine-learning alternative-classification tree analysis (CTA)-used to generate propensity scores and to estimate treatment effects in time-to-event data may identify complex relationships between covariates not found using conventional regression-based approaches. METHOD: Using empirical data, we identify all statistically valid CTA propensity score models and then use them to compute strata-specific, observation-level propensity score weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to the conventional method, by evaluating covariate balance and treatment effect estimates obtained using Cox regression and a weighted CTA outcomes model. RESULTS: All models had some imbalanced covariates. Nevertheless, treatment effect estimates were generally consistent across all weighted models. CONCLUSIONS: In the study sample, given that all approaches elicited similar results, using CTA increased confidence that bias could not be reduced any further. Because the CTA algorithm identifies all statistically valid propensity score models for a sample, it is most likely to identify a correctly specified propensity score model-and therefore should be used either to confirm results using traditional methods, or to reveal biases that may be missed by traditional methods. Moreover, given that the true treatment effect is never known in observational data, CTA should be considered for estimating outcomes because no statistical assumptions are required.

Identifying causal mechanisms in health care interventions using classification tree analysis.

RATIONALE, AIMS, AND OBJECTIVES: Mediation analysis identifies causal pathways by testing the relationships between the treatment, the outcome, and an intermediate variable that mediates the relationship between the treatment and outcome. This paper introduces classification tree analysis (CTA), a machine-learning procedure, as an alternative to conventional methods for analysing mediation effects. METHOD: Using data from the JOBS II study, we compare CTA to structural equation models (SEMs) by assessing their consistency in revealing mediation effects on 2 outcomes; reemployment (a binary variable) and depressive symptoms (a continuous variable). Because study participants were not randomized sequentially to both treatment and mediator, an additional model was generated including baseline covariates to strengthen the validity of some key identifying assumptions required of all mediation analyses. RESULTS: Using SEM, no statistically significant treatment or mediated effects were found for either outcome. In contrast, CTA found a significant treatment effect for reemployment (P = .047) and a mediated pathway for individuals in the treatment group (P = .014). No CTA model could be generated for the reemployment outcome. When covariates were added to the model, CTA found numerous interactions, while SEM found no effects. CONCLUSIONS: CTA may uncover mediation effects where conventional approaches do not, because CTA does not require any assumptions about the distribution of variables nor of the functional form of the model, and CTA will systematically identify all statistically viable interactions. The versatility of CTA enables the investigator to explore the theorized underlying causal mechanism of an intervention in a much more comprehensive manner than conventional mediation analytic approaches.