Electronic connectivity between hospital pairs: impact on emergency department-related utilization.

OBJECTIVE: To use more precise measures of which hospitals are electronically connected to determine whether health information exchange (HIE) is associated with lower emergency department (ED)-related utilization. MATERIALS AND METHODS: We combined 2018 Medicare fee-for-service claims to identify beneficiaries with 2 ED encounters within 30 days, and Definitive Healthcare and AHA IT Supplement data to identify hospital participation in HIE networks (HIOs and EHR vendor networks). We determined whether the 2 encounters for the same beneficiary occurred at: the same organization, different organizations connected by HIE, or different organizations not connected by HIE. Outcomes were: (1) whether any repeat imaging occurred during the second ED visit; (2) for beneficiaries with a treat-and-release ED visit followed by a second ED visit, whether they were admitted to the hospital after the second visit; (3) for beneficiaries discharged from the hospital followed by an ED visit, whether they were admitted to the hospital. RESULTS: In adjusted mixed effects models, for all outcomes, beneficiaries returning to the same organization had significantly lower utilization compared to those going to different organizations. Comparing only those going to different organizations, HIE was not associated with lower levels of repeat imaging. HIE was associated with lower likelihood of hospital admission following a treat-and-release ED visit (1.83 percentage points [-3.44 to -0.21]) but higher likelihood of admission following hospital discharge (2.78 percentage points [0.48-5.08]). DISCUSSION: Lower utilization for beneficiaries returning to the same organization could reflect better access to information or other factors such as aligned incentives. CONCLUSION: HIE is not consistently associated with utilization outcomes reflecting more coordinated care in the ED setting.

Association of Primary Care Engagement in Value-Based Reform Programs With Health Services Outcomes: Participation and Synergies.

Importance: Policy makers envision synergistic benefits from primary care reform programs that advance infrastructure and processes in the context of a supportive payment environment. However, these programs have been operationalized and implemented separately, raising the question of whether synergies are achieved. Objective: To evaluate associations between primary care engagement in voluntary delivery system and/or payment reform programs and health services outcomes. Design Setting and Participants: This was an observational longitudinal analysis of US ambulatory primary care organizations (PCOs) with attributed Medicare fee-for-service beneficiaries (1.6-1.9 million unique beneficiaries annually) using data for 2009, 2010, and 2015 to 2017; PCOs included multispecialty practices that delivered primary care. Data analyses were performed from January 2020 to December 2021. Exposures: Annual PCO participation in or recognition by (1) the Centers for Medicare & Medicaid's meaningful use (MU) program, (2) the National Committee for Quality Assurance's Patient-Centered Medical Home (PCMH) program, and/or (3) the Medicare Shared Savings Program (MSSP), an Accountable Care Organizations program. Main Outcomes and Measures: Independent and joint associations between an additional year of participation by a PCO in each of the 3 reform programs, and 3 types of outcomes: (1) hospital utilization (all-cause admissions, ambulatory care sensitive admissions, all-cause readmissions, all-cause emergency department visits); (2) evidence-based diabetes guideline adherence (≥1 annual glycated hemoglobin test, low-density lipoprotein cholesterol test, nephropathy screening, and eye examination); and (3) Medicare spending (total, acute inpatient, and skilled nursing facility). Results: The study sample comprised 47 880 unique PCOs (size ≤10 beneficiaries, 50%; ≤1-2 clinicians, 65%) and approximately 5.61 million unique Medicare beneficiaries (mean [SD] age, 71.4 [12.7] years; 3 207 568 [57.14%] women; 4 474 541 [79.71%] non-Hispanic White individuals) across the study years (2009, 2010, 2015-2017). Of the hospital utilization measures, only ambulatory care sensitive admission was associated with improved performance, showing a statistically significant marginal effect size for joint participation in MU and MSSP (-0.0002; 95% CI, -0.0005 to 0.0000) and MSSP alone (-0.0003; 95% CI, -0.0005 to -0.0001). For diabetes adherence, joint participation in PCMH and MU was associated with 0.06 more measures met (95% CI, 0.03 to 0.10) while participation in all 3 programs was associated with 0.05 more measures met (95% CI, 0.02 to 0.09). Stand-alone PCMH and stand-alone MU participation were also associated with improved performance. Joint participation in MU and MSSP was associated with $33.89 lower spending (95% CI, -$65.79 to -$1.99) as was stand-alone MSSP participation (-$37.04; 95% CI, -$65.73 to -$8.35). Conclusions and Relevance: This longitudinal observational study found that participation by PCOs in single or multiple reform programs was associated with better performance for only a subset of health services outcomes. More consistent and larger synergies may be realized with improved alignment of program requirements and goals.

Longitudinal participation in delivery and payment reform programs among US Primary Care Organizations.

OBJECTIVE: To assess longitudinal primary care organization participation patterns in large-scale reform programs and identify organizational characteristics associated with multiprogram participation. DATA SOURCES: Secondary data analysis of national program participation data over an eight-year period (2009-2016). STUDY DESIGN: We conducted a retrospective, observational study by creating a unique set of data linkages (including Medicare and Medicaid Meaningful Use and Medicare Shared Savings Program Accountable Care Organization (MSSP ACO) participation from CMS, Patient-Centered Medical Home (PCMH) participation from the National Committee for Quality Assurance, and organizational characteristics) to measure longitudinal participation and identify what types of organizations participate in one or more of these reform programs. We used multivariate models to identify organizational characteristics that differentiate those that participate in none, one, or two-to-three programs. DATA EXTRACTION METHODS: We used Medicare claims to identify organizations that delivered primary care services (n = 56 ,287) and then linked organizations to program participation data and characteristics. PRINCIPAL FINDINGS: No program achieved more than 50% participation across the 56,287 organizations in a given year, and participation levels flattened or decreased in later years. 36% of organizations did not participate in any program over the eight-year study period; 50% participated in one; 13% in two; and 1% in all three. 14.31% of organizations participated in five or more years of Meaningful Use while 3.84% of organizations participated in five years of the MSSP ACO Program and 0.64% participated in at least five years of PCMH. Larger organizations, those with younger providers, those with more primary care providers, and those with larger Medicare patient panels were more likely to participate in more programs. CONCLUSIONS AND RELEVANCE: Primary care transformation via use of voluntary programs, each with their own participation requirements and approach to incentives, has failed to broadly engage primary care organizations. Those that have chosen to participate in multiple programs are likely those already providing high-quality care.

Now trending: Coping with non-parallel trends in difference-in-differences analysis.

Difference-in-differences (DID) analysis is used widely to estimate the causal effects of health policies and interventions. A critical assumption in DID is "parallel trends": that pre-intervention trends in outcomes are the same between treated and comparison groups. To date, little guidance has been available to researchers who wish to use DID when the parallel trends assumption is violated. Using a Monte Carlo simulation experiment, we tested the performance of several estimators (standard DID; DID with propensity score matching; single-group interrupted time-series analysis; and multi-group interrupted time-series analysis) when the parallel trends assumption is violated. Using nationwide data from US hospitals (n = 3737) for seven data periods (four pre-interventions and three post-interventions), we used alternative estimators to evaluate the effect of a placebo intervention on common outcomes in health policy (clinical process quality and 30-day risk-standardized mortality for acute myocardial infarction, heart failure, and pneumonia). Estimator performance was assessed using mean-squared error and estimator coverage. We found that mean-squared error values were considerably lower for the DID estimator with matching than for the standard DID or interrupted time-series analysis models. The DID estimator with matching also had superior performance for estimator coverage. Our findings were robust across all outcomes evaluated.

Practice strategies to improve primary care for chronic disease patients under a pay-for-value program.

BACKGROUND: Improving primary care for patients with chronic illness is critical to advancing healthcare quality and value. Yet, little is known about what strategies are successful in helping primary care practices deliver high-quality care for this population under value-based payment models. METHODS: Double-blind interviews in 14 primary care practices in the state of Michigan, stratified based on whether they did (n = 7) or did not (n = 7) demonstrate improvement in primary care outcomes for patients with at least one reported chronic disease between 2010 and 2013. All practices participate in a statewide pay-for-performance program run by a large commercial payer. Using an implementation science framework to identify leverage points for effecting organizational change, we sought to identify, describe and compare strategies among improving and non-improving practices across three domains: (1) organizational learning opportunities, (2) approaches to motivating staff, and (3) acquisition and use of resources. RESULTS: We identified 10 strategies; 6 were "differentiating" - that is, more prevalent among improving practices. These differentiating strategies included: (1) participation in learning collaboratives, (2) accessing payer tools to monitor quality performance, (3) framing pay-for-performance as a practice transformation opportunity, (4) reinvesting earned incentive money in equitable, practice-centric improvement, (5) employing a care manager, and (6) using available technical support from local hospitals and provider organizations to support performance improvement. Implementation of these strategies varied based on organizational context and relative strengths. CONCLUSIONS: Practices that succeeded in improving care for chronic disease patients pursued a mix of strategies that helped meet immediate care delivery needs while also creating new adaptive structures and processes to better respond to changing pressures and demands. These findings help inform payers and primary care practices seeking evidence-based strategies to foster a stronger delivery system for patients with significant healthcare needs.

Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple-group ITSA. METHOD: We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)-the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)-a commonly used approach for evaluating treatment effects in ITSA studies. RESULTS: Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post-intervention period (P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave-one-out analysis to assess generalizability. CONCLUSIONS: The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model-free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance- and maximum-corrected index of classification accuracy, and cross-validation to assess generalizability.

Using forecast modelling to evaluate treatment effects in single-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. ITSA is commonly evaluated using methods which may produce biased results if model assumptions are violated. In this paper, treatment effects are alternatively assessed by using forecasting methods to closely fit the preintervention observations and then forecast the post-intervention trend. A treatment effect may be inferred if the actual post-intervention observations diverge from the forecasts by some specified amount. METHOD: The forecasting approach is demonstrated using the effect of California's Proposition 99 for reducing cigarette sales. Three forecast models are fit to the preintervention series-linear regression (REG), Holt-Winters (HW) non-seasonal smoothing, and autoregressive moving average (ARIMA)-and forecasts are generated into the post-intervention period. The actual observations are then compared with the forecasts to assess intervention effects. RESULTS: The preintervention data were fit best by HW, followed closely by ARIMA. REG fit the data poorly. The actual post-intervention observations were above the forecasts in HW and ARIMA, suggesting no intervention effect, but below the forecasts in the REG (suggesting a treatment effect), thereby raising doubts about any definitive conclusion of a treatment effect. CONCLUSIONS: In a single-group ITSA, treatment effects are likely to be biased if the model is misspecified. Therefore, evaluators should consider using forecast models to accurately fit the preintervention data and generate plausible counterfactual forecasts, thereby improving causal inference of treatment effects in single-group ITSA studies.

Using group-based trajectory modelling to enhance causal inference in interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Several enhancements have been proposed for interrupted time series analysis (ITSA) to improve causal inference. Presently, group-based trajectory modelling (GBTM) is introduced as a complement to ITSA. GBTM assumes a certain number of discrete groups in the sample have unique trajectories of the outcome. GBTM is used herein for 2 purposes: (1) to compare outcomes across all trajectory groups via a stand-alone GBTM and (2) to identify comparable non-treated units to serve as controls in the ITSA outcome model. Examples of each are offered. METHOD: The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California to other states not exposed to smoking reduction initiatives. In the stand-alone GBTM, distinct trajectory groups are identified based on cigarette sales for the entire observation period (1970-2000). In the second approach, a GBTM is generated using only baseline period observations (1970-1988), and treatment effects (difference in post-intervention trends) are then estimated for the treatment unit versus non-treated units in the treated unit's trajectory group. RESULTS: In the stand-alone GBTM, 3 distinct trajectory groups were identified: low-decreasing, medium-decreasing, and high-decreasing (California and 26 other states were in the low-decreasing group). When using baseline data for matching, California and 19 non-treated states comprised the low group. California had a significantly larger decrease in post-intervention cigarette sales than these controls (P < 0.01). CONCLUSIONS: GBTM enhances ITSA by providing perspective for the outcome trajectory in the treated unit's group versus all others and can identify non-treated units to be used for estimating treatment effects.

Using permutation tests to enhance causal inference in interrupted time series analysis.

RATIONALE, AIMS AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robustness check based on permutation tests to further improve causal inference. METHOD: We evaluate the effect of California's Proposition 99 for reducing cigarette sales by iteratively casting each nontreated state into the role of "treated," creating a comparable control group using the ITSAMATCH package in Stata, and then evaluating treatment effects using ITSA regression. If statistically significant "treatment effects" are estimated for pseudotreated states, then any significant changes in the outcome of the actual treatment unit (California) cannot be attributed to the intervention. We perform these analyses setting the cutpoint significance level to P > .40 for identifying balanced matches (the highest threshold possible for which controls could still be found for California) and use the difference in differences of trends as the treatment effect estimator. RESULTS: Only California attained a statistically significant treatment effect, strengthening confidence in the conclusion that Proposition 99 reduced cigarette sales. CONCLUSIONS: The proposed permutation testing framework provides an additional robustness check to either support or refute a treatment effect identified in for the true treated unit in ITSA. Given its value and ease of implementation, this framework should be considered as a standard robustness test in all multiple group interrupted time series analyses.

Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation.

RATIONALE, AIMS AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robust evaluation framework that combines the synthetic controls method (SYNTH) to generate a comparable control group and ITSA regression to assess covariate balance and estimate treatment effects. METHODS: We evaluate the effect of California's Proposition 99 for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. SYNTH is used to reweight nontreated units to make them comparable to the treated unit. These weights are then used in ITSA regression models to assess covariate balance and estimate treatment effects. RESULTS: Covariate balance was achieved for all but one covariate. While California experienced a significant decrease in the annual trend of cigarette sales after Proposition 99, there was no statistically significant treatment effect when compared to synthetic controls. CONCLUSIONS: The advantage of using this framework over regression alone is that it ensures that a comparable control group is generated. Additionally, it offers a common set of statistical measures familiar to investigators, the capability for assessing covariate balance, and enhancement of the evaluation with a comprehensive set of postestimation measures. Therefore, this robust framework should be considered as a primary approach for evaluating treatment effects in multiple group time series analysis.

A matching framework to improve causal inference in interrupted time-series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time-series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. When ITSA is implemented without a comparison group, the internal validity may be quite poor. Therefore, adding a comparable control group to serve as the counterfactual is always preferred. This paper introduces a novel matching framework, ITSAMATCH, to create a comparable control group by matching directly on covariates and then use these matches in the outcomes model. METHOD: We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. We compare ITSAMATCH results to 2 commonly used matching approaches, synthetic controls (SYNTH), and regression adjustment; SYNTH reweights nontreated units to make them comparable to the treated unit, and regression adjusts covariates directly. Methods are compared by assessing covariate balance and treatment effects. RESULTS: Both ITSAMATCH and SYNTH achieved covariate balance and estimated similar treatment effects. The regression model found no treatment effect and produced inconsistent covariate adjustment. CONCLUSIONS: While the matching framework achieved results comparable to SYNTH, it has the advantage of being technically less complicated, while producing statistical estimates that are straightforward to interpret. Conversely, regression adjustment may "adjust away" a treatment effect. Given its advantages, ITSAMATCH should be considered as a primary approach for evaluating treatment effects in multiple-group time-series analysis.

Estimating causal effects for survival (time-to-event) outcomes by combining classification tree analysis and propensity score weighting.

RATIONALE, AIMS AND OBJECTIVES: A common approach to assessing treatment effects in nonrandomized studies with time-to-event outcomes is to estimate propensity scores and compute weights using logistic regression, test for covariate balance, and then estimate treatment effects using Cox regression. A machine-learning alternative-classification tree analysis (CTA)-used to generate propensity scores and to estimate treatment effects in time-to-event data may identify complex relationships between covariates not found using conventional regression-based approaches. METHOD: Using empirical data, we identify all statistically valid CTA propensity score models and then use them to compute strata-specific, observation-level propensity score weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to the conventional method, by evaluating covariate balance and treatment effect estimates obtained using Cox regression and a weighted CTA outcomes model. RESULTS: All models had some imbalanced covariates. Nevertheless, treatment effect estimates were generally consistent across all weighted models. CONCLUSIONS: In the study sample, given that all approaches elicited similar results, using CTA increased confidence that bias could not be reduced any further. Because the CTA algorithm identifies all statistically valid propensity score models for a sample, it is most likely to identify a correctly specified propensity score model-and therefore should be used either to confirm results using traditional methods, or to reveal biases that may be missed by traditional methods. Moreover, given that the true treatment effect is never known in observational data, CTA should be considered for estimating outcomes because no statistical assumptions are required.

Minimizing imbalances on patient characteristics between treatment groups in randomized trials using classification tree analysis.

RATIONALE, AIMS, AND OBJECTIVES: Randomization ensures that treatment groups do not differ systematically in their characteristics, thereby reducing threats to validity that may otherwise explain differences in outcomes. Large observed imbalances in patient characteristics may indicate that selection bias is being introduced into the treatment allocation process. We introduce classification tree analysis (CTA) as a novel algorithmic approach for identifying potential imbalances in characteristics and their interactions when provisionally assigning each new participant to one or the other treatment group. The participant is then permanently assigned to the treatment group that elicits either no or less imbalance than if assigned to the alternate group. METHOD: Using data on participant characteristics from a clinical trial, we compare 3 different treatment allocation approaches: permuted block randomization (the original allocation method), minimization, and CTA. Treatment allocation performance is assessed by examining balance of all 17 patient characteristics between study groups for each of the allocation techniques. RESULTS: While all 3 treatment allocation techniques achieved excellent balance on main effect variables, Classification tree analysis further identified imbalances on interactions and in the distributions of some of the continuous variables. CONCLUSIONS: Classification tree analysis offers an algorithmic procedure that may be used with any randomization methodology to identify and then minimize linear, nonlinear, and interactive effects that induce covariate imbalance between groups. Investigators should consider using the CTA approach as a real-time complement to randomization for any clinical trial to safeguard the treatment allocation process against bias.

Using classification tree analysis to generate propensity score weights.

RATIONALE, AIMS AND OBJECTIVES: In evaluating non-randomized interventions, propensity scores (PS) estimate the probability of assignment to the treatment group given observed characteristics. Machine learning algorithms have been proposed as an alternative to conventional logistic regression for modelling PS in order to avoid limitations of linear methods. We introduce classification tree analysis (CTA) to generate PS which is a "decision-tree"-like classification model that provides accurate, parsimonious decision rules that are easy to display and interpret, reports P values derived via permutation tests, and evaluates cross-generalizability. METHOD: Using empirical data, we identify all statistically valid CTA PS models and then use them to compute strata-specific, observation-level PS weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to logistic regression and boosted regression, by evaluating covariate balance using standardized differences, model predictive accuracy, and treatment effect estimates obtained using median regression and a weighted CTA outcomes model. RESULTS: While all models had some imbalanced covariates, main-effects logistic regression yielded the lowest average standardized difference, whereas CTA yielded the greatest predictive accuracy. Nevertheless, treatment effect estimates were generally consistent across all models. CONCLUSIONS: Assessing standardized differences in means as a test of covariate balance is inappropriate for machine learning algorithms that segment the sample into two or more strata. Because the CTA algorithm identifies all statistically valid PS models for a sample, it is most likely to identify a correctly specified PS model, and should be considered as an alternative approach to modeling the PS.

Improving causal inference with a doubly robust estimator that combines propensity score stratification and weighting.

RATIONALE, AIMS AND OBJECTIVES: When a randomized controlled trial is not feasible, health researchers typically use observational data and rely on statistical methods to adjust for confounding when estimating treatment effects. These methods generally fall into 3 categories: (1) estimators based on a model for the outcome using conventional regression adjustment; (2) weighted estimators based on the propensity score (ie, a model for the treatment assignment); and (3) "doubly robust" (DR) estimators that model both the outcome and propensity score within the same framework. In this paper, we introduce a new DR estimator that utilizes marginal mean weighting through stratification (MMWS) as the basis for weighted adjustment. This estimator may prove more accurate than treatment effect estimators because MMWS has been shown to be more accurate than other models when the propensity score is misspecified. We therefore compare the performance of this new estimator to other commonly used treatment effects estimators. METHOD: Monte Carlo simulation is used to compare the DR-MMWS estimator to regression adjustment, 2 weighted estimators based on the propensity score and 2 other DR methods. To assess performance under varied conditions, we vary the level of misspecification of the propensity score model as well as misspecify the outcome model. RESULTS: Overall, DR estimators generally outperform methods that model one or the other components (eg, propensity score or outcome). The DR-MMWS estimator outperforms all other estimators when both the propensity score and outcome models are misspecified and performs equally as well as other DR estimators when only the propensity score is misspecified. CONCLUSIONS: Health researchers should consider using DR-MMWS as the principal evaluation strategy in observational studies, as this estimator appears to outperform other estimators in its class.

A comparison of approaches for stratifying on the propensity score to reduce bias.

RATIONALE, AIMS, AND OBJECTIVES: Stratification is a popular propensity score (PS) adjustment technique. It has been shown that stratifying the PS into 5 quantiles can remove over 90% of the bias due to the covariates used to generate the PS. Because of this finding, many investigators partition their data into 5 quantiles of the PS without examining whether a more robust solution (one that increases covariate balance while potentially reducing bias in the outcome analysis) can be found for their data. Two approaches (referred to herein as PSCORE and PSTRATA) obtain the optimal stratification solution by repeatedly dividing the data into strata until balance is achieved between treatment and control groups on the PS. These algorithms differ in how they partition the data, and it is not known which is better, or if either is better than a 5-quantile default approach, for reducing bias in treatment effect estimates. METHOD: Monte Carlo simulations and empirical data are used to assess whether PS strata defined by PSCORE, PSTRATA, or 5 quantiles is best at reducing bias in treatment effect estimates, when used within a marginal mean weighting framework (MMWS). These estimates are further compared to results derived using inverse probability of treatment weights (IPTW). RESULTS: PSTRATA was slightly better than PSCORE in balancing covariates and reducing bias, while both approaches outperformed the 5-quantile approach. Overall MMWS using any stratification method outperformed IPTW. CONCLUSIONS: Investigators should routinely use stratification approaches that obtain the optimal stratification solution, rather than simply partitioning the data into 5 quantiles of the PS. Moreover, MMWS (in conjunction with an optimal stratification approach) should be considered as an alternative to IPTW in studies that use PS weights.

Persistent threats to validity in single-group interrupted time series analysis with a cross over design.

RATIONALE, AIMS AND OBJECTIVES: The basic single-group interrupted time series analysis (ITSA) design has been shown to be susceptible to the most common threat to validity-history-the possibility that some other event caused the observed effect in the time series. A single-group ITSA with a crossover design (in which the intervention is introduced and withdrawn 1 or more times) should be more robust. In this paper, we describe and empirically assess the susceptibility of this design to bias from history. METHOD: Time series data from 2 natural experiments (the effect of multiple repeals and reinstatements of Louisiana's motorcycle helmet law on motorcycle fatalities and the association between the implementation and withdrawal of Gorbachev's antialcohol campaign with Russia's mortality crisis) are used to illustrate that history remains a threat to ITSA validity, even in a crossover design. RESULTS: Both empirical examples reveal that the single-group ITSA with a crossover design may be biased because of history. In the case of motorcycle fatalities, helmet laws appeared effective in reducing mortality (while repealing the law increased mortality), but when a control group was added, it was shown that this trend was similar in both groups. In the case of Gorbachev's antialcohol campaign, only when contrasting the results against those of a control group was the withdrawal of the campaign found to be the more likely culprit in explaining the Russian mortality crisis than the collapse of the Soviet Union. CONCLUSIONS: Even with a robust crossover design, single-group ITSA models remain susceptible to bias from history. Therefore, a comparable control group design should be included, whenever possible.

Challenges to validity in single-group interrupted time series analysis.

RATIONALE, AIMS AND OBJECTIVES: Single-group interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single unit of observation is studied; the outcome variable is serially ordered as a time series, and the intervention is expected to "interrupt" the level and/or trend of the time series, subsequent to its introduction. The most common threat to validity is history-the possibility that some other event caused the observed effect in the time series. Although history limits the ability to draw causal inferences from single ITSA models, it can be controlled for by using a comparable control group to serve as the counterfactual. METHOD: Time series data from 2 natural experiments (effect of Florida's 2000 repeal of its motorcycle helmet law on motorcycle fatalities and California's 1988 Proposition 99 to reduce cigarette sales) are used to illustrate how history biases results of single-group ITSA results-as opposed to when that group's results are contrasted to those of a comparable control group. RESULTS: In the first example, an external event occurring at the same time as the helmet repeal appeared to be the cause of a rise in motorcycle deaths, but was only revealed when Florida was contrasted with comparable control states. Conversely, in the second example, a decreasing trend in cigarette sales prior to the intervention raised question about a treatment effect attributed to Proposition 99, but was reinforced when California was contrasted with comparable control states. CONCLUSIONS: Results of single-group ITSA should be considered preliminary, and interpreted with caution, until a more robust study design can be implemented.

Combining machine learning and matching techniques to improve causal inference in program evaluation.

RATIONALE, AIMS AND OBJECTIVES: Program evaluations often utilize various matching approaches to emulate the randomization process for group assignment in experimental studies. Typically, the matching strategy is implemented, and then covariate balance is assessed before estimating treatment effects. This paper introduces a novel analytic framework utilizing a machine learning algorithm called optimal discriminant analysis (ODA) for assessing covariate balance and estimating treatment effects, once the matching strategy has been implemented. This framework holds several key advantages over the conventional approach: application to any variable metric and number of groups; insensitivity to skewed data or outliers; and use of accuracy measures applicable to all prognostic analyses. Moreover, ODA accepts analytic weights, thereby extending the methodology to any study design where weights are used for covariate adjustment or more precise (differential) outcome measurement. METHOD: One-to-one matching on the propensity score was used as the matching strategy. Covariate balance was assessed using standardized difference in means (conventional approach) and measures of classification accuracy (ODA). Treatment effects were estimated using ordinary least squares regression and ODA. RESULTS: Using empirical data, ODA produced results highly consistent with those obtained via the conventional methodology for assessing covariate balance and estimating treatment effects. CONCLUSIONS: When ODA is combined with matching techniques within a treatment effects framework, the results are consistent with conventional approaches. However, given that it provides additional dimensions and robustness to the analysis versus what can currently be achieved using conventional approaches, ODA offers an appealing alternative.