Problematic meta-analyses: Bayesian and frequentist perspectives on combining randomized controlled trials and non-randomized studies.

PURPOSE: In the literature, the propriety of the meta-analytic treatment-effect produced by combining randomized controlled trials (RCT) and non-randomized studies (NRS) is questioned, given the inherent confounding in NRS that may bias the meta-analysis. The current study compared an implicitly principled pooled Bayesian meta-analytic treatment-effect with that of frequentist pooling of RCT and NRS to determine how well each approach handled the NRS bias. MATERIALS & METHODS: Binary outcome Critical-Care meta-analyses, reflecting the importance of such outcomes in Critical-Care practice, combining RCT and NRS were identified electronically. Bayesian pooled treatment-effect and 95% credible-intervals (BCrI), posterior model probabilities indicating model plausibility and Bayes-factors (BF) were estimated using an informative heavy-tailed heterogeneity prior (half-Cauchy). Preference for pooling of RCT and NRS was indicated for Bayes-factors > 3 or < 0.333 for the converse. All pooled frequentist treatment-effects and 95% confidence intervals (FCI) were re-estimated using the popular DerSimonian-Laird (DSL) random effects model. RESULTS: Fifty meta-analyses were identified (2009-2021), reporting pooled estimates in 44; 29 were pharmaceutical-therapeutic and 21 were non-pharmaceutical therapeutic. Re-computed pooled DSL FCI excluded the null (OR or RR = 1) in 86% (43/50). In 18 meta-analyses there was an agreement between FCI and BCrI in excluding the null. In 23 meta-analyses where FCI excluded the null, BCrI embraced the null. BF supported a pooled model in 27 meta-analyses and separate models in 4. The highest density of the posterior model probabilities for 0.333 < Bayes factor < 1 was 0.8. CONCLUSIONS: In the current meta-analytic cohort, an integrated and multifaceted Bayesian approach gave support to including NRS in a pooled-estimate model. Conversely, caution should attend the reporting of naïve frequentist pooled, RCT and NRS, meta-analytic treatment effects.

Electronic connectivity between hospital pairs: impact on emergency department-related utilization.

OBJECTIVE: To use more precise measures of which hospitals are electronically connected to determine whether health information exchange (HIE) is associated with lower emergency department (ED)-related utilization. MATERIALS AND METHODS: We combined 2018 Medicare fee-for-service claims to identify beneficiaries with 2 ED encounters within 30 days, and Definitive Healthcare and AHA IT Supplement data to identify hospital participation in HIE networks (HIOs and EHR vendor networks). We determined whether the 2 encounters for the same beneficiary occurred at: the same organization, different organizations connected by HIE, or different organizations not connected by HIE. Outcomes were: (1) whether any repeat imaging occurred during the second ED visit; (2) for beneficiaries with a treat-and-release ED visit followed by a second ED visit, whether they were admitted to the hospital after the second visit; (3) for beneficiaries discharged from the hospital followed by an ED visit, whether they were admitted to the hospital. RESULTS: In adjusted mixed effects models, for all outcomes, beneficiaries returning to the same organization had significantly lower utilization compared to those going to different organizations. Comparing only those going to different organizations, HIE was not associated with lower levels of repeat imaging. HIE was associated with lower likelihood of hospital admission following a treat-and-release ED visit (1.83 percentage points [-3.44 to -0.21]) but higher likelihood of admission following hospital discharge (2.78 percentage points [0.48-5.08]). DISCUSSION: Lower utilization for beneficiaries returning to the same organization could reflect better access to information or other factors such as aligned incentives. CONCLUSION: HIE is not consistently associated with utilization outcomes reflecting more coordinated care in the ED setting.

Modelling of intensive care unit (ICU) length of stay as a quality measure: a problematic exercise.

BACKGROUND: Intensive care unit (ICU) length of stay (LOS) and the risk adjusted equivalent (RALOS) have been used as quality metrics. The latter measures entail either ratio or difference formulations or ICU random effects (RE), which have not been previously compared. METHODS: From calendar year 2016 data of an adult ICU registry-database (Australia & New Zealand Intensive Care Society (ANZICS) CORE), LOS predictive models were established using linear (LMM) and generalised linear (GLMM) mixed models. Model fixed effects quality-metric formulations were estimated as RALOSR for LMM (geometric mean derived from log(ICU LOS)) and GLMM (day) and observed minus expected ICU LOS (OMELOS from GLMM). Metric confidence intervals (95%CI) were estimated by bootstrapping; random effects (RE) were predicted for LMM and GLMM. Forest-plot displays of ranked quality-metric point-estimates (95%CI) were generated for ICU hospital classifications (metropolitan, private, rural/regional, and tertiary). Robust rank confidence sets (point estimate and 95%CI), both marginal (pertaining to a singular ICU) and simultaneous (pertaining to all ICU differences), were established. RESULTS: The ICU cohort was of 94,361 patients from 125 ICUs (metropolitan 16.9%, private 32.8%, rural/regional 6.4%, tertiary 43.8%). Age (mean, SD) was 61.7 (17.5) years; 58.3% were male; APACHE III severity-of-illness score 54.6 (25.7); ICU annual patient volume 1192 (702) and ICU LOS 3.2 (4.9). There was no concordance of ICU ranked model predictions, GLMM versus LMM, nor for the quality metrics used, RALOSR, OMELOS and site-specific RE for each of the ICU hospital classifications. Furthermore, there was no concordance between ICU ranking confidence sets, marginal and simultaneous for models or quality metrics. CONCLUSIONS: Inference regarding adjusted ICU LOS was dependent upon the statistical estimator and the quality index used to quantify any LOS differences across ICUs. That is, there was no "one best model"; thus, ICU "performance" is determined by model choice and any rankings thereupon should be circumspect.

Association of Primary Care Engagement in Value-Based Reform Programs With Health Services Outcomes: Participation and Synergies.

Importance: Policy makers envision synergistic benefits from primary care reform programs that advance infrastructure and processes in the context of a supportive payment environment. However, these programs have been operationalized and implemented separately, raising the question of whether synergies are achieved. Objective: To evaluate associations between primary care engagement in voluntary delivery system and/or payment reform programs and health services outcomes. Design Setting and Participants: This was an observational longitudinal analysis of US ambulatory primary care organizations (PCOs) with attributed Medicare fee-for-service beneficiaries (1.6-1.9 million unique beneficiaries annually) using data for 2009, 2010, and 2015 to 2017; PCOs included multispecialty practices that delivered primary care. Data analyses were performed from January 2020 to December 2021. Exposures: Annual PCO participation in or recognition by (1) the Centers for Medicare & Medicaid's meaningful use (MU) program, (2) the National Committee for Quality Assurance's Patient-Centered Medical Home (PCMH) program, and/or (3) the Medicare Shared Savings Program (MSSP), an Accountable Care Organizations program. Main Outcomes and Measures: Independent and joint associations between an additional year of participation by a PCO in each of the 3 reform programs, and 3 types of outcomes: (1) hospital utilization (all-cause admissions, ambulatory care sensitive admissions, all-cause readmissions, all-cause emergency department visits); (2) evidence-based diabetes guideline adherence (≥1 annual glycated hemoglobin test, low-density lipoprotein cholesterol test, nephropathy screening, and eye examination); and (3) Medicare spending (total, acute inpatient, and skilled nursing facility). Results: The study sample comprised 47 880 unique PCOs (size ≤10 beneficiaries, 50%; ≤1-2 clinicians, 65%) and approximately 5.61 million unique Medicare beneficiaries (mean [SD] age, 71.4 [12.7] years; 3 207 568 [57.14%] women; 4 474 541 [79.71%] non-Hispanic White individuals) across the study years (2009, 2010, 2015-2017). Of the hospital utilization measures, only ambulatory care sensitive admission was associated with improved performance, showing a statistically significant marginal effect size for joint participation in MU and MSSP (-0.0002; 95% CI, -0.0005 to 0.0000) and MSSP alone (-0.0003; 95% CI, -0.0005 to -0.0001). For diabetes adherence, joint participation in PCMH and MU was associated with 0.06 more measures met (95% CI, 0.03 to 0.10) while participation in all 3 programs was associated with 0.05 more measures met (95% CI, 0.02 to 0.09). Stand-alone PCMH and stand-alone MU participation were also associated with improved performance. Joint participation in MU and MSSP was associated with $33.89 lower spending (95% CI, -$65.79 to -$1.99) as was stand-alone MSSP participation (-$37.04; 95% CI, -$65.73 to -$8.35). Conclusions and Relevance: This longitudinal observational study found that participation by PCOs in single or multiple reform programs was associated with better performance for only a subset of health services outcomes. More consistent and larger synergies may be realized with improved alignment of program requirements and goals.

Longitudinal participation in delivery and payment reform programs among US Primary Care Organizations.

OBJECTIVE: To assess longitudinal primary care organization participation patterns in large-scale reform programs and identify organizational characteristics associated with multiprogram participation. DATA SOURCES: Secondary data analysis of national program participation data over an eight-year period (2009-2016). STUDY DESIGN: We conducted a retrospective, observational study by creating a unique set of data linkages (including Medicare and Medicaid Meaningful Use and Medicare Shared Savings Program Accountable Care Organization (MSSP ACO) participation from CMS, Patient-Centered Medical Home (PCMH) participation from the National Committee for Quality Assurance, and organizational characteristics) to measure longitudinal participation and identify what types of organizations participate in one or more of these reform programs. We used multivariate models to identify organizational characteristics that differentiate those that participate in none, one, or two-to-three programs. DATA EXTRACTION METHODS: We used Medicare claims to identify organizations that delivered primary care services (n = 56 ,287) and then linked organizations to program participation data and characteristics. PRINCIPAL FINDINGS: No program achieved more than 50% participation across the 56,287 organizations in a given year, and participation levels flattened or decreased in later years. 36% of organizations did not participate in any program over the eight-year study period; 50% participated in one; 13% in two; and 1% in all three. 14.31% of organizations participated in five or more years of Meaningful Use while 3.84% of organizations participated in five years of the MSSP ACO Program and 0.64% participated in at least five years of PCMH. Larger organizations, those with younger providers, those with more primary care providers, and those with larger Medicare patient panels were more likely to participate in more programs. CONCLUSIONS AND RELEVANCE: Primary care transformation via use of voluntary programs, each with their own participation requirements and approach to incentives, has failed to broadly engage primary care organizations. Those that have chosen to participate in multiple programs are likely those already providing high-quality care.

Now trending: Coping with non-parallel trends in difference-in-differences analysis.

Difference-in-differences (DID) analysis is used widely to estimate the causal effects of health policies and interventions. A critical assumption in DID is "parallel trends": that pre-intervention trends in outcomes are the same between treated and comparison groups. To date, little guidance has been available to researchers who wish to use DID when the parallel trends assumption is violated. Using a Monte Carlo simulation experiment, we tested the performance of several estimators (standard DID; DID with propensity score matching; single-group interrupted time-series analysis; and multi-group interrupted time-series analysis) when the parallel trends assumption is violated. Using nationwide data from US hospitals (n = 3737) for seven data periods (four pre-interventions and three post-interventions), we used alternative estimators to evaluate the effect of a placebo intervention on common outcomes in health policy (clinical process quality and 30-day risk-standardized mortality for acute myocardial infarction, heart failure, and pneumonia). Estimator performance was assessed using mean-squared error and estimator coverage. We found that mean-squared error values were considerably lower for the DID estimator with matching than for the standard DID or interrupted time-series analysis models. The DID estimator with matching also had superior performance for estimator coverage. Our findings were robust across all outcomes evaluated.

Using randomization tests to assess treatment effects in multiple-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple-group ITSA typically has small sample sizes, and parametric methods for multiple-group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non-parametric, distribution-free option for computing exact P values. METHOD: The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)-the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)-a commonly used parametric approach for evaluating treatment effects in ITSA studies. RESULTS: Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P < 0.01). CONCLUSIONS: In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.

Practice strategies to improve primary care for chronic disease patients under a pay-for-value program.

BACKGROUND: Improving primary care for patients with chronic illness is critical to advancing healthcare quality and value. Yet, little is known about what strategies are successful in helping primary care practices deliver high-quality care for this population under value-based payment models. METHODS: Double-blind interviews in 14 primary care practices in the state of Michigan, stratified based on whether they did (n = 7) or did not (n = 7) demonstrate improvement in primary care outcomes for patients with at least one reported chronic disease between 2010 and 2013. All practices participate in a statewide pay-for-performance program run by a large commercial payer. Using an implementation science framework to identify leverage points for effecting organizational change, we sought to identify, describe and compare strategies among improving and non-improving practices across three domains: (1) organizational learning opportunities, (2) approaches to motivating staff, and (3) acquisition and use of resources. RESULTS: We identified 10 strategies; 6 were "differentiating" - that is, more prevalent among improving practices. These differentiating strategies included: (1) participation in learning collaboratives, (2) accessing payer tools to monitor quality performance, (3) framing pay-for-performance as a practice transformation opportunity, (4) reinvesting earned incentive money in equitable, practice-centric improvement, (5) employing a care manager, and (6) using available technical support from local hospitals and provider organizations to support performance improvement. Implementation of these strategies varied based on organizational context and relative strengths. CONCLUSIONS: Practices that succeeded in improving care for chronic disease patients pursued a mix of strategies that helped meet immediate care delivery needs while also creating new adaptive structures and processes to better respond to changing pressures and demands. These findings help inform payers and primary care practices seeking evidence-based strategies to foster a stronger delivery system for patients with significant healthcare needs.

Using machine learning to evaluate treatment effects in multiple-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time, and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. In this paper, we introduce a novel machine learning approach using optimal discriminant analysis (ODA) to evaluate treatment effects in multiple-group ITSA. METHOD: We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California (CA) to Montana (MT)-the best matching control state not exposed to any smoking reduction initiatives. We contrast results from ODA to those of ITSA regression (ITSAREG)-a commonly used approach for evaluating treatment effects in ITSA studies. RESULTS: Both approaches found CA and MT to be comparable on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the post-intervention period (P < 0.0001). The ODA model achieved very high effect strength of sensitivity (a measure of classification accuracy) of 91.67%, which remained high (75.00%) after conducting leave-one-out analysis to assess generalizability. CONCLUSIONS: The ODA framework achieved results comparable to ITSAREG, bolstering confidence in the intervention effect. In addition, ODA confers several advantages over conventional approaches that may make it a better approach to use in multiple group ITSA studies: insensitivity to skewed data, model-free permutation tests to derive P values, identification of the threshold value which best discriminates intervention and control groups, a chance- and maximum-corrected index of classification accuracy, and cross-validation to assess generalizability.

Using forecast modelling to evaluate treatment effects in single-group interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. ITSA is commonly evaluated using methods which may produce biased results if model assumptions are violated. In this paper, treatment effects are alternatively assessed by using forecasting methods to closely fit the preintervention observations and then forecast the post-intervention trend. A treatment effect may be inferred if the actual post-intervention observations diverge from the forecasts by some specified amount. METHOD: The forecasting approach is demonstrated using the effect of California's Proposition 99 for reducing cigarette sales. Three forecast models are fit to the preintervention series-linear regression (REG), Holt-Winters (HW) non-seasonal smoothing, and autoregressive moving average (ARIMA)-and forecasts are generated into the post-intervention period. The actual observations are then compared with the forecasts to assess intervention effects. RESULTS: The preintervention data were fit best by HW, followed closely by ARIMA. REG fit the data poorly. The actual post-intervention observations were above the forecasts in HW and ARIMA, suggesting no intervention effect, but below the forecasts in the REG (suggesting a treatment effect), thereby raising doubts about any definitive conclusion of a treatment effect. CONCLUSIONS: In a single-group ITSA, treatment effects are likely to be biased if the model is misspecified. Therefore, evaluators should consider using forecast models to accurately fit the preintervention data and generate plausible counterfactual forecasts, thereby improving causal inference of treatment effects in single-group ITSA studies.

Using group-based trajectory modelling to enhance causal inference in interrupted time series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Several enhancements have been proposed for interrupted time series analysis (ITSA) to improve causal inference. Presently, group-based trajectory modelling (GBTM) is introduced as a complement to ITSA. GBTM assumes a certain number of discrete groups in the sample have unique trajectories of the outcome. GBTM is used herein for 2 purposes: (1) to compare outcomes across all trajectory groups via a stand-alone GBTM and (2) to identify comparable non-treated units to serve as controls in the ITSA outcome model. Examples of each are offered. METHOD: The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California to other states not exposed to smoking reduction initiatives. In the stand-alone GBTM, distinct trajectory groups are identified based on cigarette sales for the entire observation period (1970-2000). In the second approach, a GBTM is generated using only baseline period observations (1970-1988), and treatment effects (difference in post-intervention trends) are then estimated for the treatment unit versus non-treated units in the treated unit's trajectory group. RESULTS: In the stand-alone GBTM, 3 distinct trajectory groups were identified: low-decreasing, medium-decreasing, and high-decreasing (California and 26 other states were in the low-decreasing group). When using baseline data for matching, California and 19 non-treated states comprised the low group. California had a significantly larger decrease in post-intervention cigarette sales than these controls (P < 0.01). CONCLUSIONS: GBTM enhances ITSA by providing perspective for the outcome trajectory in the treated unit's group versus all others and can identify non-treated units to be used for estimating treatment effects.

Using permutation tests to enhance causal inference in interrupted time series analysis.

RATIONALE, AIMS AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied serially over time and the intervention is expected to "interrupt" the level and/or trend of that outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robustness check based on permutation tests to further improve causal inference. METHOD: We evaluate the effect of California's Proposition 99 for reducing cigarette sales by iteratively casting each nontreated state into the role of "treated," creating a comparable control group using the ITSAMATCH package in Stata, and then evaluating treatment effects using ITSA regression. If statistically significant "treatment effects" are estimated for pseudotreated states, then any significant changes in the outcome of the actual treatment unit (California) cannot be attributed to the intervention. We perform these analyses setting the cutpoint significance level to P > .40 for identifying balanced matches (the highest threshold possible for which controls could still be found for California) and use the difference in differences of trends as the treatment effect estimator. RESULTS: Only California attained a statistically significant treatment effect, strengthening confidence in the conclusion that Proposition 99 reduced cigarette sales. CONCLUSIONS: The proposed permutation testing framework provides an additional robustness check to either support or refute a treatment effect identified in for the true treated unit in ITSA. Given its value and ease of implementation, this framework should be considered as a standard robustness test in all multiple group interrupted time series analyses.

Combining synthetic controls and interrupted time series analysis to improve causal inference in program evaluation.

RATIONALE, AIMS AND OBJECTIVES: Interrupted time series analysis (ITSA) is an evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome. The internal validity is strengthened considerably when the treated unit is contrasted with a comparable control group. In this paper, we introduce a robust evaluation framework that combines the synthetic controls method (SYNTH) to generate a comparable control group and ITSA regression to assess covariate balance and estimate treatment effects. METHODS: We evaluate the effect of California's Proposition 99 for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. SYNTH is used to reweight nontreated units to make them comparable to the treated unit. These weights are then used in ITSA regression models to assess covariate balance and estimate treatment effects. RESULTS: Covariate balance was achieved for all but one covariate. While California experienced a significant decrease in the annual trend of cigarette sales after Proposition 99, there was no statistically significant treatment effect when compared to synthetic controls. CONCLUSIONS: The advantage of using this framework over regression alone is that it ensures that a comparable control group is generated. Additionally, it offers a common set of statistical measures familiar to investigators, the capability for assessing covariate balance, and enhancement of the evaluation with a comprehensive set of postestimation measures. Therefore, this robust framework should be considered as a primary approach for evaluating treatment effects in multiple group time series analysis.

Estimating causal effects for survival (time-to-event) outcomes by combining classification tree analysis and propensity score weighting.

RATIONALE, AIMS AND OBJECTIVES: A common approach to assessing treatment effects in nonrandomized studies with time-to-event outcomes is to estimate propensity scores and compute weights using logistic regression, test for covariate balance, and then estimate treatment effects using Cox regression. A machine-learning alternative-classification tree analysis (CTA)-used to generate propensity scores and to estimate treatment effects in time-to-event data may identify complex relationships between covariates not found using conventional regression-based approaches. METHOD: Using empirical data, we identify all statistically valid CTA propensity score models and then use them to compute strata-specific, observation-level propensity score weights that are subsequently applied in outcomes analyses. We compare findings obtained using this framework to the conventional method, by evaluating covariate balance and treatment effect estimates obtained using Cox regression and a weighted CTA outcomes model. RESULTS: All models had some imbalanced covariates. Nevertheless, treatment effect estimates were generally consistent across all weighted models. CONCLUSIONS: In the study sample, given that all approaches elicited similar results, using CTA increased confidence that bias could not be reduced any further. Because the CTA algorithm identifies all statistically valid propensity score models for a sample, it is most likely to identify a correctly specified propensity score model-and therefore should be used either to confirm results using traditional methods, or to reveal biases that may be missed by traditional methods. Moreover, given that the true treatment effect is never known in observational data, CTA should be considered for estimating outcomes because no statistical assumptions are required.

A matching framework to improve causal inference in interrupted time-series analysis.

RATIONALE, AIMS, AND OBJECTIVES: Interrupted time-series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. When ITSA is implemented without a comparison group, the internal validity may be quite poor. Therefore, adding a comparable control group to serve as the counterfactual is always preferred. This paper introduces a novel matching framework, ITSAMATCH, to create a comparable control group by matching directly on covariates and then use these matches in the outcomes model. METHOD: We evaluate the effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales, by comparing California to other states not exposed to smoking reduction initiatives. We compare ITSAMATCH results to 2 commonly used matching approaches, synthetic controls (SYNTH), and regression adjustment; SYNTH reweights nontreated units to make them comparable to the treated unit, and regression adjusts covariates directly. Methods are compared by assessing covariate balance and treatment effects. RESULTS: Both ITSAMATCH and SYNTH achieved covariate balance and estimated similar treatment effects. The regression model found no treatment effect and produced inconsistent covariate adjustment. CONCLUSIONS: While the matching framework achieved results comparable to SYNTH, it has the advantage of being technically less complicated, while producing statistical estimates that are straightforward to interpret. Conversely, regression adjustment may "adjust away" a treatment effect. Given its advantages, ITSAMATCH should be considered as a primary approach for evaluating treatment effects in multiple-group time-series analysis.

Precommitting to choose wisely about low-value services: a stepped wedge cluster randomised trial.

BACKGROUND: Little is known about how to discourage clinicians from ordering low-value services. Our objective was to test whether clinicians committing their future selves (ie, precommitting) to follow Choosing Wisely recommendations with decision supports could decrease potentially low-value orders. METHODS: We conducted a 12-month stepped wedge cluster randomised trial among 45 primary care physicians and advanced practice providers in six adult primary care clinics of a US community group practice.Clinicians were invited to precommit to Choosing Wisely recommendations against imaging for uncomplicated low back pain, imaging for uncomplicated headaches and unnecessary antibiotics for acute sinusitis. Clinicians who precommitted received 1-6 months of point-of-care precommitment reminders as well as patient education handouts and weekly emails with resources to support communication about low-value services.The primary outcome was the difference between control and intervention period percentages of visits with potentially low-value orders. Secondary outcomes were differences between control and intervention period percentages of visits with possible alternate orders, and differences between control and 3-month postintervention follow-up period percentages of visits with potentially low-value orders. RESULTS: The intervention was not associated with a change in the percentage of visits with potentially low-value orders overall, for headaches or for acute sinusitis, but was associated with a 1.7% overall increase in alternate orders (p=0.01). For low back pain, the intervention was associated with a 1.2% decrease in the percentage of visits with potentially low-value orders (p=0.001) and a 1.9% increase in the percentage of visits with alternate orders (p=0.007). No changes were sustained in follow-up. CONCLUSION: Clinician precommitment to follow Choosing Wisely recommendations was associated with a small, unsustained decrease in potentially low-value orders for only one of three targeted conditions and may have increased alternate orders. TRIAL REGISTRATION NUMBER: NCT02247050; Pre-results.

Identifying causal mechanisms in health care interventions using classification tree analysis.

RATIONALE, AIMS, AND OBJECTIVES: Mediation analysis identifies causal pathways by testing the relationships between the treatment, the outcome, and an intermediate variable that mediates the relationship between the treatment and outcome. This paper introduces classification tree analysis (CTA), a machine-learning procedure, as an alternative to conventional methods for analysing mediation effects. METHOD: Using data from the JOBS II study, we compare CTA to structural equation models (SEMs) by assessing their consistency in revealing mediation effects on 2 outcomes; reemployment (a binary variable) and depressive symptoms (a continuous variable). Because study participants were not randomized sequentially to both treatment and mediator, an additional model was generated including baseline covariates to strengthen the validity of some key identifying assumptions required of all mediation analyses. RESULTS: Using SEM, no statistically significant treatment or mediated effects were found for either outcome. In contrast, CTA found a significant treatment effect for reemployment (P = .047) and a mediated pathway for individuals in the treatment group (P = .014). No CTA model could be generated for the reemployment outcome. When covariates were added to the model, CTA found numerous interactions, while SEM found no effects. CONCLUSIONS: CTA may uncover mediation effects where conventional approaches do not, because CTA does not require any assumptions about the distribution of variables nor of the functional form of the model, and CTA will systematically identify all statistically viable interactions. The versatility of CTA enables the investigator to explore the theorized underlying causal mechanism of an intervention in a much more comprehensive manner than conventional mediation analytic approaches.

Minimizing imbalances on patient characteristics between treatment groups in randomized trials using classification tree analysis.

RATIONALE, AIMS, AND OBJECTIVES: Randomization ensures that treatment groups do not differ systematically in their characteristics, thereby reducing threats to validity that may otherwise explain differences in outcomes. Large observed imbalances in patient characteristics may indicate that selection bias is being introduced into the treatment allocation process. We introduce classification tree analysis (CTA) as a novel algorithmic approach for identifying potential imbalances in characteristics and their interactions when provisionally assigning each new participant to one or the other treatment group. The participant is then permanently assigned to the treatment group that elicits either no or less imbalance than if assigned to the alternate group. METHOD: Using data on participant characteristics from a clinical trial, we compare 3 different treatment allocation approaches: permuted block randomization (the original allocation method), minimization, and CTA. Treatment allocation performance is assessed by examining balance of all 17 patient characteristics between study groups for each of the allocation techniques. RESULTS: While all 3 treatment allocation techniques achieved excellent balance on main effect variables, Classification tree analysis further identified imbalances on interactions and in the distributions of some of the continuous variables. CONCLUSIONS: Classification tree analysis offers an algorithmic procedure that may be used with any randomization methodology to identify and then minimize linear, nonlinear, and interactive effects that induce covariate imbalance between groups. Investigators should consider using the CTA approach as a real-time complement to randomization for any clinical trial to safeguard the treatment allocation process against bias.

Modeling time-to-event (survival) data using classification tree analysis.

RATIONALE, AIMS, AND OBJECTIVES: Time to the occurrence of an event is often studied in health research. Survival analysis differs from other designs in that follow-up times for individuals who do not experience the event by the end of the study (called censored) are accounted for in the analysis. Cox regression is the standard method for analysing censored data, but the assumptions required of these models are easily violated. In this paper, we introduce classification tree analysis (CTA) as a flexible alternative for modelling censored data. Classification tree analysis is a "decision-tree"-like classification model that provides parsimonious, transparent (ie, easy to visually display and interpret) decision rules that maximize predictive accuracy, derives exact P values via permutation tests, and evaluates model cross-generalizability. METHOD: Using empirical data, we identify all statistically valid, reproducible, longitudinally consistent, and cross-generalizable CTA survival models and then compare their predictive accuracy to estimates derived via Cox regression and an unadjusted naïve model. Model performance is assessed using integrated Brier scores and a comparison between estimated survival curves. RESULTS: The Cox regression model best predicts average incidence of the outcome over time, whereas CTA survival models best predict either relatively high, or low, incidence of the outcome over time. CONCLUSIONS: Classification tree analysis survival models offer many advantages over Cox regression, such as explicit maximization of predictive accuracy, parsimony, statistical robustness, and transparency. Therefore, researchers interested in accurate prognoses and clear decision rules should consider developing models using the CTA-survival framework.