A New Look at P Values for Randomized Clinical Trials.

BACKGROUND: We have examined the primary efficacy results of 23,551 randomized clinical trials from the Cochrane Database of Systematic Reviews. METHODS: We estimate that the great majority of trials have much lower statistical power for actual effects than the 80 or 90% for the stated effect sizes. Consequently, "statistically significant" estimates tend to seriously overestimate actual treatment effects, "nonsignificant" results often correspond to important effects, and efforts to replicate often fail to achieve "significance" and may even appear to contradict initial results. To address these issues, we reinterpret the P value in terms of a reference population of studies that are, or could have been, in the Cochrane Database. RESULTS: This leads to an empirical guide for the interpretation of an observed P value from a "typical" clinical trial in terms of the degree of overestimation of the reported effect, the probability of the effect's sign being wrong, and the predictive power of the trial. CONCLUSIONS: Such an interpretation provides additional insight about the effect under study and can guard medical researchers against naive interpretations of the P value and overoptimistic effect sizes. Because many research fields suffer from low power, our results are also relevant outside the medical domain. (Funded by the U.S. Office of Naval Research.)

Pharmacogenetic testing among patients with mood and anxiety disorders is associated with decreased utilization and cost: A propensity-score matched study.

BACKGROUND: Naturalistic and small randomized trials have suggested that pharmacogenetic testing may improve treatment outcomes in depression, but its cost-effectiveness is not known. There is growing enthusiasm for personalized medicine, relying on genetic variation as a contributor to heterogeneity of treatment effects. We sought to examine the relationship between a commercial pharmacogenetic test for psychotropic medications and 6-month cost of care and utilization in a large commercial health plan. METHODS: We performed a propensity-score matched case-control analysis of longitudinal health claims data from a large US insurer. Individuals with a mood or anxiety disorder diagnosis (N = 817) who received genetic testing for pharmacokinetic and pharmacodynamic variation were matched to 2,745 individuals who did not receive such testing. Outcomes included number of outpatient visits, inpatient hospitalizations, emergency room visits, and prescriptions, as well as associated costs over 6 months. RESULTS: On average, individuals who underwent testing experienced 40% fewer all-cause emergency room visits (mean difference 0.13 visits; P < 0.0001) and 58% fewer inpatient all-cause hospitalizations (mean difference 0.10 visits; P < 0.0001) than individuals in the control group. The two groups did not differ significantly in number of psychotropic medications prescribed or mood-disorder related hospitalizations. Overall 6-month costs were estimated to be $1,948 (SE 611) lower in the tested group. CONCLUSIONS: Pharmacogenetic testing represents a promising strategy to reduce costs and utilization among patients with mood and anxiety disorders.

Addressing unmeasured confounding in comparative observational research.

PURPOSE: Observational pharmacoepidemiological studies can provide valuable information on the effectiveness or safety of interventions in the real world, but one major challenge is the existence of unmeasured confounder(s). While many analytical methods have been developed for dealing with this challenge, they appear under-utilized, perhaps due to the complexity and varied requirements for implementation. Thus, there is an unmet need to improve understanding the appropriate course of action to address unmeasured confounding under a variety of research scenarios. METHODS: We implemented a stepwise search strategy to find articles discussing the assessment of unmeasured confounding in electronic literature databases. Identified publications were reviewed and characterized by the applicable research settings and information requirements required for implementing each method. We further used this information to develop a best practice recommendation to help guide the selection of appropriate analytical methods for assessing the potential impact of unmeasured confounding. RESULTS: Over 100 papers were reviewed, and 15 methods were identified. We used a flowchart to illustrate the best practice recommendation which was driven by 2 critical components: (1) availability of information on the unmeasured confounders; and (2) goals of the unmeasured confounding assessment. Key factors for implementation of each method were summarized in a checklist to provide further assistance to researchers for implementing these methods. CONCLUSION: When assessing comparative effectiveness or safety in observational research, the impact of unmeasured confounding should not be ignored. Instead, we suggest quantitatively evaluating the impact of unmeasured confounding and provided a best practice recommendation for selecting appropriate analytical methods.

Recursive partitioning for heterogeneous causal effects.

In this paper we propose methods for estimating heterogeneity in causal effects in experimental and observational studies and for conducting hypothesis tests about the magnitude of differences in treatment effects across subsets of the population. We provide a data-driven approach to partition the data into subpopulations that differ in the magnitude of their treatment effects. The approach enables the construction of valid confidence intervals for treatment effects, even with many covariates relative to the sample size, and without "sparsity" assumptions. We propose an "honest" approach to estimation, whereby one sample is used to construct the partition and another to estimate treatment effects for each subpopulation. Our approach builds on regression tree methods, modified to optimize for goodness of fit in treatment effects and to account for honest estimation. Our model selection criterion anticipates that bias will be eliminated by honest estimation and also accounts for the effect of making additional splits on the variance of treatment effect estimates within each subpopulation. We address the challenge that the "ground truth" for a causal effect is not observed for any individual unit, so that standard approaches to cross-validation must be modified. Through a simulation study, we show that for our preferred method honest estimation results in nominal coverage for 90% confidence intervals, whereas coverage ranges between 74% and 84% for nonhonest approaches. Honest estimation requires estimating the model with a smaller sample size; the cost in terms of mean squared error of treatment effects for our preferred method ranges between 7-22%.

Propensity score matching and subclassification in observational studies with multi-level treatments.

In this article, we develop new methods for estimating average treatment effects in observational studies, in settings with more than two treatment levels, assuming unconfoundedness given pretreatment variables. We emphasize propensity score subclassification and matching methods which have been among the most popular methods in the binary treatment literature. Whereas the literature has suggested that these particular propensity-based methods do not naturally extend to the multi-level treatment case, we show, using the concept of weak unconfoundedness and the notion of the generalized propensity score, that adjusting for a scalar function of the pretreatment variables removes all biases associated with observed pretreatment variables. We apply the proposed methods to an analysis of the effect of treatments for fibromyalgia. We also carry out a simulation study to assess the finite sample performance of the methods relative to previously proposed methods.

An economist's perspective on Shadish (2010) and West and Thoemmes (2010).

In Shadish (2010) and West and Thoemmes (2010), the authors contrasted 2 approaches to causality. The first originated in the psychology literature and is associated with work by Campbell (e.g., Shadish, Cook, & Campbell, 2002), and the second has its roots in the statistics literature and is associated with work by Rubin (e.g., Rubin, 2006). In this article, I discuss some of the issues raised by Shadish and by West and Thoemmes. I focus mostly on the impact the 2 approaches have had on research in a 3rd field, economics. In economics, the ideas of both Campbell and Rubin have been very influential, with some of the methods they developed now routinely taught in graduate programs and routinely used in empirical work and other methods receiving much less attention. At the same time, economists have added to the understanding of these methods and through these extensions have further improved researchers' ability to draw causal inferences in observational studies.

Analyzing a randomized trial on breast self-examination with noncompliance and missing outcomes.

Recently, instrumental variables methods have been used to address non-compliance in randomized experiments. Complicating such analyses is often the presence of missing data. The standard model for missing data, missing at random (MAR), has some unattractive features in this context. In this paper we compare MAR-based estimates of the complier average causal effect (CACE) with an estimator based on an alternative, nonignorable model for the missing data process, developed by Frangakis and Rubin (1999, Biometrika, 86, 365-379). We also introduce a new missing data model that, like the Frangakis-Rubin model, is specially suited for models with instrumental variables, but makes different substantive assumptions. We analyze these issues in the context of a randomized trial of breast self-examination (BSE). In the study two methods of teaching BSE, consisting of either mailed information about BSE (the standard treatment) or the attendance of a course involving theoretical and practical sessions (the new treatment), were compared with the aim of assessing whether teaching programs could increase BSE practice and improve examination skills. The study was affected by the two sources of bias mentioned above: only 55% of women assigned to receive the new treatment complied with their assignment and 35% of the women did not respond to the post-test questionnaire. Comparing the causal estimand of the new treatment using the MAR, Frangakis-Rubin, and our new approach, the results suggest that for these data the MAR assumption appears least plausible, and that the new model appears most plausible among the three choices.