Efficient methods for signal detection from correlated adverse events in clinical trials.

It is an important and yet challenging task to identify true signals from many adverse events that may be reported during the course of a clinical trial. One unique feature of drug safety data from clinical trials, unlike data from post-marketing spontaneous reporting, is that many types of adverse events are reported by only very few patients leading to rare events. Due to the limited study size, the p-values of testing whether the rate is higher in the treatment group across all types of adverse events are in general not uniformly distributed under the null hypothesis that there is no difference between the treatment group and the placebo group. A consequence is that typically fewer than 100α percent of the hypotheses are rejected under the null at the nominal significance level of α . The other challenge is multiplicity control. Adverse events from the same body system may be correlated. There may also be correlations between adverse events from different body systems. To tackle these challenging issues, we develop Monte-Carlo-based methods for the signal identification from patient-reported adverse events in clinical trials. The proposed methodologies account for the rare events and arbitrary correlation structures among adverse events within and/or between body systems. Extensive simulation studies demonstrate that the proposed method can accurately control the family-wise error rate and is more powerful than existing methods under many practical situations. Application to two real examples is provided.

Controlling false discovery proportion in identification of drug-related adverse events from multiple system organ classes.

Analyzing safety data from clinical trials to detect safety signals worth further examination involves testing multiple hypotheses, one for each observed adverse event (AE) type. There exists certain hierarchical structure for these hypotheses due to the classification of the AEs into system organ classes, and these AEs are also likely correlated. Many approaches have been proposed to identify safety signals under the multiple testing framework and tried to achieve control of false discovery rate (FDR). The FDR control concerns the expectation of the false discovery proportion (FDP). In practice, the control of the actual random variable FDP could be more relevant and has recently drawn much attention. In this paper, we proposed a two-stage procedure for safety signal detection with direct control of FDP, through a permutation-based approach for screening groups of AEs and a permutation-based approach of constructing simultaneous upper bounds for false discovery proportion. Our simulation studies showed that this new approach has controlled FDP. We demonstrate our approach using data sets derived from a drug clinical trial.

Improved power of familywise error rate procedures for discrete data under dependency.

In many applications where it is necessary to test multiple hypotheses simultaneously, the data encountered are discrete. In such cases, it is important for multiplicity adjustment to take into account the discreteness of the distributions of the p-values, to assure that the procedure is not overly conservative. In this paper, we review some known multiple testing procedures for discrete data that control the familywise error rate, the probability of making any false rejection. Taking advantage of the fact that the exact permutation or exact pairwise permutation distributions of the p-values can often be determined when the sample size is small, we investigate procedures that incorporate the dependence structure through the exact permutation distribution and propose two new procedures that incorporate the exact pairwise permutation distributions. A step-up procedure is also proposed that accounts for the discreteness of the data. The performance of the proposed procedures is investigated through simulation studies and two applications. The results show that by incorporating both discreteness and dependency of p-value distributions, gains in power can be achieved.

Multiple testing with discrete data: Proportion of true null hypotheses and two adaptive FDR procedures.

We consider multiple testing with false discovery rate (FDR) control when p values have discrete and heterogeneous null distributions. We propose a new estimator of the proportion of true null hypotheses and demonstrate that it is less upwardly biased than Storey's estimator and two other estimators. The new estimator induces two adaptive procedures, that is, an adaptive Benjamini-Hochberg (BH) procedure and an adaptive Benjamini-Hochberg-Heyse (BHH) procedure. We prove that the adaptive BH (aBH) procedure is conservative nonasymptotically. Through simulation studies, we show that these procedures are usually more powerful than their nonadaptive counterparts and that the adaptive BHH procedure is usually more powerful than the aBH procedure and a procedure based on randomized p-value. The adaptive procedures are applied to a study of HIV vaccine efficacy, where they identify more differentially polymorphic positions than the BH procedure at the same FDR level.

On inference of control-based imputation for analysis of repeated binary outcomes with missing data.

Missing data are common in longitudinal clinical trials. How to handle missing data is critical for both sponsors and regulatory agencies to assess treatment effect from the trials. Recently, a control-based imputation has been proposed, where the missing data are imputed based on the assumption that patients who discontinued the test drug will have a similar response profile to the patients in the control group. Under control-based imputation, the variance estimation may be biased using Rubin's formula which could produce biased statistical inferences. We evaluate several statistical methods for obtaining appropriate variances under control-based imputation for analysis of repeated binary outcomes with monotone missing data and show that both the analytical method developed by Robins & Wang and the nonparametric bootstrap method provide more appropriate variance estimates under various simulation settings. We use the methods in an application of an antidepressant Phase III clinical trial and give discussion and recommendations on method performance and preference.

Control-based imputation for sensitivity analyses in informative censoring for recurrent event data.

In clinical trials, missing data commonly arise through nonadherence to the randomized treatment or to study procedure. For trials in which recurrent event endpoints are of interests, conventional analyses using the proportional intensity model or the count model assume that the data are missing at random, which cannot be tested using the observed data alone. Thus, sensitivity analyses are recommended. We implement the control-based multiple imputation as sensitivity analyses for the recurrent event data. We model the recurrent event using a piecewise exponential proportional intensity model with frailty and sample the parameters from the posterior distribution. We impute the number of events after dropped out and correct the variance estimation using a bootstrap procedure. We apply the method to an application of sitagliptin study.

Bayesian probability of success for clinical trials using historical data.

Developing sophisticated statistical methods for go/no-go decisions is crucial for clinical trials, as planning phase III or phase IV trials is costly and time consuming. In this paper, we develop a novel Bayesian methodology for determining the probability of success of a treatment regimen on the basis of the current data of a given trial. We introduce a new criterion for calculating the probability of success that allows for inclusion of covariates as well as allowing for historical data based on the treatment regimen, and patient characteristics. A new class of prior distributions and covariate distributions is developed to achieve this goal. The methodology is quite general and can be used with univariate or multivariate continuous or discrete data, and it generalizes Chuang-Stein's work. This methodology will be invaluable for informing the scientist on the likelihood of success of the compound, while including the information of covariates for patient characteristics in the trial population for planning future pre-market or post-market trials.

Medical records-based postmarketing safety evaluation of rare events with uncertain status.

We develop a simple statistic for comparing rates of rare adverse events between treatment groups in postmarketing safety studies where the events have uncertain status. In this setting, the statistic is asymptotically equivalent to the logrank statistic, but the limiting distribution has Poisson and binomial components instead of being Gaussian. We develop two new procedures for computing critical values, a Gaussian approximation and a parametric bootstrap. Both numerical and asymptotic properties of the procedures are studied. The test procedures are demonstrated on a postmarketing safety study of the RotaTeq vaccine. This vaccine was developed to reduce the incidence of severe diarrhea in infants.

Medical records-based postmarketing safety evaluation of rare events with uncertain status.

We develop a simple statistic for comparing rates of rare adverse events between treatment groups in postmarketing safety studies where the events have uncertain status. In this setting, the statistic is asymptotically equivalent to the logrank statistic, but the limiting distribution has Poisson and binomial components instead of being Gaussian. We develop two new procedures for computing critical values: a Gaussian approximation and a parametric bootstrap. Both numerical and asymptotic properties of the procedures are studied. The test procedures are demonstrated on a postmarketing safety study of the RotaTeq vaccine. This vaccine was developed to reduce the incidence of severe diarrhea in infants.

cSSMD: assessing collective activity for addressing off-target effects in genome-scale RNA interference screens.

MOTIVATION: Off-target activity commonly exists in RNA interference (RNAi) screens and often generates false positives. Existing analytic methods for addressing the off-target effects are demonstrably inadequate in RNAi confirmatory screens. RESULTS: Here, we present an analytic method assessing the collective activity of multiple short interfering RNAs (siRNAs) targeting a gene. Using this method, we can not only reduce the impact of off-target activities, but also evaluate the specific effect of an siRNA, thus providing information about potential off-target effects. Using in-house RNAi screens, we demonstrate that our method obtains more reasonable and sensible results than current methods such as the redundant siRNA activity (RSA) method, the RNAi gene enrichment ranking (RIGER) method, the frequency approach and the t-test. CONTACT: xiaohua_zhang@merck.com SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Efficient Multiple Imputation for Sensitivity Analysis of Recurrent Events Data with Informative Censoring.

Missing data are commonly encountered in clinical trials due to dropout or nonadherence to study procedures. In trials in which recurrent events are of interest, the observed count can be an undercount of the events if a patient drops out before the end of the study. In many applications, the data are not necessarily missing at random and it is often not possible to test the missing at random assumption. Consequently, it is critical to conduct sensitivity analysis. We develop a control-based multiple imputation method for recurrent events data, where patients who drop out of the study are assumed to have a similar response profile to those in the control group after dropping out. Specifically, we consider the copy reference approach and the jump to reference approach. We model the recurrent event data using a semiparametric proportional intensity frailty model with the baseline hazard function completely unspecified. We develop nonparametric maximum likelihood estimation and inference procedures. We then impute the missing data based on the large sample distribution of the resulting estimators. The variance estimation is corrected by a bootstrap procedure. Simulation studies demonstrate the proposed method performs well in practical settings. We provide applications to two clinical trials.

Age dependence of the risk of intussusception following [corrected] rhesus-human reassortant rotavirus tetravalent vaccine: is it beyond doubt?

The relation between the risk of intussusception and age at the time of receipt of the first dose of rhesus-human reassortant rotavirus tetravalent vaccine (RRV-TV) has been studied extensively on the basis of Centers for Disease Control and Prevention (CDC) matched case-control study data, using various statistical methods, including conditional logistic regression and quadratic smoothing splines. However, different conclusions have been reported in published analyses regarding the dependence of the risk of intussusception on age at first dose. The authors reanalyzed the CDC matched case-control data set using unrestricted and restricted quadratic smoothing spline methods for various exposure windows (i.e., intervals postvaccination). These analyses indicated that the use of different models may lead to different conclusions. The restricted quadratic smoothing spline with appropriately chosen knot locations showed a statistically significant increased risk of intussusception associated with RRV-TV for the exposure window 3-14 days after the first dose at an age as young as 49 days, the youngest age in the data set at which vaccine was administered; this implies an increased risk of intussusception associated with RRV-TV at all ages studied.

Determination of sample size in genome-scale RNAi screens.

MOTIVATION: For genome-scale RNAi research, it is critical to investigate sample size required for the achievement of reasonably low false negative rate (FNR) and false positive rate. RESULTS: The analysis in this article reveals that current design of sample size contributes to the occurrence of low signal-to-noise ratio in genome-scale RNAi projects. The analysis suggests that (i) an arrangement of 16 wells per plate is acceptable and an arrangement of 20-24 wells per plate is preferable for a negative control to be used for hit selection in a primary screen without replicates; (ii) in a confirmatory screen or a primary screen with replicates, a sample size of 3 is not large enough, and there is a large reduction in FNRs when sample size increases from 3 to 4. To search a tradeoff between benefit and cost, any sample size between 4 and 11 is a reasonable choice. If the main focus is the selection of siRNAs with strong effects, a sample size of 4 or 5 is a good choice. If we want to have enough power to detect siRNAs with moderate effects, sample size needs to be 8, 9, 10 or 11. These discoveries about sample size bring insight to the design of a genome-scale RNAi screen experiment.

Sequential generalized likelihood ratio tests for vaccine safety evaluation.

The evaluation of vaccine safety involves pre-clinical animal studies, pre-licensure randomized clinical trials, and post-licensure safety studies. Sequential design and analysis are of particular interest because they allow early termination of the trial or quick detection that the vaccine exceeds a prescribed bound on the adverse event rate. After a review of the recent developments in this area, we propose a new class of sequential generalized likelihood ratio tests for evaluating adverse event rates in two-armed pre-licensure clinical trials and single-armed post-licensure studies. The proposed approach is illustrated using data from the Rotavirus Efficacy and Safety Trial. Simulation studies of the performance of the proposed approach and other methods are also given.

Hit selection with false discovery rate control in genome-scale RNAi screens.

RNA interference (RNAi) is a modality in which small double-stranded RNA molecules (siRNAs) designed to lead to the degradation of specific mRNAs are introduced into cells or organisms. siRNA libraries have been developed in which siRNAs targeting virtually every gene in the human genome are designed, synthesized and are presented for introduction into cells by transfection in a microtiter plate array. These siRNAs can then be transfected into cells using high-throughput screening (HTS) methodologies. The goal of RNAi HTS is to identify a set of siRNAs that inhibit or activate defined cellular phenotypes. The commonly used analysis methods including median +/- kMAD have issues about error rates in multiple hypothesis testing and plate-wise versus experiment-wise analysis. We propose a methodology based on a Bayesian framework to address these issues. Our approach allows for sharing of information across plates in a plate-wise analysis, which obviates the need for choosing either a plate-wise or experimental-wise analysis. The proposed approach incorporates information from reliable controls to achieve a higher power and a balance between the contribution from the samples and control wells. Our approach provides false discovery rate (FDR) control to address multiple testing issues and it is robust to outliers.

Safety and efficacy of a pentavalent human-bovine (WC3) reassortant rotavirus vaccine.

BACKGROUND: Rotavirus is a leading cause of childhood gastroenteritis and death worldwide. METHODS: We studied healthy infants approximately 6 to 12 weeks old who were randomly assigned to receive three oral doses of live pentavalent human-bovine (WC3 strain) reassortant rotavirus vaccine containing human serotypes G1, G2, G3, G4, and P[8] or placebo at 4-to-10-week intervals in a blinded fashion. Active surveillance was used to identify subjects with serious adverse and other events. RESULTS: The 34,035 infants in the vaccine group and 34,003 in the placebo group were monitored for serious adverse events. Intussusception occurred in 12 vaccine recipients and 15 placebo recipients within one year after the first dose including six vaccine recipients and five placebo recipients within 42 days after any dose (relative risk, 1.6; 95 percent confidence interval, 0.4 to 6.4). The vaccine reduced hospitalizations and emergency department visits related to G1-G4 rotavirus gastroenteritis occurring 14 or more days after the third dose by 94.5 percent (95 percent confidence interval, 91.2 to 96.6 percent). In a nested substudy, efficacy against any G1-G4 rotavirus gastroenteritis through the first full rotavirus season after vaccination was 74.0 percent (95 percent confidence interval, 66.8 to 79.9 percent); efficacy against severe gastroenteritis was 98.0 percent (95 percent confidence interval, 88.3 to 100 percent). The vaccine reduced clinic visits for G1-G4 rotavirus gastroenteritis by 86.0 percent (95 percent confidence interval, 73.9 to 92.5 percent). CONCLUSIONS: This vaccine was efficacious in preventing rotavirus gastroenteritis, decreasing severe disease and health care contacts. The risk of intussusception was similar in vaccine and placebo recipients. (ClinicalTrials.gov number, NCT00090233.)

Integrating experimental and analytic approaches to improve data quality in genome-wide RNAi screens.

RNA interference (RNAi) not only plays an important role in drug discovery but can also be developed directly into drugs. RNAi high-throughput screening (HTS) biotechnology allows us to conduct genome-wide RNAi research. A central challenge in genome-wide RNAi research is to integrate both experimental and computational approaches to obtain high quality RNAi HTS assays. Based on our daily practice in RNAi HTS experiments, we propose the implementation of 3 experimental and analytic processes to improve the quality of data from RNAi HTS biotechnology: (1) select effective biological controls; (2) adopt appropriate plate designs to display and/or adjust for systematic errors of measurement; and (3) use effective analytic metrics to assess data quality. The applications in 5 real RNAi HTS experiments demonstrate the effectiveness of integrating these processes to improve data quality. Due to the effectiveness in improving data quality in RNAi HTS experiments, the methods and guidelines contained in the 3 experimental and analytic processes are likely to have broad utility in genome-wide RNAi research.

Periodic benefit-risk assessment using Bayesian stochastic multi-criteria acceptability analysis.

Benefit-risk (BR) assessment is essential to ensure the best decisions are made for a medical product in the clinical development process, regulatory marketing authorization, post-market surveillance, and coverage and reimbursement decisions. One challenge of BR assessment in practice is that the benefit and risk profile may keep evolving while new evidence is accumulating. Regulators and the International Conference on Harmonization (ICH) recommend performing periodic benefit-risk evaluation report (PBRER) through the product's lifecycle. In this paper, we propose a general statistical framework for periodic benefit-risk assessment, in which Bayesian meta-analysis and stochastic multi-criteria acceptability analysis (SMAA) will be combined to synthesize the accumulating evidence. The proposed approach allows us to compare the acceptability of different drugs dynamically and effectively and accounts for the uncertainty of clinical measurements and imprecise or incomplete preference information of decision makers. We apply our approaches to two real examples in a post-hoc way for illustration purpose. The proposed method may easily be modified for other pre and post market settings, and thus be an important complement to the current structured benefit-risk assessment (sBRA) framework to improve the transparent and consistency of the decision-making process.

The use of strictly standardized mean difference for hit selection in primary RNA interference high-throughput screening experiments.

RNA interference (RNAi) high-throughput screening (HTS) has been hailed as the 2nd genomics wave following the 1st genomics wave of gene expression microarrays and single-nucleotide polymorphism discovery platforms. Following an RNAi HTS, the authors are interested in identifying short interfering RNA (siRNA) hits with large inhibition/activation effects. For hit selection, the z-score method and its variants are commonly used in primary RNAi HTS experiments. Recently, strictly standardized mean difference (SSMD) has been proposed to measure the siRNA effect represented by the magnitude of difference between an siRNA and a negative reference group. The links between SSMD and d+-probability offer a clear interpretation of siRNA effects from a probability perspective. Hence, SSMD can be used as a ranking metric for hit selection. In this article, the authors investigated both the SSMD-based testing process and the use of SSMD as a ranking metric for hit selection in 2 primary siRNA HTS experiments. The analysis results showed that, as a ranking metric, SSMD was more stable and reliable than percentage inhibition and led to more robust hit selection results. Using the SSMD -based testing method, the false-negative rate can more readily be obtained. More important, the use of the SSMD-based method can result in a reduction in both the false-negative and false-positive rates. The applications presented in this article demonstrate that the SSMD method addresses scientific questions and fills scientific needs better than both percentage inhibition and the commonly used z-score method for hit selection.

Impact of indinavir on the quality of life in patients with advanced HIV infection treated with zidovudine and lamivudine.

OBJECTIVE: In AIDS Clinical Trial Group (ACTG) study 320, triple-combination antiretroviral therapy including indinavir significantly slowed progression to acquired immunodeficiency syndrome or death, compared with treatment with dual nucleoside reverse-transcriptase inhibitors (NRTIs) alone, in zidovudine-experienced patients with advanced human immunodeficiency virus (HIV) infection. We examined the impact of indinavir on quality of life in participants from this study. METHODS: A total of 1156 protease inhibitor- and lamivudine-naive patients stratified by CD4 cell count (