Identifying individual predictive factors for treatment efficacy.

Given the heterogeneous responses to therapy and the high cost of treatments, there is an increasing interest in identifying pretreatment predictors of therapeutic effect. Clearly, the success of such an endeavor will depend on the amount of information that the patient-specific variables convey about the individual causal treatment effect on the response of interest. In the present work, using causal inference and information theory, a strategy is proposed to evaluate individual predictive factors for cancer immunotherapy efficacy. In a first step, the methodology proposes a causal inference model to describe the joint distribution of the pretreatment predictors and the individual causal treatment effect. Further, in a second step, the so-called predictive causal information (PCI), a metric that quantifies the amount of information the pretreatment predictors convey on the individual causal treatment effects, is introduced and its properties are studied. The methodology is applied to identify predictors of therapeutic success for a therapeutic vaccine in advanced lung cancer. A user-friendly R library EffectTreat is provided to carry out the necessary calculations.

Implementing the meta-analytic approach for the evaluation of surrogate endpoints in SAS and R: a word of caution.

The meta-analytic approach has become the gold-standard methodology for the evaluation of surrogate endpoints and several implementations are currently available in SAS and R. The methodology is based on hierarchical models that are numerically demanding and, when the amount of data is limited, maximum likelihood algorithms may not converge or may converge to an ill-conditioned maximum such as a boundary solution. This may produce misleading conclusions and have negative implications for the evaluation of new drugs. In the present work, we explore the use of two distinct functions in R (lme and lmer) and the MIXED procedure in SAS to assess the validity of putative surrogate endpoints in the meta-analytic framework, via simulations and the analysis of a real case study. We describe some problems found with the lmer function in R that led to a poorer performance as compared with the lme function and MIXED procedure.

Identifying predictive biomarkers of CIMAvaxEGF success in non-small cell lung cancer patients.

BACKGROUND: Immunosenescence biomarkers and peripheral blood parameters are evaluated separately as possible predictive markers of immunotherapy. Here, we illustrate the use of a causal inference model to identify predictive biomarkers of CIMAvaxEGF success in the treatment of Non-Small Cell Lung Cancer Patients. METHODS: Data from a controlled clinical trial evaluating the effect of CIMAvax-EGF were analyzed retrospectively, following a causal inference approach. Pre-treatment potential predictive biomarkers included basal serum EGF concentration, peripheral blood parameters and immunosenescence biomarkers. The proportion of CD8 + CD28- T cells, CD4+ and CD8+ T cells, CD4/CD8 ratio and CD19+ B cells. The 33 patients with complete information were included. The predictive causal information (PCI) was calculated for all possible models. The model with a minimum number of predictors, but with high prediction accuracy (PCI > 0.7) was selected. Good, rare and poor responder patients were identified using the predictive probability of treatment success. RESULTS: The mean of PCI increased from 0.486, when only one predictor is considered, to 0.98 using the multivariate approach with all predictors. The model considering the proportion of CD4+ T cell, basal Epidermal Growth Factor (EGF) concentration, neutrophil to lymphocyte ratio, Monocytes, and Neutrophils as predictors were selected (PCI > 0.74). Patients predicted as good responders according to the pre-treatment biomarkers values treated with CIMAvax-EGF had a significant higher observed survival compared with the control group (p = 0.03). No difference was observed for bad responders. CONCLUSIONS: Peripheral blood parameters and immunosenescence biomarkers together with basal EGF concentration in serum resulted in good predictors of the CIMAvax-EGF success in advanced NSCLC. Future research should explore molecular and genetic profile as biomarkers for CIMAvax-EGF and it combination with immune-checkpoint inhibitors. The study illustrates the application of a new methodology, based on causal inference, to evaluate multivariate pre-treatment predictors. The multivariate approach allows realistic predictions of the clinical benefit of patients and should be introduced in daily clinical practice.

On the relationship between association and surrogacy when both the surrogate and true endpoint are binary outcomes.

The relationship between association and surrogacy has been the focus of much debate in the surrogate marker literature. Recently, the individual causal association (ICA) has been introduced as a metric of surrogacy in the causal inference framework, when both the surrogate and the true endpoint are normally distributed and when both are binary. Earlier work on the normal case has demonstrated that, although the ICA and the adjusted association are related metrics, their relationship strongly depends on unidentifiable parameters and, consequently, the association between both endpoints conveys little information on the validity of the surrogate. In addition, in the normal setting, the magnitude of the ICA does not depend on the mean of the outcomes. The latter implies that identifiable parameters such as mean responses and treatment effects provide no information on the validity of the surrogate. In the present work it is shown that this is fundamentally different in the binary case. We demonstrate that the observed association between the outcomes as well as the success rates in both treatment groups are quite predictive for the ICA. It is shown that finding a good surrogate will be more likely when the association between the endpoints is large, there are sizeable treatment effects and the success rates for both endpoints are similar in both treatment groups. These results are demonstrated using extensive simulations and illustrated on a case study in multi-drug resistant tuberculosis.

A reflection on the causal interpretation of individual-level surrogacy.

At the beginning of the 21st century, a new paradigm was introduced for the evaluation of surrogate endpoints based on meta-analysis. In this paradigm, the putative surrogate is assessed at two different levels, the so-called, trial and individual level. Trial level surrogacy is defined as the association between the expected causal treatment effects across different trials populations, whereas the individual level is defined as the association between the surrogate and true endpoints, after adjusting by trial and treatment. It has been argued that the individual level surrogacy does not have a causal interpretation and, consequently, it is a poor metric of surrogacy. In the present work, an alternative definition of individual level surrogacy is introduced based on individual causal treatment effects. In addition, using the maximum entropy principle, a direct link between the individual level surrogacy, as defined in the meta-analytic approach, and the newly proposed definition is established. This new perspective sets the individual level surrogacy in a more coherent framework with respect to the trial level and bridges the two main schools of thought in this domain, namely, the causal inference and meta-analytic schools.

A reflection on the possibility of finding a good surrogate.

Surrogate endpoints need to be statistically evaluated before they can be used as substitutes of true endpoints in clinical studies. However, even though several evaluation methods have been introduced over the last decades, the identification of good surrogate endpoints remains practically and conceptually challenging. In the present work, the question regarding the existence of a good surrogate is addressed using information-theoretic concepts, within a causal-inference framework. The methodology can help practitioners to assess, given a clinically relevant true endpoint and a treatment of interest, the chances of finding a good surrogate endpoint in the first place. The methodology focuses on binary outcomes and is illustrated using data from the Initial Glaucoma Treatment Study. Furthermore, a newly developed and user friendly R package Surrogate is provided to carry out the necessary calculations.

Assessing the predictive value of a binary surrogate for a binary true endpoint based on the minimum probability of a prediction error.

The individual causal association (ICA) has recently been introduced as a metric of surrogacy in a causal-inference framework. The ICA is defined on the unit interval and quantifies the association between the individual causal effect on the surrogate (ΔS) and true (ΔT) endpoint. In addition, the ICA offers a general assessment of the surrogate predictive value, taking value 1 when there is a deterministic relationship between ΔT and ΔS, and value 0 when both causal effects are independent. However, when one moves away from the previous two extreme scenarios, the interpretation of the ICA becomes challenging. In the present work, a new metric of surrogacy, the minimum probability of a prediction error (PPE), is introduced when both endpoints are binary, ie, the probability of erroneously predicting the value of ΔT using ΔS. Although the PPE has a more straightforward interpretation than the ICA, its magnitude is bounded above by a quantity that depends on the true endpoint. For this reason, the reduction in prediction error (RPE) attributed to the surrogate is defined. The RPE always lies in the unit interval, taking value 1 if prediction is perfect and 0 if ΔS conveys no information on ΔT. The methodology is illustrated using data from two clinical trials and a user-friendly R package Surrogate is provided to carry out the validation exercise.

A maximum entropy approach for the evaluation of surrogate endpoints based on causal inference.

The maximum entropy principle offers a constructive criterion for setting up probability distributions on the basis of partial knowledge. In the present work, the principle is applied to tackle an important problem in the surrogate marker field, namely, the evaluation of a binary outcome as a putative surrogate for a binary true endpoint within a causal inference framework. In the first step, the maximum entropy principle is used to determine the relative frequencies associated with the values of the vector of potential outcomes. Subsequently, in the second step, these relative frequencies are used in combination with two newly proposed metrics of surrogacy, the so-called individual causal association and the surrogate predictive function, to assess the validity of the surrogate. The procedure is conceptually similar to the use of noninformative or reference priors in Bayesian statistics. Additionally, approximate, identifiable bounds are proposed for the estimands of interest, and their performance is studied via simulations. The methods are illustrated using data from a clinical trial involving schizophrenic patients, and a newly developed and user-friendly R package Surrogate is provided to carry out the validation exercise.

Assessing a surrogate predictive value: a causal inference approach.

Several methods have been developed for the evaluation of surrogate endpoints within the causal-inference and meta-analytic paradigms. In both paradigms, much effort has been made to assess the capacity of the surrogate to predict the causal treatment effect on the true endpoint. In the present work, the so-called surrogate predictive function (SPF) is introduced for that purpose, using potential outcomes. The relationship between the SPF and the individual causal association, a new metric of surrogacy recently proposed in the literature, is studied in detail. It is shown that the SPF, in conjunction with the individual causal association, can offer an appealing quantification of the surrogate predictive value. However, neither the distribution of the potential outcomes nor the SPF are identifiable from the data. These identifiability issues are tackled using a two-step procedure. In the first step, the region of the parametric space of the distribution of the potential outcomes, compatible with the data at hand, is geometrically characterized. Further, in a second step, a Monte Carlo approach is used to study the behavior of the SPF on the previous region. The method is illustrated using data from a clinical trial involving schizophrenic patients and a newly developed and user friendly R package Surrogate is provided to carry out the validation exercise. Copyright © 2016 John Wiley & Sons, Ltd.

An information-theoretic approach for the evaluation of surrogate endpoints based on causal inference.

In this work a new metric of surrogacy, the so-called individual causal association (ICA), is introduced using information-theoretic concepts and a causal inference model for a binary surrogate and true endpoint. The ICA has a simple and appealing interpretation in terms of uncertainty reduction and, in some scenarios, it seems to provide a more coherent assessment of the validity of a surrogate than existing measures. The identifiability issues are tackled using a two-step procedure. In the first step, the region of the parametric space of the distribution of the potential outcomes, compatible with the data at hand, is geometrically characterized. Further, in a second step, a Monte Carlo approach is proposed to study the behavior of the ICA on the previous region. The method is illustrated using data from the Collaborative Initial Glaucoma Treatment Study. A newly developed and user-friendly R package Surrogate is provided to carry out the evaluation exercise.

Exploring the relationship between the causal-inference and meta-analytic paradigms for the evaluation of surrogate endpoints.

Nowadays, two main frameworks for the evaluation of surrogate endpoints, based on causal-inference and meta-analysis, dominate the scene. Earlier work showed that the metrics of surrogacy introduced in both paradigms are related, although in a complex way that is difficult to study analytically. In the present work, this relationship is further examined using simulations and the analysis of a case study. The results indicate that the extent to which both paradigms lead to similar conclusions regarding the validity of the surrogate, depends on a complex interplay between multiple factors like the ratio of the between and within trial variability and the unidentifiable correlations between the potential outcomes. All the analyses were carried out using the newly developed R package Surrogate, which is freely available via CRAN.

Statistical evaluation of surrogate endpoints with examples from cancer clinical trials.

A surrogate endpoint is intended to replace a clinical endpoint for the evaluation of new treatments when it can be measured more cheaply, more conveniently, more frequently, or earlier than that clinical endpoint. A surrogate endpoint is expected to predict clinical benefit, harm, or lack of these. Besides the biological plausibility of a surrogate, a quantitative assessment of the strength of evidence for surrogacy requires the demonstration of the prognostic value of the surrogate for the clinical outcome, and evidence that treatment effects on the surrogate reliably predict treatment effects on the clinical outcome. We focus on these two conditions, and outline the statistical approaches that have been proposed to assess the extent to which these conditions are fulfilled. When data are available from a single trial, one can assess the "individual level association" between the surrogate and the true endpoint. When data are available from several trials, one can additionally assess the "trial level association" between the treatment effect on the surrogate and the treatment effect on the true endpoint. In the latter case, the "surrogate threshold effect" can be estimated as the minimum effect on the surrogate endpoint that predicts a statistically significant effect on the clinical endpoint. All these concepts are discussed in the context of randomized clinical trials in oncology, and illustrated with two meta-analyses in gastric cancer.

On the relationship between the causal-inference and meta-analytic paradigms for the validation of surrogate endpoints.

The increasing cost of drug development has raised the demand for surrogate endpoints when evaluating new drugs in clinical trials. However, over the years, it has become clear that surrogate endpoints need to be statistically evaluated and deemed valid, before they can be used as substitutes of "true" endpoints in clinical studies. Nowadays, two paradigms, based on causal-inference and meta-analysis, dominate the scene. Nonetheless, although the literature emanating from these paradigms is wide, till now the relationship between them has largely been left unexplored. In the present work, we discuss the conceptual framework underlying both approaches and study the relationship between them using theoretical elements and the analysis of a real case study. Furthermore, we show that the meta-analytic approach can be embedded within a causal-inference framework on the one hand and that it can be heuristically justified why surrogate endpoints successfully evaluated using this approach will often be appealing from a causal-inference perspective as well, on the other. A newly developed and user friendly R package Surrogate is provided to carry out the evaluation exercise.

A new modeling approach for quantifying expert opinion in the drug discovery process.

Expert opinion plays an important role when choosing clusters of chemical compounds for further investigation. Often, the process by which the clusters are assigned to the experts for evaluation, the so-called selection process, and the qualitative ratings given by the experts to the clusters (chosen/not chosen) need to be jointly modeled to avoid bias. This approach is referred to as the joint modeling approach. However, misspecifying the selection model may impact the estimation and inferences on parameters in the rating model, which are of most scientific interest. We propose to incorporate the selection process into the analysis by adding a new set of random effects to the rating model and, in this way, avoid the need to model it parametrically. This approach is referred to as the combined model approach. Through simulations, the performance of the combined and joint models was compared in terms of bias and confidence interval coverage. The estimates from the combined model were nearly unbiased, and the derived confidence intervals had coverage probability around 95% in all scenarios considered. In contrast, the estimates from the joint model were severely biased under some form of misspecification of the selection model, and fitting the model was often numerically challenging. The results show that the combined model may offer a safer alternative on which to base inferences when there are doubts about the validity of the selection model. Importantly, thanks to its greater numerical stability, the combined model may outperform the joint model even when the latter is correctly specified.

Impact of selection bias on the evaluation of clusters of chemical compounds in the drug discovery process.

Expert opinion plays an important role when selecting promising clusters of chemical compounds in the drug discovery process. Indeed, experts can qualitatively assess the potential of each cluster, and with appropriate statistical methods, these qualitative assessments can be quantified into a success probability for each of them. However, one crucial element often overlooked is the procedure by which the clusters are assigned to/selected by the experts for evaluation. In the present work, the impact such a procedure may have on the statistical analysis and the entire evaluation process is studied. It has been shown that some implementations of the selection procedure may seriously compromise the validity of the evaluation even when the rating and selection processes are independent. Consequently, the fully random allocation of the clusters to the experts is strongly advocated.

Ignoring overdispersion in hierarchical loglinear models: Possible problems and solutions.

Poisson data frequently exhibit overdispersion; and, for univariate models, many options exist to circumvent this problem. Nonetheless, in complex scenarios, for example, in longitudinal studies, accounting for overdispersion is a more challenging task. Recently, Molenberghs et.al, presented a model that accounts for overdispersion by combining two sets of random effects. However, introducing a new set of random effects implies additional distributional assumptions for intrinsically unobservable variables, which has not been considered before. Using the combined model as a framework, we explored the impact of ignoring overdispersion in complex longitudinal settings via simulations. Furthermore, we evaluated the effect of misspecifying the random-effects distribution on both the combined model and the classical Poisson hierarchical model. Our results indicate that even though inferences may be affected by ignored overdispersion, the combined model is a promising tool in this scenario.

An information-theoretic approach to surrogate-marker evaluation with failure time endpoints.

Over the last decades, the evaluation of potential surrogate endpoints in clinical trials has steadily been growing in importance, not only thanks to the availability of ever more potential markers and surrogate endpoints, also because more methodological development has become available. While early work has been devoted, to a large extent, to Gaussian, binary, and longitudinal endpoints, the case of time-to-event endpoints is in need of careful scrutiny as well, owing to the strong presence of such endpoints in oncology and beyond. While work had been done in the past, it was often cumbersome to use such tools in practice, because of the need for fitting copula or frailty models that were further embedded in a hierarchical or two-stage modeling approach. In this paper, we present a methodologically elegant and easy-to-use approach based on information theory. We resolve essential issues, including the quantification of "surrogacy" based on such an approach. Our results are put to the test in a simulation study and are applied to data from clinical trials in oncology. The methodology has been implemented in R.

A unified approach to multi-item reliability.

The reliability of multi-item scales has received a lot of attention in the psychometric literature, where a myriad of measures like the Cronbach's α or the Spearman-Brown formula have been proposed. Most of these measures, however, are based on very restrictive models that apply only to unidimensional instruments. In this article, we introduce two measures to quantify the reliability of multi-item scales based on a more general model. We show that they capture two different aspects of the reliability problem and satisfy a minimum set of intuitive properties. The relevance and complementary value of the measures is studied and earlier approaches are placed in a broader theoretical framework. Finally, we apply them to investigate the reliability of the Positive and Negative Syndrome Scale, a rating scale for the assessment of the severity of schizophrenia.

The Functional Living Index-Cancer: estimating its reliability based on clinical trial data.

PURPOSE: The Functional Living Index-Cancer was developed to measure quality of life in cancer trials as an adjunct to the usual clinical outcomes. The scale is considered conceptually good, since it covers a broad range of relevant aspects of quality of life, but the main criticism has been that its reliability has never been properly investigated. In this paper, we investigate the reliability of the FLIC. METHODS: We apply a new methodology based on linear mixed models that allows estimating reliability from real clinical data. The reliability of the FLIC is estimated using data coming from a longitudinal study in breast cancer. With this new approach, we avoid the need for additional data collection on which classical reliability studies are based. RESULTS: The average reliability of the FLIC over the repeated measurements is satisfactory, even though the initial measurement in the study showed a somewhat lower value. Taking into account the longitudinal character of the measurements, we show that highly reliable information can be obtained with a relatively small number of measurements per patient. CONCLUSION: The FLIC provides reliable quality of life measurements in patients with breast cancer. Additional studies would be welcome to validate these results in other populations.

Evaluation of biomarkers for pharmacological activity.

In recent years the cost of drug development has increased the demands on efficiency in the selection of suitable drug candidates. Biomarkers for efficacy and safety could be a plausible strategy to improve this selection process. In the present work, we focus on the study and evaluation of different physiological variables as biomarkers for pharmacological activity. We proposed three different approaches using multivariate and univariate techniques. We note that even though one could argue that the multivariate procedure is more powerful than the other alternatives, the univariate methods also offer a great flexibility to answer interesting scientific questions. The three approaches were used to analyze a crossover study involving an opioid antagonist.