An information-theoretic approach for the assessment of a continuous outcome as a surrogate for a binary true endpoint based on causal inference: Application to vaccine evaluation.

Within the causal association paradigm, a method is proposed to assess the validity of a continuous outcome as a surrogate for a binary true endpoint. The methodology is based on a previously introduced information-theoretic definition of surrogacy and has two main steps. In the first step, a new model is proposed to describe the joint distribution of the potential outcomes associated with the putative surrogate and the true endpoint of interest. The identifiability issues inherent to this type of models are handled via sensitivity analysis. In the second step, a metric of surrogacy new to this setting, the so-called individual causal association is presented. The methodology is studied in detail using theoretical considerations, some simulations, and data from a randomized clinical trial evaluating an inactivated quadrivalent influenza vaccine. A user-friendly R package Surrogate is provided to carry out the evaluation exercise.

Identifying individual predictive factors for treatment efficacy.

Given the heterogeneous responses to therapy and the high cost of treatments, there is an increasing interest in identifying pretreatment predictors of therapeutic effect. Clearly, the success of such an endeavor will depend on the amount of information that the patient-specific variables convey about the individual causal treatment effect on the response of interest. In the present work, using causal inference and information theory, a strategy is proposed to evaluate individual predictive factors for cancer immunotherapy efficacy. In a first step, the methodology proposes a causal inference model to describe the joint distribution of the pretreatment predictors and the individual causal treatment effect. Further, in a second step, the so-called predictive causal information (PCI), a metric that quantifies the amount of information the pretreatment predictors convey on the individual causal treatment effects, is introduced and its properties are studied. The methodology is applied to identify predictors of therapeutic success for a therapeutic vaccine in advanced lung cancer. A user-friendly R library EffectTreat is provided to carry out the necessary calculations.

Implementing the meta-analytic approach for the evaluation of surrogate endpoints in SAS and R: a word of caution.

The meta-analytic approach has become the gold-standard methodology for the evaluation of surrogate endpoints and several implementations are currently available in SAS and R. The methodology is based on hierarchical models that are numerically demanding and, when the amount of data is limited, maximum likelihood algorithms may not converge or may converge to an ill-conditioned maximum such as a boundary solution. This may produce misleading conclusions and have negative implications for the evaluation of new drugs. In the present work, we explore the use of two distinct functions in R (lme and lmer) and the MIXED procedure in SAS to assess the validity of putative surrogate endpoints in the meta-analytic framework, via simulations and the analysis of a real case study. We describe some problems found with the lmer function in R that led to a poorer performance as compared with the lme function and MIXED procedure.

The individual-level surrogate threshold effect in a causal-inference setting with normally distributed endpoints.

In the meta-analytic surrogate evaluation framework, the trial-level coefficient of determination Rtrial2 quantifies the strength of the association between the expected causal treatment effects on the surrogate (S) and the true (T) endpoints. Burzykowski and Buyse supplemented this metric of surrogacy with the surrogate threshold effect (STE), which is defined as the minimum value of the causal treatment effect on S for which the predicted causal treatment effect on T exceeds zero. The STE supplements Rtrial2 with a more direct clinically interpretable metric of surrogacy. Alonso et al. proposed to evaluate surrogacy based on the strength of the association between the individual (rather than expected) causal treatment effects on S and T. In the current paper, the individual-level surrogate threshold effect (ISTE) is introduced in the setting where S and T are normally distributed variables. ISTE is defined as the minimum value of the individual causal treatment effect on S for which the lower limit of the prediction interval around the individual causal treatment effect on T exceeds zero. The newly proposed methodology is applied in a case study, and it is illustrated that ISTE has an appealing clinical interpretation. The R package surrogate implements the methodology and a web appendix (supporting information) that details how the analyses can be conducted in practice is provided.

Identifying predictive biomarkers of CIMAvaxEGF success in non-small cell lung cancer patients.

BACKGROUND: Immunosenescence biomarkers and peripheral blood parameters are evaluated separately as possible predictive markers of immunotherapy. Here, we illustrate the use of a causal inference model to identify predictive biomarkers of CIMAvaxEGF success in the treatment of Non-Small Cell Lung Cancer Patients. METHODS: Data from a controlled clinical trial evaluating the effect of CIMAvax-EGF were analyzed retrospectively, following a causal inference approach. Pre-treatment potential predictive biomarkers included basal serum EGF concentration, peripheral blood parameters and immunosenescence biomarkers. The proportion of CD8 + CD28- T cells, CD4+ and CD8+ T cells, CD4/CD8 ratio and CD19+ B cells. The 33 patients with complete information were included. The predictive causal information (PCI) was calculated for all possible models. The model with a minimum number of predictors, but with high prediction accuracy (PCI > 0.7) was selected. Good, rare and poor responder patients were identified using the predictive probability of treatment success. RESULTS: The mean of PCI increased from 0.486, when only one predictor is considered, to 0.98 using the multivariate approach with all predictors. The model considering the proportion of CD4+ T cell, basal Epidermal Growth Factor (EGF) concentration, neutrophil to lymphocyte ratio, Monocytes, and Neutrophils as predictors were selected (PCI > 0.74). Patients predicted as good responders according to the pre-treatment biomarkers values treated with CIMAvax-EGF had a significant higher observed survival compared with the control group (p = 0.03). No difference was observed for bad responders. CONCLUSIONS: Peripheral blood parameters and immunosenescence biomarkers together with basal EGF concentration in serum resulted in good predictors of the CIMAvax-EGF success in advanced NSCLC. Future research should explore molecular and genetic profile as biomarkers for CIMAvax-EGF and it combination with immune-checkpoint inhibitors. The study illustrates the application of a new methodology, based on causal inference, to evaluate multivariate pre-treatment predictors. The multivariate approach allows realistic predictions of the clinical benefit of patients and should be introduced in daily clinical practice.

On the relationship between association and surrogacy when both the surrogate and true endpoint are binary outcomes.

The relationship between association and surrogacy has been the focus of much debate in the surrogate marker literature. Recently, the individual causal association (ICA) has been introduced as a metric of surrogacy in the causal inference framework, when both the surrogate and the true endpoint are normally distributed and when both are binary. Earlier work on the normal case has demonstrated that, although the ICA and the adjusted association are related metrics, their relationship strongly depends on unidentifiable parameters and, consequently, the association between both endpoints conveys little information on the validity of the surrogate. In addition, in the normal setting, the magnitude of the ICA does not depend on the mean of the outcomes. The latter implies that identifiable parameters such as mean responses and treatment effects provide no information on the validity of the surrogate. In the present work it is shown that this is fundamentally different in the binary case. We demonstrate that the observed association between the outcomes as well as the success rates in both treatment groups are quite predictive for the ICA. It is shown that finding a good surrogate will be more likely when the association between the endpoints is large, there are sizeable treatment effects and the success rates for both endpoints are similar in both treatment groups. These results are demonstrated using extensive simulations and illustrated on a case study in multi-drug resistant tuberculosis.

A reflection on the causal interpretation of individual-level surrogacy.

At the beginning of the 21st century, a new paradigm was introduced for the evaluation of surrogate endpoints based on meta-analysis. In this paradigm, the putative surrogate is assessed at two different levels, the so-called, trial and individual level. Trial level surrogacy is defined as the association between the expected causal treatment effects across different trials populations, whereas the individual level is defined as the association between the surrogate and true endpoints, after adjusting by trial and treatment. It has been argued that the individual level surrogacy does not have a causal interpretation and, consequently, it is a poor metric of surrogacy. In the present work, an alternative definition of individual level surrogacy is introduced based on individual causal treatment effects. In addition, using the maximum entropy principle, a direct link between the individual level surrogacy, as defined in the meta-analytic approach, and the newly proposed definition is established. This new perspective sets the individual level surrogacy in a more coherent framework with respect to the trial level and bridges the two main schools of thought in this domain, namely, the causal inference and meta-analytic schools.

A reflection on the possibility of finding a good surrogate.

Surrogate endpoints need to be statistically evaluated before they can be used as substitutes of true endpoints in clinical studies. However, even though several evaluation methods have been introduced over the last decades, the identification of good surrogate endpoints remains practically and conceptually challenging. In the present work, the question regarding the existence of a good surrogate is addressed using information-theoretic concepts, within a causal-inference framework. The methodology can help practitioners to assess, given a clinically relevant true endpoint and a treatment of interest, the chances of finding a good surrogate endpoint in the first place. The methodology focuses on binary outcomes and is illustrated using data from the Initial Glaucoma Treatment Study. Furthermore, a newly developed and user friendly R package Surrogate is provided to carry out the necessary calculations.

Assessing the predictive value of a binary surrogate for a binary true endpoint based on the minimum probability of a prediction error.

The individual causal association (ICA) has recently been introduced as a metric of surrogacy in a causal-inference framework. The ICA is defined on the unit interval and quantifies the association between the individual causal effect on the surrogate (ΔS) and true (ΔT) endpoint. In addition, the ICA offers a general assessment of the surrogate predictive value, taking value 1 when there is a deterministic relationship between ΔT and ΔS, and value 0 when both causal effects are independent. However, when one moves away from the previous two extreme scenarios, the interpretation of the ICA becomes challenging. In the present work, a new metric of surrogacy, the minimum probability of a prediction error (PPE), is introduced when both endpoints are binary, ie, the probability of erroneously predicting the value of ΔT using ΔS. Although the PPE has a more straightforward interpretation than the ICA, its magnitude is bounded above by a quantity that depends on the true endpoint. For this reason, the reduction in prediction error (RPE) attributed to the surrogate is defined. The RPE always lies in the unit interval, taking value 1 if prediction is perfect and 0 if ΔS conveys no information on ΔT. The methodology is illustrated using data from two clinical trials and a user-friendly R package Surrogate is provided to carry out the validation exercise.

A maximum entropy approach for the evaluation of surrogate endpoints based on causal inference.

The maximum entropy principle offers a constructive criterion for setting up probability distributions on the basis of partial knowledge. In the present work, the principle is applied to tackle an important problem in the surrogate marker field, namely, the evaluation of a binary outcome as a putative surrogate for a binary true endpoint within a causal inference framework. In the first step, the maximum entropy principle is used to determine the relative frequencies associated with the values of the vector of potential outcomes. Subsequently, in the second step, these relative frequencies are used in combination with two newly proposed metrics of surrogacy, the so-called individual causal association and the surrogate predictive function, to assess the validity of the surrogate. The procedure is conceptually similar to the use of noninformative or reference priors in Bayesian statistics. Additionally, approximate, identifiable bounds are proposed for the estimands of interest, and their performance is studied via simulations. The methods are illustrated using data from a clinical trial involving schizophrenic patients, and a newly developed and user-friendly R package Surrogate is provided to carry out the validation exercise.

Predictive factors influencing pregnancy rates after intrauterine insemination with frozen donor semen: a prospective cohort study.

The extent to which certain parameters can influence pregnancy rates after intrauterine insemination with frozen donor semen was examined prospectively. Between July 2011 and September 2015, 402 women received 1264 IUI cycles with frozen donor semen in a tertiary referral infertility centre. A case report form was used to collect data prospectively. The primary outcome measure was clinical pregnancy rate (CPR), confirmed by detection of a gestational sac and fetal heartbeat using ultrasonography at 7-8 weeks of gestation. Statistical analysis was carried out using generalized estimating equations (GEE) to account for the correlation between observations from the same patient. Overall, CPR per cycle was 17.2%. Multivariate GEE analysis revealed the following parameters as predictive for a successful pregnancy outcome: female age (P = 0.0003), non-smoking or smoking fewer than 15 cigarettes a day (P = 0.0470 and P = 0.0235, respectively), secondary infertility (P = 0.0062), low progesterone levels at day zero of the cycle (P = 0.0164) and use of ovarian stimulation with HMG and recombinant FSH compared with clomiphene citrate and natural cycle (P = 0.0006 and P = 0.0004, respectively). These parameters were the most important factors influencing the success rate in a sperm donation programme.

Predictive value of different covariates influencing pregnancy rate following intrauterine insemination with homologous semen: a prospective cohort study.

The aim was to examine the value of different covariates in the prediction of intrauterine insemination (IUI) success. Between July 2011 and September 2015, data from 1401 IUI cycles with homologous semen in 556 couples were collected prospectively, by questionnaire, in a tertiary referral infertility centre. Statistical analysis was performed using generalized estimating equations (GEEs). GEEs were used instead of an ordinary logistic regression model to take into account the correlation between observations from the same person. The primary outcome parameter was clinical pregnancy rate (CPR), confirmed with a gestational sac and fetal heartbeat on ultrasonography at 7-8 weeks. An overall CPR of 9.5% per cycle was observed. Univariate statistical analysis revealed female and male age, male smoking, female body mass index, ovarian stimulation and inseminating motile count (IMC) as covariates significantly influencing CPR per cycle. Multivariate GEE analysis revealed that the only valuable prognostic covariates included female age, male smoking and infertility status (i.e. primary/secondary infertility). IMC showed a significant curvilinear relationship, with first an increase and then a decrease in pregnancy rate.

Assessing a surrogate predictive value: a causal inference approach.

Several methods have been developed for the evaluation of surrogate endpoints within the causal-inference and meta-analytic paradigms. In both paradigms, much effort has been made to assess the capacity of the surrogate to predict the causal treatment effect on the true endpoint. In the present work, the so-called surrogate predictive function (SPF) is introduced for that purpose, using potential outcomes. The relationship between the SPF and the individual causal association, a new metric of surrogacy recently proposed in the literature, is studied in detail. It is shown that the SPF, in conjunction with the individual causal association, can offer an appealing quantification of the surrogate predictive value. However, neither the distribution of the potential outcomes nor the SPF are identifiable from the data. These identifiability issues are tackled using a two-step procedure. In the first step, the region of the parametric space of the distribution of the potential outcomes, compatible with the data at hand, is geometrically characterized. Further, in a second step, a Monte Carlo approach is used to study the behavior of the SPF on the previous region. The method is illustrated using data from a clinical trial involving schizophrenic patients and a newly developed and user friendly R package Surrogate is provided to carry out the validation exercise. Copyright © 2016 John Wiley & Sons, Ltd.

An information-theoretic approach for the evaluation of surrogate endpoints based on causal inference.

In this work a new metric of surrogacy, the so-called individual causal association (ICA), is introduced using information-theoretic concepts and a causal inference model for a binary surrogate and true endpoint. The ICA has a simple and appealing interpretation in terms of uncertainty reduction and, in some scenarios, it seems to provide a more coherent assessment of the validity of a surrogate than existing measures. The identifiability issues are tackled using a two-step procedure. In the first step, the region of the parametric space of the distribution of the potential outcomes, compatible with the data at hand, is geometrically characterized. Further, in a second step, a Monte Carlo approach is proposed to study the behavior of the ICA on the previous region. The method is illustrated using data from the Collaborative Initial Glaucoma Treatment Study. A newly developed and user-friendly R package Surrogate is provided to carry out the evaluation exercise.

Exploring the relationship between the causal-inference and meta-analytic paradigms for the evaluation of surrogate endpoints.

Nowadays, two main frameworks for the evaluation of surrogate endpoints, based on causal-inference and meta-analysis, dominate the scene. Earlier work showed that the metrics of surrogacy introduced in both paradigms are related, although in a complex way that is difficult to study analytically. In the present work, this relationship is further examined using simulations and the analysis of a case study. The results indicate that the extent to which both paradigms lead to similar conclusions regarding the validity of the surrogate, depends on a complex interplay between multiple factors like the ratio of the between and within trial variability and the unidentifiable correlations between the potential outcomes. All the analyses were carried out using the newly developed R package Surrogate, which is freely available via CRAN.

Estimating the reliability of repeatedly measured endpoints based on linear mixed-effects models. A tutorial.

There are various settings in which researchers are interested in the assessment of the correlation between repeated measurements that are taken within the same subject (i.e., reliability). For example, the same rating scale may be used to assess the symptom severity of the same patients by multiple physicians, or the same outcome may be measured repeatedly over time in the same patients. Reliability can be estimated in various ways, for example, using the classical Pearson correlation or the intra-class correlation in clustered data. However, contemporary data often have a complex structure that goes well beyond the restrictive assumptions that are needed with the more conventional methods to estimate reliability. In the current paper, we propose a general and flexible modeling approach that allows for the derivation of reliability estimates, standard errors, and confidence intervals - appropriately taking hierarchies and covariates in the data into account. Our methodology is developed for continuous outcomes together with covariates of an arbitrary type. The methodology is illustrated in a case study, and a Web Appendix is provided which details the computations using the R package CorrMixed and the SAS software. Copyright © 2016 John Wiley & Sons, Ltd.

Statistical evaluation of surrogate endpoints with examples from cancer clinical trials.

A surrogate endpoint is intended to replace a clinical endpoint for the evaluation of new treatments when it can be measured more cheaply, more conveniently, more frequently, or earlier than that clinical endpoint. A surrogate endpoint is expected to predict clinical benefit, harm, or lack of these. Besides the biological plausibility of a surrogate, a quantitative assessment of the strength of evidence for surrogacy requires the demonstration of the prognostic value of the surrogate for the clinical outcome, and evidence that treatment effects on the surrogate reliably predict treatment effects on the clinical outcome. We focus on these two conditions, and outline the statistical approaches that have been proposed to assess the extent to which these conditions are fulfilled. When data are available from a single trial, one can assess the "individual level association" between the surrogate and the true endpoint. When data are available from several trials, one can additionally assess the "trial level association" between the treatment effect on the surrogate and the treatment effect on the true endpoint. In the latter case, the "surrogate threshold effect" can be estimated as the minimum effect on the surrogate endpoint that predicts a statistically significant effect on the clinical endpoint. All these concepts are discussed in the context of randomized clinical trials in oncology, and illustrated with two meta-analyses in gastric cancer.

On the relationship between the causal-inference and meta-analytic paradigms for the validation of surrogate endpoints.

The increasing cost of drug development has raised the demand for surrogate endpoints when evaluating new drugs in clinical trials. However, over the years, it has become clear that surrogate endpoints need to be statistically evaluated and deemed valid, before they can be used as substitutes of "true" endpoints in clinical studies. Nowadays, two paradigms, based on causal-inference and meta-analysis, dominate the scene. Nonetheless, although the literature emanating from these paradigms is wide, till now the relationship between them has largely been left unexplored. In the present work, we discuss the conceptual framework underlying both approaches and study the relationship between them using theoretical elements and the analysis of a real case study. Furthermore, we show that the meta-analytic approach can be embedded within a causal-inference framework on the one hand and that it can be heuristically justified why surrogate endpoints successfully evaluated using this approach will often be appealing from a causal-inference perspective as well, on the other. A newly developed and user friendly R package Surrogate is provided to carry out the evaluation exercise.

Association between white matter microstructure, executive functions, and processing speed in older adults: the impact of vascular health.

Cerebral white matter damage is not only a commonly reported consequence of healthy aging, but is also associated with cognitive decline and dementia. The aetiology of this damage is unclear; however, individuals with hypertension have a greater burden of white matter signal abnormalities (WMSA) on MR imaging than those without hypertension. It is therefore possible that elevated blood pressure (BP) impacts white matter tissue structure which in turn has a negative impact on cognition. However, little information exists about whether vascular health indexed by BP mediates the relationship between cognition and white matter tissue structure. We used diffusion tensor imaging to examine the impact of vascular health on regional associations between white matter integrity and cognition in healthy older adults spanning the normotensive to moderate-severe hypertensive BP range (43-87 years; N = 128). We examined how white matter structure was associated with performance on tests of two cognitive domains, executive functioning (EF) and processing speed (PS), and how patterns of regional associations were modified by BP and WMSA. Multiple linear regression and structural equation models demonstrated associations between tissue structure, EF and PS in frontal, temporal, parietal, and occipital white matter regions. Radial diffusivity was more prominently associated with performance than axial diffusivity. BP only minimally influenced the relationship between white matter integrity, EF and PS. However, WMSA volume had a major impact on neurocognitive associations. This suggests that, although BP and WMSA are causally related, these differential metrics of vascular health may act via independent pathways to influence brain structure, EF and PS.

Establishing normative data for repeated cognitive assessment: a comparison of different statistical methods.

Serial cognitive assessment is conducted to monitor changes in the cognitive abilities of patients over time. At present, mainly the regression-based change and the ANCOVA approaches are used to establish normative data for serial cognitive assessment. These methods are straightforward, but they have some severe drawbacks. For example, they can only consider the data of two measurement occasions. In this article, we propose three alternative normative methods that are not hampered by these problems-that is, multivariate regression, the standard linear mixed model (LMM), and the linear mixed model combined with multiple imputation (LMM with MI) approaches. The multivariate regression method is primarily useful when a small number of repeated measurements are taken at fixed time points. When the data are more unbalanced, the standard LMM and the LMM with MI methods are more appropriate because they allow for a more adequate modeling of the covariance structure. The standard LMM has the advantage that it is easier to conduct and that it does not require a Monte Carlo component. The LMM with MI, on the other hand, has the advantage that it can flexibly deal with missing responses and missing covariate values at the same time. The different normative methods are illustrated on the basis of the data of a large longitudinal study in which a cognitive test (the Stroop Color Word Test) was administered at four measurement occasions (i.e., at baseline and 3, 6, and 12 years later). The results are discussed and suggestions for future research are provided.