Effective sample size: A measure of individual uncertainty in predictions.

Clinical prediction models are estimated using a sample of limited size from the target population, leading to uncertainty in predictions, even when the model is correctly specified. Generally, not all patient profiles are observed uniformly in model development. As a result, sampling uncertainty varies between individual patients' predictions. We aimed to develop an intuitive measure of individual prediction uncertainty. The variance of a patient's prediction can be equated to the variance of the sample mean outcome in n ∗ $$ {n}_{\ast } $$ hypothetical patients with the same predictor values. This hypothetical sample size n ∗ $$ {n}_{\ast } $$ can be interpreted as the number of similar patients n eff $$ {n}_{\mathrm{eff}} $$ that the prediction is effectively based on, given that the model is correct. For generalized linear models, we derived analytical expressions for the effective sample size. In addition, we illustrated the concept in patients with acute myocardial infarction. In model development, n eff $$ {n}_{\mathrm{eff}} $$ can be used to balance accuracy versus uncertainty of predictions. In a validation sample, the distribution of n eff $$ {n}_{\mathrm{eff}} $$ indicates which patients were more and less represented in the development data, and whether predictions might be too uncertain for some to be practically meaningful. In a clinical setting, the effective sample size may facilitate communication of uncertainty about predictions. We propose the effective sample size as a clinically interpretable measure of uncertainty in individual predictions. Its implications should be explored further for the development, validation and clinical implementation of prediction models.

Landmarking 2.0: Bridging the gap between joint models and landmarking.

The problem of dynamic prediction with time-dependent covariates, given by biomarkers, repeatedly measured over time, has received much attention over the last decades. Two contrasting approaches have become in widespread use. The first is joint modeling, which attempts to jointly model the longitudinal markers and the event time. The second is landmarking, a more pragmatic approach that avoids modeling the marker process. Landmarking has been shown to be less efficient than correctly specified joint models in simulation studies, when data are generated from the joint model. When the mean model is misspecified, however, simulation has shown that joint models may be inferior to landmarking. The objective of this article is to develop methods that improve the predictive accuracy of landmarking, while retaining its relative simplicity and robustness. We start by fitting a working longitudinal model for the biomarker, including a temporal correlation structure. Based on that model, we derive a predictable time-dependent process representing the expected value of the biomarker after the landmark time, and we fit a time-dependent Cox model based on the predictable time-dependent covariate. Dynamic predictions based on this approach for new patients can be obtained by first deriving the expected values of the biomarker, given the measured values before the landmark time point, and then calculating the predicted probabilities based on the time-dependent Cox model. We illustrate the approach in predicting overall survival in liver cirrhosis patients based on prothrombin index.

Analysis of time-to-event for observational studies: Guidance to the use of intensity models.

This paper provides guidance for researchers with some mathematical background on the conduct of time-to-event analysis in observational studies based on intensity (hazard) models. Discussions of basic concepts like time axis, event definition and censoring are given. Hazard models are introduced, with special emphasis on the Cox proportional hazards regression model. We provide check lists that may be useful both when fitting the model and assessing its goodness of fit and when interpreting the results. Special attention is paid to how to avoid problems with immortal time bias by introducing time-dependent covariates. We discuss prediction based on hazard models and difficulties when attempting to draw proper causal conclusions from such models. Finally, we present a series of examples where the methods and check lists are exemplified. Computational details and implementation using the freely available R software are documented in Supplementary Material. The paper was prepared as part of the STRATOS initiative.

Prediction meets causal inference: the role of treatment in clinical prediction models.

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a 'predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.

On the relation between the cause-specific hazard and the subdistribution rate for competing risks data: The Fine-Gray model revisited.

The Fine-Gray proportional subdistribution hazards model has been puzzling many people since its introduction. The main reason for the uneasy feeling is that the approach considers individuals still at risk for an event of cause 1 after they fell victim to the competing risk of cause 2. The subdistribution hazard and the extended risk sets, where subjects who failed of the competing risk remain in the risk set, are generally perceived as unnatural . One could say it is somewhat of a riddle why the Fine-Gray approach yields valid inference. To take away these uneasy feelings, we explore the link between the Fine-Gray and cause-specific approaches in more detail. We introduce the reduction factor as representing the proportion of subjects in the Fine-Gray risk set that has not yet experienced a competing event. In the presence of covariates, the dependence of the reduction factor on a covariate gives information on how the effect of the covariate on the cause-specific hazard and the subdistribution hazard relate. We discuss estimation and modeling of the reduction factor, and show how they can be used in various ways to estimate cumulative incidences, given the covariates. Methods are illustrated on data of the European Society for Blood and Marrow Transplantation.

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration.

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

MAGE-A3 immunotherapeutic as adjuvant therapy for patients with resected, MAGE-A3-positive, stage III melanoma (DERMA): a double-blind, randomised, placebo-controlled, phase 3 trial.

BACKGROUND: Despite newly approved treatments, metastatic melanoma remains a life-threatening condition. We aimed to evaluate the efficacy of the MAGE-A3 immunotherapeutic in patients with stage IIIB or IIIC melanoma in the adjuvant setting. METHODS: DERMA was a phase 3, double-blind, randomised, placebo-controlled trial done in 31 countries and 263 centres. Eligible patients were 18 years or older and had histologically proven, completely resected, stage IIIB or IIIC, MAGE-A3-positive cutaneous melanoma with macroscopic lymph node involvement and an Eastern Cooperative Oncology Group performance score of 0 or 1. Randomisation and treatment allocation at the investigator sites were done centrally via the internet. We randomly assigned patients (2:1) to receive up to 13 intramuscular injections of recombinant MAGE-A3 with AS15 immunostimulant (MAGE-A3 immunotherapeutic; 300 µg MAGE-A3 antigen plus 420 µg CpG 7909 reconstituted in AS01B to a total volume of 0·5 mL), or placebo, over a 27-month period: five doses at 3-weekly intervals, followed by eight doses at 12-weekly intervals. The co-primary outcomes were disease-free survival in the overall population and in patients with a potentially predictive gene signature (GS-positive) identified previously and validated here via an adaptive signature design. The final analyses included all patients who had received at least one dose of study treatment; analyses for efficacy were in the as-randomised population and for safety were in the as-treated population. This trial is registered with ClinicalTrials.gov, number NCT00796445. FINDINGS: Between Dec 1, 2008, and Sept 19, 2011, 3914 patients were screened, 1391 randomly assigned, and 1345 started treatment (n=895 for MAGE-A3 and n=450 for placebo). At final analysis (data cutoff May 23, 2013), median follow-up was 28·0 months [IQR 23·3-35·5] in the MAGE-A3 group and 28·1 months [23·7-36·9] in the placebo group. Median disease-free survival was 11·0 months (95% CI 10·0-11·9) in the MAGE-A3 group and 11·2 months (8·6-14·1) in the placebo group (hazard ratio [HR] 1·01, 0·88-1·17, p=0·86). In the GS-positive population, median disease-free survival was 9·9 months (95% CI 5·7-17·6) in the MAGE-A3 group and 11·6 months (5·6-22·3) in the placebo group (HR 1·11, 0·83-1·49, p=0·48). Within the first 31 days of treatment, adverse events of grade 3 or worse were reported by 126 (14%) of 894 patients in the MAGE-A3 group and 56 (12%) of 450 patients in the placebo group, treatment-related adverse events of grade 3 or worse by 36 (4%) patients given MAGE-A3 vs six (1%) patients given placebo, and at least one serious adverse event by 14% of patients in both groups (129 patients given MAGE-A3 and 64 patients given placebo). The most common adverse events of grade 3 or worse were neoplasms (33 [4%] patients in the MAGE-A3 group vs 17 [4%] patients in the placebo group), general disorders and administration site conditions (25 [3%] for MAGE-A3 vs four [<1%] for placebo) and infections and infestations (17 [2%] for MAGE-A3 vs seven [2%] for placebo). No deaths were related to treatment. INTERPRETATION: An antigen-specific immunotherapeutic alone was not efficacious in this clinical setting. Based on these findings, development of the MAGE-A3 immunotherapeutic for use in melanoma has been stopped. FUNDING: GlaxoSmithKline Biologicals SA.

Testing interaction between treatment and high-dimensional covariates in randomized clinical trials.

In this paper, we considered different methods to test the interaction between treatment and a potentially large number (p) of covariates in randomized clinical trials. The simplest approach was to fit univariate (marginal) models and to combine the univariate statistics or p-values (e.g., minimum p-value). Another possibility was to reduce the dimension of the covariates using the principal components (PCs) and to test the interaction between treatment and PCs. Finally, we considered the Goeman global test applied to the high-dimensional interaction matrix, adjusted for the main (treatment and covariates) effects. These tests can be used for personalized medicine to test if a large set of biomarkers can be useful to identify a subset of patients who may be more responsive to treatment. We evaluated the performance of these methods on simulated data and we applied them on data from two early phases oncology clinical trials.

Dynamic frailty models based on compound birth-death processes.

Frailty models are used in survival analysis to model unobserved heterogeneity. They accommodate such heterogeneity by the inclusion of a random term, the frailty, which is assumed to multiply the hazard of a subject (individual frailty) or the hazards of all subjects in a cluster (shared frailty). Typically, the frailty term is assumed to be constant over time. This is a restrictive assumption and extensions to allow for time-varying or dynamic frailties are of interest. In this paper, we extend the auto-correlated frailty models of Henderson and Shimakura and of Fiocco, Putter and van Houwelingen, developed for longitudinal count data and discrete survival data, to continuous survival data. We present a rigorous construction of the frailty processes in continuous time based on compound birth-death processes. When the frailty processes are used as mixtures in models for survival data, we derive the marginal hazards and survival functions and the marginal bivariate survival functions and cross-ratio function. We derive distributional properties of the processes, conditional on observed data, and show how to obtain the maximum likelihood estimators of the parameters of the model using a (stochastic) expectation-maximization algorithm. The methods are applied to a publicly available data set.

Comparison of stopped Cox regression with direct methods such as pseudo-values and binomial regression.

By far the most popular model to obtain survival predictions for individual patients is the Cox model. The Cox model does not make any assumptions on the underlying hazard, but it relies heavily on the proportional hazards assumption. The most common ways to circumvent this robustness problem are 1) to categorize patients based on their prognostic risk score and to base predictions on Kaplan-Meier curves for the risk categories, or 2) to include interactions with the covariates and suitable functions of time. Robust estimators of the t(0)-year survival probabilities can also be obtained from a "stopped Cox" regression model, in which all observations are administratively censored at t(0). Other recent approaches to solve this robustness problem, originally proposed in the context of competing risks, are pseudo-values and direct binomial regression, based on unbiased estimating equations. In this paper stopped Cox regression is compared with these direct approaches. This is done by means of a simulation study to assess the biases of the different approaches and an analysis of breast cancer data to get some feeling for the performance in practice. The tentative conclusion is that stopped Cox and direct models agree well if the follow-up is not too long. There are larger differences for long-term follow-up data. There stopped Cox might be more efficient, but less robust.

From model building to validation and back: a plea for robustness.

This paper is the written version of the President's invited lecture speaker at the International Society for Clinical Biostatistics conference in Munich in 2013. The paper takes the stand of clinician and patient who are in need of a reliable prognostic model for the planning of treatment and patient care during the follow-up after the initial treatment. The paper discusses (i) the need for grouping of data; (ii) the lack of robustness of the Cox model; (iii) the robust approach to repeated measures; and (iv) the robust handling of time-dependent covariates (biomarkers) in dynamic survival analysis.

The role of empirical Bayes methodology as a leading principle in modern medical statistics.

This paper reviews and discusses the role of Empirical Bayes methodology in medical statistics in the last 50 years. It gives some background on the origin of the empirical Bayes approach and its link with the famous Stein estimator. The paper describes the application in four important areas in medical statistics: disease mapping, health care monitoring, meta-analysis, and multiple testing. It ends with a warning that the application of the outcome of an empirical Bayes analysis to the individual "subjects" is a delicate matter that should be handled with prudence and care.

Validation, calibration and refitting of a familial breast cancer model in sisterships: a case study in the Swedish sisters data.

In Sweden, a unique data set has been compiled with breast cancer incidence in all sisterships with at least two sisters born between 1932 and 2001, and the effect of family history has been analyzed by standard epidemiological methods. Such data are ideal to explore the validity of existing models for familial breast cancer. This paper explores the validity of the Jonker model that adds a hypothetical gene to the well-known BRCA1 and BRCA2 genes. The validity of the model for the Swedish data is checked by using a calibration model for breast cancer incidence given the (retrospective) family history as assessed at the end of the study period. This enables the validity of the overall incidence and the effect of family history to be assessed in the same model. The conclusion is that the existing model does reasonably well for the effect of family history but is seriously wrong for the early incidence rate. Therefore, the model is refitted in the Swedish data. Finally, the calibration of the refitted model is checked when using current family history as used in the epidemiological studies. The refitted Jonker model fits the data well and shows good agreement with the epidemiological findings.

Dealing with heterogeneity between cohorts in genomewide SNP association studies.

In Genomewide association (GWA) studies investigating thousands of SNPs, large sample sizes are needed to obtain a reasonable power after correction for multiple testing. To obtain the necessary sample sizes, data from different populations/cohorts are combined. The problem of pooling evidence across cohorts bears some resemblance with meta-analysis of clinical trials, and in fact classical meta-analytic methodologies from that field are typically used in GWAs. However, in genetics, it can be expected that the cohorts show some amount of heterogeneity in the association measures that are used for significance testing. In this paper, we demonstrate how it is possible to exploit this heterogeneity to improve our ability to detect influential genetic variants. We also discuss how pathway analysis based on summary data can help resolve heterogeneity. The current standard method for testing SNPs across cohorts in GWAs will miss heterogeneous but important genetic variants affecting complex diseases. Our new testing strategy has the potential to detect them while maintaining sensitivity to variants with homogeneous effects.

Surgery versus prolonged conservative treatment for sciatica.

BACKGROUND: Lumbar-disk surgery often is performed in patients who have sciatica that does not resolve within 6 weeks, but the optimal timing of surgery is not known. METHODS: We randomly assigned 283 patients who had had severe sciatica for 6 to 12 weeks to early surgery or to prolonged conservative treatment with surgery if needed. The primary outcomes were the score on the Roland Disability Questionnaire, the score on the visual-analogue scale for leg pain, and the patient's report of perceived recovery during the first year after randomization. Repeated-measures analysis according to the intention-to-treat principle was used to estimate the outcome curves for both groups. RESULTS: Of 141 patients assigned to undergo early surgery, 125 (89%) underwent microdiskectomy after a mean of 2.2 weeks. Of 142 patients designated for conservative treatment, 55 (39%) were treated surgically after a mean of 18.7 weeks. There was no significant overall difference in disability scores during the first year (P=0.13). Relief of leg pain was faster for patients assigned to early surgery (P<0.001). Patients assigned to early surgery also reported a faster rate of perceived recovery (hazard ratio, 1.97; 95% confidence interval, 1.72 to 2.22; P<0.001). In both groups, however, the probability of perceived recovery after 1 year of follow-up was 95%. CONCLUSIONS: The 1-year outcomes were similar for patients assigned to early surgery and those assigned to conservative treatment with eventual surgery if needed, but the rates of pain relief and of perceived recovery were faster for those assigned to early surgery. (Current Controlled Trials number, ISRCTN26872154 [controlled-trials.com].).

Vertical modeling: a pattern mixture approach for competing risks modeling.

We study an alternative approach for estimation in the competing risks framework, called vertical modeling. It is motivated by a decomposition of the joint distribution of time and cause of failure. The two elements of this decomposition are (1) the time of failure and (2) the cause of failure condition on time of failure. Both elements of the model are based on observable quantities, namely the total hazard and the relative cause-specific hazards. The model can be implemented using the standard software. The relative cause-specific hazards are flexibly estimated using multinomial logistic regression and smoothing splines. We show estimates of cumulative incidences from vertical modeling to be more efficient statistically than those obtained from the standard nonparametric model. We illustrate our methods using data of 8966 leukemia patients from the European Group for Blood and Marrow Transplantation.

A simple method for co-segregation analysis to evaluate the pathogenicity of unclassified variants; BRCA1 and BRCA2 as an example.

BACKGROUND: Assessment of the clinical significance of unclassified variants (UVs) identified in BRCA1 and BRCA2 is very important for genetic counselling. The analysis of co-segregation of the variant with the disease in families is a powerful tool for the classification of these variants. Statistical methods have been described in literature but these methods are not always easy to apply in a diagnostic setting. METHODS: We have developed an easy to use method which calculates the likelihood ratio (LR) of an UV being deleterious, with penetrance as a function of age of onset, thereby avoiding the use of liability classes. The application of this algorithm is publicly available http://www.msbi.nl/cosegregation. It can easily be used in a diagnostic setting since it requires only information on gender, genotype, present age and/or age of onset for breast and/or ovarian cancer. RESULTS: We have used the algorithm to calculate the likelihood ratio in favour of causality for 3 UVs in BRCA1 (p.M18T, p.S1655F and p.R1699Q) and 5 in BRCA2 (p.E462G p.Y2660D, p.R2784Q, p.R3052W and p.R3052Q). Likelihood ratios varied from 0.097 (BRCA2, p.E462G) to 230.69 (BRCA2, p.Y2660D). Typing distantly related individuals with extreme phenotypes (i.e. very early onset cancer or old healthy individuals) are most informative and give the strongest likelihood ratios for or against causality. CONCLUSION: Although co-segregation analysis on itself is in most cases insufficient to prove pathogenicity of an UV, this method simplifies the use of co-segregation as one of the key features in a multifactorial approach considerably.

Assessment of global phase uncertainty in case-control studies.

BACKGROUND: In haplotype-based candidate gene studies a problem is that the genotype data are unphased, which results in haplotype ambiguity. The R(h)(2) measure 1 quantifies haplotype predictability from genotype data. It is computed for each individual haplotype, and for a measure of global relative efficiency a minimum R(h)(2) value is suggested. Alternatively, we developed methods directly based on the information content of haplotype frequency estimates to obtain global relative efficiency measures: R(A)(2) and R(D)(2) based on A- and D-optimality, respectively. All three methods are designed for single populations; they can be applied in cases only, controls only or the whole data. Therefore they are not necessarily optimal for haplotype testing in case-control studies. RESULTS: A new global relative efficiency measure R(T)(2) was derived to maximize power of a simple test statistic that compares haplotype frequencies in cases and controls. Application to real data showed that our proposed method R(T)(2) gave a clear and summarizing measure for the case-control study conducted. Additionally this measure might be used for selection of individuals, who have the highest potential for improving power by resolving phase ambiguity. CONCLUSION: Instead of using relative efficiency measure for cases only, controls only or their combined data, we link uncertainty measure to case-control studies directly. Hence, our global efficiency measure might be useful to assess whether data are informative or have enough power for estimation of a specific haplotype risk.