Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model.

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.

Inverse probability of censoring weighting for visual predictive checks of time-to-event models with time-varying covariates.

When constructing models to summarize clinical data to be used for simulations, it is good practice to evaluate the models for their capacity to reproduce the data. This can be done by means of Visual Predictive Checks (VPC), which consist of several reproductions of the original study by simulation from the model under evaluation, calculating estimates of interest for each simulated study and comparing the distribution of those estimates with the estimate from the original study. This procedure is a generic method that is straightforward to apply, in general. Here we consider the application of the method to time-to-event data and consider the special case when a time-varying covariate is not known or cannot be approximated after event time. In this case, simulations cannot be conducted beyond the end of the follow-up time (event or censoring time) in the original study. Thus, the simulations must be censored at the end of the follow-up time. Since this censoring is not random, the standard KM estimates from the simulated studies and the resulting VPC will be biased. We propose to use inverse probability of censoring weighting (IPoC) method to correct the KM estimator for the simulated studies and obtain unbiased VPCs. For analyzing the Cantos study, the IPoC weighting as described here proved valuable and enabled the generation of VPCs to qualify PKPD models for simulations. Here, we use a generated data set, which allows illustration of the different situations and evaluation against the known truth.

A practical perspective on the concordance index for the evaluation and selection of prognostic time-to-event models.

Developing a prognostic model for biomedical applications typically requires mapping an individual's set of covariates to a measure of the risk that he or she may experience the event to be predicted. Many scenarios, however, especially those involving adverse pathological outcomes, are better described by explicitly accounting for the timing of these events, as well as their probability. As a result, in these cases, traditional classification or ranking metrics may be inadequate to inform model evaluation or selection. To address this limitation, it is common practice to reframe the problem in the context of survival analysis, and resort, instead, to the concordance index (C-index), which summarises how well a predicted risk score describes an observed sequence of events. A practically meaningful interpretation of the C-index, however, may present several difficulties and pitfalls. Specifically, we identify two main issues: i) the C-index remains implicitly, and subtly, dependent on time, and ii) its relationship with the number of subjects whose risk was incorrectly predicted is not straightforward. Failure to consider these two aspects may introduce undesirable and unwanted biases in the evaluation process, and even result in the selection of a suboptimal model. Hence, here, we discuss ways to obtain a meaningful interpretation in spite of these difficulties. Aiming to assist experimenters regardless of their familiarity with the C-index, we start from an introductory-level presentation of its most popular estimator, highlighting the latter's temporal dependency, and suggesting how it might be correctly used to inform model selection. We also address the nonlinearity of the C-index with respect to the number of correct risk predictions, elaborating a simplified framework that may enable an easier interpretation and quantification of C-index improvements or deteriorations.

Multi-dimensional predictors of first drinking initiation and regular drinking onset in adolescence: A prospective longitudinal study.

Early adolescent drinking onset is linked to myriad negative consequences. Using the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA) baseline to year 8 data, this study (1) leveraged best subsets selection and Cox Proportional Hazards regressions to identify the most robust predictors of adolescent first and regular drinking onset, and (2) examined the clinical utility of drinking onset in forecasting later binge drinking and withdrawal effects. Baseline predictors included youth psychodevelopmental characteristics, cognition, brain structure, family, peer, and neighborhood domains. Participants (N=538) were alcohol-naïve at baseline. The strongest predictors of first and regular drinking onset were positive alcohol expectancies (Hazard Ratios [HRs]=1.67-1.87), easy home alcohol access (HRs=1.62-1.67), more parental solicitation (e.g., inquiring about activities; HRs=1.72-1.76), and less parental control and knowledge (HRs=.72-.73). Robust linear regressions showed earlier first and regular drinking onset predicted earlier transition into binge and regular binge drinking (ßs=0.57-0.95). Zero-inflated Poisson regressions revealed that delayed first and regular drinking increased the likelihood (Incidence Rate Ratios [IRR]=1.62 and IRR=1.29, respectively) of never experiencing withdrawal. Findings identified behavioral and environmental factors predicting temporal paths to youthful drinking, dissociated first from regular drinking initiation, and revealed adverse sequelae of younger drinking initiation, supporting efforts to delay drinking onset.

Developing and Validating Models to Predict Progression to Proliferative Diabetic Retinopathy.

Purpose: To develop models for progression of nonproliferative diabetic retinopathy (NPDR) to proliferative diabetic retinopathy (PDR) and determine if incorporating updated information improves model performance. Design: Retrospective cohort study. Participants: Electronic health record (EHR) data from a tertiary academic center, University of California San Francisco (UCSF), and a safety-net hospital, Zuckerberg San Francisco General (ZSFG) Hospital were used to identify patients with a diagnosis of NPDR, age ≥ 18 years, a diagnosis of type 1 or 2 diabetes mellitus, ≥ 6 months of ophthalmology follow-up, and no prior diagnosis of PDR before the index date (date of first NPDR diagnosis in the EHR). Methods: Four survival models were developed: Cox proportional hazards, Cox with backward selection, Cox with LASSO regression and Random Survival Forest. For each model, three variable sets were compared to determine the impact of including updated clinical information: Static0 (data up to the index date), Static6m (data updated 6 months after the index date), and Dynamic (data in Static0 plus data change during the 6-month period). The UCSF data were split into 80% training and 20% testing (internal validation). The ZSFG data were used for external validation. Model performance was evaluated by the Harrell's concordance index (C-Index). Main Outcome Measures: Time to PDR. Results: The UCSF cohort included 1130 patients and 92 (8.1%) patients progressed to PDR. The ZSFG cohort included 687 patients and 30 (4.4%) patients progressed to PDR. All models performed similarly (C-indices ∼ 0.70) in internal validation. The random survival forest with Static6m set performed best in external validation (C-index 0.76). Insurance and age were selected or ranked as highly important by all models. Other key predictors were NPDR severity, diabetic neuropathy, number of strokes, mean Hemoglobin A1c, and number of hospital admissions. Conclusions: Our models for progression of NPDR to PDR achieved acceptable predictive performance and validated well in an external setting. Updating the baseline variables with new clinical information did not consistently improve the predictive performance. Financial Disclosures: Proprietary or commercial disclosure may be found after the references.

Standardizing the CAP Score in Huntington's Disease by Predicting Age-at-Onset.

BACKGROUND: Huntington's disease (HD) is an autosomal dominant, neurological disease caused by an expanded CAG repeat near the N-terminus of the huntingtin (HTT) gene. A leading theory concerning the etiology of HD is that both onset and progression are driven by cumulative exposure to the effects of mutant (or CAG expanded) huntingtin (mHTT). The CAG-Age-Product (CAP) score (i.e., the product of excess CAG length and age) is a commonly used measure of this cumulative exposure. CAP score has been widely used as a predictor of a variety of disease state variables in HD. The utility of the CAP score has been somewhat diminished, however, by a lack of agreement on its precise definition. The most commonly used forms of the CAP score are highly correlated so that, for purposes of prediction, it makes little difference which is used. However, reported values of CAP scores, based on commonly used definitions, differ substantially in magnitude when applied to the same data. This complicates the process of inter-study comparison. OBJECTIVE: In this paper, we propose a standardized definition for the CAP score which will resolve this difficulty. Our standardization is chosen so that CAP = 100 at the expected age of diagnosis. METHODS: Statistical methods include novel survival analysis methodology applied to the 13 disease landmarks taken from the Enroll-HD database (PDS 5) and comparisons with the existing, gold standard, onset model. RESULTS: Useful by-products of our work include up-to-date, age-at-onset (AO) results and a refined AO model suitable for use in other contexts, a discussion of several useful properties of the CAP score that have not previously been noted in the literature and the introduction of the concept of a toxicity onset model. CONCLUSION: We suggest that taking L = 30 and K = 6.49 provides a useful standardization of the CAP score, suitable for use in the routine modeling of clinical data in HD.

Generalised weibull model-based approaches to detect non-constant hazard to signal adverse drug reactions in longitudinal data.

Pharmacovigilance is the process of monitoring the emergence of harm from a medicine once it has been licensed and is in use. The aim is to identify new adverse drug reactions (ADRs) or changes in frequency of known ADRs. The last decade has seen increased interest for the use of electronic health records (EHRs) in pharmacovigilance. The causal mechanism of an ADR will often result in the occurrence being time dependent. We propose identifying signals for ADRs based on detecting a variation in hazard of an event using a time-to-event approach. Cornelius et al. proposed a method based on the Weibull Shape Parameter (WSP) and demonstrated this to have optimal performance for ADRs occurring shortly after taking treatment or delayed ADRs, and introduced censoring at varying time points to increase performance for intermediate ADRs. We now propose two new approaches which combined perform equally well across all time periods. The performance of this new approach is illustrated through an EHR Bisphosphonates dataset and a simulation study. One new approach is based on the power generalised Weibull distribution (pWSP) introduced by Bagdonavicius and Nikulin alongside an extended version of the WSP test, which includes one censored dataset resulting in improved detection across time period (dWSP). In the Bisphosphonates example, the pWSP and dWSP tests correctly signalled two known ADRs, and signal one adverse event for which no evidence of association with the drug exist. A combined test involving both pWSP and dWSP is reliable independently of the time of occurrence of ADRs.

The association between the pattern of change in N-terminal pro-B-type natriuretic peptide and short-term outcomes in children undergoing surgery for congenital heart disease.

OBJECTIVES: This study aimed to determine whether changes in perioperative N-terminal pro-B-type natriuretic peptide (NT-proBNP) are associated with short-term outcomes in children undergoing surgery for congenital heart disease (CHD). METHODS: We retrospectively included 873 consecutive children with CHD after cardiac surgery. NT-proBNP concentrations were collected from each child prior to and at 1, 12, 36 and 72 h after surgery. The patients had postsurgical follow-ups at 30, 90 and 180 days. The end point was postoperative composite adverse events. RESULTS: The patients were classified into 3 groups using joint latent class mixture time-to-event models: (i) relatively stable (86.7%), (ii) decreasing (7.2%) and (iii) increasing (6.1%). In total, 257 (29.4%) adverse events occurred. The joint latent class mixture time-to-event models showed that increasing NT-proBNP was strongly associated with adverse events, with adjusted hazard ratio of 2.33 (95% confidence interval 1.52-3.60). Multinomial logistic regression showed that the variables associated with the pattern of change were age, weight at surgery, mode of delivery and cardiopulmonary bypass time. CONCLUSIONS: The pattern of dynamic postsurgical changes in NT-proBNP may facilitate outcome stratification and identification of a high risk for adverse events.

Case-control study investigating parameters affecting ventilator-associated events in mechanically ventilated patients.

We analyzed a set of clinical parameters using Cox proportional hazard regressions to yield significant factors associated with the development of ventilator-associated events. In our study, intubation site, certain comorbidities, morphine, prednisone, and nutrition emerged as factors. Additionally, we presented potential mechanisms that require further research to validate.

Kernel-Based Visual Hazard Comparison (kbVHC): a Simulation-Free Diagnostic for Parametric Repeated Time-to-Event Models.

Repeated time-to-event (RTTE) models are the preferred method to characterize the repeated occurrence of clinical events. Commonly used diagnostics for parametric RTTE models require representative simulations, which may be difficult to generate in situations with dose titration or informative dropout. Here, we present a novel simulation-free diagnostic tool for parametric RTTE models; the kernel-based visual hazard comparison (kbVHC). The kbVHC aims to evaluate whether the mean predicted hazard rate of a parametric RTTE model is an adequate approximation of the true hazard rate. Because the true hazard rate cannot be directly observed, the predicted hazard is compared to a non-parametric kernel estimator of the hazard rate. With the degree of smoothing of the kernel estimator being determined by its bandwidth, the local kernel bandwidth is set to the lowest value that results in a bootstrap coefficient of variation (CV) of the hazard rate that is equal to or lower than a user-defined target value (CVtarget). The kbVHC was evaluated in simulated scenarios with different number of subjects, hazard rates, CVtarget values, and hazard models (Weibull, Gompertz, and circadian-varying hazard). The kbVHC was able to distinguish between Weibull and Gompertz hazard models, even when the hazard rate was relatively low (< 2 events per subject). Additionally, it was more sensitive than the Kaplan-Meier VPC to detect circadian variation of the hazard rate. An additional useful feature of the kernel estimator is that it can be generated prior to model development to explore the shape of the hazard rate function.

Habitat-mediated timing of migration in polar bears: an individual perspective.

Migration phenology is largely determined by how animals respond to seasonal changes in environmental conditions. Our perception of the relationship between migratory behavior and environmental cues can vary depending on the spatial scale at which these interactions are measured. Understanding the behavioral mechanisms behind population-scale movements requires knowledge of how individuals respond to local cues. We show how time-to-event models can be used to predict what factors are associated with the timing of an individual's migratory behavior using data from GPS collared polar bears (Ursus maritimus) that move seasonally between sea ice and terrestrial habitats. We found the concentration of sea ice that bears experience at a local level, along with the duration of exposure to these conditions, was most associated with individual migration timing. Our results corroborate studies that assume thresholds of >50% sea ice concentration are necessary for suitable polar bear habitat; however, continued periods (e.g., days to weeks) of exposure to suboptimal ice concentrations during seasonal melting were required before the proportion of bears migrating to land increased substantially. Time-to-event models are advantageous for examining individual movement patterns because they account for the idea that animals make decisions based on an accumulation of knowledge from the landscapes they move through and not simply the environment they are exposed to at the time of a decision. Understanding the migration behavior of polar bears moving between terrestrial and marine habitat, at multiple spatiotemporal scales, will be a major aspect of quantifying observed and potential demographic responses to climate-induced environmental changes.