Evaluating dynamic and predictive discrimination for recurrent event models: use of a time-dependent C-index.

Interest in analyzing recurrent event data has increased over the past few decades. One essential aspect of a risk prediction model for recurrent event data is to accurately distinguish individuals with different risks of developing a recurrent event. Although the concordance index (C-index) effectively evaluates the overall discriminative ability of a regression model for recurrent event data, a local measure is also desirable to capture dynamic performance of the regression model over time. Therefore, in this study, we propose a time-dependent C-index measure for inferring the model's discriminative ability locally. We formulated the C-index as a function of time using a flexible parametric model and constructed a concordance-based likelihood for estimation and inference. We adapted a perturbation-resampling procedure for variance estimation. Extensive simulations were conducted to investigate the proposed time-dependent C-index's finite-sample performance and estimation procedure. We applied the time-dependent C-index to three regression models of a study of re-hospitalization in patients with colorectal cancer to evaluate the models' discriminative capability.

Any infection among patients with hip fracture: Predictive ability of Charlson, Elixhauser, Rx-Risk, and Nordic comorbidity indices.

BACKGROUND: In studies on infection after hip fracture surgery, a common and serious complication, it remains unknown which comorbidity index is best for case-mix confounder adjustment. We evaluated the predictive ability of Charlson Comorbidity Index (CCI), Elixhauser Comorbidity Index (ECI), Rx-Risk Index (Rx-Risk), and Nordic Multimorbidity Index (NMI) for any infection up to 1 year from discharge after hip fracture surgery. METHODS: Using Danish medical registries, we included 92,600 patients (mean age 83 years) surgically treated for hip fracture between 2004 and 2018. Comorbidity-index scores were calculated using prevalence of diagnosis codes, prescription codes, or both. Lookback periods of 1, 5, and 10 years were applied. Logistic regression was used to calculate c-index to assess discrimination of comorbidity indices individually and in combination with a base model of age and sex. Outcome was any infection (not only surgical site infection) in-hospital and 1 year after discharge. RESULTS: At 10-year lookback period, the c-index for individual comorbidity indices for in-hospital infections varied from 0.53 to 0.56, similar to base model alone (0.56). The predictive ability of comorbidity indices in combination with base model varied from 0.56 to 0.57. Within 1 year after discharge, NMI in combination with base model had best predictive ability for infection (c-index = 0.62), followed by CCI and ECI (c-index = 0.60) and Rx-Risk (c-index = 0.58). Discrimination was similar for all lookback periods. CONCLUSIONS: Comorbidity indices have low predictive ability for any infection up to 1 year after hip fracture surgery, similar to that of age and sex alone. For case-mix adjustment, evaluated comorbidity indices are of equal value.

Predicting the Onset of Hepatitis B Virus-Related Acute-on-Chronic Liver Failure.

BACKGROUND & AIMS: Acute-on-chronic liver failure (ACLF) is a life-threatening syndrome with rapid progression. This study aimed to develop and validate a prognostic score to predict the onset of ACLF in hepatitis B virus (HBV) etiology. METHODS: The prospective clinical data of 1373 patients with acute deterioration of HBV-related chronic liver disease were used to identify clinical characteristics and develop a prognostic score for the onset of ACLF. RESULTS: Of the patients assessed using the Chinese Group on the Study of Severe Hepatitis B (COSSH)-ACLF criteria, 903 patients with non-ACLF at admission (1 received transplantation at 5 days) were stratified: 71 with progression to ACLF and 831 without progression to ACLF at 7 days. Four predictors (total bilirubin, international normalized ratio, alanine aminotransferase, and ferritin) were associated significantly with ACLF onset at 7 days. The COSSH-onset-ACLF score was constituted as follows: (0.101 × ln [alanine aminotransferase] + 0.819 × ln [total bilirubin] + 2.820 × ln [international normalized ratio] + 0.016 × ln [ferritin]). The C-indexes of the new score for 7-/14-/28-day onset (0.928/0.925/0.913) were significantly higher than those of 5 other scores (Chronic Liver Failure Consortium ACLF development score/Model for End-stage Liver Disease score/Model for End-stage Liver Disease sodium score/COSSH-ACLF score/Chronic liver failure Consortium ACLF score; all P < .001). The improvement in predictive errors, time-dependent receiver operating characteristic, probability density function evaluation, and calibration curves of the new score showed the highest predictive value for ACLF onset at 7/14/28 days. Risk stratification of the new score showed 2 strata with high and low risk (≥6.3/<6.3) of ACLF onset. The external validation group further confirmed the earlier results. CONCLUSIONS: A new prognostic score based on 4 predictors can accurately predict the 7-/14-/28-day onset of ACLF in patients with acute deterioration of HBV-related chronic liver disease and might be used to guide clinical management.

Concordance indices with left-truncated and right-censored data.

In the context of time-to-event analysis, a primary objective is to model the risk of experiencing a particular event in relation to a set of observed predictors. The Concordance Index (C-Index) is a statistic frequently used in practice to assess how well such models discriminate between various risk levels in a population. However, the properties of conventional C-Index estimators when applied to left-truncated time-to-event data have not been well studied, despite the fact that left-truncation is commonly encountered in observational studies. We show that the limiting values of the conventional C-Index estimators depend on the underlying distribution of truncation times, which is similar to the situation with right-censoring as discussed in Uno et al. (2011) [On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine 30(10), 1105-1117]. We develop a new C-Index estimator based on inverse probability weighting (IPW) that corrects for this limitation, and we generalize this estimator to settings with left-truncated and right-censored data. The proposed IPW estimators are highly robust to the underlying truncation distribution and often outperform the conventional methods in terms of bias, mean squared error, and coverage probability. We apply these estimators to evaluate a predictive survival model for mortality among patients with end-stage renal disease.

Conicity-index predicts all-cause mortality in Chinese older people: a 10-year community follow-up.

BACKGROUND: Abdominal obesity (AO) has been regarded as the most dangerous type of obesity. The Conicity-index (C-index) had a high ability to discriminate underlying AO. The purpose of this study was to determine the ability of C-index to predict all-cause mortality among non-cancer Chinese older people. METHODS: The participants were residents of the Wanshou Road community in Beijing, China. Receiver operating curve (ROC) curves were used to determine the sensitivity and specificity of the best cut-off values for different anthropometric measures for predicting all-cause mortality. The area under the curve (AUC) of the ROC curves were calculated to compare the relative ability of various anthropometric measures to correctly identify older people in the community where all-cause mortality occurs. Included subjects were grouped according to C-index tertiles. The association between C-index and all-cause mortality was verified using Kaplan-Meier survival analysis and different Cox regression models. RESULTS: During a mean follow-up period of 9.87 years, 1821 subjects completed follow-up. The average age was 71.21 years, of which 59.4% were female. The ROC curve results showed that the AUC of the C-index in predicting all-cause mortality was 0.633. Kaplan-Meier survival curves showed a clear dose-response relationship between C-index and all-cause mortality. With the increase of C-index, the survival rate of the study population showed a significant downward trend (P < 0.05). Adjusted for age, gender, hip circumference, systolic blood pressure, diastolic blood pressure, fasting blood glucose (FBG), 2-h postprandial blood glucose (2hPG), glycosylated hemoglobin, high-density lipids protein (LDL), triglyceride, serum creatinine, serum uric acid, urine albumin-creatinine ratio (UACR), Mini-Mental State Examination (MMSE), smoking history, and drinking history, COX regression analysis showed that in the model adjusted for all covariates, the risk of all-cause mortality in tertile 3 was 1.505 times that in tertile 1, and the difference was statistically significant. CONCLUSIONS: The C-index is an independent risk factor for all-cause mortality in the non-cancer Chinese older people.

A prognostic system for epithelial ovarian carcinomas using machine learning.

INTRODUCTION: Integrating additional factors into the International Federation of Gynecology and Obstetrics (FIGO) staging system is needed for accurate patient classification and survival prediction. In this study, we tested machine learning as a novel tool for incorporating additional prognostic parameters into the conventional FIGO staging system for stratifying patients with epithelial ovarian carcinomas and evaluating their survival. MATERIAL AND METHODS: Cancer-specific survival data for epithelial ovarian carcinomas were extracted from the Surveillance, Epidemiology, and End Results (SEER) program. Two datasets were constructed based upon the year of diagnosis. Dataset 1 (39 514 cases) was limited to primary tumor (T), regional lymph nodes (N) and distant metastasis (M). Dataset 2 (25 291 cases) included additional parameters of age at diagnosis (A) and histologic type and grade (H). The Ensemble Algorithm for Clustering Cancer Data (EACCD) was applied to generate prognostic groups with depiction in dendrograms. C-indices provided dendrogram cutoffs and comparisons of prediction accuracy. RESULTS: Dataset 1 was stratified into nine epithelial ovarian carcinoma prognostic groups, contrasting with 10 groups from FIGO methodology. The EACCD grouping had a slightly higher accuracy in survival prediction than FIGO staging (C-index = 0.7391 vs 0.7371, increase in C-index = 0.0020, 95% confidence interval [CI] 0.0012-0.0027, p = 1.8 × 10-7 ). Nevertheless, there remained a strong inter-system association between EACCD and FIGO (rank correlation = 0.9480, p = 6.1 × 10-15 ). Analysis of Dataset 2 demonstrated that A and H could be smoothly integrated with the T, N and M criteria. Survival data were stratified into nine prognostic groups with an even higher prediction accuracy (C-index = 0.7605) than when using only T, N and M. CONCLUSIONS: EACCD was successfully applied to integrate A and H with T, N and M for stratification and survival prediction of epithelial ovarian carcinoma patients. Additional factors could be advantageously incorporated to test the prognostic impact of emerging diagnostic or therapeutic advances.

Development and validation of a Super learner-based model for predicting survival in Chinese Han patients with resected colorectal cancer.

OBJECTIVE: Improved prognostic prediction for patients with colorectal cancer stays an important challenge. This study aimed to develop an effective prognostic model for predicting survival in resected colorectal cancer patients through the implementation of the Super learner. METHODS: A total of 2333 patients who met the inclusion criteria were enrolled in the cohort. We used multivariate Cox regression analysis to identify significant prognostic factors and Super learner to construct prognostic models. Prediction models were internally validated by 10-fold cross-validation and externally validated with a dataset from The Cancer Genome Atlas. Discrimination and calibration were evaluated by Harrell concordence index (C-index) and calibration plots, respectively. RESULTS: Age, T stage, N stage, histological type, tumor location, lymph-vascular invasion, preoperative carcinoembryonic antigen and sample lymph nodes were integrated into prediction models. The concordance index of Super learner-based prediction model (SLM) was 0.792 (95% confidence interval: 0.767-0.818), which is higher than that of the seventh edition American Joint Committee on Cancer TNM staging system 0.689 (95% confidence interval: 0.672-0.703) for predicting overall survival (P < 0.05). In the external validation, the concordance index of the SLM for predicting overall survival was also higher than that of tumor-node-metastasis (TNM) stage system (0.764 vs. 0.682, respectively; P < 0.001). In addition, the SLM showed good calibration properties. CONCLUSIONS: We developed and externally validated an effective prognosis prediction model based on Super learner, which offered more reliable and accurate prognosis prediction and may be used to more accurately identify high-risk patients who need more active surveillance in patients with resected colorectal cancer.

Pretreatment Hemoglobin as an Independent Prognostic Factor in Primary Central Nervous System Lymphomas.

BACKGROUND: Primary central nervous system lymphoma (PCNSL) is a rare subtype of extranodal lymphoma. Despite established clinical prognostic scoring such as that of the Memorial Sloan Kettering Cancer Center (MSKCC) and the International Extranodal Lymphoma Study Group, outcome prediction needs to be improved. Several studies have indicated an association between changes in hematologic laboratory parameters with patient outcomes in PCNSL. We sought to assess the association between hematological parameters and overall survival (OS) in patients with PCNSL. METHODS: Pretreatment blood tests were analyzed in patients with newly diagnosed PCNSL (n = 182), and we divided the analysis into two cohorts (A and B, both n = 91). OS was evaluated using the Cox proportional hazards models and log-rank test. Furthermore, the accuracy of the different multivariate models was assessed by Harrell's concordance index (C-index). RESULTS: Using prechemotherapy blood tests, anemia was found in 38 patients (41.8%) in cohort A and 34 patients (37.4%) in cohort B. In univariate analysis, anemia (<12 g/dL in women and <13 g/dL in men) was significantly associated with OS. None of the other blood tests parameters (neutrophils, lymphocyte, or platelets counts) or their ratios (neutrophil-to-lymphocyte ratio and neutrophil-to-platelets ratio) were associated with OS. In multivariate analysis, after adjusting by MSKCC score, anemia remained an independent prognostic factor. Interestingly, the prediction accuracy of OS using Harrell's C-index was similar using anemia or MSKCC (mean C-index, 0.6) and was increased to 0.67 when combining anemia and MSKCC. CONCLUSION: The presence of anemia was associated with poor prognosis in both cohorts of PCNSL. Validation of these results and biologic role of hemoglobin levels in PCNSL requires further investigation. IMPLICATIONS FOR PRACTICE: The prediction of the outcome of primary central nervous system lymphoma (PCNSL) using the most frequently used scores (i.e., Memorial Sloan Kettering Cancer Center [MSKCC] or International Extranodal Lymphoma Study Group) needs to be improved. We analyzed a large cohort of PCNSL to dissect the potential prognostic value of blood tests in this rare entity. We found anemia as an independent predictor for overall survival in PCNSL. Interestingly, the accuracy to predict PCNSL outcome was improved using hemoglobin level. This improvement was additional to the currently used clinical score (i.e., MSKCC). Finally, none of the other blood tests parameters or their ratios had a prognostic impact in this study.

Concordance measure and discriminatory accuracy in transformation cure models.

Many populations of early-stage cancer patients have non-negligible latent cure fractions that can be modeled using transformation cure models. However, there is a lack of statistical metrics to evaluate prognostic utility of biomarkers in this context due to the challenges associated with unknown cure status and heavy censorship. In this article, we develop general concordance measures as evaluation metrics for the discriminatory accuracy of transformation cure models including the so-called promotion time cure models and mixture cure models. We introduce explicit formulas for the consistent estimates of the concordance measures, and show that their asymptotically normal distributions do not depend on the unknown censoring distribution. The estimates work for both parametric and semiparametric transformation models as well as transformation cure models. Numerical feasibility of the estimates and their robustness to the censoring distributions are illustrated via simulation studies and demonstrated using a melanoma data set.

A C-index for recurrent event data: Application to hospitalizations among dialysis patients.

We propose a C-index (index of concordance) applicable to recurrent event data. The present work addresses the dearth of measures for quantifying a regression model's ability to discriminate with respect to recurrent event risk. The data which motivated the methods arise from the Dialysis Outcomes and Practice Patterns Study (DOPPS), a long-running prospective international study of end-stage renal disease patients on hemodialysis. We derive the theoretical properties of the measure under the proportional rates model (Lin et al., 2000), and propose computationally convenient inference procedures based on perturbed influence functions. The methods are shown through simulations to perform well in moderate samples. Analysis of hospitalizations among a cohort of DOPPS patients reveals substantial improvement in discrimination upon adding country indicators to a model already containing basic clinical and demographic covariates, and further improvement upon adding a relatively large set of comorbidity indicators.

Validation of the American Joint Committee on Cancer eighth edition staging system in patients undergoing hepatectomy for hepatocellular carcinoma: a US population-based study.

BACKGROUND: The American Joint Committee on Cancer (AJCC) eighth edition staging system for hepatocellular carcinoma (HCC) has incorporated several significant changes. This study aims to evaluate the newly proposed staging system and assess its strengths and weaknesses. MATERIALS AND METHODS: Using the Surveillance, Epidemiology, and End Results database, we identified patients with pathologically confirmed stage I-III HCC diagnosed between 2004 and 2014. RESULTS: After all exclusion criteria were applied, AJCC seventh and eighth edition staging was possible in 4931 patients. According to the AJCC eighth edition staging system, stages IB and II did not differ significantly in terms of overall survival (OS) and cause-specific survival (CSS) (P = 0.928 and 0.872, respectively). On the basis of the above results, we reclassified T1a, T1b, and T2 into several subgroups. According to the modified AJCC eighth edition staging system, OS and CSS among the five groups of patients differed significantly. For OS predication, the Akaike information criterion values for the AJCC seventh, eighth, and modified eighth edition staging systems were 29,288.24, 29,282.85, and 27,182.21, respectively, and the c-indices for the AJCC seventh, eighth, and modified eighth edition staging systems were 0.5716, 0.5805, and 0.6082, respectively. Regarding CSS, the Akaike information criterion values for the AJCC seventh, eighth, and modified eighth edition staging systems were 21,701.11, 21,682.12, and 20,313.26, respectively, and the c-indices for the AJCC seventh, eighth, and modified eighth edition staging systems were 0.5983, 0.6117, and 0.6436, respectively. CONCLUSIONS: This is the first large-scale validation of the AJCC eighth edition staging system for patients undergoing hepatectomy. Our study revealed that there was a lack of discrepancy in the outcomes for stage IB and II tumors for AJCC eighth edition staging system of HCC. Our modified AJCC eighth edition staging system allows for finer stratification of patients undergoing hepatectomy according to more detailed tumor size and vascular invasion classifications.

The use of repeated blood pressure measures for cardiovascular risk prediction: a comparison of statistical models in the ARIC study.

Many prediction models have been developed for the risk assessment and the prevention of cardiovascular disease in primary care. Recent efforts have focused on improving the accuracy of these prediction models by adding novel biomarkers to a common set of baseline risk predictors. Few have considered incorporating repeated measures of the common risk predictors. Through application to the Atherosclerosis Risk in Communities study and simulations, we compare models that use simple summary measures of the repeat information on systolic blood pressure, such as (i) baseline only; (ii) last observation carried forward; and (iii) cumulative mean, against more complex methods that model the repeat information using (iv) ordinary regression calibration; (v) risk-set regression calibration; and (vi) joint longitudinal and survival models. In comparison with the baseline-only model, we observed modest improvements in discrimination and calibration using the cumulative mean of systolic blood pressure, but little further improvement from any of the complex methods. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

On comparing 2 correlated C indices with censored survival data.

As new biomarkers and risk prediction procedures are in rapid development, it is of great interest to develop valid methods for comparing predictive power of 2 biomarkers or risk score systems. Harrell C statistic has been routinely used as a global adequacy assessment of a risk score system, and the difference of 2 Harrell C statistics as a test statistic has been suggested in recent literature for comparison of predictive power of 2 biomarkers for censored outcome. In this study, we showed that such a test can have severely inflated type I errors as the difference between the 2 Harrell C statistics does not converge to zero under the null hypothesis of equal predictive power measured by concordance probabilities, as illustrated by 2 counterexamples and corresponding numerical simulations. We further investigate a necessary and sufficient condition under which the difference of 2 Harrell C statistics converges to zero under the null hypothesis.

A unified Bayesian semiparametric approach to assess discrimination ability in survival analysis.

The discriminatory ability of a marker for censored survival data is routinely assessed by the time-dependent ROC curve and the c-index. The time-dependent ROC curve evaluates the ability of a biomarker to predict whether a patient lives past a particular time t. The c-index measures the global concordance of the marker and the survival time regardless of the time point. We propose a Bayesian semiparametric approach to estimate these two measures. The proposed estimators are based on the conditional distribution of the survival time given the biomarker and the empirical biomarker distribution. The conditional distribution is estimated by a linear-dependent Dirichlet process mixture model. The resulting ROC curve is smooth as it is estimated by a mixture of parametric functions. The proposed c-index estimator is shown to be more efficient than the commonly used Harrell's c-index since it uses all pairs of data rather than only informative pairs. The proposed estimators are evaluated through simulations and illustrated using a lung cancer dataset.

Estimating the concordance probability in a survival analysis with a discrete number of risk groups.

A clinical risk classification system is an important component of a treatment decision algorithm. A measure used to assess the strength of a risk classification system is discrimination, and when the outcome is survival time, the most commonly applied global measure of discrimination is the concordance probability. The concordance probability represents the pairwise probability of lower patient risk given longer survival time. The c-index and the concordance probability estimate have been used to estimate the concordance probability when patient-specific risk scores are continuous. In the current paper, the concordance probability estimate and an inverse probability censoring weighted c-index are modified to account for discrete risk scores. Simulations are generated to assess the finite sample properties of the concordance probability estimate and the weighted c-index. An application of these measures of discriminatory power to a metastatic prostate cancer risk classification system is examined.

Random survival forests for competing risks.

We introduce a new approach to competing risks using random forests. Our method is fully non-parametric and can be used for selecting event-specific variables and for estimating the cumulative incidence function. We show that the method is highly effective for both prediction and variable selection in high-dimensional problems and in settings such as HIV/AIDS that involve many competing risks.

Concordance for prognostic models with competing risks.

The concordance probability is a widely used measure to assess discrimination of prognostic models with binary and survival endpoints. We formally define the concordance probability for a prognostic model of the absolute risk of an event of interest in the presence of competing risks and relate it to recently proposed time-dependent area under the receiver operating characteristic curve measures. For right-censored data, we investigate inverse probability of censoring weighted (IPCW) estimates of a truncated concordance index based on a working model for the censoring distribution. We demonstrate consistency and asymptotic normality of the IPCW estimate if the working model is correctly specified and derive an explicit formula for the asymptotic variance under independent censoring. The small sample properties of the estimator are assessed in a simulation study also against misspecification of the working model. We further illustrate the methods by computing the concordance probability for a prognostic model of coronary heart disease (CHD) events in the presence of the competing risk of non-CHD death.

Comparing two correlated C indices with right-censored survival outcome: a one-shot nonparametric approach.

The area under the receiver operating characteristic curve is often used as a summary index of the diagnostic ability in evaluating biomarkers when the clinical outcome (truth) is binary. When the clinical outcome is right-censored survival time, the C index, motivated as an extension of area under the receiver operating characteristic curve, has been proposed by Harrell as a measure of concordance between a predictive biomarker and the right-censored survival outcome. In this work, we investigate methods for statistical comparison of two diagnostic or predictive systems, of which they could either be two biomarkers or two fixed algorithms, in terms of their C indices. We adopt a U-statistics-based C estimator that is asymptotically normal and develop a nonparametric analytical approach to estimate the variance of the C estimator and the covariance of two C estimators. A z-score test is then constructed to compare the two C indices. We validate our one-shot nonparametric method via simulation studies in terms of the type I error rate and power. We also compare our one-shot method with resampling methods including the jackknife and the bootstrap. Simulation results show that the proposed one-shot method provides almost unbiased variance estimations and has satisfactory type I error control and power. Finally, we illustrate the use of the proposed method with an example from the Framingham Heart Study.

Key concepts and limitations of statistical methods for evaluating biomarkers of kidney disease.

Interest in developing and using novel markers of kidney injury is increasing. To maintain scientific rigour in these endeavors, a comprehensive understanding of statistical methodology is required to rigorously assess the incremental value of novel biomarkers in existing clinical risk prediction models. Such knowledge is especially relevant, because no single statistical method is sufficient to evaluate a novel biomarker. In this review, we highlight the strengths and limitations of various traditional and novel statistical methods used in the literature for biomarker studies and use biomarkers of AKI as examples to show limitations of some popular statistical methods.

Covariate-adjusted measures of discrimination for survival data.

MOTIVATION: Discrimination statistics describe the ability of a survival model to assign higher risks to individuals who experience earlier events: examples are Harrell's C-index and Royston and Sauerbrei's D, which we call the D-index. Prognostic covariates whose distributions are controlled by the study design (e.g. age and sex) influence discrimination and can make it difficult to compare model discrimination between studies. Although covariate adjustment is a standard procedure for quantifying disease-risk factor associations, there are no covariate adjustment methods for discrimination statistics in censored survival data. OBJECTIVE: To develop extensions of the C-index and D-index that describe the prognostic ability of a model adjusted for one or more covariate(s). METHOD: We define a covariate-adjusted C-index and D-index for censored survival data, propose several estimators, and investigate their performance in simulation studies and in data from a large individual participant data meta-analysis, the Emerging Risk Factors Collaboration. RESULTS: The proposed methods perform well in simulations. In the Emerging Risk Factors Collaboration data, the age-adjusted C-index and D-index were substantially smaller than unadjusted values. The study-specific standard deviation of baseline age was strongly associated with the unadjusted C-index and D-index but not significantly associated with the age-adjusted indices. CONCLUSIONS: The proposed estimators improve meta-analysis comparisons, are easy to implement and give a more meaningful clinical interpretation.