Performance of Cox regression models for composite time-to-event endpoints with component-wise censoring in randomized trials.

BACKGROUND: Composite time-to-event endpoints are beneficial for assessing related outcomes jointly in clinical trials, but components of the endpoint may have different censoring mechanisms. For example, in the PRagmatic EValuation of evENTs And Benefits of Lipid-lowering in oldEr adults (PREVENTABLE) trial, the composite outcome contains one endpoint that is right censored (all-cause mortality) and two endpoints that are interval censored (dementia and persistent disability). Although Cox regression is an established method for time-to-event outcomes, it is unclear how models perform under differing component-wise censoring schemes for large clinical trial data. The goal of this article is to conduct a simulation study to investigate the performance of Cox models under different scenarios for composite endpoints with component-wise censoring. METHODS: We simulated data by varying the strength and direction of the association between treatment and outcome for the two component types, the proportion of events arising from the components of the outcome (right censored and interval censored), and the method for including the interval-censored component in the Cox model (upper value and midpoint of the interval). Under these scenarios, we compared the treatment effect estimate bias, confidence interval coverage, and power. RESULTS: Based on the simulation study, Cox models generally have adequate power to achieve statistical significance for comparing treatments for composite outcomes with component-wise censoring. In our simulation study, we did not observe substantive bias for scenarios under the null hypothesis or when the treatment has a similar relative effect on each component outcome. Performance was similar regardless of if the upper value or midpoint of the interval-censored part of the composite outcome was used. CONCLUSION: Cox regression is a suitable method for analysis of clinical trial data with composite time-to-event endpoints subject to different component-wise censoring mechanisms.

High-Sensitivity C-Reactive Protein Modifies the Cardiovascular Risk of Lipoprotein(a): Multi-Ethnic Study of Atherosclerosis.

BACKGROUND: Little is known about the relationship between lipoprotein (a) [Lp(a)] and high-sensitivity C-reactive protein (hsCRP) and their joint association with atherosclerotic cardiovascular disease (ASCVD). OBJECTIVES: The purpose of this study was to assess whether Lp(a)-associated ASCVD risk is modified by hsCRP in the context of primary prevention. METHODS: The current study included 4,679 participants from the MESA (Multi-Ethnic Study of Atherosclerosis) Apolipoprotein ancillary data set. Cox proportional hazards models and Kaplan-Meier curves were used to assess the association among Lp(a), hsCRP, and time to cardiovascular disease (CVD) events. RESULTS: During a mean follow-up of 13.6 years, 684 CVD events occurred. A significant interaction was observed between Lp(a) and hsCRP (P = 0.04). With hsCRP <2 mg/L, no significant CVD risk was observed at any level of Lp(a) from <50 mg/dL to >100 mg/dL. However, with hsCRP ≥2 mg/L, a significant CVD risk was observed with Lp(a) of 50-99.9 mg/dL (HR: 1.36; 95% CI: 1.02-1.81) and Lp(a) ≥100 mg/dL (HR: 2.09; 95% CI: 1.40-3.13). Isolated elevations of either Lp(a) or hsCRP were not associated with increased CVD risk. In contrast, the combination of elevated Lp(a) (≥50 mg/dL) and hsCRP (≥2 mg/L) was independently associated with significant CVD risk (HR: 1.62; 95% CI: 1.25-2.10) and all-cause mortality (HR: 1.39; 95% CI: 1.12-1.72). CONCLUSIONS: Lp(a)-associated ASCVD risk is observed only with concomitant elevation of hsCRP. Individuals with concomitant presence of elevated Lp(a) and systemic inflammation have greater ASCVD risk and all-cause mortality, and thus may merit closer surveillance and more aggressive ASCVD risk management.

A random forest method with feature selection for developing medical prediction models with clustered and longitudinal data.

BACKGROUND: Machine learning methodologies are gaining popularity for developing medical prediction models for datasets with a large number of predictors, particularly in the setting of clustered and longitudinal data. Binary Mixed Model (BiMM) forest is a promising machine learning algorithm which may be applied to develop prediction models for clustered and longitudinal binary outcomes. Although machine learning methods for clustered and longitudinal methods such as BiMM forest exist, feature selection has not been analyzed via data simulations. Feature selection improves the practicality and ease of use of prediction models for clinicians by reducing the burden of data collection. Thus, feature selection procedures are not only beneficial, but are often necessary for development of medical prediction models. In this study, we aim to assess feature selection within the BiMM forest setting for modeling clustered and longitudinal binary outcomes. METHODS: We conducted a simulation study to compare BiMM forest with feature selection (backward elimination or stepwise selection) to standard generalized linear mixed model feature selection methods (shrinkage and backward elimination). We also evaluated feature selection methods to develop models predicting mobility disability in older adults using the Health, Aging and Body Composition Study dataset as an example utilization of the proposed methodology. RESULTS: BiMM forest with backward elimination generally offered higher computational efficiency, similar or higher predictive performance (accuracy and area under the receiver operating curve), and similar or higher ability to identify correct features compared to linear methods for the different simulated scenarios. For predicting mobility disability in older adults, methods generally performed similarly in terms of accuracy, area under the receiver operating curve, and specificity; however, BiMM forest with backward elimination had the highest sensitivity. CONCLUSIONS: This study is novel because it is the first investigation of feature selection for developing random forest prediction models for clustered and longitudinal binary outcomes. Results from the simulation study reveal that BiMM forest with backward elimination has the highest accuracy (performance and identification of correct features) and lowest computation time compared to other feature selection methods in some scenarios and similar performance in other scenarios. Many informatics datasets have clustered and longitudinal outcomes and results from this study suggest that BiMM forest with backward elimination may be beneficial for developing medical prediction models.

Machine Learning in Aging: An Example of Developing Prediction Models for Serious Fall Injury in Older Adults.

BACKGROUND: Advances in computational algorithms and the availability of large datasets with clinically relevant characteristics provide an opportunity to develop machine learning prediction models to aid in diagnosis, prognosis, and treatment of older adults. Some studies have employed machine learning methods for prediction modeling, but skepticism of these methods remains due to lack of reproducibility and difficulty in understanding the complex algorithms that underlie models. We aim to provide an overview of two common machine learning methods: decision tree and random forest. We focus on these methods because they provide a high degree of interpretability. METHOD: We discuss the underlying algorithms of decision tree and random forest methods and present a tutorial for developing prediction models for serious fall injury using data from the Lifestyle Interventions and Independence for Elders (LIFE) study. RESULTS: Decision tree is a machine learning method that produces a model resembling a flow chart. Random forest consists of a collection of many decision trees whose results are aggregated. In the tutorial example, we discuss evaluation metrics and interpretation for these models. Illustrated using data from the LIFE study, prediction models for serious fall injury were moderate at best (area under the receiver operating curve of 0.54 for decision tree and 0.66 for random forest). CONCLUSIONS: Machine learning methods offer an alternative to traditional approaches for modeling outcomes in aging, but their use should be justified and output should be carefully described. Models should be assessed by clinical experts to ensure compatibility with clinical practice.

BiMM tree: A decision tree method for modeling clustered and longitudinal binary outcomes.

Clustered binary outcomes are frequently encountered in clinical research (e.g. longitudinal studies). Generalized linear mixed models (GLMMs) for clustered endpoints have challenges for some scenarios (e.g. data with multi-way interactions and nonlinear predictors unknown a priori). We develop an alternative, data-driven method called Binary Mixed Model (BiMM) tree, which combines decision tree and GLMM within a unified framework. Simulation studies show that BiMM tree achieves slightly higher or similar accuracy compared to standard methods. The method is applied to a real dataset from the Acute Liver Failure Study Group.

BiMM forest: A random forest method for modeling clustered and longitudinal binary outcomes.

Clustered binary outcomes and datasets with many predictor variables are frequently encountered in clinical research (e.g. longitudinal studies). Generalized linear mixed models (GLMMs) typically employed for clustered endpoints have challenges for some scenarios, particularly for complex datasets which contain many interactions among predictors and nonlinear predictors of outcome. We propose a new method called Binary Mixed Model (BiMM) forest, which combines random forest and GLMM methodology. BiMM forest offers a flexible and stable method which naturally models interactions among predictors and can be employed in the setting of clustered data. Simulation studies show that BiMM forest achieves similar or superior prediction accuracy compared to standard random forest, GLMMs and its tree counterpart (BiMM tree) for clustered binary outcomes. The method is applied to a real dataset from the Acute Liver Failure Study Group. BiMM forest offers an alternative method for modeling clustered binary outcomes which may be applied in myriad research settings.

Predicting daily outcomes in acetaminophen-induced acute liver failure patients with machine learning techniques.

BACKGROUND/OBJECTIVE: Assessing prognosis for acetaminophen-induced acute liver failure (APAP-ALF) patients during the first week of hospitalization often presents significant challenges. Current models such as the King's College Criteria (KCC) and the Acute Liver Failure Study Group (ALFSG) Prognostic Index are developed to predict outcome using only a single time point on hospital admission. Models using longitudinal data are not currently available for APAP-ALF patients. We aim to develop and compare performance of prediction models for outcomes during the first week of hospitalization for APAP-ALF patients. METHODS: Models are developed for the ALFSG registry data to predict longitudinal outcomes for 1042 APAP-ALF patients enrolled 01/1998-02/2016. The primary outcome is defined as daily low versus high coma grade. Accuracy in prediction of outcome (AC), sensitivity (SN), specificity (SP) and area under the receiver operating curve (AUC) are compared between the following models: classification and regression tree, random forest, frequentist generalized linear mixed model (GLMM), Bayesian GLMM, BiMM tree, and BiMM forest using original and imputed datasets. RESULTS: BiMM tree offers predictive (test set) 63% AC, 72% SP and 53% SN for the original dataset, whereas BiMM forest offers predictive (test set) 69% AC, 63% SP and 74% SN for the imputed dataset. BiMM tree has the highest AUC for the original testing dataset (0.697), whereas BiMM forest and standard random forest have the highest AUC for the imputed testing dataset (0.749). The three most important predictors of daily outcome for the BiMM tree are pressor use, bilirubin and creatinine. The BiMM forest model identifies lactate, ammonia and ALT as the three most important predictors of outcome. CONCLUSIONS: BiMM tree offers a prognostic tool for APAP-ALF patients, which has good accuracy and simple interpretation of predictors which are consistent with clinical observations. BiMM tree and forest models are developed using the first week of in-patient data and are appropriate for predicting outcome over time. While the BiMM forest has slightly higher predictive AC, the BiMM tree model is simpler to use at the bedside.

A Comparison of Random Forest Variable Selection Methods for Classification Prediction Modeling.

Random forest classification is a popular machine learning method for developing prediction models in many research settings. Often in prediction modeling, a goal is to reduce the number of variables needed to obtain a prediction in order to reduce the burden of data collection and improve efficiency. Several variable selection methods exist for the setting of random forest classification; however, there is a paucity of literature to guide users as to which method may be preferable for different types of datasets. Using 311 classification datasets freely available online, we evaluate the prediction error rates, number of variables, computation times and area under the receiver operating curve for many random forest variable selection methods. We compare random forest variable selection methods for different types of datasets (datasets with binary outcomes, datasets with many predictors, and datasets with imbalanced outcomes) and for different types of methods (standard random forest versus conditional random forest methods and test based versus performance based methods). Based on our study, the best variable selection methods for most datasets are Jiang's method and the method implemented in the VSURF R package. For datasets with many predictors, the methods implemented in the R packages varSelRF and Boruta are preferable due to computational efficiency. A significant contribution of this study is the ability to assess different variable selection techniques in the setting of random forest classification in order to identify preferable methods based on applications in expert and intelligent systems.

Protective effect of soft contact lenses after Boston keratoprosthesis.

PURPOSE: To evaluate associations between preoperative diagnosis, soft contact lens (SCL) retention and complications. METHODS: A retrospective chart review was conducted of 92 adult patients (103 eyes) who received a Boston keratoprosthesis type I at the Massachusetts's Eye and Ear Infirmary or the Flaum Eye Institute. Records were reviewed for preoperative diagnosis, SCL retention and subsequent complications. Preoperative categories included 16 autoimmune (Stevens-Johnson syndrome, ocular cicatricial pemphigoid, rheumatoid arthritis and uveitis), 9 chemical injury and 67 'other' (aniridia, postoperative infection, dystrophies, keratopathies) patients. RESULTS: 50% of the lenses had been lost the first time after about a year. A subset (n=17) experienced more than 2 SCL losses per year; this group is comprised of 1 patient with autoimmune diseases, 2 patients with chemical injuries and 14 patients with 'other' diseases. The preoperative diagnosis was not predictive of contact lens retention. However, multivariate analysis demonstrated that the absence of a contact lens was an independent risk factor for postoperative complications, such as corneal melts with or without aqueous humour leak/extrusion and infections. CONCLUSIONS: Presence of a contact lens after Boston keratoprosthesis implantation decreases the risk of postoperative complications; this has been clinically experienced by ophthalmologists, but never before has the benefit of contact lens use in this patient population been statistically documented.

Predicting outcome on admission and post-admission for acetaminophen-induced acute liver failure using classification and regression tree models.

BACKGROUND/AIM: Assessing prognosis for acetaminophen-induced acute liver failure (APAP-ALF) patients often presents significant challenges. King's College (KCC) has been validated on hospital admission, but little has been published on later phases of illness. We aimed to improve determinations of prognosis both at the time of and following admission for APAP-ALF using Classification and Regression Tree (CART) models. METHODS: CART models were applied to US ALFSG registry data to predict 21-day death or liver transplant early (on admission) and post-admission (days 3-7) for 803 APAP-ALF patients enrolled 01/1998-09/2013. Accuracy in prediction of outcome (AC), sensitivity (SN), specificity (SP), and area under receiver-operating curve (AUROC) were compared between 3 models: KCC (INR, creatinine, coma grade, pH), CART analysis using only KCC variables (KCC-CART) and a CART model using new variables (NEW-CART). RESULTS: Traditional KCC yielded 69% AC, 90% SP, 27% SN, and 0.58 AUROC on admission, with similar performance post-admission. KCC-CART at admission offered predictive 66% AC, 65% SP, 67% SN, and 0.74 AUROC. Post-admission, KCC-CART had predictive 82% AC, 86% SP, 46% SN and 0.81 AUROC. NEW-CART models using MELD (Model for end stage liver disease), lactate and mechanical ventilation on admission yielded predictive 72% AC, 71% SP, 77% SN and AUROC 0.79. For later stages, NEW-CART (MELD, lactate, coma grade) offered predictive AC 86%, SP 91%, SN 46%, AUROC 0.73. CONCLUSION: CARTs offer simple prognostic models for APAP-ALF patients, which have higher AUROC and SN than KCC, with similar AC and negligibly worse SP. Admission and post-admission predictions were developed. KEY POINTS: • Prognostication in acetaminophen-induced acute liver failure (APAP-ALF) is challenging beyond admission • Little has been published regarding the use of King's College Criteria (KCC) beyond admission and KCC has shown limited sensitivity in subsequent studies • Classification and Regression Tree (CART) methodology allows the development of predictive models using binary splits and offers an intuitive method for predicting outcome, using processes familiar to clinicians • Data from the ALFSG registry suggested that CART prognosis models for the APAP population offer improved sensitivity and model performance over traditional regression-based KCC, while maintaining similar accuracy and negligibly worse specificity • KCC-CART models offered modest improvement over traditional KCC, with NEW-CART models performing better than KCC-CART particularly at late time points.

Random forest classification of etiologies for an orphan disease.

Classification of objects into pre-defined groups based on known information is a fundamental problem in the field of statistics. Although approaches for solving this problem exist, finding an accurate classification method can be challenging in an orphan disease setting, where data are minimal and often not normally distributed. The purpose of this paper is to illustrate the application of the random forest (RF) classification procedure in a real clinical setting and discuss typical questions that arise in the general classification framework as well as offer interpretations of RF results. This paper includes methods for assessing predictive performance, importance of predictor variables, and observation-specific information.