Outcome risk model development for heterogeneity of treatment effect analyses: a comparison of non-parametric machine learning methods and semi-parametric statistical methods.

BACKGROUND: In randomized clinical trials, treatment effects may vary, and this possibility is referred to as heterogeneity of treatment effect (HTE). One way to quantify HTE is to partition participants into subgroups based on individual's risk of experiencing an outcome, then measuring treatment effect by subgroup. Given the limited availability of externally validated outcome risk prediction models, internal models (created using the same dataset in which heterogeneity of treatment analyses also will be performed) are commonly developed for subgroup identification. We aim to compare different methods for generating internally developed outcome risk prediction models for subject partitioning in HTE analysis. METHODS: Three approaches were selected for generating subgroups for the 2,441 participants from the United States enrolled in the ASPirin in Reducing Events in the Elderly (ASPREE) randomized controlled trial. An extant proportional hazards-based outcomes predictive risk model developed on the overall ASPREE cohort of 19,114 participants was identified and was used to partition United States' participants by risk of experiencing a composite outcome of death, dementia, or persistent physical disability. Next, two supervised non-parametric machine learning outcome classifiers, decision trees and random forests, were used to develop multivariable risk prediction models and partition participants into subgroups with varied risks of experiencing the composite outcome. Then, we assessed how the partitioning from the proportional hazard model compared to those generated by the machine learning models in an HTE analysis of the 5-year absolute risk reduction (ARR) and hazard ratio for aspirin vs. placebo in each subgroup. Cochran's Q test was used to detect if ARR varied significantly by subgroup. RESULTS: The proportional hazard model was used to generate 5 subgroups using the quintiles of the estimated risk scores; the decision tree model was used to generate 6 subgroups (6 automatically determined tree leaves); and the random forest model was used to generate 5 subgroups using the quintiles of the prediction probability as risk scores. Using the semi-parametric proportional hazards model, the ARR at 5 years was 15.1% (95% CI 4.0-26.3%) for participants with the highest 20% of predicted risk. Using the random forest model, the ARR at 5 years was 13.7% (95% CI 3.1-24.4%) for participants with the highest 20% of predicted risk. The highest outcome risk group in the decision tree model also exhibited a risk reduction, but the confidence interval was wider (5-year ARR = 17.0%, 95% CI= -5.4-39.4%). Cochran's Q test indicated ARR varied significantly only by subgroups created using the proportional hazards model. The hazard ratio for aspirin vs. placebo therapy did not significantly vary by subgroup in any of the models. The highest risk groups for the proportional hazards model and random forest model contained 230 participants each, while the highest risk group in the decision tree model contained 41 participants. CONCLUSIONS: The choice of technique for internally developed models for outcome risk subgroups influences HTE analyses. The rationale for the use of a particular subgroup determination model in HTE analyses needs to be explicitly defined based on desired levels of explainability (with features importance), uncertainty of prediction, chances of overfitting, and assumptions regarding the underlying data structure. Replication of these analyses using data from other mid-size clinical trials may help to establish guidance for selecting an outcomes risk prediction modelling technique for HTE analyses.

Toward Understanding the Size Dependence of Shape Features for Predicting Spiculation in Lung Nodules for Computer-Aided Diagnosis.

We analyze the importance of shape features for predicting spiculation ratings assigned by radiologists to lung nodules in computed tomography (CT) scans. Using the Lung Image Database Consortium (LIDC) data and classification models based on decision trees, we demonstrate that the importance of several shape features increases disproportionately relative to other image features with increasing size of the nodule. Our shaped-based classification results show an area under the receiver operating characteristic (ROC) curve of 0.65 when classifying spiculation for small nodules and an area of 0.91 for large nodules, resulting in a 26% difference in classification performance using shape features. An analysis of the results illustrates that this change in performance is driven by features that measure boundary complexity, which perform well for large nodules but perform relatively poorly and do no better than other features for small nodules. For large nodules, the roughness of the segmented boundary maps well to the semantic concept of spiculation. For small nodules, measuring directly the complexity of hard segmentations does not yield good results for predicting spiculation due to limits imposed by spatial resolution and the uncertainty in boundary location. Therefore, a wider range of features, including shape, texture, and intensity features, are needed to predict spiculation ratings for small nodules. A further implication is that the efficacy of shape features for a particular classifier used to create computer-aided diagnosis systems depends on the distribution of nodule sizes in the training and testing sets, which may not be consistent across different research studies.

Commercial quality "awards" are not a strong indicator of quality surgical care.

BACKGROUND: This study aimed to determine whether publicized hospital rankings can be used to predict surgical outcomes. METHODS: Patients undergoing one of nine surgical procedures were identified, using the Healthcare Cost and Utilization Project State Inpatient Database for Florida and New York 2011-2013 and merged with hospital data from the American Hospital Association Annual Survey. Nine quality designations were analyzed as possible predictors of inpatient mortality and postoperative complications, using logistic regression, decision trees, and support vector machines. RESULTS: We identified 229,657 patients within 177 hospitals. Decision trees were the highest performing machine learning algorithm for predicting inpatient mortality and postoperative complications (accuracy 0.83, P<.001). The top 3 variables associated with low surgical mortality (relative impact) were Hospital Compare (42), total procedure volume (16) and, Joint Commission (12). When analyzed separately for each individual procedure, hospital quality awards were not predictors of postoperative complications for 7 of the 9 studied procedures. However, when grouping together procedures with a volume-outcome relationship, hospital ranking becomes a significant predictor of postoperative complications. CONCLUSION: Hospital quality rankings are not a reliable indicator of quality for all surgical procedures. Hospital and provider quality must be evaluated with an emphasis on creating consistent, reliable, and accurate measures of quality that translate to improved patient outcomes.

Building an ensemble system for diagnosing masses in mammograms.

PURPOSE: Classification of a suspicious mass (region of interest, ROI) in a mammogram as malignant or benign may be achieved using mass shape features. An ensemble system was built for this purpose and tested. METHODS: Multiple contours were generated from a single ROI using various parameter settings of the image enhancement functions for the segmentation. For each segmented contour, the mass shape features were computed. For classification, the dataset was partitioned into four subsets based on the patient age (young/old) and the ROI size (large/small). We built an ensemble learning system consisting of four single classifiers, where each classifier is a specialist, trained specifically for one of the subsets. Those specialist classifiers are also an optimal classifier for the subset, selected from several candidate classifiers through preliminary experiment. In this scheme, the final diagnosis (malignant or benign) of an instance is the classification produced by the classifier trained for the subset to which the instance belongs. RESULTS: The Digital Database for Screening Mammography (DDSM) from the University of South Florida was used to test the ensemble system for classification of masses, which achieved a 72% overall accuracy. This ensemble of specialist classifiers achieved better performance than single classification (56%). CONCLUSION: An ensemble classifier for mammography-detected masses may provide superior performance to any single classifier in distinguishing benign from malignant cases.