Statistical models versus machine learning for competing risks: development and validation of prognostic models.

BACKGROUND: In health research, several chronic diseases are susceptible to competing risks (CRs). Initially, statistical models (SM) were developed to estimate the cumulative incidence of an event in the presence of CRs. As recently there is a growing interest in applying machine learning (ML) for clinical prediction, these techniques have also been extended to model CRs but literature is limited. Here, our aim is to investigate the potential role of ML versus SM for CRs within non-complex data (small/medium sample size, low dimensional setting). METHODS: A dataset with 3826 retrospectively collected patients with extremity soft-tissue sarcoma (eSTS) and nine predictors is used to evaluate model-predictive performance in terms of discrimination and calibration. Two SM (cause-specific Cox, Fine-Gray) and three ML techniques are compared for CRs in a simple clinical setting. ML models include an original partial logistic artificial neural network for CRs (PLANNCR original), a PLANNCR with novel specifications in terms of architecture (PLANNCR extended), and a random survival forest for CRs (RSFCR). The clinical endpoint is the time in years between surgery and disease progression (event of interest) or death (competing event). Time points of interest are 2, 5, and 10 years. RESULTS: Based on the original eSTS data, 100 bootstrapped training datasets are drawn. Performance of the final models is assessed on validation data (left out samples) by employing as measures the Brier score and the Area Under the Curve (AUC) with CRs. Miscalibration (absolute accuracy error) is also estimated. Results show that the ML models are able to reach a comparable performance versus the SM at 2, 5, and 10 years regarding both Brier score and AUC (95% confidence intervals overlapped). However, the SM are frequently better calibrated. CONCLUSIONS: Overall, ML techniques are less practical as they require substantial implementation time (data preprocessing, hyperparameter tuning, computational intensity), whereas regression methods can perform well without the additional workload of model training. As such, for non-complex real life survival data, these techniques should only be applied complementary to SM as exploratory tools of model's performance. More attention to model calibration is urgently needed.

A modern approach to aortic valve insufficiency: Aortic root restoration via HAART 300 internal annuloplasty ring.

BACKGROUND AND AIM OF THE STUDY: HAART 300 is an internal geometric annuloplasty ring. The safety and efficacy of this novel device in aortic valve (AV) repair in a single referral center are reported. METHODS: Twenty patients with trileaflet AV insufficiency with ascending aorta and/or aortic root enlargement were included. Subannular implantation was performed to correct annular dilatation, whereas concomitant leaflet repair was performed whenever required. All but two patients also received ascending aorta replacement, whereas selective sinus replacement was performed in all but five patients. RESULTS: Follow-up was for a maximum of 3.8 years and a mean of 2.2 years. Mean age was 54.2 years old. Moderate to severe preoperative AV insufficiency was noted in 75% of patients, whereas 70% of them had an ascending aorta over 45 mm. One patient was lost from follow-up. Overall mortality as well as major complication rates were zero. Early postoperatively, no more than mild AV regurgitation was detected, whereas only one patient appeared with moderate AV regurgitation during our 2.2-year follow-up. New York Heart Association class was also significantly lower compared to preoperative values and valve gradients remained low at last follow-up. CONCLUSIONS: Geometric ring annuloplasty is a safe and effective valve sparing approach to deal with AV insufficiency contributing to overall root reconstruction. Short-term results are excellent rendering this easily reproducible and versatile method very attractive.

Survival prediction models since liver transplantation - comparisons between Cox models and machine learning techniques.

BACKGROUND: Predicting survival of recipients after liver transplantation is regarded as one of the most important challenges in contemporary medicine. Hence, improving on current prediction models is of great interest.Nowadays, there is a strong discussion in the medical field about machine learning (ML) and whether it has greater potential than traditional regression models when dealing with complex data. Criticism to ML is related to unsuitable performance measures and lack of interpretability which is important for clinicians. METHODS: In this paper, ML techniques such as random forests and neural networks are applied to large data of 62294 patients from the United States with 97 predictors selected on clinical/statistical grounds, over more than 600, to predict survival from transplantation. Of particular interest is also the identification of potential risk factors. A comparison is performed between 3 different Cox models (with all variables, backward selection and LASSO) and 3 machine learning techniques: a random survival forest and 2 partial logistic artificial neural networks (PLANNs). For PLANNs, novel extensions to their original specification are tested. Emphasis is given on the advantages and pitfalls of each method and on the interpretability of the ML techniques. RESULTS: Well-established predictive measures are employed from the survival field (C-index, Brier score and Integrated Brier Score) and the strongest prognostic factors are identified for each model. Clinical endpoint is overall graft-survival defined as the time between transplantation and the date of graft-failure or death. The random survival forest shows slightly better predictive performance than Cox models based on the C-index. Neural networks show better performance than both Cox models and random survival forest based on the Integrated Brier Score at 10 years. CONCLUSION: In this work, it is shown that machine learning techniques can be a useful tool for both prediction and interpretation in the survival context. From the ML techniques examined here, PLANN with 1 hidden layer predicts survival probabilities the most accurately, being as calibrated as the Cox model with all variables. TRIAL REGISTRATION: Retrospective data were provided by the Scientific Registry of Transplant Recipients under Data Use Agreement number 9477 for analysis of risk factors after liver transplantation.

Assessment of bias in scoring of AI-based radiotherapy segmentation and planning studies using modified TRIPOD and PROBAST guidelines as an example.

BACKGROUND AND PURPOSE: Studies investigating the application of Artificial Intelligence (AI) in the field of radiotherapy exhibit substantial variations in terms of quality. The goal of this study was to assess the amount of transparency and bias in scoring articles with a specific focus on AI based segmentation and treatment planning, using modified PROBAST and TRIPOD checklists, in order to provide recommendations for future guideline developers and reviewers. MATERIALS AND METHODS: The TRIPOD and PROBAST checklist items were discussed and modified using a Delphi process. After consensus was reached, 2 groups of 3 co-authors scored 2 articles to evaluate usability and further optimize the adapted checklists. Finally, 10 articles were scored by all co-authors. Fleiss' kappa was calculated to assess the reliability of agreement between observers. RESULTS: Three of the 37 TRIPOD items and 5 of the 32 PROBAST items were deemed irrelevant. General terminology in the items (e.g., multivariable prediction model, predictors) was modified to align with AI-specific terms. After the first scoring round, further improvements of the items were formulated, e.g., by preventing the use of sub-questions or subjective words and adding clarifications on how to score an item. Using the final consensus list to score the 10 articles, only 2 out of the 61 items resulted in a statistically significant kappa of 0.4 or more demonstrating substantial agreement. For 41 items no statistically significant kappa was obtained indicating that the level of agreement among multiple observers is due to chance alone. CONCLUSION: Our study showed low reliability scores with the adapted TRIPOD and PROBAST checklists. Although such checklists have shown great value during development and reporting, this raises concerns about the applicability of such checklists to objectively score scientific articles for AI applications. When developing or revising guidelines, it is essential to consider their applicability to score articles without introducing bias.

Neural Networks for Survival Prediction in Medicine Using Prognostic Factors: A Review and Critical Appraisal.

Survival analysis deals with the expected duration of time until one or more events of interest occur. Time to the event of interest may be unobserved, a phenomenon commonly known as right censoring, which renders the analysis of these data challenging. Over the years, machine learning algorithms have been developed and adapted to right-censored data. Neural networks have been repeatedly employed to build clinical prediction models in healthcare with a focus on cancer and cardiology. We present the first ever attempt at a large-scale review of survival neural networks (SNNs) with prognostic factors for clinical prediction in medicine. This work provides a comprehensive understanding of the literature (24 studies from 1990 to August 2021, global search in PubMed). Relevant manuscripts are classified as methodological/technical (novel methodology or new theoretical model; 13 studies) or applications (11 studies). We investigate how researchers have used neural networks to fit survival data for prediction. There are two methodological trends: either time is added as part of the input features and a single output node is specified, or multiple output nodes are defined for each time interval. A critical appraisal of model aspects that should be designed and reported more carefully is performed. We identify key characteristics of prediction models (i.e., number of patients/predictors, evaluation measures, calibration), and compare ANN's predictive performance to the Cox proportional hazards model. The median sample size is 920 patients, and the median number of predictors is 7. Major findings include poor reporting (e.g., regarding missing data, hyperparameters) as well as inaccurate model development/validation. Calibration is neglected in more than half of the studies. Cox models are not developed to their full potential and claims for the performance of SNNs are exaggerated. Light is shed on the current state of art of SNNs in medicine with prognostic factors. Recommendations are made for the reporting of clinical prediction models. Limitations are discussed, and future directions are proposed for researchers who seek to develop existing methodology.

How do cancer clinicians perceive real-world data and the evidence derived therefrom? Findings from an international survey of the European Organisation for Research and Treatment of Cancer.

Background: The role of real-world evidence (RWE) in the development of anticancer therapies has been gradually growing over time. Regulators, payers and health technology assessment agencies, spurred by the rise of the precision medicine model, are increasingly incorporating RWE into their decision-making regarding the authorization and reimbursement of novel antineoplastic treatments. However, it remains unclear how this trend is viewed by clinicians in the field. This study aimed to investigate the opinions of these stakeholders with respect to RWE and its suitability for informing regulatory, reimbursement-related and clinical decisions in oncology. Methods: An online survey was disseminated to clinicians belonging to the network of the European Organisation for Research and Treatment of Cancer between May and July 2021. Results: In total, 557 clinicians across 30 different countries participated in the survey, representing 13 distinct cancer domains. Despite seeing the methodological challenges associated with its interpretation as difficult to overcome, the respondents mostly (75.0%) perceived RWE positively, and believed such evidence could be relatively strong, depending on the designs and data sources of the studies from which it is produced. Few (4.6%) saw a future expansion of its influence on decision-makers as a negative evolution. Furthermore, nearly all (94.0%) participants were open to the idea of sharing anonymized or pseudonymized electronic health data of their patients with external parties for research purposes. Nevertheless, most clinicians (77.0%) still considered randomized controlled trials (RCTs) to be the gold standard for generating clinical evidence in oncology, and a plurality (49.2%) thought that RWE cannot fully address the knowledge gaps that remain after a new antitumor intervention has entered the market. Moreover, a majority of respondents (50.7%) expressed that they relied more heavily on RCT-derived evidence than on RWE for their own decision-making. Conclusion: While cancer clinicians have positive opinions about RWE and want to contribute to its generation, they also continue to hold RCTs in high regard as sources of actionable evidence.

Prognostic Significance of Bone Metastasis in Soft Tissue Sarcoma Patients Receiving Palliative Systemic Therapy: An Explorative, Retrospective Pooled Analysis of the EORTC-Soft Tissue and Bone Sarcoma Group (STBSG) Database.

Background: Soft-tissue sarcomas (STS) constitute a rare group of heterogeneous mesenchymal tumours containing more than 100 histologic subtypes. Here, we investigate whether, and if so, to what extent, skeletal metastases affect the outcome of patients with advanced or metastatic disease. Materials and Methods: Selected patients participated in five clinical trials of EORTC-STBSG. Individuals were included if they started treatment with an active drug and had advanced/metastatic STS. The endpoints of interest were overall survival (OS) and progression-free survival (PFS). Univariate and multivariate pooled analyses (after correcting for 12 covariates) were employed with Kaplan-Meier and Cox regression to model the impact of bone metastasis at presentation per treatment line stratified by study. For the subset of patients with bone metastasis, the impact of another metastatic organ site was explored with multivariate Cox regression models. Results: 565 out of 1034 (54.6%) patients received first-line systemic treatment for locally advanced or metastatic disease. Bone metastases were present in 140 patients (77 first-line, 63 second-line or higher). The unadjusted difference in OS/PFS with or without bone metastasis was statistically significant only for first-line patients. For OS, the adjusted hazard ratios for bone metastasis presence were 1.33 (95%-CI: 0.99-1.78) and 1.11 (95%-CI: 0.81-1.52) for first-line/second-line or higher treated patients, respectively. Likewise, the adjusted hazard ratios for PFS were 1.31 (95%-CI: 1.00-1.73) and 1.07 (95%-CI: 0.80-1.43). Effects were not statistically significant, despite a trend in first-line patients for both endpoints. Subgroup analyses indicated bone and lymph node metastasis as the most detrimental combination for OS and bone and lung metastasis for PFS. Conclusions: Adult STS patients receiving palliative systemic therapy with bone metastasis carried an overall worse prognosis than STS patients without bone metastases. Skeletal metastasis was detrimental for both OS and PFS, independent of the treatment line. Findings may have implications for the management of these patients.

New benchmarks to design clinical trials with advanced or metastatic liposarcoma or synovial sarcoma patients: An EORTC - Soft Tissue and Bone Sarcoma Group (STBSG) meta-analysis based on a literature review for soft-tissue sarcomas.

BACKGROUND: Recently, we performed a meta-analysis based on a literature review for STS trials (published 2003-2018, ≥10 adult patients) to update long-standing reference values for leiomyosarcomas. This work is extended for liposarcomas (LPS) and synovial sarcomas (SS). MATERIALS AND METHODS: Study endpoints were progression-free survival rates (PFSRs) at 3 and 6 months. Trial-specific estimates were pooled per treatment line (first-line or pre-treated) with random effects meta-analyses. The choice of the therapeutic benefit to target in future trials was guided by the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). RESULTS: Information was acquired for 1030 LPS patients (25 trials; 7 first-line, 17 pre-treated, 1 both) and 348 SS patients (13 trials; 3 first-line, 10 pre-treated). For LPS, the overall pooled first-line PFSRs were 69% (95%-CI 60-77%) and 56% (95%-CI 45-67%) at 3 and 6 months, respectively. These rates were 49% (95%-CI 40-57%)/28% (95%-CI 22-34%) for >1 lines. For SS, first-line PFSRs were 74% (95%-CI 58-86%)/56% (95%-CI 31-78%) at 3 and 6 months, and pre-treated rates were 45% (95%-CI 34-57%)/25% (95%-CI 16-36%). Following ESMO-MCBS guidelines, the minimum values to target are 79% and 69% for first-line LPS (82% and 69% for SS) at 3 and 6 months. For pre-treated LPS, recommended PFSRs at 3 and 6 months suggesting drug activity are 63% and 44% (60% and 41% for SS). CONCLUSIONS: New benchmarks are proposed for advanced/metastatic LPS or SS to design future histology-specific phase II trials. More data are needed to provide definitive thresholds for the different LPS subtypes.

A Simulation Study to Compare the Predictive Performance of Survival Neural Networks with Cox Models for Clinical Trial Data.

BACKGROUND: Studies focusing on prediction models are widespread in medicine. There is a trend in applying machine learning (ML) by medical researchers and clinicians. Over the years, multiple ML algorithms have been adapted to censored data. However, the choice of methodology should be motivated by the real-life data and their complexity. Here, the predictive performance of ML techniques is compared with statistical models in a simple clinical setting (small/moderate sample size and small number of predictors) with Monte-Carlo simulations. METHODS: Synthetic data (250 or 1000 patients) were generated that closely resembled 5 prognostic factors preselected based on a European Osteosarcoma Intergroup study (MRC BO06/EORTC 80931). Comparison was performed between 2 partial logistic artificial neural networks (PLANNs) and Cox models for 20, 40, 61, and 80% censoring. Survival times were generated from a log-normal distribution. Models were contrasted in terms of the C-index, Brier score at 0-5 years, integrated Brier score (IBS) at 5 years, and miscalibration at 2 and 5 years (usually neglected). The endpoint of interest was overall survival. RESULTS: PLANNs original/extended were tuned based on the IBS at 5 years and the C-index, achieving a slightly better performance with the IBS. Comparison with Cox models showed that PLANNs can reach similar predictive performance on simulated data for most scenarios with respect to the C-index, Brier score, or IBS. However, Cox models were frequently less miscalibrated. Performance was robust in scenario data where censored patients were removed before 2 years or curtailing at 5 years was performed (on training data). CONCLUSION: Survival neural networks reached a comparable predictive performance with Cox models but were generally less well calibrated. All in all, researchers should be aware of burdensome aspects of ML techniques such as data preprocessing, tuning of hyperparameters, and computational intensity that render them disadvantageous against conventional regression models in a simple clinical setting.

Efficacy thresholds for clinical trials with advanced or metastatic leiomyosarcoma patients: A European Organisation for Research and Treatment of Cancer Soft Tissue and Bone Sarcoma Group meta-analysis based on a literature review for soft-tissue sarcomas.

BACKGROUND: In 2002, the European Organisation for Research and Treatment of Cancer Soft Tissue and Bone Sarcoma Group reported well-established values for conducting phase II trials for soft-tissue sarcomas. An update is provided for leiomyosarcoma (LMS). MATERIALS AND METHODS: Clinical trials with advanced or metastatic LMS were identified via literature review in PubMed (published 2003-2018, ≥10 adult LMS patients). End-points were 3- and 6-month progression-free survival rates (PFSR-3m and PFSR-6m). When estimates could not be derived from publications, data requests were sent out. Treatments were classified as recommended (R-T) or non-recommended (NR-T) according to the ESMO 2018 guidelines. A random effects meta-analysis was used to pool trial-specific estimates for first-line (1L) or pre-treated (2L+) patients separately. The ESMO Magnitude of Clinical Benefit Scale was used to guide the treatment effect to target in future trials. RESULTS: From 47 studies identified, we obtained information on 7 1L and 16 2L+ trials for 1500 LMS patients. Overall, in 1L, PFSR-3m and PFSR-6m were 74% (95% confidence interval [CI] 64-82%) and 58% (95% CI 50-66%), respectively. For 2L+, PFSR-3m was 48% (95% CI 41-54%), and PFSR-6m was 28% (95% CI 22-34%). No difference was observed between R-T and NR-T for first or later lines. Under the alternative that the true benefit amounts to a hazard ratio of 0.65, a PFSR-6m ≥70% can be considered to suggest drug activity in 1L. For 2L+, a PFSR-3m ≥62% or PFSR-6m ≥44% would suggest drug activity. Specific results are also provided for uterine LMS. CONCLUSIONS: This work provides a new benchmark for designing phase II studies for advanced or metastatic LMS.