Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis.

Background: The Society of Thoracic Surgeons and European System for Cardiac Operative Risk Evaluation (EuroSCORE) II risk scores are the most commonly used risk prediction models for in-hospital mortality after adult cardiac surgery. However, they are prone to miscalibration over time and poor generalization across data sets; thus, their use remains controversial. Despite increased interest, a gap in understanding the effect of data set drift on the performance of machine learning (ML) over time remains a barrier to its wider use in clinical practice. Data set drift occurs when an ML system underperforms because of a mismatch between the data it was developed from and the data on which it is deployed. Objective: In this study, we analyzed the extent of performance drift using models built on a large UK cardiac surgery database. The objectives were to (1) rank and assess the extent of performance drift in cardiac surgery risk ML models over time and (2) investigate any potential influence of data set drift and variable importance drift on performance drift. Methods: We conducted a retrospective analysis of prospectively, routinely gathered data on adult patients undergoing cardiac surgery in the United Kingdom between 2012 and 2019. We temporally split the data 70:30 into a training and validation set and a holdout set. Five novel ML mortality prediction models were developed and assessed, along with EuroSCORE II, for relationships between and within variable importance drift, performance drift, and actual data set drift. Performance was assessed using a consensus metric. Results: A total of 227,087 adults underwent cardiac surgery during the study period, with a mortality rate of 2.76% (n=6258). There was strong evidence of a decrease in overall performance across all models (P<.0001). Extreme gradient boosting (clinical effectiveness metric [CEM] 0.728, 95% CI 0.728-0.729) and random forest (CEM 0.727, 95% CI 0.727-0.728) were the overall best-performing models, both temporally and nontemporally. EuroSCORE II performed the worst across all comparisons. Sharp changes in variable importance and data set drift from October to December 2017, from June to July 2018, and from December 2018 to February 2019 mirrored the effects of performance decrease across models. Conclusions: All models show a decrease in at least 3 of the 5 individual metrics. CEM and variable importance drift detection demonstrate the limitation of logistic regression methods used for cardiac surgery risk prediction and the effects of data set drift. Future work will be required to determine the interplay between ML models and whether ensemble models could improve on their respective performance advantages.

Stroke after Cardiac Surgery: A Risk Factor Analysis of 580,117 Patients from UK National Adult Cardiac Surgical Audit Cohort.

Cerebrovascular accident is the most ominous complication observed after cardiac surgery, carrying an increased risk of morbidity and mortality. Analysis of the problem shows its multidimensional nature. In this study, we aimed to identify major determinants among classic variables, either demographic, clinical or type of surgical procedure, based on the analysis of a large dataset of 580,117 patients from the UK National Adult Cardiac Surgical Audit (NACSA). For this purpose, univariate and multivariate logistic regression models were utilized to determine associations between predictors and dependent variable (Stroke after cardiac surgery). Odds ratios (ORs) and 95% confidence intervals (CIs) were constructed for each independent variable. Statistical analysis allows us to confirm with greater certainty the predictive value of some variables such as age, gender, diabetes mellitus (diabetes treated with insulin OR = 1.37, 95%CI = 1.23-1.53), and systemic arterial hypertension (OR = 1.11, 95%CI = 1.05-1.16);, to emphasize the role of preoperative atrial fibrillation (OR = 1.10, 95%CI = 1.03-1.16) extracardiac arteriopathy (OR = 1.70, 95%CI = 1.58-1.82), and previous cerebral vascular accident (OR 1.71, 95%CI = 1.6-1.9), and to reappraise others like smoking status (crude OR = 1.00, 95%CI = 0.93-1.07 for current smokers) or BMI (OR = 0.98, 95%CI = 0.97-0.98). This could allow for better preoperative risk stratification. In addition, identifying those surgical procedures (for example thoracic aortic surgery associated with a crude OR of 3.72 and 95%CI = 3.53-3.93) burdened by a high risk of neurological complications may help broaden the field of preventive and protective techniques.

Left Atrial Strain to Predict Postoperative Atrial Fibrillation in Patients Undergoing Coronary Artery Bypass Grafting.

BACKGROUND: Postoperative atrial fibrillation (POAF) is a common complication after cardiac surgery being associated with poorer outcomes. Revealing before the operation of left atrial subtle structural/functional abnormalities may help to identify patients at increased risk of POAF. We investigated the role of left atrial strain parameters by preoperative speckle tracking echocardiography as independent predictors of POAF in patients undergoing coronary artery bypass graft. METHODS: Consecutive patients undergoing isolated coronary artery bypass graft were prospectively enrolled at three Italian centers. All patients underwent transthoracic echocardiography before the operation. The occurrence of POAF up to discharge was monitored. RESULTS: Overall, a total of 310 patients were included. POAF was demonstrated in 103 patients (33%). At receiver operating characteristic curve analysis, lower global peak atrial longitudinal strain (PALS) values significantly predicted the risk of POAF (area under the curve, 0.74; P<0.001). The optimal cutoff value for the arrhythmia prediction was a global PALS value <28%, with a specificity of 86% and a sensitivity of 36%. The incidence of POAF was 51% in patients with global PALS <28% versus 14% in those with PALS ≥28% (P<0.001), with a POAF-free survival at Kaplan-Meier analysis of 45.4% and 85.7%, respectively (P<0.001). At multivariate analysis, a global PALS <28% carried a 3.6-fold higher risk of POAF (hazard ratio, 3.6 [95% CI, 2.2-5.9]; P<0.001). The risk increase was even higher when PALS <28% was associated with age ≥70 years (adjusted hazard ratio, 11.2 [4.7-26.6], P<0.001). CONCLUSIONS: A presurgery global PALS <28% is a specific parameter to stratify patients at increased risk of POAF after coronary artery bypass graft. This assessment can be useful to identify patients at higher arrhythmic risk in whom perioperative preventive strategies and stricter monitoring aimed at early diagnosing and treating POAF may be applied.

Red blood cell transfusion and mortality after transcatheter aortic valve implantation via transapical approach: A propensity-matched comparison from the TRITAVI registry.

Objective: Bleeding is frequent during transcatheter aortic valve implantation (TAVI), especially when performed through a transapical approach (TA), and is associated with a worse prognosis. The present study aims to test the implication of red blood cell (RBC) transfusion and the optimal transfusion strategy in this context. Methods: Among 11,265 participants in the multicenter TRITAVI (Transfusion Requirements in Transcatheter Aortic Valve Implantation) registry, 548 patients (4.9%) who received TA-TAVI at 19 European centers were included. One-to-one propensity score matching was performed to reduce treatment selection bias and potential confounding among transfused versus non-transfused patients. The primary endpoint of the study was the 30-day occurrence of all-cause mortality. Results: 209 patients (38 %) received RBC transfusions. The primary endpoint occurred in 47 (8.6 %) patients. Propensity score matching identified 188 pairs of patients with and without RBC transfusion. In the propensity score-matched analysis, RBC transfusion was associated with increased 30-day mortality (HR 3.35, 95 % CI 1.51 - 7.39; p = 0.002). At multivariable cox regression analysis, RBC transfusion was an independent predictor of 30-day mortality (HR 3.07, 95 % CI 1.01-9.41, p = 0.048), as well as baseline ejection fraction (HR 0.96, 95 % CI 0.92-0.99, p = 0.043), and acute kidney injury (HR 3.95, 95 % CI 1.11-14.05, p = 0.034). Conclusions: RBC transfusion is an independent predictor of short-term mortality in patients undergoing TA-TAVI, regardless of major bleeding.Clinical trial registration: https://www.clinicaltrials.gov Unique identifier: NCT03740425.

Risk Score for Prediction of Dialysis After Transcatheter Aortic Valve Replacement.

BACKGROUND: Dialysis is a rare but serious complication after transcatheter aortic valve replacement. We analyzed the large multicenter TRITAVI (transfusion requirements in transcatheter aortic valve implantation) registry in order to develop and validate a clinical score assessing this risk. METHODS AND RESULTS: A total of 10 071 consecutive patients were enrolled in 19 European centers. Patients were randomly assigned (2:1) to a derivation and validation cohort. Two scores were developed, 1 including only preprocedural variables (TRITAVIpre) and 1 also including procedural variables (TRITAVIpost). In the 6714 patients of the derivation cohort (age 82±6 years, 48% men), preprocedural factors independently associated with dialysis and included in the TRITAVIpre score were male sex, diabetes, prior coronary artery bypass graft, anemia, nonfemoral access, and creatinine clearance <30 mL/min per m2. Additional independent predictors among procedural features were volume of contrast, need for transfusion, and major vascular complications. Both scores showed a good discrimination power for identifying risk for dialysis with C-statistic 0.78 for TRITAVIpre and C-statistic 0.88 for TRITAVIpost score. Need for dialysis increased from the lowest to the highest of 3 risk score groups (from 0.3% to 3.9% for TRITAVIpre score and from 0.1% to 6.2% for TRITAVIpost score). Analysis of the 3357 patients of the validation cohort (age 82±7 years, 48% men) confirmed the good discrimination power of both scores (C-statistic 0.80 for TRITAVIpre and 0.81 for TRITAVIpost score). Need for dialysis was associated with a significant increase in 1-year mortality (from 6.9% to 54.4%; P=0.0001). CONCLUSIONS: A simple preprocedural clinical score can help predict the risk of dialysis after transcatheter aortic valve replacement.