ABSTRACT
Aims: Spectrum bias can arise when a diagnostic test is derived from study populations with different disease spectra than the target population, resulting in poor generalizability. We used a real-world artificial intelligence (AI)-derived algorithm to detect severe aortic stenosis (AS) to experimentally assess the effect of spectrum bias on test performance. Methods and results: All adult patients at the Mayo Clinic between 1 January 1989 and 30 September 2019 with transthoracic echocardiograms within 180 days after electrocardiogram (ECG) were identified. Two models were developed from two distinct patient cohorts: a whole-spectrum cohort comparing severe AS to any non-severe AS and an extreme-spectrum cohort comparing severe AS to no AS at all. Model performance was assessed. Overall, 258 607 patients had valid ECG and echocardiograms pairs. The area under the receiver operator curve was 0.87 and 0.91 for the whole-spectrum and extreme-spectrum models, respectively. Sensitivity and specificity for the whole-spectrum model was 80% and 81%, respectively, while for the extreme-spectrum model it was 84% and 84%, respectively. When applying the AI-ECG derived from the extreme-spectrum cohort to patients in the whole-spectrum cohort, the sensitivity, specificity, and area under the curve dropped to 83%, 73%, and 0.86, respectively. Conclusion: While the algorithm performed robustly in identifying severe AS, this study shows that limiting datasets to clearly positive or negative labels leads to overestimation of test performance when testing an AI algorithm in the setting of classifying severe AS using ECG data. While the effect of the bias may be modest in this example, clinicians should be aware of the existence of such a bias in AI-derived algorithms.
ABSTRACT
Aims: The current gold standard comprehensive assessment of coronary microvascular dysfunction (CMD) is through a limited-access invasive catheterization lab procedure. We aimed to develop a point-of-care tool to assist clinical guidance in patients presenting with chest pain and/or an abnormal cardiac functional stress test and with non-obstructive coronary artery disease (NOCAD). Methods and results: This study included 1893 NOCAD patients (<50% angiographic stenosis) who underwent CMD evaluation as well as an electrocardiogram (ECG) up to 1-year prior. Endothelial-independent CMD was defined by coronary flow reserve (CFR) ≤2.5 in response to intracoronary adenosine. Endothelial-dependent CMD was defined by a maximal percent increase in coronary blood flow (%ΔCBF) ≤50% in response to intracoronary acetylcholine infusion. We trained algorithms to distinguish between the following outcomes: CFR ≤2.5, %ΔCBF ≤50, and the combination of both. Two classes of algorithms were trained, one depending on ECG waveforms as input, and another using tabular clinical data. Mean age was 51 ± 12 years and 66% were females (n = 1257). Area under the curve values ranged from 0.49 to 0.67 for all the outcomes. The best performance in our analysis was for the outcome CFR ≤2.5 with clinical variables. Area under the curve and accuracy were 0.67% and 60%. When decreasing the threshold of positivity, sensitivity and negative predictive value increased to 92% and 90%, respectively, while specificity and positive predictive value decreased to 25% and 29%, respectively. Conclusion: An artificial intelligence-enabled algorithm may be able to assist clinical guidance by ruling out CMD in patients presenting with chest pain and/or an abnormal functional stress test. This algorithm needs to be prospectively validated in different cohorts.
ABSTRACT
Background: We have demonstrated that a neural network is able to predict a person's age from the electrocardiogram (ECG) [artificial intelligence (AI) ECG age]. However, some discrepancies were observed between ECG-derived and chronological ages. We assessed whether the difference between AI ECG and chronological age (Age-Gap) represents biological ageing and predicts long-term outcomes. Methods and results: We previously developed a convolutional neural network to predict chronological age from ECGs. In this study, we used the network to analyse standard digital 12-lead ECGs in a cohort of 25 144 subjects ≥30 years who had primary care outpatient visits from 1997 to 2003. Subjects with coronary artery disease, stroke, and atrial fibrillation were excluded. We tested whether Age-Gap was correlated with total and cardiovascular mortality. Of 25 144 subjects tested (54% females, 95% Caucasian) followed for 12.4 ± 5.3 years, the mean chronological age was 53.7 ± 11.6 years and ECG-derived age was 54.6 ± 11 years (R 2 = 0.79, P < 0.0001). The mean Age-Gap was small at 0.88 ± 7.4 years. Compared to those whose ECG-derived age was within 1 standard deviation (SD) of their chronological age, patients with Age-Gap ≥1 SD had higher all-cause and cardiovascular disease (CVD) mortality. Conversely, subjects whose Age-Gap was ≤1 SD had lower all-cause and CVD mortality. Results were unchanged after adjusting for CVD risk factors and other survival influencing factors. Conclusion: The difference between AI ECG and chronological age is an independent predictor of all-cause and cardiovascular mortality. Discrepancies between these possibly reflect disease independent biological ageing.
