Machine learning models' assessment: trust and performance.

The common black box nature of machine learning models is an obstacle to their application in health care context. Their widespread application is limited by a significant "lack of trust." So, the main goal of this work is the development of an evaluation approach that can assess, simultaneously, trust and performance. Trust assessment is based on (i) model robustness (stability assessment), (ii) confidence (95% CI of geometric mean), and (iii) interpretability (comparison of respective features ranking with clinical evidence). Performance is assessed through geometric mean. For validation, in patients' stratification in cardiovascular risk assessment, a Portuguese dataset (N=1544) was applied. Five different models were compared: (i) GRACE score, the most common risk assessment tool in Portugal for patients with acute coronary syndrome; (ii) logistic regression; (iii) Naïve Bayes; (iv) decision trees; and (v) rule-based approach, previously developed by this team. The obtained results confirm that the simultaneous assessment of trust and performance can be successfully implemented. The rule-based approach seems to have potential for clinical application. It provides a high level of trust in the respective operation while outperformed the GRACE model's performance, enhancing the required physicians' acceptance. This may increase the possibility to effectively aid the clinical decision.

An interpretable machine learning approach to estimate the influence of inflammation biomarkers on cardiovascular risk assessment.

BACKGROUND AND OBJECTIVE: Cardiovascular disease has a huge impact on health care services, originating unsustainable costs at clinical, social, and economic levels. In this context, patients' risk stratification tools are central to support clinical decisions contributing to the implementation of effective preventive health care. Although useful, these tools present some limitations, in particular, some lack of performance as well as the impossibility to consider new risk factors potentially important in the prognosis of severe cardiac events. Moreover, the actual use of these tools in the daily practice requires the physicians' trust. The main goal of this work addresses these two issues: (i) evaluate the importance of inflammation biomarkers when combined with a risk assessment tool; (ii) incorporation of personalization and interpretability as key elements of that assessment. METHODS: Firstly, machine learning based models were created to assess the potential of the inflammation biomarkers applied in secondary prevention, namely in the prediction of the six month risk of death/myocardial infarction. Then, an approach based on three main phases was created: (i) set of interpretable rules supported by clinical evidence; (ii) selection based on a machine learning classifier able to identify for a given patient the most suitable subset of rules; (iii) an ensemble scheme combining the previous subset of rules in the estimation of the patient cardiovascular risk. All the results were statistically validated (t-test, Wilcoxon-signed rank test) according to a previous verification of data normality (Shapiro-Wilk). RESULTS: The proposed methodology was applied to a real acute coronary syndrome patients dataset (N = 1544) from the Cardiology Unit of Coimbra Hospital and Universitary centre. The first assessment was based on the GRACE tool and a Random Forest classifier, the incorporation of inflammation biomarkers achieved SE=0.83; SP=0.84 whereas the original GRACE risk factors reached SE=0.75; SP=0.85. In the second phase, the proposed approach with inflammation biomarkers achieved SE=0.763 and SP=0.778. CONCLUSIONS: This approach confirms the potential of combining inflammation markers with the GRACE score, increasing SE and SP, when compared with the original GRACE. Additionally, it assures interpretability and personalization, which are critical issues to allow its application in the daily clinical practice.

Rationale and design of the iCORONARY trial: improving the cost-effectiveness of coronary artery disease diagnosis.

BACKGROUND: In patients with stable coronary artery disease (CAD), revascularisation decisions are based mainly on the visual grading of the severity of coronary stenosis on invasive coronary angiography (ICA). However, invasive fractional flow reserve (FFR) is the current standard to determine the haemodynamic significance of coronary stenosis. Non-invasive and less-invasive imaging techniques such as computed-tomography-derived FFR (FFR-CT) and angiography-derived FFR (QFR) combine both anatomical and functional information in complex algorithms to calculate FFR. TRIAL DESIGN: The iCORONARY trial is a prospective, multicentre, non-inferiority randomised controlled trial (RCT) with a blinded endpoint evaluation. It investigates the costs, effects and outcomes of different diagnostic strategies to evaluate the presence of CAD and the need for revascularisation in patients with stable angina pectoris who undergo coronary computed tomography angiography. Those with a Coronary Artery Disease-Reporting and Data System (CAD-RADS) score between 0-2 and 5 will be included in a prospective registry, whereas patients with CAD-RADS 3 or 4A will be enrolled in the RCT. The RCT consists of three randomised groups: (1) FFR-CT-guided strategy, (2) QFR-guided strategy or (3) standard of care including ICA and invasive pressure measurements for all intermediate stenoses. The primary endpoint will be the occurrence of major adverse cardiac events (death, myocardial infarction and repeat revascularisation) at 1 year. CLINICALTRIALS: gov-identifier: NCT04939207. CONCLUSION: The iCORONARY trial will assess whether a strategy of FFR-CT or QFR is non-inferior to invasive angiography to guide the need for revascularisation in patients with stable CAD. Non-inferiority to the standard of care implies that these techniques are attractive, less-invasive alternatives to current diagnostic pathways.

A Clinical Interpretable Approach Applied to Cardiovascular Risk Assessment.

The effectiveness of predictive models in supporting the Clinical Decision is closely related with their clinical interpretability, i.e.the model should provide clear information on how to reach a specific classification/decision. In fact, the development of interpretable and accurate predictive models assumes a key importance as these tools can be very useful in Clinical Decision Support Systems (CDSS). The development of those models may comprise two main perspectives; existent clinical knowledge (clinical expert knowledge, clinical guidelines, current models, etc.) as well as data driven approaches able to extract (new) knowledge from recent clinical datasets. This work focuses in knowledge extraction from recent datasets (data driven) based on computational intelligence techniques. The main hypothesis that supports this work is that individuals with similar characteristics present a similar risk prof ile. Thus, this work addresses the development of stratification models able to learn distinct groups (or classes) of subjects assessing the similarity between characterizing variables. In particular, in the current study a data-driven supervised cluster approach is proposed aiming the derivation of meaningful rules directly from the dataset. The validation was performed based on the largest Portuguese coronary artery disease patient's dataset, provided by the Portuguese Society of Cardiology and comprising 13902 acute coronary syndrome patients. The goal was to assess the risk of death 30 days after admission. The models' performance was assessed through the sensitivity, specificity and geometric mean values. The obtained results show the potential of this approach, as they represent an acceptable performance (GM= 72%) while the clinical interpretability of the model is assured through the derived rules. Despite the achieved results, there are several research directions to be followed in order to enhance this work.

Pattern discovery and similarity assessment for robust heart sound segmentation.

Heart Sound Segmentation plays a fundamental role in pathology detection in Phonocardiogram (PCG) signals. This matter of study has been widely studied in the past decades, however the majority of algorithms' results correspond only to small databases, composed by only quality signals or signals specific to one acquisition system. In this work we proposed a robust segmentation algorithm integrated with clinical information, based on a pattern recognition approach for segmentation of the fundamental heart sounds, which is validated in several databases from different countries and with different acquisition instrumentations. The database comprises a total of 3153 recordings from 764 patients with a variety of pathological conditions. The general results were 95% and 96% of sensitivity and positive predictivity, respectively. Based on the results the algorithm is able to perform with accuracy maintaining generalization capabilities.

An interpretable data-driven approach for rules construction: Application to cardiovascular risk assessment.

The development of models able to produce an understandable decision by the clinicians is of great importance to support their decision. Therefore, the research of methodologies able to extract useful knowledge from existing datasets, as well as to integrate this knowledge into the current clinical evidence, is a key aspect in the enhancement of the clinical decision. This work focuses on the development of interpretable models to assess the patient's condition based on supervised clustering theories, enabling the discovery of a set of features that best represents that condition. At the same time, the technique is supported on a structure that enables the formulation of simple and interpretable rules. Despite its general applicability, the proposed methodology is applied to coronary artery disease (CAD), particularly, in the risk of death assessment (30 days after the admission) of patients that have been admitted to the emergency unit. The validation is performed using a real dataset with Acute Coronary Syndromes, provided by the Portuguese Society of Cardiology. While the methodology produces simple and interpretable rules, the performance achieves an improvement of 7% in relation to geometric mean, when compared with GRACE model (commonly used in Portugal).

Pattern discovery and similarity assessment for robust Heart Sound Segmentation.

Heart Sound Segmentation plays a fundamental role in pathology detection in Phonocardiogram (PCG) signals. This matter of study has been widely studied in the past decades, however the majority of algorithms' results correspond only to small databases, composed by only quality signals or signals specific to one acquisition system. In this work we proposed a robust segmentation algorithm integrated with clinical information, based on a pattern recognition approach for segmentation of the fundamental heart sounds, which is validated in several databases from different countries and with different acquisition instrumentations. The database comprises a total of 3153 recordings from 764 patients with a variety of pathological conditions. The general results were 95% and 96% of sensitivity and positive predictivity, respectively. Based on the results the algorithm is able to perform with accuracy maintaining generalization capabilities.

Detection of crackle events using a multi-feature approach.

The automatic detection of adventitious lung sounds is a valuable tool to monitor respiratory diseases like chronic obstructive pulmonary disease. Crackles are adventitious and explosive respiratory sounds that are usually associated with the inflammation or infection of the small bronchi, bronchioles and alveoli. In this study a multi-feature approach is proposed for the detection of events, in the frame space, that contain one or more crackles. The performance of thirty-five features was tested. These features include thirty-one features usually used in the context of Music Information Retrieval, a wavelet based feature as well as the Teager energy and the entropy. The classification was done using a logistic regression classifier. Data from seventeen patients with manifestations of adventitious sounds and three healthy volunteers were used to evaluate the performance of the proposed method. The dataset includes crackles, wheezes and normal lung sounds. The optimal detection parameters, such as the number of features, were chosen based on a grid search. The performance of the detection was studied taking into account the sensitivity and the positive predictive value. For the conditions tested, the best results were obtained for the frame size equal to 128 ms and twenty-seven features.

Assessment of cardiovascular function from multi-Gaussian fitting of a finger photoplethysmogram.

Monitoring of cardiovascular function on a beat-to-beat basis is fundamental for protecting patients in different settings including emergency medicine and interventional cardiology, but still faces technical challenges and several limitations. In the present study, we propose a new method for the extraction of cardiovascular performance surrogates from analysis of the photoplethysmographic (PPG) signal alone.We propose using a multi-Gaussian (MG) model consisting of five Gaussian functions to decompose the PPG pulses into its main physiological components. From the analysis of these components, we aim to extract estimators of the left ventricular ejection time, blood pressure and vascular tone changes. Using a multi-derivative analysis of the components related with the systolic ejection, we investigate which are the characteristic points that best define the left ventricular ejection time (LVET). Six LVET estimates were compared with the echocardiographic LVET in a database comprising 68 healthy and cardiovascular diseased volunteers. The best LVET estimate achieved a low absolute error (15.41 ± 13.66 ms), and a high correlation (ρ = 0.78) with the echocardiographic reference.To assess the potential use of the temporal and morphological characteristics of the proposed MG model components as surrogates for blood pressure and vascular tone, six parameters have been investigated: the stiffness index (SI), the T1_d and T1_2 (defined as the time span between the MG model forward and reflected waves), the reflection index (RI), the R1_d and the R1_2 (defined as their amplitude ratio). Their association to reference values of blood pressure and total peripheral resistance was investigated in 43 volunteers exhibiting hemodynamic instability. A good correlation was found between the majority of the extracted and reference parameters, with an exception to R1_2 (amplitude ratio between the main forward wave and the first reflection wave), which correlated low with all the reference parameters. The highest correlation ([Formula: see text] = 0.45) was found between T1_2 and the total peripheral resistance index (TPRI); while in the patients that experienced syncope, the highest agreement ([Formula: see text] = 0.57) was found between SI and systolic blood pressure (SBP) and mean blood pressure (MBP).In conclusion, the presented method for the extraction of surrogates of cardiovascular performance might improve patient monitoring and warrants further investigation.

Integration of Different Risk Assessment Tools to Improve Stratification of Patients with Coronary Artery Disease.

Cardiovascular disease (CVD) causes unaffordable social and health costs that tend to increase as the European population ages. In this context, clinical guidelines recommend the use of risk scores to predict the risk of a cardiovascular disease event. Some useful tools have been developed to predict the risk of occurrence of a cardiovascular disease event (e.g. hospitalization or death). However, these tools present some drawbacks. These problems are addressed through two methodologies: (i) combination of risk assessment tools: fusion of naïve Bayes classifiers complemented with a genetic optimization algorithm and (ii) personalization of risk assessment: subtractive clustering applied to a reduced-dimensional space to create groups of patients. Validation was performed based on two ACS-NSTEMI patient data sets. This work improved the performance in relation to current risk assessment tools, achieving maximum values of sensitivity, specificity, and geometric mean of, respectively, 79.8, 83.8, and 80.9 %. Additionally, it assured clinical interpretability, ability to incorporate of new risk factors, higher capability to deal with missing risk factors and avoiding the selection of a standard CVD risk assessment tool to be applied in the clinical practice.

Expert knowledge integration in the data mining process with application to cardiovascular risk assessment.

The data mining process, when applied to clinical databases, suffers from critical data problems, from noisy acquisitions to missing or incomplete data points. Expert knowledge, in the form of practitioners' experience and clinical guidelines, is already used to manually correct some of these problems, while enhancing expert's confidence in such systems. In this work, we propose the Knowledge-Biased Tree (KB3), a knowledge biased decision tree inducer that is able to exploit IF THEN rules to guide the tree inducing process. The KB3 approach was tested against its unbiased counterpart, the C5.0 algorithm in the cardiovascular risk assessment task. Using a clinical dataset provided by the hospital of Sta Cruz (Lisbon, Portugal) the performance of the proposed algorithm is compared against the unbiased C5.0 and the state of the art risk score used in clinical practice (GRACE risk score).

Assessment of cardiovascular risk based on a data-driven knowledge discovery approach.

The cardioRisk project addresses the development of personalized risk assessment tools for patients who have been admitted to the hospital with acute myocardial infarction. Although there are models available that assess the short-term risk of death/new events for such patients, these models were established in circumstances that do not take into account the present clinical interventions and, in some cases, the risk factors used by such models are not easily available in clinical practice. The integration of the existing risk tools (applied in the clinician's daily practice) with data-driven knowledge discovery mechanisms based on data routinely collected during hospitalizations, will be a breakthrough in overcoming some of these difficulties. In this context, the development of simple and interpretable models (based on recent datasets), unquestionably will facilitate and will introduce confidence in this integration process. In this work, a simple and interpretable model based on a real dataset is proposed. It consists of a decision tree model structure that uses a reduced set of six binary risk factors. The validation is performed using a recent dataset provided by the Portuguese Society of Cardiology (11113 patients), which originally comprised 77 risk factors. A sensitivity, specificity and accuracy of, respectively, 80.42%, 77.25% and 78.80% were achieved showing the effectiveness of the approach.

Personalization algorithms applied to cardiovascular disease risk assessment.

Cardiovascular disease (CVD) is the major cause of death in the world. Clinical guidelines recommend the use of risk assessment tools (scores) to identify the CVD risk of each patient as the correct stratification of patients may significantly contribute to the optimization of the health care strategies. This work further explores the personalization of CVD risk assessment, supported on the evidence that a specific CVD risk assessment tool may have good performance within a given group of patients and might perform poorly within other groups. Two main personalization methods based on the proper creation of groups of patients are presented: i) clustering patients approach; ii) similarity measures approach. These two methodologies were validated in a Portuguese population (460 Acute Coronary Syndrome with non-ST segment elevation (ACS-NSTEMI) patients). The similarity measures approach had the best performance, achieving maximum values of sensitivity, specificity and geometric mean of, respectively, 77.7%, 63.2%, 69.7%. These values represent an enhancement in relation to the best performance obtained with current CVD risk assessment tools applied in clinical practice (78.5%, 53.2%, 64.4%).

Cardiovascular disease risk assessment innovative approaches developed in HeartCycle project.

Two innovative CVD event risk assessment strategies were developed in the scope of HeartCycle project: i) combination of individual risk assessment tools; ii) personalization of risk assessment based on grouping of patients. These approaches aimed to defeat some of the major limitations of the tools currently applied in the daily clinical practice, namely to: i) improve the risk prediction performance when comparing it to the one achieved by the individual current risk assessment tools; ii) consider the available knowledge provided by other risk assessment tools; iii) cope with missing risk factors; iv) incorporate additional clinical knowledge. Two different real patients' datasets were applied to validate the developed strategies: i) Santa Cruz Hospital, Portugal, N=460 ACS-NSTEMI1 patients; ii) Leiria Pombal Hospital Centre, Portugal, N=99 ACS-NSTEMI. Based on the gathered results, we propose a new strategy in order to improve patients' stratification.

Multi-Gaussian fitting for the assessment of left ventricular ejection time from the photoplethysmogram.

The Left ventricular ejection time (LVET) is one of the primary surrogates of the left ventricular contractility and stroke volume. Its continuous monitoring is considered to be a valuable hypovolumia prognostic parameter and an important risk predictor in cardiovascular diseases such as cardiac and light chain amyloidosis. In this paper, we present a novel methodology for the assessment of LVET based the Photoplethysmographic (PPG) waveform. We propose the use of Gaussian functions to model both systolic and diastolic phases of the PPG beat and consequently determine the onset and offset of the systolic ejection from the analysis of the systolic phase 3(rd) derivative. The results achieved by the proposed methodology were compared with the algorithm proposed by Chan et al. [1], revealing better estimation of LVET (15.84 ± 13.56 ms vs 23.01 ± 14.60 ms), and similar correlation with the echocardiographic reference (0.73 vs 0.75).

Improvement of CVD risk assessment tools' performance through innovative patients' grouping strategies.

There are available in the clinical community several practical risk tools to assess the risk of occurrence of a cardiovascular event. Although valuable, these tools typically present some lack of performance (low sensitivity/low specificity) when applied to a general (average) patient. This flaw is addressed in this work through an innovative personalization strategy that is supported on the evidence that current risk assessment tools perform differently among different populations/groups of patients. The proposed methodology is based on two main hypotheses: i) patients are grouped through a proper dimension reduction technique complemented with an unsupervised learning algorithm, ii) for each group the most suitable risk assessment tool can be selected improving the risk prediction performance. As a result, risk personalization is simply achieved by the identification of the group that patients belong to. The validation of the strategy is carried out through the combination of three current risk assessment tools (GRACE, TIMI, PURSUIT) developed to predict the risk of an event in coronary artery disease patients. The combination of these tools is validated with a real patient testing dataset: Santa Cruz Hospital, Portugal, N=460 ACS-NSTEMI patients. Considering the obtained results with the available dataset it is possible to state that the main objective of this work was achieved.

Fusion of risk assessment models with application to coronary artery disease patients.

Several risk score models are available in literature to predict death/myocardial infarction event for coronary artery disease (CAD) patients, within a short period of time. However, the choice of the most adequate model is not straightforward since there might not be a consensus about the best model to use in clinical practice Moreover, individually, these models present some weaknesses, such as the inability to deal with missing information. This work addresses these problems, proposing a Bayesian classifier strategy enabling the simultaneous use of several models (models' fusion). Thus, a higher number of risk factors can be used in the common model, while it can deal with missing information. The validation of the strategy is carried out through the combination of three current risk score models (GRACE, TIMI, PURSUIT). Results were obtained based on a dataset that comprises 460 consecutive patients admitted to the Cardiology Department of Santa Cruz Hospital, Lisbon, from 1999 to 2001. A comparison with the voting scheme, which considers exclusively the outputs of models to combine (models output combination) is also carried out. The proposed Bayesian approach had very satisfactory results, confirming the potential of its application to the clinical practice.

Cardiovascular risk and status assessment.

This work focuses on the development of models to support the assessment of a patient's global cardiovascular condition. Three types of models, based on different types of information, have been developed: long term cardiovascular risk models, that evaluate the risk of occurring of cardiovascular event within a long period of time (years); short term cardiovascular risk models, to assess the risk of death within a short period of time (months); cardiovascular status assessment models, to estimate the current cardiovascular condition of a patient. Three major drawbacks of current cardiovascular tools are addressed: reduced number of risk factors considered by each individual tool, inappropriateness of these tools to incorporate empirical clinical expertise and incapacity of these tools to deal with incomplete information. Methodologies and preliminary results, obtained under FP7 HeartCycle project, as well as future directions of research are also presented in this paper.

Ischemia detection in the context of a cardiovascular status assessment tool.

In this work a new strategy for ischemic episodes automatic detection is proposed, considering ST segment deviation and T wave and QRS morphology characteristics. A new measure of ST deviation based on time-frequency analysis, and the use of the expansion in Hermite functions technique for T wave and QRS complex morphology characterization, are the key points of the proposed methodology. HeartCycle is a European project that aims to improve life quality of coronary artery disease (CAD) and heart failure (HF) patients. Within this project, the Medical Risk Assessment module is responsible for develop models to assess cardiovascular (CV) risk and status of referred patients. The present work was performed under the context of CV status models, where myocardial ischemia plays a central role. For algorithms validation purposes, the European Society of Cardiology (ESC) ST-T database was used. A sensitivity of 96.7% and a positive predictivity of 96.2% reveal the capacity of the proposed strategy to perform ischemic episodes identification.

Ventricular arrhythmias assessment.

An integrated framework for ventricular arrhythmias (VA) assessment, composed of two levels, is proposed in this work. The first level consists of four independent neural networks (NN), designed for specific detection tasks: signal quality, premature ventricular contractions (PVC), ventricular tachycardia (VT) and ventricular fibrillation (VF). Time and frequency domain features, obtained from the electrocardiogram (ECG) and selected through a correlation analysis procedure, form the inputs to the neural modules. The outputs feed the second layer, which consists of a global classifier (ANFIS structure), returns the global result for the VA assessment scheme. Sensitivity and specificity values, evaluated from public MIT-BIH databases, show the effectiveness of the proposed strategy.