Evaluation of a Machine Learning-Based Dysphagia Prediction Tool in Clinical Routine: A Prospective Observational Cohort Study.

Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients' risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.

Machine Learning-Based Prediction of Malnutrition in Surgical In-Patients: A Validation Pilot Study.

BACKGROUND: Malnutrition in hospitalised patients can lead to serious complications, worse patient outcomes and longer hospital stays. State-of-the-art screening methods rely on scores, which need additional manual assessments causing higher workload. OBJECTIVES: The aim of this prospective study was to validate a machine learning (ML)-based approach for an automated prediction of malnutrition in hospitalised patients. METHODS: For 159 surgical in-patients, an assessment of malnutrition by dieticians was compared to the ML-based prediction conducted in the evening of admission. RESULTS: The model achieved an accuracy of 83.0% and an AUROC of 0.833 in the prospective validation cohort. CONCLUSION: The results of this pilot study indicate that an automated malnutrition screening could replace manual screening tools in hospitals.

Introduction and Comparison of Novel Decentral Learning Schemes with Multiple Data Pools for Privacy-Preserving ECG Classification.

Artificial intelligence and machine learning have led to prominent and spectacular innovations in various scenarios. Application in medicine, however, can be challenging due to privacy concerns and strict legal regulations. Methods that centralize knowledge instead of data could address this issue. In this work, 6 different decentralized machine learning algorithms are applied to 12-lead ECG classification and compared to conventional, centralized machine learning. The results show that state-of-the-art federated learning leads to reasonable losses of classification performance compared to a standard, central model (-0.054 AUROC) while providing a significantly higher level of privacy. A proposed weighted variant of federated learning (-0.049 AUROC) and an ensemble (-0.035 AUROC) outperformed the standard federated learning algorithm. Overall, considering multiple metrics, the novel batch-wise sequential learning scheme performed best (-0.036 AUROC to baseline). Although, the technical aspects of implementing them in a real-world application are to be carefully considered, the described algorithms constitute a way forward towards preserving-preserving AI in medicine.

A Transformer-Based Model Trained on Large Scale Claims Data for Prediction of Severe COVID-19 Disease Progression.

In situations like the COVID-19 pandemic, healthcare systems are under enormous pressure as they can rapidly collapse under the burden of the crisis. Machine learning (ML) based risk models could lift the burden by identifying patients with a high risk of severe disease progression. Electronic Health Records (EHRs) provide crucial sources of information to develop these models because they rely on routinely collected healthcare data. However, EHR data is challenging for training ML models because it contains irregularly timestamped diagnosis, prescription, and procedure codes. For such data, transformer-based models are promising. We extended the previously published Med-BERT model by including age, sex, medications, quantitative clinical measures, and state information. After pre-training on approximately 988 million EHRs from 3.5 million patients, we developed models to predict Acute Respiratory Manifestations (ARM) risk using the medical history of 80,211 COVID-19 patients. Compared to Random Forests, XGBoost, and RETAIN, our transformer-based models more accurately forecast the risk of developing ARM after COVID-19 infection. We used Integrated Gradients and Bayesian networks to understand the link between the essential features of our model. Finally, we evaluated adapting our model to Austrian in-patient data. Our study highlights the promise of predictive transformer-based models for precision medicine.

Frail People in LABLand: Development of an Easy-to-Use Machine Learning Model to Identify Frail People in Hospitals Based on Laboratory Data.

BACKGROUND: Frail individuals are very vulnerable to stressors, which often lead to adverse outcomes. To ensure an adequate therapy, a holistic diagnostic approach is needed which is provided in geriatric wards. It is important to identify frail individuals outside the geriatric ward as well to ensure that they also benefit from the holistic approach. OBJECTIVES: The goal of this study was to develop a machine learning model to identify frail individuals in hospitals. The model should be applicable without additional effort, quickly and in many different places in the healthcare system. METHODS: We used Gradient Boosting Decision Trees (GBDT) to predict a frailty target derived from a gold standard assessment. The used features were laboratory values, age and sex. We also identified the most important features. RESULTS: The best GBDT achieved an AUROC of 0.696. The most important laboratory values are urea, creatinine, granulocytes, chloride and calcium. CONCLUSION: The model performance is acceptable, but insufficient for clinical use. Additional laboratory values or the laboratory history could improve the performance.

Long-term risk of recurrent cerebrovascular events after patent foramen ovale closure: Results from a real-world stroke cohort.

INTRODUCTION: Patent foramen ovale (PFO)-closure is recommended for stroke prevention in selected patients with suspected PFO-associated stroke. However, studies on cerebrovascular event recurrence after PFO-closure are limited by relatively short follow-up periods and information on the underlying aetiology of recurrent events is scarce. PATIENTS AND METHODS: All consecutive patients with a cerebral ischaemic event and PFO-closure at the University Hospital Graz were prospectively identified from 2004 to 2021. Indication for PFO-closure was based on a neurological-cardiological PFO board decision. Patients underwent standardized clinical and echocardiographic follow-up 6 months after PFO-closure. Recurrent cerebrovascular events were assessed via electronical health records. RESULTS: PFO-closure was performed in 515 patients (median age: 49 years; Amplatzer PFO occluder: 42%). Over a median follow-up of 11 years (range: 2-18 years, 5141 total patient-years), recurrent ischaemic cerebrovascular events were observed in 34 patients (ischaemic stroke: n = 22, TIA: n = 12) and associated with age, hyperlipidaemia and smoking in multivariable analysis (p < 0.05 each). Large artery atherosclerosis and small vessel disease were the most frequent aetiologies of recurrent stroke/TIA (27% and 24% respectively), and only two events were related to atrial fibrillation (AF). Recurrent ischaemic cerebrovascular event rates and incident AF were comparable in patients treated with different PFO occluders (p > 0.1). DISCUSSION AND CONCLUSION: In this long-term follow-up-study of patients with a cerebral ischaemic event who had received PFO-closure with different devices, rates of recurrent stroke/TIA were low and largely related to large artery atherosclerosis and small vessel disease. Thorough vascular risk factor control seems crucial for secondary stroke prevention in patients treated for PFO-related stroke.

Development of an Architecture to Implement Machine Learning Based Risk Prediction in Clinical Routine: A Service-Oriented Approach.

BACKGROUND: Patients at risk of developing a disease have to be identified at an early stage to enable prevention. One way of early detection is the use of machine learning based prediction models trained on electronic health records. OBJECTIVES: The aim of this project was to develop a software solution to predict cardiovascular and nephrological events using machine learning models. In addition, a risk verification interface for health care professionals was established. METHODS: In order to meet the requirements, different tools were analysed. Based on this, a software architecture was created, which was designed to be as modular as possible. RESULTS: A software was realised that is able to automatically calculate and display risks using machine learning models. Furthermore, predictions can be verified via an interface adapted to the need of health care professionals, which shows data required for prediction. CONCLUSION: Due to the modularised software architecture and the status-based calculation process, different technologies could be applied. This facilitates the installation of the software at multiple health care providers, for which adjustments need to be carried out at one part of the software only.

Towards Automated Screening of Literature on Artificial Intelligence in Nursing.

We evaluate the performance of multiple text classification methods used to automate the screening of article abstracts in terms of their relevance to a topic of interest. The aim is to develop a system that can be first trained on a set of manually screened article abstracts before using it to identify additional articles on the same topic. Here the focus is on articles related to the topic "artificial intelligence in nursing". Eight text classification methods are tested, as well as two simple ensemble systems. The results indicate that it is feasible to use text classification technology to support the manual screening process of article abstracts when conducting a literature review. The best results are achieved by an ensemble system, which achieves a F1-score of 0.41, with a sensitivity of 0.54 and a specificity of 0.96. Future work directions are discussed.

External Validation of a Machine Learning Based Delirium Prediction Software in Clinical Routine.

BACKGROUND: Various machine learning (ML) models have been developed for the prediction of clinical outcomes, but there is missing evidence on their performance in clinical routine and external validation. OBJECTIVES: Our aim was to deploy and prospectively evaluate an already developed delirium prediction software in clinical routine of an external hospital. METHODS: We compared updated ML models of the software and models re-trained with the external hospital's data. The best models were deployed in clinical routine for one month, and risk predictions for all admitted patients were compared to the risk ratings of a senior physician. After using the software, clinicians completed a questionnaire assessing technology acceptance. RESULTS: Re-trained models achieved a high discriminative performance (AUROC > 0.92). Compared to clinical risk ratings, the software achieved a sensitivity of 100.0% and a specificity of 90.6%. Usefulness, ease of use and output quality were rated positively by the users. CONCLUSION: A ML based delirium prediction software achieved a high discriminative performance and high technology acceptance at an external hospital using re-trained ML models.

Machine Learning Based Risk Prediction for Major Adverse Cardiovascular Events.

BACKGROUND: Patients with major adverse cardiovascular events (MACE) such as myocardial infarction or stroke suffer from frequent hospitalizations and have high mortality rates. By identifying patients at risk at an early stage, MACE can be prevented with the right interventions. OBJECTIVES: The aim of this study was to develop machine learning-based models for the 5-year risk prediction of MACE. METHODS: The data used for modelling included electronic medical records of more than 128,000 patients including 29,262 patients with MACE. A feature selection based on filter and embedded methods resulted in 826 features for modelling. Different machine learning methods were used for modelling on the training data. RESULTS: A random forest model achieved the best calibration and discriminative performance on a separate test data set with an AUROC of 0.88. CONCLUSION: The developed risk prediction models achieved an excellent performance in the test data. Future research is needed to determine the performance of these models and their clinical benefit in prospective settings.

How to Improve Information Technology to Support Healthcare to Address the COVID-19 Pandemic: an International Survey with Health Informatics Experts.

OBJECTIVES: To identify the ways in which healthcare information and communication technologies can be improved to address the challenges raised by the COVID-19 pandemic. METHODS: The study population included health informatics experts who had been involved with the planning, development and deployment of healthcare information and communication technologies in healthcare settings in response to the challenges presented by the COVID-19 pandemic. Data were collected via an online survey. A non-probability convenience sampling strategy was employed. Data were analyzed with content analysis. RESULTS: A total of 65 participants from 16 countries responded to the conducted survey. The four major themes regarding recommended improvements identified from the content analysis included: improved technology availability, improved interoperability, intuitive user interfaces and adoption of standards of care. Respondents also identified several key healthcare information and communication technologies that can help to provide better healthcare to patients during the COVID-19 pandemic, including telehealth, advanced software, electronic health records, remote work technologies (e.g., remote desktop computer access), and clinical decision support tools. CONCLUSIONS: Our results help to identify several important healthcare information and communication technologies, recommended by health informatics experts, which can help to provide better care to patients during the COVID-19 pandemic. The results also highlight the need for improved interoperability, intuitive user interfaces and advocating the adoption of standards of care.

Predictive Modelling and Its Visualization for Telehealth Data - Concept and Implementation of an Interactive Viewer.

BACKGROUND: Predictive modelling is becoming increasingly important in the healthcare sector. A comprehensive understanding of obtained models and their predictions is indispensable for the development and later acceptance of such systems. OBJECTIVES: A general concept of a toolset that supports data scientists in the development of predictive models in the telehealth context had to be developed and subsequently implemented. METHODS: Based on surveys the user requirements were determined. The concept development was based on the data model of the 'HerzMobil Tirol' telehealth program. The implementation was conducted in MATLAB. RESULTS: A list of requirements was identified, based on which a viewer was implemented. CONCLUSION: The developed viewer concept and its implementation facilitate a deeper insight and a better understanding of the development process of predictive models in the telehealth context.

An Improvised Classification Model for Predicting Delirium.

With the vast increase of digital healthcare data, there is an opportunity to mine the data for understanding inherent health patterns. Although machine-learning techniques demonstrated their applications in healthcare to answer several questions, there is still room for improvement in every aspect. In this paper, we are demonstrating a method that improves the performance of a delirium prediction model using random forest in combination with logistic regression.

Information Adapted Machine Learning Models for Prediction in Clinical Workflow.

BACKGROUND: In a database of electronic health records, the amount of available information varies widely between patients. In a real-time prediction scenario, a machine learning model may receive limited information for some patients. OBJECTIVES: Our aim was to evaluate the influence of missing data on real-time prediction of delirium, and detect changes in prediction performance when training separate models for patients with missing data. METHODS: We compared a model trained specifically on data with missing values to the currently implemented model predicting delirium. Also, we simulated five test data sets with different amount of missing data and compared the prediction results to the prediction on complete data set when using the same model. RESULTS: For patients with missing laboratory and nursing assessment data, a model trained especially for this scenario performed significantly better than the implemented model. The combination of procedure data and demographic data achieved the closest results to a prediction with a complete data set. CONCLUSION: An ongoing evaluation of real-time prediction is indispensable. Additional models adapted to the information available might improve prediction performance.

Is Regular Re-Training of a Predictive Delirium Model Necessary After Deployment in Routine Care?

Adoption of electronic medical records in hospitals generates a large amount of data. Health care professionals can easily lose their sight on the important insights of the patients' clinical and medical history. Although machine learning algorithms have already proved their significance in healthcare research, remains a challenge translation and dissemination of fully automated prediction algorithms from research to decision support at the point of care. In this paper, we address the effect of changes in the characteristics of data over time on the performance of deployed models for the use case of predicting delirium in hospitalised patients. We have analysed the stability of models trained with subsets of data from one single year (2012, 2013...2016, respectively), and tested the models with data from 2017. Our results show that in the case of delirium prediction, the models were stable over time, indicating that re-training the models is not necessary e.g. once per year might be more than sufficient.

Patient Record Linkage for Data Quality Assessment Based on Time Series Matching.

BACKGROUND: Huge amounts of data are collected by healthcare providers and other institutions. However, there are data protection regulations, which limit their utilisation for secondary use, e.g. RESEARCH: In scenarios, where several data sources are obtained without universal identifiers, record linkage methods need to be applied to obtain a comprehensive dataset. OBJECTIVES: In this study, we had the objective to link two datasets comprising data from ergometric performance tests in order to have reference values to free text annotations for assessing their data quality. METHODS: We applied an iterative, distance-based time series record linkage algorithm to find corresponding entries in the two given datasets. Subsequently, we assessed the resulting matching rate. The implementation was done in Matlab. RESULTS: The matching rate of our record linkage algorithm was 74.5% for matching patients' records with their ergometry records. The highest rate of appropriate free text annotations was 87.9%. CONCLUSION: For the given scenario, our algorithm matched 74.5% of the patients. However, we had no gold standard for validating our results. Most of the free text annotations contained the expected values.

Utilising Information of the Case Fee Catalogue to Enhance 30-Day Readmission Prediction in the German DRG System.

Unplanned hospital readmissions are a burden to the healthcare system and to the patients. To lower the readmission rates, machine learning approaches can be used to create predictive models, with the intention to provide actionable information for caregivers. According to the German Diagnosis Related Groups (G-DRG) system, for every stay in a German hospital, data are collected for the subsequent reimbursement calculations. After statistical evaluation, these data are summarised in the yearly updated Case Fee Catalogue, which not only contains the weights for the reimbursement calculations, but also the expected length of stay values. The aim of the present paper was to evaluate potential enhancements of the prediction accuracy of our 30-day readmission prediction model by utilising additional information from the Case Fee Catalogue. A bagged ensemble of 25 regression trees was applied to §21 datasets from five independent German hospitals from 2013 to 2017, resulting in 422,597 cases. The overall model showed an area under the receiver operating characteristics curve of 0.812. Three of the top five features ranked by out of bag feature importance emerged from the Case Fee Catalogue. We conclude, that additional information from the Case Fee Catalogue can enhance the accuracy of 30-day readmission prediction.

Prediction of Readmissions in the German DRG System Based on §21 Datasets.

Hospital readmissions receive increasing interest, since they are burdensome for patients and costly for healthcare providers. For the calculation of reimbursement fees, in Germany there is the German-Diagnosis Related Groups (G-DRG) system. For every hospital stay, data are collected as a so-called "case", as the basis for the subsequent reimbursement calculations ("§21 dataset"). Merging rules lead to a loss of information in §21 datasets. We applied machine learning to §21 datasets and evaluated the influence of case merging for the resulting accuracy of readmission risk prediction. Data from 478,966 cases were analysed by applying a random forest. Many cases with readmissions within 30 days had been merged and thus their prediction required additional data. Using 10-fold cross validation, the prediction for readmissions within 31-60 days showed no notable difference in the area under the ROC curves comparing unedited §21 datasets with §21 datasets with restored original cases. The achieved AUC values of 0.69 lie in a similar range as the values of comparable state-of-the-art models. We conclude that dealing with merged cases, i.e. adding data, is required for 30-day-readmission prediction, whereas un-merging brings no improvement for the readmission prediction of period beyond 30 days.

Effect of Nursing Assessment on Predictive Delirium Models in Hospitalised Patients.

Delirium is an acute neuropsychiatric syndrome which is common in elderly patients during their hospitalisation and is associated with an increased mortality and morbidity. Since delirium is a) often underdiagnosed and b) preventable if early signs are detected,igh expectations are set in delirium risk assessment during hospital admission. In our latest studies, we showed that delirium prediction using machine learning algorithms is possible based on the patients' health history. The aim of this study is to compare the influence of nursing assessment data on prediction models with clinical and demographic data. We approached the problem by a) comparing the performance of predictive models including nursing data with models based on clinical and demographic data only and b) analysing the feature importance of all available features. From our results we concluded that nursing assessment data can improve the performance of delirium prediction models better than demographic, laboratory, diagnosis, procedures, and previous transfers' data alone.

On the Representation of Machine Learning Results for Delirium Prediction in a Hospital Information System in Routine Care.

Digitalisation of health care for the purpose of medical documentation lead to huge amounts of data, hence having an opportunity to derive knowledge and associations of different attributes recorded. Many health care events can be prevented when identified. Machine learning algorithms could identify such events but there is ambiguity in understanding the suggestions especially in clinical setup. In this paper we are presenting how we explain the decision based on random forest to health care professionals in the course of the project predicting delirium during hospitalisation on the day of admission.