Clinical evaluation of a machine learning-based dysphagia risk prediction tool.

PURPOSE: The rise of digitization promotes the development of screening and decision support tools. We sought to validate the results from a machine learning based dysphagia risk prediction tool with clinical evaluation. METHODS: 149 inpatients in the ENT department were evaluated in real time by the risk prediction tool, as well as clinically over a 3-week period. Patients were classified by both as patients at risk/no risk. RESULTS: The AUROC, reflecting the discrimination capability of the algorithm, was 0.97. The accuracy achieved 92.6% given an excellent specificity as well as sensitivity of 98% and 82.4% resp. Higher age, as well as male sex and the diagnosis of oropharyngeal malignancies were found more often in patients at risk of dysphagia. CONCLUSION: The proposed dysphagia risk prediction tool proved to have an outstanding performance in discriminating risk from no risk patients in a prospective clinical setting. It is likely to be particularly useful in settings where there is a lower incidence of patients with dysphagia and less awareness among staff.

Evaluation of a Machine Learning-Based Dysphagia Prediction Tool in Clinical Routine: A Prospective Observational Cohort Study.

Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients' risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.

Technology Acceptance of a Machine Learning Algorithm Predicting Delirium in a Clinical Setting: a Mixed-Methods Study.

Early identification of patients with life-threatening risks such as delirium is crucial in order to initiate preventive actions as quickly as possible. Despite intense research on machine learning for the prediction of clinical outcomes, the acceptance of the integration of such complex models in clinical routine remains unclear. The aim of this study was to evaluate user acceptance of an already implemented machine learning-based application predicting the risk of delirium for in-patients. We applied a mixed methods design to collect opinions and concerns from health care professionals including physicians and nurses who regularly used the application. The evaluation was framed by the Technology Acceptance Model assessing perceived ease of use, perceived usefulness, actual system use and output quality of the application. Questionnaire results from 47 nurses and physicians as well as qualitative results of four expert group meetings rated the overall usefulness of the delirium prediction positively. For healthcare professionals, the visualization and presented information was understandable, the application was easy to use and the additional information for delirium management was appreciated. The application did not increase their workload, but the actual system use was still low during the pilot study. Our study provides insights into the user acceptance of a machine learning-based application supporting delirium management in hospitals. In order to improve quality and safety in healthcare, computerized decision support should predict actionable events and be highly accepted by users.

A novel hybrid modeling approach for the evaluation of integrated care and economic outcome in heart failure treatment.

BACKGROUND: Demographic changes, increased life expectancy and the associated rise in chronic diseases pose challenges to public health care systems. Optimized treatment methods and integrated concepts of care are potential solutions to overcome increasing financial burdens and improve quality of care. In this context modeling is a powerful tool to evaluate potential benefits of different treatment procedures on health outcomes as well as health care budgets. This work presents a novel modeling approach for simulating different treatment procedures of heart failure patients based on extensive data sets from outpatient and inpatient care. METHODS: Our hybrid heart failure model is based on discrete event and agent based methodologies and facilitates the incorporation of different therapeutic procedures for outpatient and inpatient care on patient individual level. The state of health is modeled with the functional classification of the New York Heart Association (NYHA), strongly affecting discrete state transition probabilities alongside age and gender. Cooperation with Austrian health care and health insurance providers allowed the realization of a detailed model structure based on clinical data of more than 25,000 patients. RESULTS: Simulation results of conventional care and a telemonitoring program underline the unfavorable prognosis for heart failure and reveal the correlation of NYHA classes with health and economic outcomes. Average expenses for the treatment of NYHA class IV patients of €10,077 ± €165 were more than doubled compared to other classes. The selected use case of a telemonitoring program demonstrated potential cost savings within two years of application. NYHA classes II and III revealed most potential for additional treatment measures. CONCLUSION: The presented model allows performing extensive simulations of established treatment procedures for heart failure patients and evaluating new holistic methods of care and innovative study designs. This approach offers health care providers a unique, adaptable and comprehensive tool for decision making in the complex and socioeconomically challenging field of cardiovascular diseases.

EHR problem list clustering for improved topic-space navigation.

BACKGROUND: The amount of patient-related information within clinical information systems accumulates over time, especially in cases where patients suffer from chronic diseases with many hospitalizations and consultations. The diagnosis or problem list is an important feature of the electronic health record, which provides a dynamic account of a patient's current illness and past history. In the case of an Austrian hospital network, problem list entries are limited to fifty characters and are potentially linked to ICD-10. The requirement of producing ICD codes at each hospital stay, together with the length limitation of list items leads to highly redundant problem lists, which conflicts with the physicians' need of getting a good overview of a patient in short time. This paper investigates a method, by which problem list items can be semantically grouped, in order to allow for fast navigation through patient-related topic spaces. METHODS: We applied a minimal language-dependent preprocessing strategy and mapped problem list entries as tf-idf weighted character 3-grams into a numerical vector space. Based on this representation we used the unweighted pair group method with arithmetic mean (UPGMA) clustering algorithm with cosine distances and inferred an optimal boundary in order to form semantically consistent topic spaces, taking into consideration different levels of dimensionality reduction via latent semantic analysis (LSA). RESULTS: With the proposed clustering approach, evaluated via an intra- and inter-patient scenario in combination with a natural language pipeline, we achieved an average compression rate of 80% of the initial list items forming consistent semantic topic spaces with an F-measure greater than 0.80 in both cases. The average number of identified topics in the intra-patient case (µIntra = 78.4) was slightly lower than in the inter-patient case (µInter = 83.4). LSA-based feature space reduction had no significant positive performance impact in our investigations. CONCLUSIONS: The investigation presented here is centered on a data-driven solution to the known problem of information overload, which causes ineffective human-computer interactions at clinicians' work places. This problem is addressed by navigable disease topic spaces where related items are grouped and the topics can be more easily accessed.

Machine learning-based delirium prediction in surgical in-patients: a prospective validation study.

Objective: Delirium is a syndrome that leads to severe complications in hospitalized patients, but is considered preventable in many cases. One of the biggest challenges is to identify patients at risk in a hectic clinical routine, as most screening tools cause additional workload. The aim of this study was to validate a machine learning (ML)-based delirium prediction tool on surgical in-patients undergoing a systematic assessment of delirium. Materials and Methods: 738 in-patients of a vascular surgery, a trauma surgery and an orthopedic surgery department were screened for delirium using the DOS scale twice a day over their hospital stay. Concurrently, delirium risk was predicted by the ML algorithm in real-time for all patients at admission and evening of admission. The prediction was performed automatically based on existing EHR data and without any additional documentation needed. Results: 103 patients (14.0%) were screened positive for delirium using the DOS scale. Out of them, 85 (82.5%) were correctly identified by the ML algorithm. Specificity was slightly lower, detecting 463 (72.9%) out of 635 patients without delirium. The AUROC of the algorithm was 0.883 (95% CI, 0.8523-0.9147). Discussion: In this prospective validation study, the implemented machine-learning algorithm was able to detect patients with delirium in surgical departments with high discriminative performance. Conclusion: In future, this tool or similar decision support systems may help to replace time-intensive screening tools and enable efficient prevention of delirium.

Machine Learning-Based Prediction of Malnutrition in Surgical In-Patients: A Validation Pilot Study.

BACKGROUND: Malnutrition in hospitalised patients can lead to serious complications, worse patient outcomes and longer hospital stays. State-of-the-art screening methods rely on scores, which need additional manual assessments causing higher workload. OBJECTIVES: The aim of this prospective study was to validate a machine learning (ML)-based approach for an automated prediction of malnutrition in hospitalised patients. METHODS: For 159 surgical in-patients, an assessment of malnutrition by dieticians was compared to the ML-based prediction conducted in the evening of admission. RESULTS: The model achieved an accuracy of 83.0% and an AUROC of 0.833 in the prospective validation cohort. CONCLUSION: The results of this pilot study indicate that an automated malnutrition screening could replace manual screening tools in hospitals.

Secondary Use of Clinical Problem List Entries for Neural Network-Based Disease Code Assignment.

Clinical information systems have become large repositories for semi-structured and partly annotated electronic health record data, which have reached a critical mass that makes them interesting for supervised data-driven neural network approaches. We explored automated coding of 50 character long clinical problem list entries using the International Classification of Diseases (ICD-10) and evaluated three different types of network architectures on the top 100 ICD-10 three-digit codes. A fastText baseline reached a macro-averaged F1-score of 0.83, followed by a character-level LSTM with a macro-averaged F1-score of 0.84. The top performing approach used a downstreamed RoBERTa model with a custom language model, yielding a macro-averaged F1-score of 0.88. A neural network activation analysis together with an investigation of the false positives and false negatives unveiled inconsistent manual coding as a main limiting factor.

A Transformer-Based Model Trained on Large Scale Claims Data for Prediction of Severe COVID-19 Disease Progression.

In situations like the COVID-19 pandemic, healthcare systems are under enormous pressure as they can rapidly collapse under the burden of the crisis. Machine learning (ML) based risk models could lift the burden by identifying patients with a high risk of severe disease progression. Electronic Health Records (EHRs) provide crucial sources of information to develop these models because they rely on routinely collected healthcare data. However, EHR data is challenging for training ML models because it contains irregularly timestamped diagnosis, prescription, and procedure codes. For such data, transformer-based models are promising. We extended the previously published Med-BERT model by including age, sex, medications, quantitative clinical measures, and state information. After pre-training on approximately 988 million EHRs from 3.5 million patients, we developed models to predict Acute Respiratory Manifestations (ARM) risk using the medical history of 80,211 COVID-19 patients. Compared to Random Forests, XGBoost, and RETAIN, our transformer-based models more accurately forecast the risk of developing ARM after COVID-19 infection. We used Integrated Gradients and Bayesian networks to understand the link between the essential features of our model. Finally, we evaluated adapting our model to Austrian in-patient data. Our study highlights the promise of predictive transformer-based models for precision medicine.

Machine Learning Based Risk Prediction for Major Adverse Cardiovascular Events for ELGA-Authorized Clinics1.

BACKGROUND: Artificial Intelligence (AI) has had an important impact on many industries as well as the field of medical diagnostics. In healthcare, AI techniques such as case-based reasoning and data driven machine learning (ML) algorithms have been used to support decision-making processes for complex tasks. This is used to assist medical professionals in making clinical decisions. A way of supporting clinicians is providing predicted prognoses of various ML models. OBJECTIVES: Training an ML model based on the data of a hospital and using it on another hospital have some challenges. METHODS: In this research, we applied data analysis to discover required data filters on a hospital's EHR data for training a model for another hospital. RESULTS: We applied experiments on real-world data of ELGA (Austrian health record system) and KAGes (a public healthcare provider of 20+ hospitals in Austria). In this scenario, we train the prediction model for ELGA- authorized health service providers using the KAGes data since we do not have access to the complete ELGA data. CONCLUSION: Finally, we observed that filtering the data with both feature and value selection increases the classification performance of the prediction model, which is trained for another system.

Frail People in LABLand: Development of an Easy-to-Use Machine Learning Model to Identify Frail People in Hospitals Based on Laboratory Data.

BACKGROUND: Frail individuals are very vulnerable to stressors, which often lead to adverse outcomes. To ensure an adequate therapy, a holistic diagnostic approach is needed which is provided in geriatric wards. It is important to identify frail individuals outside the geriatric ward as well to ensure that they also benefit from the holistic approach. OBJECTIVES: The goal of this study was to develop a machine learning model to identify frail individuals in hospitals. The model should be applicable without additional effort, quickly and in many different places in the healthcare system. METHODS: We used Gradient Boosting Decision Trees (GBDT) to predict a frailty target derived from a gold standard assessment. The used features were laboratory values, age and sex. We also identified the most important features. RESULTS: The best GBDT achieved an AUROC of 0.696. The most important laboratory values are urea, creatinine, granulocytes, chloride and calcium. CONCLUSION: The model performance is acceptable, but insufficient for clinical use. Additional laboratory values or the laboratory history could improve the performance.

Long-term risk of recurrent cerebrovascular events after patent foramen ovale closure: Results from a real-world stroke cohort.

INTRODUCTION: Patent foramen ovale (PFO)-closure is recommended for stroke prevention in selected patients with suspected PFO-associated stroke. However, studies on cerebrovascular event recurrence after PFO-closure are limited by relatively short follow-up periods and information on the underlying aetiology of recurrent events is scarce. PATIENTS AND METHODS: All consecutive patients with a cerebral ischaemic event and PFO-closure at the University Hospital Graz were prospectively identified from 2004 to 2021. Indication for PFO-closure was based on a neurological-cardiological PFO board decision. Patients underwent standardized clinical and echocardiographic follow-up 6 months after PFO-closure. Recurrent cerebrovascular events were assessed via electronical health records. RESULTS: PFO-closure was performed in 515 patients (median age: 49 years; Amplatzer PFO occluder: 42%). Over a median follow-up of 11 years (range: 2-18 years, 5141 total patient-years), recurrent ischaemic cerebrovascular events were observed in 34 patients (ischaemic stroke: n = 22, TIA: n = 12) and associated with age, hyperlipidaemia and smoking in multivariable analysis (p < 0.05 each). Large artery atherosclerosis and small vessel disease were the most frequent aetiologies of recurrent stroke/TIA (27% and 24% respectively), and only two events were related to atrial fibrillation (AF). Recurrent ischaemic cerebrovascular event rates and incident AF were comparable in patients treated with different PFO occluders (p > 0.1). DISCUSSION AND CONCLUSION: In this long-term follow-up-study of patients with a cerebral ischaemic event who had received PFO-closure with different devices, rates of recurrent stroke/TIA were low and largely related to large artery atherosclerosis and small vessel disease. Thorough vascular risk factor control seems crucial for secondary stroke prevention in patients treated for PFO-related stroke.

Development of an Architecture to Implement Machine Learning Based Risk Prediction in Clinical Routine: A Service-Oriented Approach.

BACKGROUND: Patients at risk of developing a disease have to be identified at an early stage to enable prevention. One way of early detection is the use of machine learning based prediction models trained on electronic health records. OBJECTIVES: The aim of this project was to develop a software solution to predict cardiovascular and nephrological events using machine learning models. In addition, a risk verification interface for health care professionals was established. METHODS: In order to meet the requirements, different tools were analysed. Based on this, a software architecture was created, which was designed to be as modular as possible. RESULTS: A software was realised that is able to automatically calculate and display risks using machine learning models. Furthermore, predictions can be verified via an interface adapted to the need of health care professionals, which shows data required for prediction. CONCLUSION: Due to the modularised software architecture and the status-based calculation process, different technologies could be applied. This facilitates the installation of the software at multiple health care providers, for which adjustments need to be carried out at one part of the software only.

External Validation of a Machine Learning Based Delirium Prediction Software in Clinical Routine.

BACKGROUND: Various machine learning (ML) models have been developed for the prediction of clinical outcomes, but there is missing evidence on their performance in clinical routine and external validation. OBJECTIVES: Our aim was to deploy and prospectively evaluate an already developed delirium prediction software in clinical routine of an external hospital. METHODS: We compared updated ML models of the software and models re-trained with the external hospital's data. The best models were deployed in clinical routine for one month, and risk predictions for all admitted patients were compared to the risk ratings of a senior physician. After using the software, clinicians completed a questionnaire assessing technology acceptance. RESULTS: Re-trained models achieved a high discriminative performance (AUROC > 0.92). Compared to clinical risk ratings, the software achieved a sensitivity of 100.0% and a specificity of 90.6%. Usefulness, ease of use and output quality were rated positively by the users. CONCLUSION: A ML based delirium prediction software achieved a high discriminative performance and high technology acceptance at an external hospital using re-trained ML models.

Machine Learning Based Risk Prediction for Major Adverse Cardiovascular Events.

BACKGROUND: Patients with major adverse cardiovascular events (MACE) such as myocardial infarction or stroke suffer from frequent hospitalizations and have high mortality rates. By identifying patients at risk at an early stage, MACE can be prevented with the right interventions. OBJECTIVES: The aim of this study was to develop machine learning-based models for the 5-year risk prediction of MACE. METHODS: The data used for modelling included electronic medical records of more than 128,000 patients including 29,262 patients with MACE. A feature selection based on filter and embedded methods resulted in 826 features for modelling. Different machine learning methods were used for modelling on the training data. RESULTS: A random forest model achieved the best calibration and discriminative performance on a separate test data set with an AUROC of 0.88. CONCLUSION: The developed risk prediction models achieved an excellent performance in the test data. Future research is needed to determine the performance of these models and their clinical benefit in prospective settings.

Multivariable Risk Prediction of Dysphagia in Hospitalized Patients Using Machine Learning.

BACKGROUND: Dysphagia is a dysfunction of the swallowing act and is highly prevalent in acute post-stroke patients and patients with chronic neurological diseases. Dysphagia is associated with several potentially life threatening complications. Thus, an early identification and treatment could reduce morbidity and mortality rates. OBJECTIVES: The aim of the study was to develop a multivariable model predicting the individual risk of dysphagia in hospitalized patients. METHODS: We trained different machine learning algorithms on the electronic health records of over 33,000 patients. RESULTS: The tree-based Random Forest Classifier and Adaboost Classifier algorithms achieved an area under the receiver operating characteristic curve of 0.94. CONCLUSION: The developed models outperformed previously published models predicting dysphagia. In future, an implementation in the clinical workflow is needed to determine the clinical benefit.

Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study.

OBJECTIVE: Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest-based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting. MATERIALS AND METHODS: Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting. RESULTS: During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r = 0.81) and nonblinded (r = 0.62) settings. A major advantage of our setting was the timely prediction without additional data entry. DISCUSSION: The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals. CONCLUSIONS: Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.

Waste Disposal Sites as All-You-Can Eat Buffets for Carrion Crows (Corvus corone).

In cities and densely populated areas, several corvid species are considered nuisance animals. In Austria, particularly carrion (Corvus corone) and hooded crows (C. cornix) are regarded as pests by the general public that frequently cause damage to crops, feed on human waste, and thus spread trash. We conducted a detailed one-year field survey to estimate the abundance of carrion crows in relation to potential anthropogenic food sources and reference sites in the Austrian Rhine valley. Our results demonstrated that the number and proximity of waste management facilities, animal feeding areas, and agricultural areas, and the productive capacity of agricultural areas, predominantly influenced habitat choice and abundance of carrion crows. In the current study, the probability of observing more than two carrion crows at a survey site decreased with increasing human population density. Moreover, the abundance of crows increased despite a continuous increase in crow hunting kills registered during the past 25 years. Our study suggests a regionally comprehensive waste management plan could serve as a promising strategy to manage nuisance birds. A reduction in anthropogenic food supply through improved waste management practices is required for long-term, sustainable management to limit the abundance of crow populations in and close to urban environments.

Development of a Machine Learning Model Predicting an ICU Admission for Patients with Elective Surgery and Its Prospective Validation in Clinical Practice.

Frequent utilization of the Intensive Care Unit (ICU) is associated with higher costs and decreased availability for patients who urgently need it. Common risk assessment tool, like the ASA score, lack objectivity and do account only for some influencing parameters. The aim of our study was (1) to develop a reliable machine learning model predicting ICU admission risk after elective surgery, and (2) to implement it in a clinical workflow. We used electronic medical records from more than 61,000 patients for modelling. A random forest model outperformed other methods with an area under the curve of 0.91 in the retrospective test set. In the prospective implementation, the model achieved a sensitivity of 73.3% and a specificity of 80.8%. Further research is essential to determine physicians' attitudes to machine learning models and assess the long term improvement of ICU management.