Training and Interpreting Machine Learning Algorithms to Evaluate Fall Risk After Emergency Department Visits.

BACKGROUND: Machine learning is increasingly used for risk stratification in health care. Achieving accurate predictive models do not improve outcomes if they cannot be translated into efficacious intervention. Here we examine the potential utility of automated risk stratification and referral intervention to screen older adults for fall risk after emergency department (ED) visits. OBJECTIVE: This study evaluated several machine learning methodologies for the creation of a risk stratification algorithm using electronic health record data and estimated the effects of a resultant intervention based on algorithm performance in test data. METHODS: Data available at the time of ED discharge were retrospectively collected and separated into training and test datasets. Algorithms were developed to predict the outcome of a return visit for fall within 6 months of an ED index visit. Models included random forests, AdaBoost, and regression-based methods. We evaluated models both by the area under the receiver operating characteristic (ROC) curve, also referred to as area under the curve (AUC), and by projected clinical impact, estimating number needed to treat (NNT) and referrals per week for a fall risk intervention. RESULTS: The random forest model achieved an AUC of 0.78, with slightly lower performance in regression-based models. Algorithms with similar performance, when evaluated by AUC, differed when placed into a clinical context with the defined task of estimated NNT in a real-world scenario. CONCLUSION: The ability to translate the results of our analysis to the potential tradeoff between referral numbers and NNT offers decisionmakers the ability to envision the effects of a proposed intervention before implementation.

Human pluripotent stem cell-derived neural constructs for predicting neural toxicity.

Human pluripotent stem cell-based in vitro models that reflect human physiology have the potential to reduce the number of drug failures in clinical trials and offer a cost-effective approach for assessing chemical safety. Here, human embryonic stem (ES) cell-derived neural progenitor cells, endothelial cells, mesenchymal stem cells, and microglia/macrophage precursors were combined on chemically defined polyethylene glycol hydrogels and cultured in serum-free medium to model cellular interactions within the developing brain. The precursors self-assembled into 3D neural constructs with diverse neuronal and glial populations, interconnected vascular networks, and ramified microglia. Replicate constructs were reproducible by RNA sequencing (RNA-Seq) and expressed neurogenesis, vasculature development, and microglia genes. Linear support vector machines were used to construct a predictive model from RNA-Seq data for 240 neural constructs treated with 34 toxic and 26 nontoxic chemicals. The predictive model was evaluated using two standard hold-out testing methods: a nearly unbiased leave-one-out cross-validation for the 60 training compounds and an unbiased blinded trial using a single hold-out set of 10 additional chemicals. The linear support vector produced an estimate for future data of 0.91 in the cross-validation experiment and correctly classified 9 of 10 chemicals in the blinded trial.

Operationalizing a real-time scoring model to predict fall risk among older adults in the emergency department.

Predictive models are increasingly being developed and implemented to improve patient care across a variety of clinical scenarios. While a body of literature exists on the development of models using existing data, less focus has been placed on practical operationalization of these models for deployment in real-time production environments. This case-study describes challenges and barriers identified and overcome in such an operationalization for a model aimed at predicting risk of outpatient falls after Emergency Department (ED) visits among older adults. Based on our experience, we provide general principles for translating an EHR-based predictive model from research and reporting environments into real-time operation.

Machine learning-assisted screening for cognitive impairment in the emergency department.

BACKGROUND/OBJECTIVES: Despite a high prevalence and association with poor outcomes, screening to identify cognitive impairment (CI) in the emergency department (ED) is uncommon. Identification of high-risk subsets of older adults is a critical challenge to expanding screening programs. We developed and evaluated an automated screening tool to identify a subset of patients at high risk for CI. METHODS: In this secondary analysis of existing data collected for a randomized control trial, we developed machine-learning models to identify patients at higher risk of CI using only variables available in electronic health record (EHR). We used records from 1736 community-dwelling adults age > 59 being discharged from three EDs. Potential CI was determined based on the Blessed Orientation Memory Concentration (BOMC) test, administered in the ED. A nested cross-validation framework was used to evaluate machine-learning algorithms, comparing area under the receiver-operator curve (AUC) as the primary metric of performance. RESULTS: Based on BOMC scores, 121 of 1736 (7%) participants screened positive for potential CI at the time of their ED visit. The best performing algorithm, an XGBoost model, predicted BOMC positivity with an AUC of 0.72. With a classification threshold of 0.4, this model had a sensitivity of 0.73, a specificity of 0.64, a negative predictive value of 0.97, and a positive predictive value of 0.13. In a hypothetical ED with 200 older adult visits per week, the use of this model would lead to a decrease in the in-person screening burden from 200 to 77 individuals in order to detect 10 of 14 patients who would fail a BOMC. CONCLUSION: This study demonstrates that an algorithm based on EHR data can define a subset of patients at higher risk for CI. Incorporating such an algorithm into a screening workflow could allow screening efforts and resources to be focused where they have the most impact.

Collaborative design and implementation of a clinical decision support system for automated fall-risk identification and referrals in emergency departments.

Of the 3 million older adults seeking fall-related emergency care each year, nearly one-third visited the Emergency Department (ED) in the previous 6 months. ED providers have a great opportunity to refer patients for fall prevention services at these initial visits, but lack feasible tools for identifying those at highest-risk. Existing fall screening tools have been poorly adopted due to ED staff/provider burden and lack of workflow integration. To address this, we developed an automated clinical decision support (CDS) system for identifying and referring older adult ED patients at risk of future falls. We engaged an interdisciplinary design team (ED providers, health services researchers, information technology/predictive analytics professionals, and outpatient Falls Clinic staff) to collaboratively develop a system that successfully met user requirements and integrated seamlessly into existing ED workflows. Our rapid-cycle development and evaluation process employed a novel combination of human-centered design, implementation science, and patient experience strategies, facilitating simultaneous design of the CDS tool and intervention implementation strategies. This included defining system requirements, systematically identifying and resolving usability problems, assessing barriers and facilitators to implementation (e.g., data accessibility, lack of time, high patient volumes, appointment availability) from multiple vantage points, and refining protocols for communicating with referred patients at discharge. ED physician, nurse, and patient stakeholders were also engaged through online surveys and user testing. Successful CDS design and implementation required integration of multiple new technologies and processes into existing workflows, necessitating interdisciplinary collaboration from the onset. By using this iterative approach, we were able to design and implement an intervention meeting all project goals. Processes used in this Clinical-IT-Research partnership can be applied to other use cases involving automated risk-stratification, CDS development, and EHR-facilitated care coordination.

Iterative processes: a review of semi-supervised machine learning in rehabilitation science.

Purpose: To define semi-supervised machine learning (SSML) and explore current and potential applications of this analytic strategy in rehabilitation research.Method: We conducted a scoping review using PubMed, GoogleScholar and Medline. Studies were included if they: (1) described a semi-supervised approach to apply machine learning algorithms during data analysis and (2) examined constructs encompassed by the International Classification of Functioning, Disability and Health (ICF). The first two authors reviewed identified articles and recorded study and participant characteristics. The ICF domain used in each study was also identified.Results: After combining information from the eight studies, we established that SSML was a feasible approach for analysis of complex data in rehabilitation research. We also determined that semi-supervised approaches may be more accurate than supervised machine learning approaches.Conclusions: A semi-supervised approach to machine learning has potential to enhance our understanding of complex data sets in rehabilitation science. SSML mirrors the iterative process of rehabilitation, making this approach ideal for calibrating devices, classifying activities or identifying just-in-time interventions. Rehabilitation scientists who are interested in conducting SSML should collaborate with data scientists to advance the application of this approach within our field.Implications for rehabilitationSemi-supervised machine learning applications may be a feasible approach for analyses of complex data sets in rehabilitation research.Semi-supervised machine learning approaches uses a combination of labelled and unlabelled data to produce accurate predictive models, thereby requiring less user-input data than other machine learning approaches (i.e., supervised, unsupervised), reducing resource cost and user-burden.Semi-supervised machine learning is an iterative process that, when applied to rehabilitation assessment and outcomes, could produce accurate personalized models for treatment.Rehabilitation researchers and data scientists should collaborate to implement semi-supervised machine learning approaches in rehabilitation research, optimizing the power of large datasets that are becoming more readily available within the field (e.g., EEG signals, sensors, smarthomes).