Grains of Sand to Clinical Pearls: Realizing the Potential of Wearable Data.

Despite the rapid growth of wearables as a consumer technology sector and a growing evidence base supporting their use, they have been slow to be adopted by the health system into clinical care. As regulatory, reimbursement, and technical barriers recede, a persistent challenge remains how to make wearable data actionable for clinicians-transforming disconnected grains of wearable data into meaningful clinical "pearls". In order to bridge this adoption gap, wearable data must become visible, interpretable, and actionable for the clinician. We showcase emerging trends and best practices that illustrate these 3 pillars, and offer some recommendations on how the ecosystem can move forward.

User-centred design for machine learning in health care: a case study from care management.

OBJECTIVES: Few machine learning (ML) models are successfully deployed in clinical practice. One of the common pitfalls across the field is inappropriate problem formulation: designing ML to fit the data rather than to address a real-world clinical pain point. METHODS: We introduce a practical toolkit for user-centred design consisting of four questions covering: (1) solvable pain points, (2) the unique value of ML (eg, automation and augmentation), (3) the actionability pathway and (4) the model's reward function. This toolkit was implemented in a series of six participatory design workshops with care managers in an academic medical centre. RESULTS: Pain points amenable to ML solutions included outpatient risk stratification and risk factor identification. The endpoint definitions, triggering frequency and evaluation metrics of the proposed risk scoring model were directly influenced by care manager workflows and real-world constraints. CONCLUSIONS: Integrating user-centred design early in the ML life cycle is key for configuring models in a clinically actionable way. This toolkit can guide problem selection and influence choices about the technical setup of the ML problem.

Expanding the Secondary Use of Prostate Cancer Real World Data: Automated Classifiers for Clinical and Pathological Stage.

Background: Explicit documentation of stage is an endorsed quality metric by the National Quality Forum. Clinical and pathological cancer staging is inconsistently recorded within clinical narratives but can be derived from text in the Electronic Health Record (EHR). To address this need, we developed a Natural Language Processing (NLP) solution for extraction of clinical and pathological TNM stages from the clinical notes in prostate cancer patients. Methods: Data for patients diagnosed with prostate cancer between 2010 and 2018 were collected from a tertiary care academic healthcare system's EHR records in the United States. This system is linked to the California Cancer Registry, and contains data on diagnosis, histology, cancer stage, treatment and outcomes. A randomly selected sample of patients were manually annotated for stage to establish the ground truth for training and validating the NLP methods. For each patient, a vector representation of clinical text (written in English) was used to train a machine learning model alongside a rule-based model and compared with the ground truth. Results: A total of 5,461 prostate cancer patients were identified in the clinical data warehouse and over 30% were missing stage information. Thirty-three to thirty-six percent of patients were missing a clinical stage and the models accurately imputed the stage in 21-32% of cases. Twenty-one percent had a missing pathological stage and using NLP 71% of missing T stages and 56% of missing N stages were imputed. For both clinical and pathological T and N stages, the rule-based NLP approach out-performed the ML approach with a minimum F1 score of 0.71 and 0.40, respectively. For clinical M stage the ML approach out-performed the rule-based model with a minimum F1 score of 0.79 and 0.88, respectively. Conclusions: We developed an NLP pipeline to successfully extract clinical and pathological staging information from clinical narratives. Our results can serve as a proof of concept for using NLP to augment clinical and pathological stage reporting in cancer registries and EHRs to enhance the secondary use of these data.

Development and Implementation of a Real-time Bundle-adherence Dashboard for Central Line-associated Bloodstream Infections.

INTRODUCTION: Central line-associated bloodstream infections (CLABSIs) are the most common hospital-acquired infection in pediatric patients. High adherence to the CLABSI bundle mitigates CLABSIs. At our institution, there did not exist a hospital-wide system to measure bundle-adherence. We developed an electronic dashboard to monitor CLABSI bundle-adherence across the hospital and in real time. METHODS: Institutional stakeholders and areas of opportunity were identified through interviews and data analyses. We created a data pipeline to pull adherence data from twice-daily bundle checks and populate a dashboard in the electronic health record. The dashboard was developed to allow visualization of overall and individual element bundle-adherence across units. Monthly dashboard accesses and element-level bundle-adherence were recorded, and the nursing staff's feedback about the dashboard was obtained. RESULTS: Following deployment in September 2018, the dashboard was primarily accessed by quality improvement, clinical effectiveness and analytics, and infection prevention and control. Quality improvement and infection prevention and control specialists presented dashboard data at improvement meetings to inform unit-level accountability initiatives. All-element adherence across the hospital increased from 25% in September 2018 to 44% in December 2019, and average adherence to each bundle element increased between 2018 and 2019. CONCLUSIONS: CLABSI bundle-adherence, overall and by element, increased across the hospital following the deployment of a real-time electronic data dashboard. The dashboard enabled population-level surveillance of CLABSI bundle-adherence that informed bundle accountability initiatives. Data transparency enabled by electronic dashboards promises to be a useful tool for infectious disease control.

Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records.

Early prediction of patient outcomes is important for targeting preventive care. This protocol describes a practical workflow for developing deep-learning risk models that can predict various clinical and operational outcomes from structured electronic health record (EHR) data. The protocol comprises five main stages: formal problem definition, data pre-processing, architecture selection, calibration and uncertainty, and generalizability evaluation. We have applied the workflow to four endpoints (acute kidney injury, mortality, length of stay and 30-day hospital readmission). The workflow can enable continuous (e.g., triggered every 6 h) and static (e.g., triggered at 24 h after admission) predictions. We also provide an open-source codebase that illustrates some key principles in EHR modeling. This protocol can be used by interdisciplinary teams with programming and clinical expertise to build deep-learning prediction models with alternate data sources and prediction tasks.

Reporting of demographic data and representativeness in machine learning models using electronic health records.

OBJECTIVE: The development of machine learning (ML) algorithms to address a variety of issues faced in clinical practice has increased rapidly. However, questions have arisen regarding biases in their development that can affect their applicability in specific populations. We sought to evaluate whether studies developing ML models from electronic health record (EHR) data report sufficient demographic data on the study populations to demonstrate representativeness and reproducibility. MATERIALS AND METHODS: We searched PubMed for articles applying ML models to improve clinical decision-making using EHR data. We limited our search to papers published between 2015 and 2019. RESULTS: Across the 164 studies reviewed, demographic variables were inconsistently reported and/or included as model inputs. Race/ethnicity was not reported in 64%; gender and age were not reported in 24% and 21% of studies, respectively. Socioeconomic status of the population was not reported in 92% of studies. Studies that mentioned these variables often did not report if they were included as model inputs. Few models (12%) were validated using external populations. Few studies (17%) open-sourced their code. Populations in the ML studies include higher proportions of White and Black yet fewer Hispanic subjects compared to the general US population. DISCUSSION: The demographic characteristics of study populations are poorly reported in the ML literature based on EHR data. Demographic representativeness in training data and model transparency is necessary to ensure that ML models are deployed in an equitable and reproducible manner. Wider adoption of reporting guidelines is warranted to improve representativeness and reproducibility.

Predicting the Incidence of Pressure Ulcers in the Intensive Care Unit Using Machine Learning.

BACKGROUND: Reducing hospital-acquired pressure ulcers (PUs) in intensive care units (ICUs) has emerged as an important quality metric for health systems internationally. Limited work has been done to characterize the profile of PUs in the ICU using observational data from the electronic health record (EHR). Consequently, there are limited EHR-based prognostic tools for determining a patient's risk of PU development, with most institutions relying on nurse-calculated risk scores such as the Braden score to identify high-risk patients. METHODS AND RESULTS: Using EHR data from 50,851 admissions in a tertiary ICU (MIMIC-III), we show that the prevalence of PUs at stage 2 or above is 7.8 percent. For the 1,690 admissions where a PU was recorded on day 2 or beyond, we evaluated the prognostic value of the Braden score measured within the first 24 hours. A high-risk Braden score (<=12) had precision 0.09 and recall 0.50 for the future development of a PU. We trained a range of machine learning algorithms using demographic parameters, diagnosis codes, laboratory values and vitals available from the EHR within the first 24 hours. A weighted linear regression model showed precision 0.09 and recall 0.71 for future PU development. Classifier performance was not improved by integrating Braden score elements into the model. CONCLUSION: We demonstrate that an EHR-based model can outperform the Braden score as a screening tool for PUs. This may be a useful tool for automatic risk stratification early in an admission, helping to guide quality protocols in the ICU, including the allocation and timing of prophylactic interventions.

Machine Learning Approaches for Extracting Stage from Pathology Reports in Prostate Cancer.

Clinical and pathological stage are defining parameters in oncology, which direct a patient's treatment options and prognosis. Pathology reports contain a wealth of staging information that is not stored in structured form in most electronic health records (EHRs). Therefore, we evaluated three supervised machine learning methods (Support Vector Machine, Decision Trees, Gradient Boosting) to classify free-text pathology reports for prostate cancer into T, N and M stage groups.

Merging heterogeneous clinical data to enable knowledge discovery.

The vision of precision medicine relies on the integration of large-scale clinical, molecular and environmental datasets. Data integration may be thought of along two axes: data fusion across institutions, and data fusion across modalities. Cross-institutional data sharing that maintains semantic integrity hinges on the adoption of data standards and a push toward ontology-driven integration. The goal should be the creation of query-able data repositories spanning primary and tertiary care providers, disease registries, research organizations etc. to produce rich longitudinal datasets. Cross-modality sharing involves the integration of multiple data streams, from structured EHR data (diagnosis codes, laboratory tests) to genomics, imaging, monitors and patient-generated data including wearable devices. This integration presents unique technical, semantic, and ethical challenges; however recent work suggests that multi-modal clinical data can significantly improve the performance of phenotyping and prediction algorithms, powering knowledge discovery at the patient- and population-level.

Distribution of global health measures from routinely collected PROMIS surveys in patients with breast cancer or prostate cancer.

BACKGROUND: The collection of patient-reported outcomes (PROs) is an emerging priority internationally, guiding clinical care, quality improvement projects and research studies. After the deployment of Patient-Reported Outcomes Measurement Information System (PROMIS) surveys in routine outpatient workflows at an academic cancer center, electronic health record data were used to evaluate survey completion rates and self-reported global health measures across 2 tumor types: breast and prostate cancer. METHODS: This study retrospectively analyzed 11,657 PROMIS surveys from patients with breast cancer and 4411 surveys from patients with prostate cancer, and it calculated survey completion rates and global physical health (GPH) and global mental health (GMH) scores between 2013 and 2018. RESULTS: A total of 36.6% of eligible patients with breast cancer and 23.7% of patients with prostate cancer completed at least 1 survey, with completion rates lower among black patients for both tumor types (P < .05). The mean T scores (calibrated to a general population mean of 50) for GPH were 48.4 ± 9 for breast cancer and 50.6 ± 9 for prostate cancer, and the GMH scores were 52.7 ± 8 and 52.1 ± 9, respectively. GPH and GMH were frequently lower among ethnic minorities, patients without private health insurance, and those with advanced disease. CONCLUSIONS: This analysis provides important baseline data on patient-reported global health in breast and prostate cancer. Demonstrating that PROs can be integrated into clinical workflows, this study shows that supportive efforts may be needed to improve PRO collection and global health endpoints in vulnerable populations.

Architecture and Implementation of a Clinical Research Data Warehouse for Prostate Cancer.

BACKGROUND: Electronic health record (EHR) based research in oncology can be limited by missing data and a lack of structured data elements. Clinical research data warehouses for specific cancer types can enable the creation of more robust research cohorts. METHODS: We linked data from the Stanford University EHR with the Stanford Cancer Institute Research Database (SCIRDB) and the California Cancer Registry (CCR) to create a research data warehouse for prostate cancer. The database was supplemented with information from clinical trials, natural language processing of clinical notes and surveys on patient-reported outcomes. RESULTS: 11,898 unique prostate cancer patients were identified in the Stanford EHR, of which 3,936 were matched to the Stanford cancer registry and 6153 in the CCR. 7158 patients with EHR data and at least one of SCIRDB and CCR data were initially included in the warehouse. CONCLUSIONS: A disease-specific clinical research data warehouse combining multiple data sources can facilitate secondary data use and enhance observational research in oncology.

HealthNavigator: a mobile application for chronic disease screening and linkage to services at an urban Primary Health Network.

Mobile applications (apps) are promising tools to support chronic disease screening and linkage to health services. They have the potential to increase healthcare access for vulnerable populations. The HealthNavigator app was developed to provide chronic disease risk assessments, linkage to local general practitioners (GPs) and lifestyle programs, and a personalised health report for discussion with a GP. Assessments were either self-administered or facilitated by community health workers through a Primary Health Network (PHN) initiative targeting ethnically diverse communities. In total, 1492 assessments (80.4% self-administered, 19.6% facilitated) were conducted over a 12-month period in Queensland, Australia. Of these, 26% of people screened came from postcodes representing the lowest quartile of socioeconomic disadvantage. When compared against self-administered assessments, subjects screened by the facilitated program were more likely to be born outside Australia (80.5 v. 33.2%, P<0.001), and to fall within a high risk category based on cardiovascular risk scores (19.8 v. 13.7%, P<0.01) and type 2 diabetes mellitus risk scores (58.0 v. 40.1%, P<0.001). Mobile apps embedded into PHN programs may be a useful adjunct for the implementation of community screening programs. Further research is needed to determine their effect on health service access and health outcomes.

Identifying Cases of Metastatic Prostate Cancer Using Machine Learning on Electronic Health Records.

Cancer stage is rarely captured in structured form in the electronic health record (EHR). We evaluate the performance of a classifier, trained on structured EHR data, in identifying prostate cancer patients with metastatic disease. Using EHR data for a cohort of 5,861 prostate cancer patients mapped to the Observational Health Data Sciences and Informatics (OHDSI) data model, we constructed feature vectors containing frequency counts of conditions, procedures, medications, observations and laboratory values. Staging information from the California Cancer Registry was used as the ground-truth. For identifying patients with metastatic disease, a random forest model achieved precision and recall of 0.90, 0.40 using data within 12 months of diagnosis. This compared to precision 0.33, recall 0.54 for an ICD code-based query. High-precision classifiers using hundreds of structured data elements significantly outperform ICD queries, and may assist in identifying cohorts for observational research or clinical trial matching.