Who needs what (features) when? Personalizing engagement with data-driven self-management to improve health equity.

OBJECTIVES: To examine the feasibility of promoting engagement with data-driven self-management of health among individuals from minoritized medically underserved communities by tailoring the design of self-management interventions to individuals' type of motivation and regulation in accordance with the Self-Determination Theory. METHODS: Fifty-three individuals with type 2 diabetes from an impoverished minority community were randomly assigned to four different versions of an mHealth app for data-driven self-management with the focus on nutrition, Platano; each version was tailored to a specific type of motivation and regulation within the SDT self-determination continuum. These versions included financial rewards (external regulation), feedback from expert registered dietitians (RDF, introjected regulation), self-assessment of attainment of one's nutritional goals (SA, identified regulation), and personalized meal-time nutrition decision support with post-meal blood glucose forecasts (FORC, integrated regulation). We used qualitative interviews to examine interaction between participants' experiences with the app and their motivation type (internal-external). RESULTS: As hypothesized, we found a clear interaction between the type of motivation and Platano features that users responded to and benefited from. For example, those with more internal motivation reported more positive experience with SA and FORC than those with more external motivation. However, we also found that Platano features that aimed to specifically address the needs of individuals with external regulation did not create the desired experience. We attribute this to a mismatch in emphasis on informational versus emotional support, particularly evident in RDF. In addition, we found that for participants recruited from an economically disadvantaged community, internal factors, such as motivation and regulation, interacted with external factors, most notably with limited health literacy and limited access to resources. CONCLUSIONS: The study suggests feasibility of using SDT to tailor design of mHealth interventions for promoting data-driven self-management to individuals' motivation and regulation. However, further research is needed to better align design solutions with different levels of self-determination continuum, to incorporate stronger emphasis on emotional support for individuals with external regulation, and to address unique needs and challenges of underserved communities, with particular attention to limited health literacy and access to resources.

A simple modeling framework for prediction in the human glucose-insulin system.

Forecasting blood glucose (BG) levels with routinely collected data is useful for glycemic management. BG dynamics are nonlinear, complex, and nonstationary, which can be represented by nonlinear models. However, the sparsity of routinely collected data creates parameter identifiability issues when high-fidelity complex models are used, thereby resulting in inaccurate forecasts. One can use models with reduced physiological fidelity for robust and accurate parameter estimation and forecasting with sparse data. For this purpose, we approximate the nonlinear dynamics of BG regulation by a linear stochastic differential equation: we develop a linear stochastic model, which can be specialized to different settings: type 2 diabetes mellitus (T2DM) and intensive care unit (ICU), with different choices of appropriate model functions. The model includes deterministic terms quantifying glucose removal from the bloodstream through the glycemic regulation system and representing the effect of nutrition and externally delivered insulin. The stochastic term encapsulates the BG oscillations. The model output is in the form of an expected value accompanied by a band around this value. The model parameters are estimated patient-specifically, leading to personalized models. The forecasts consist of values for BG mean and variation, quantifying possible high and low BG levels. Such predictions have potential use for glycemic management as part of control systems. We present experimental results on parameter estimation and forecasting in T2DM and ICU settings. We compare the model's predictive capability with two different nonlinear models built for T2DM and ICU contexts to have a sense of the level of prediction achieved by this model.

Enabling personalized decision support with patient-generated data and attributable components.

Decision-making related to health is complex. Machine learning (ML) and patient generated data can identify patterns and insights at the individual level, where human cognition falls short, but not all ML-generated information is of equal utility for making health-related decisions. We develop and apply attributable components analysis (ACA), a method inspired by optimal transport theory, to type 2 diabetes self-monitoring data to identify patterns of association between nutrition and blood glucose control. In comparison with linear regression, we found that ACA offers a number of characteristics that make it promising for use in decision support applications. For example, ACA was able to identify non-linear relationships, was more robust to outliers, and offered broader and more expressive uncertainty estimates. In addition, our results highlight a tradeoff between model accuracy and interpretability, and we discuss implications for ML-driven decision support systems.

Ensemble Kalman Methods With Constraints.

Ensemble Kalman methods constitute an increasingly important tool in both state and parameter estimation problems. Their popularity stems from the derivative-free nature of the methodology which may be readily applied when computer code is available for the underlying state-space dynamics (for state estimation) or for the parameter-to-observable map (for parameter estimation). There are many applications in which it is desirable to enforce prior information in the form of equality or inequality constraints on the state or parameter. This paper establishes a general framework for doing so, describing a widely applicable methodology, a theory which justifies the methodology, and a set of numerical experiments exemplifying it.

Methodological variations in lagged regression for detecting physiologic drug effects in EHR data.

We studied how lagged linear regression can be used to detect the physiologic effects of drugs from data in the electronic health record (EHR). We systematically examined the effect of methodological variations ((i) time series construction, (ii) temporal parameterization, (iii) intra-subject normalization, (iv) differencing (lagged rates of change achieved by taking differences between consecutive measurements), (v) explanatory variables, and (vi) regression models) on performance of lagged linear methods in this context. We generated two gold standards (one knowledge-base derived, one expert-curated) for expected pairwise relationships between 7 drugs and 4 labs, and evaluated how the 64 unique combinations of methodological perturbations reproduce the gold standards. Our 28 cohorts included patients in the Columbia University Medical Center/NewYork-Presbyterian Hospital clinical database, and ranged from 2820 to 79,514 patients with between 8 and 209 average time points per patient. The most accurate methods achieved AUROC of 0.794 for knowledge-base derived gold standard (95%CI [0.741, 0.847]) and 0.705 for expert-curated gold standard (95% CI [0.629, 0.781]). We observed a mean AUROC of 0.633 (95%CI [0.610, 0.657], expert-curated gold standard) across all methods that re-parameterize time according to sequence and use either a joint autoregressive model with time-series differencing or an independent lag model without differencing. The complement of this set of methods achieved a mean AUROC close to 0.5, indicating the importance of these choices. We conclude that time-series analysis of EHR data will likely rely on some of the beneficial pre-processing and modeling methodologies identified, and will certainly benefit from continued careful analysis of methodological perturbations. This study found that methodological variations, such as pre-processing and representations, have a large effect on results, exposing the importance of thoroughly evaluating these components when comparing machine-learning methods.

From Reflection to Action: Combining Machine Learning with Expert Knowledge for Nutrition Goal Recommendations.

Self-tracking can help personalize self-management interventions for chronic conditions like type 2 diabetes (T2D), but reflecting on personal data requires motivation and literacy. Machine learning (ML) methods can identify patterns, but a key challenge is making actionable suggestions based on personal health data. We introduce GlucoGoalie, which combines ML with an expert system to translate ML output into personalized nutrition goal suggestions for individuals with T2D. In a controlled experiment, participants with T2D found that goal suggestions were understandable and actionable. A 4-week in-the-wild deployment study showed that receiving goal suggestions augmented participants' self-discovery, choosing goals highlighted the multifaceted nature of personal preferences, and the experience of following goals demonstrated the importance of feedback and context. However, we identified tensions between abstract goals and concrete eating experiences and found static text too ambiguous for complex concepts. We discuss implications for ML-based interventions and the need for systems that offer more interactivity, feedback, and negotiation.

Real-Time Electronic Health Record Mortality Prediction During the COVID-19 Pandemic: A Prospective Cohort Study.

BACKGROUND: The SARS-CoV-2 virus has infected millions of people, overwhelming critical care resources in some regions. Many plans for rationing critical care resources during crises are based on the Sequential Organ Failure Assessment (SOFA) score. The COVID-19 pandemic created an emergent need to develop and validate a novel electronic health record (EHR)-computable tool to predict mortality. RESEARCH QUESTIONS: To rapidly develop, validate, and implement a novel real-time mortality score for the COVID-19 pandemic that improves upon SOFA. STUDY DESIGN AND METHODS: We conducted a prospective cohort study of a regional health system with 12 hospitals in Colorado between March 2020 and July 2020. All patients >14 years old hospitalized during the study period without a do not resuscitate order were included. Patients were stratified by the diagnosis of COVID-19. From this cohort, we developed and validated a model using stacked generalization to predict mortality using data widely available in the EHR by combining five previously validated scores and additional novel variables reported to be associated with COVID-19-specific mortality. We compared the area under the receiver operator curve (AUROC) for the new model to the SOFA score and the Charlson Comorbidity Index. RESULTS: We prospectively analyzed 27,296 encounters, of which 1,358 (5.0%) were positive for SARS-CoV-2, 4,494 (16.5%) included intensive care unit (ICU)-level care, 1,480 (5.4%) included invasive mechanical ventilation, and 717 (2.6%) ended in death. The Charlson Comorbidity Index and SOFA scores predicted overall mortality with an AUROC of 0.72 and 0.90, respectively. Our novel score predicted overall mortality with AUROC 0.94. In the subset of patients with COVID-19, we predicted mortality with AUROC 0.90, whereas SOFA had AUROC of 0.85. INTERPRETATION: We developed and validated an accurate, in-hospital mortality prediction score in a live EHR for automatic and continuous calculation using a novel model, that improved upon SOFA. TAKE HOME POINTS: Study Question: Can we improve upon the SOFA score for real-time mortality prediction during the COVID-19 pandemic by leveraging electronic health record (EHR) data?Results: We rapidly developed and implemented a novel yet SOFA-anchored mortality model across 12 hospitals and conducted a prospective cohort study of 27,296 adult hospitalizations, 1,358 (5.0%) of which were positive for SARS-CoV-2. The Charlson Comorbidity Index and SOFA scores predicted all-cause mortality with AUROCs of 0.72 and 0.90, respectively. Our novel score predicted mortality with AUROC 0.94.Interpretation: A novel EHR-based mortality score can be rapidly implemented to better predict patient outcomes during an evolving pandemic.

Real-time electronic health record mortality prediction during the COVID-19 pandemic: a prospective cohort study.

OBJECTIVE: To rapidly develop, validate, and implement a novel real-time mortality score for the COVID-19 pandemic that improves upon sequential organ failure assessment (SOFA) for decision support for a Crisis Standards of Care team. MATERIALS AND METHODS: We developed, verified, and deployed a stacked generalization model to predict mortality using data available in the electronic health record (EHR) by combining 5 previously validated scores and additional novel variables reported to be associated with COVID-19-specific mortality. We verified the model with prospectively collected data from 12 hospitals in Colorado between March 2020 and July 2020. We compared the area under the receiver operator curve (AUROC) for the new model to the SOFA score and the Charlson Comorbidity Index. RESULTS: The prospective cohort included 27 296 encounters, of which 1358 (5.0%) were positive for SARS-CoV-2, 4494 (16.5%) required intensive care unit care, 1480 (5.4%) required mechanical ventilation, and 717 (2.6%) ended in death. The Charlson Comorbidity Index and SOFA scores predicted mortality with an AUROC of 0.72 and 0.90, respectively. Our novel score predicted mortality with AUROC 0.94. In the subset of patients with COVID-19, the stacked model predicted mortality with AUROC 0.90, whereas SOFA had AUROC of 0.85. DISCUSSION: Stacked regression allows a flexible, updatable, live-implementable, ethically defensible predictive analytics tool for decision support that begins with validated models and includes only novel information that improves prediction. CONCLUSION: We developed and validated an accurate in-hospital mortality prediction score in a live EHR for automatic and continuous calculation using a novel model that improved upon SOFA.

A new approach to integrating patient-generated data with expert knowledge for personalized goal setting: A pilot study.

INTRODUCTION: Self-monitoring technologies produce patient-generated data that could be leveraged to personalize nutritional goal setting to improve population health; however, most computational approaches are limited when applied to individual-level personalization with sparse and irregular self-monitoring data. We applied informatics methods from expert suggestion systems to a challenging clinical problem: generating personalized nutrition goals from patient-generated diet and blood glucose data. MATERIALS AND METHODS: We applied qualitative process coding and decision tree modeling to understand how registered dietitians translate patient-generated data into recommendations for dietary self-management of diabetes (i.e., knowledge model). We encoded this process in a set of functions that take diet and blood glucose data as an input and output diet recommendations (i.e., inference engine). Dietitians assessed face validity. Using four patient datasets, we compared our inference engine's output to clinical narratives and gold standards developed by expert clinicians. RESULTS: To dietitians, the knowledge model represented how recommendations from patient data are made. Inference engine recommendations were 63 % consistent with the gold standard (range = 42 %-75 %) and 74 % consistent with narrative clinical observations (range = 63 %-83 %). DISCUSSION: Qualitative modeling and automating how dietitians reason over patient data resulted in a knowledge model representing clinical knowledge. However, our knowledge model was less consistent with gold standard than narrative clinical recommendations, raising questions about how best to evaluate approaches that integrate patient-generated data with expert knowledge. CONCLUSION: New informatics approaches that integrate data-driven methods with expert decision making for personalized goal setting, such as the knowledge base and inference engine presented here, demonstrate the potential to extend the reach of patient-generated data by synthesizing it with clinical knowledge. However, important questions remain about the strengths and weaknesses of computer algorithms developed to discern signal from patient-generated data compared to human experts.

The parameter Houlihan: A solution to high-throughput identifiability indeterminacy for brutally ill-posed problems.

One way to interject knowledge into clinically impactful forecasting is to use data assimilation, a nonlinear regression that projects data onto a mechanistic physiologic model, instead of a set of functions, such as neural networks. Such regressions have an advantage of being useful with particularly sparse, non-stationary clinical data. However, physiological models are often nonlinear and can have many parameters, leading to potential problems with parameter identifiability, or the ability to find a unique set of parameters that minimize forecasting error. The identifiability problems can be minimized or eliminated by reducing the number of parameters estimated, but reducing the number of estimated parameters also reduces the flexibility of the model and hence increases forecasting error. We propose a method, the parameter Houlihan, that combines traditional machine learning techniques with data assimilation, to select the right set of model parameters to minimize forecasting error while reducing identifiability problems. The method worked well: the data assimilation-based glucose forecasts and estimates for our cohort using the Houlihan-selected parameter sets generally also minimize forecasting errors compared to other parameter selection methods such as by-hand parameter selection. Nevertheless, the forecast with the lowest forecast error does not always accurately represent physiology, but further advancements of the algorithm provide a path for improving physiologic fidelity as well. Our hope is that this methodology represents a first step toward combining machine learning with data assimilation and provides a lower-threshold entry point for using data assimilation with clinical data by helping select the right parameters to estimate.

Effect of vocabulary mapping for conditions on phenotype cohorts.

Objective: To study the effect on patient cohorts of mapping condition (diagnosis) codes from source billing vocabularies to a clinical vocabulary. Materials and Methods: Nine International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) concept sets were extracted from eMERGE network phenotypes, translated to Systematized Nomenclature of Medicine - Clinical Terms concept sets, and applied to patient data that were mapped from source ICD9-CM and ICD10-CM codes to Systematized Nomenclature of Medicine - Clinical Terms codes using Observational Health Data Sciences and Informatics (OHDSI) Observational Medical Outcomes Partnership (OMOP) vocabulary mappings. The original ICD9-CM concept set and a concept set extended to ICD10-CM were used to create patient cohorts that served as gold standards. Results: Four phenotype concept sets were able to be translated to Systematized Nomenclature of Medicine - Clinical Terms without ambiguities and were able to perform perfectly with respect to the gold standards. The other 5 lost performance when 2 or more ICD9-CM or ICD10-CM codes mapped to the same Systematized Nomenclature of Medicine - Clinical Terms code. The patient cohorts had a total error (false positive and false negative) of up to 0.15% compared to querying ICD9-CM source data and up to 0.26% compared to querying ICD9-CM and ICD10-CM data. Knowledge engineering was required to produce that performance; simple automated methods to generate concept sets had errors up to 10% (one outlier at 250%). Discussion: The translation of data from source vocabularies to Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) resulted in very small error rates that were an order of magnitude smaller than other error sources. Conclusion: It appears possible to map diagnoses from disparate vocabularies to a single clinical vocabulary and carry out research using a single set of definitions, thus improving efficiency and transportability of research.

Mechanistic machine learning: how data assimilation leverages physiologic knowledge using Bayesian inference to forecast the future, infer the present, and phenotype.

We introduce data assimilation as a computational method that uses machine learning to combine data with human knowledge in the form of mechanistic models in order to forecast future states, to impute missing data from the past by smoothing, and to infer measurable and unmeasurable quantities that represent clinically and scientifically important phenotypes. We demonstrate the advantages it affords in the context of type 2 diabetes by showing how data assimilation can be used to forecast future glucose values, to impute previously missing glucose values, and to infer type 2 diabetes phenotypes. At the heart of data assimilation is the mechanistic model, here an endocrine model. Such models can vary in complexity, contain testable hypotheses about important mechanics that govern the system (eg, nutrition's effect on glucose), and, as such, constrain the model space, allowing for accurate estimation using very little data.

A visual analytics approach for pattern-recognition in patient-generated data.

Objective: To develop and test a visual analytics tool to help clinicians identify systematic and clinically meaningful patterns in patient-generated data (PGD) while decreasing perceived information overload. Methods: Participatory design was used to develop Glucolyzer, an interactive tool featuring hierarchical clustering and a heatmap visualization to help registered dietitians (RDs) identify associative patterns between blood glucose levels and per-meal macronutrient composition for individuals with type 2 diabetes (T2DM). Ten RDs participated in a within-subjects experiment to compare Glucolyzer to a static logbook format. For each representation, participants had 25 minutes to examine 1 month of diabetes self-monitoring data captured by an individual with T2DM and identify clinically meaningful patterns. We compared the quality and accuracy of the observations generated using each representation. Results: Participants generated 50% more observations when using Glucolyzer (98) than when using the logbook format (64) without any loss in accuracy (69% accuracy vs 62%, respectively, p = .17). Participants identified more observations that included ingredients other than carbohydrates using Glucolyzer (36% vs 16%, p = .027). Fewer RDs reported feelings of information overload using Glucolyzer compared to the logbook format. Study participants displayed variable acceptance of hierarchical clustering. Conclusions: Visual analytics have the potential to mitigate provider concerns about the volume of self-monitoring data. Glucolyzer helped dietitians identify meaningful patterns in self-monitoring data without incurring perceived information overload. Future studies should assess whether similar tools can support clinicians in personalizing behavioral interventions that improve patient outcomes.

Comparing lagged linear correlation, lagged regression, Granger causality, and vector autoregression for uncovering associations in EHR data.

Time series analysis methods have been shown to reveal clinical and biological associations in data collected in the electronic health record. We wish to develop reliable high-throughput methods for identifying adverse drug effects that are easy to implement and produce readily interpretable results. To move toward this goal, we used univariate and multivariate lagged regression models to investigate associations between twenty pairs of drug orders and laboratory measurements. Multivariate lagged regression models exhibited higher sensitivity and specificity than univariate lagged regression in the 20 examples, and incorporating autoregressive terms for labs and drugs produced more robust signals in cases of known associations among the 20 example pairings. Moreover, including inpatient admission terms in the model attenuated the signals for some cases of unlikely associations, demonstrating how multivariate lagged regression models' explicit handling of context-based variables can provide a simple way to probe for health-care processes that confound analyses of EHR data.

Data-driven health management: reasoning about personally generated data in diabetes with information technologies.

OBJECTIVE: To investigate how individuals with diabetes and diabetes educators reason about data collected through self-monitoring and to draw implications for the design of data-driven self-management technologies. MATERIALS AND METHODS: Ten individuals with diabetes (six type 1 and four type 2) and 2 experienced diabetes educators were presented with a set of self-monitoring data captured by an individual with type 2 diabetes. The set included digital images of meals and their textual descriptions, and blood glucose (BG) readings captured before and after these meals. The participants were asked to review a set of meals and associated BG readings, explain differences in postprandial BG levels for these meals, and predict postprandial BG levels for the same individual for a different set of meals. Researchers compared conclusions and predictions reached by the participants with those arrived at by quantitative analysis of the collected data. RESULTS: The participants used both macronutrient composition of meals, most notably the inclusion of carbohydrates, and names of dishes and ingredients to reason about changes in postprandial BG levels. Both individuals with diabetes and diabetes educators reported difficulties in generating predictions of postprandial BG; their predictions varied in their correlations with the actual captured readings from r = 0.008 to r = 0.75. CONCLUSION: Overall, the study showed that identifying trends in the data collected with self-monitoring is a complex process, and that conclusions reached by both individuals with diabetes and diabetes educators are not always reliable. This suggests the need for new ways to facilitate individuals' reasoning with informatics interventions.