Development and validation of machine learning models to identify high-risk surgical patients using automatically curated electronic health record data (Pythia): A retrospective, single-site study.

BACKGROUND: Pythia is an automated, clinically curated surgical data pipeline and repository housing all surgical patient electronic health record (EHR) data from a large, quaternary, multisite health institute for data science initiatives. In an effort to better identify high-risk surgical patients from complex data, a machine learning project trained on Pythia was built to predict postoperative complication risk. METHODS AND FINDINGS: A curated data repository of surgical outcomes was created using automated SQL and R code that extracted and processed patient clinical and surgical data across 37 million clinical encounters from the EHRs. A total of 194 clinical features including patient demographics (e.g., age, sex, race), smoking status, medications, comorbidities, procedure information, and proxies for surgical complexity were constructed and aggregated. A cohort of 66,370 patients that had undergone 99,755 invasive procedural encounters between January 1, 2014, and January 31, 2017, was studied further for the purpose of predicting postoperative complications. The average complication and 30-day postoperative mortality rates of this cohort were 16.0% and 0.51%, respectively. Least absolute shrinkage and selection operator (lasso) penalized logistic regression, random forest models, and extreme gradient boosted decision trees were trained on this surgical cohort with cross-validation on 14 specific postoperative outcome groupings. Resulting models had area under the receiver operator characteristic curve (AUC) values ranging between 0.747 and 0.924, calculated on an out-of-sample test set from the last 5 months of data. Lasso penalized regression was identified as a high-performing model, providing clinically interpretable actionable insights. Highest and lowest performing lasso models predicted postoperative shock and genitourinary outcomes with AUCs of 0.924 (95% CI: 0.901, 0.946) and 0.780 (95% CI: 0.752, 0.810), respectively. A calculator requiring input of 9 data fields was created to produce a risk assessment for the 14 groupings of postoperative outcomes. A high-risk threshold (15% risk of any complication) was determined to identify high-risk surgical patients. The model sensitivity was 76%, with a specificity of 76%. Compared to heuristics that identify high-risk patients developed by clinical experts and the ACS NSQIP calculator, this tool performed superiorly, providing an improved approach for clinicians to estimate postoperative risk for patients. Limitations of this study include the missingness of data that were removed for analysis. CONCLUSIONS: Extracting and curating a large, local institution's EHR data for machine learning purposes resulted in models with strong predictive performance. These models can be used in clinical settings as decision support tools for identification of high-risk patients as well as patient evaluation and care management. Further work is necessary to evaluate the impact of the Pythia risk calculator within the clinical workflow on postoperative outcomes and to optimize this data flow for future machine learning efforts.

Impact of family-centred postnatal training on maternal and neonatal health and care practices in district hospitals in two states in India: a pre-post study.

BACKGROUND AND OBJECTIVES: The Care Companion Program (CCP) is an in-hospital multitopic skill-based training programme provided to families to improve postdischarge maternal and neonatal health. The states of Punjab and Karnataka in India piloted the programme in 12 district hospitals in July 2017, and no study to date has evaluated its impact. METHODS: We compared telephonically self-reported maternal and neonatal care practices and health outcomes before and after the launch of the CCP programme in 11 facilities. Families in the preintervention group delivered between May to June 2017 (N=1474) while those in the intervention group delivered between August and October 2017 (N=3510). Programme effects were expressed as adjusted risk ratios obtained from logistic regression models. RESULTS: At 2-week postdelivery, the practice of dry cord care improved by 4% (RR=1.04, 95% CI 1.02 to 1.06) and skin-to-skin care by 78% (RR=1.78, 95% CI 1.37 to 2.27) in the postintervention group as compared with preintervention group. Furthermore, newborn complications reduced by 16% (RR=0.84, 95% CI 0.76 to 0.91), mother complications by 12% (RR=0.88, 95% CI 0.79 to 0.97) and newborn readmissions by 56% (RR=0.44, 95% CI 0.31 to 0.61). Outpatient visits increased by 27% (RR=1.27, 95% CI 1.10 to 1.46). However, the practice of exclusive breastfeeding, unrestricted maternal diet, hand-hygiene and being instructed on warning signs were not statistically different. CONCLUSION: Postnatal care should incorporate predischarge training of families. Our findings demonstrate that it is possible to improve maternal and neonatal care practices and outcomes through a family-centered programme integrated into public health facilities in low and middle-income countries.

A survey of extant organizational and computational setups for deploying predictive models in health systems.

OBJECTIVE: Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs. MATERIALS AND METHODS: We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care. RESULTS: We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n = 10, 50%), IT Department led (n = 2, 10%), and AI in Healthcare (AIHC) Team (n = 8, 40%). DISCUSSION: No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS. CONCLUSION: Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.

Impact of diagnosis code grouping method on clinical prediction model performance: A multi-site retrospective observational study.

OBJECTIVE: The primary purpose of this work is to systematically assess the performance trade-offs on clinical prediction tasks of four diagnosis code groupings: AHRQ-Elixhauser, Single-level CCS, truncated ICD-9-CM codes, and raw ICD-9-CM codes. MATERIALS AND METHODS: We used two distinct datasets from different geographic regions and patient populations and train models for three prediction tasks: 1-year mortality following an ICU stay, 30-day mortality following surgery, and 30-day complication following surgery. We run multiple commonly-used binary classification models including penalized logistic regression, random forest, and gradient boosted trees. Model performance is evaluated using the Area Under the Receiver Operating Characteristic (AUROC) and the Area Under the Precision-Recall Curve (AUCPR). RESULTS: Single-level CCS, truncated codes, and raw codes significantly outperformed AHRQ-Elixhauser ICD grouping when predicting 30-day postoperative complication and one-year mortality after ICU admission. The performance across groupings was more similar in the 30-day postoperative mortality prediction task. DISCUSSION: Single-level CCS groupings represent aggregations of raw codes into meaningful clinical concepts and consistently balance interoperability between ICD-9-CM and ICD-10-CM while maintaining strong model performance as measured by AUROC and AUCPR. Key limitations include experimentation across two datasets and three prediction tasks, which although were well labeled and sufficiently prevalent, do not encompass all modeling tasks and outcomes. CONCLUSION: Single-level CCS groupings may serve as a good baseline for future models that incorporate diagnosis codes as features in clinical prediction tasks. Code and a compute environment summary are provided along with the analyses to enable reproducibility and to support future research.

A framework for making predictive models useful in practice.

OBJECTIVE: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality. MATERIALS AND METHODS: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP. RESULTS: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care. DISCUSSION: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit. CONCLUSION: An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Measure what matters: Counts of hospitalized patients are a better metric for health system capacity planning for a reopening.

OBJECTIVE: Responding to the COVID-19 pandemic requires accurate forecasting of health system capacity requirements using readily available inputs. We examined whether testing and hospitalization data could help quantify the anticipated burden on the health system given shelter-in-place (SIP) order. MATERIALS AND METHODS: 16,103 SARS-CoV-2 RT-PCR tests were performed on 15,807 patients at Stanford facilities between March 2 and April 11, 2020. We analyzed the fraction of tested patients that were confirmed positive for COVID-19, the fraction of those needing hospitalization, and the fraction requiring ICU admission over the 40 days between March 2nd and April 11th 2020. RESULTS: We find a marked slowdown in the hospitalization rate within ten days of SIP even as cases continued to rise. We also find a shift towards younger patients in the age distribution of those testing positive for COVID-19 over the four weeks of SIP. The impact of this shift is a divergence between increasing positive case confirmations and slowing new hospitalizations, both of which affects the demand on health systems. CONCLUSION: Without using local hospitalization rates and the age distribution of positive patients, current models are likely to overestimate the resource burden of COVID-19. It is imperative that health systems start using these data to quantify effects of SIP and aid reopening planning.

Just-in-time postnatal education programees to improve newborn care practices: needs and opportunities in low-resource settings.

Worldwide, many newborns die in the first month of life, with most deaths happening in low/middle-income countries (LMICs). Families' use of evidence-based newborn care practices in the home and timely care-seeking for illness can save newborn lives. Postnatal education is an important investment to improve families' use of evidence-based newborn care practices, yet there are gaps in the literature on postnatal education programees that have been evaluated to date. Recent findings from a 13 000+ person survey in 3 states in India show opportunities for improvement in postnatal education for mothers and families and their use of newborn care practices in the home. Our survey data and the literature suggest the need to incorporate the following strategies into future postnatal education programming: implement structured predischarge education with postdischarge reinforcement, using a multipronged teaching approach to reach whole families with education on multiple newborn care practices. Researchers need to conduct robust evaluation on postnatal education models incorporating these programee elements in the LMIC context, as well as explore whether this type of education model can work for other health areas that are critical for families to survive and thrive.