Megastudies improve the impact of applied behavioural science.

Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens' decisions and outcomes1. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals2. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy-a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research3-6. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.

Key language markers of depression on social media depend on race.

Depression has robust natural language correlates and can increasingly be measured in language using predictive models. However, despite evidence that language use varies as a function of individual demographic features (e.g., age, gender), previous work has not systematically examined whether and how depression's association with language varies by race. We examine how race moderates the relationship between language features (i.e., first-person pronouns and negative emotions) from social media posts and self-reported depression, in a matched sample of Black and White English speakers in the United States. Our findings reveal moderating effects of race: While depression severity predicts I-usage in White individuals, it does not in Black individuals. White individuals use more belongingness and self-deprecation-related negative emotions. Machine learning models trained on similar amounts of data to predict depression severity performed poorly when tested on Black individuals, even when they were trained exclusively using the language of Black individuals. In contrast, analogous models tested on White individuals performed relatively well. Our study reveals surprising race-based differences in the expression of depression in natural language and highlights the need to understand these effects better, especially before language-based models for detecting psychological phenomena are integrated into clinical practice.

A 680,000-person megastudy of nudges to encourage vaccination in pharmacies.

Encouraging vaccination is a pressing policy problem. To assess whether text-based reminders can encourage pharmacy vaccination and what kinds of messages work best, we conducted a megastudy. We randomly assigned 689,693 Walmart pharmacy patients to receive one of 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination or to a business-as-usual control condition that received no messages. We found that the reminder texts that we tested increased pharmacy vaccination rates by an average of 2.0 percentage points, or 6.8%, over a 3-mo follow-up period. The most-effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top-performing intervention included two texts delivered 3 d apart and communicated to patients that a vaccine was "waiting for you." Neither experts nor lay people anticipated that this would be the best-performing treatment, underscoring the value of simultaneously testing many different nudges in a highly powered megastudy.

A joint deep learning model enables simultaneous batch effect correction, denoising, and clustering in single-cell transcriptomics.

Recent developments of single-cell RNA-seq (scRNA-seq) technologies have led to enormous biological discoveries. As the scale of scRNA-seq studies increases, a major challenge in analysis is batch effects, which are inevitable in studies involving human tissues. Most existing methods remove batch effects in a low-dimensional embedding space. Although useful for clustering, batch effects are still present in the gene expression space, leaving downstream gene-level analysis susceptible to batch effects. Recent studies have shown that batch effect correction in the gene expression space is much harder than in the embedding space. Methods such as Seurat 3.0 rely on the mutual nearest neighbor (MNN) approach to remove batch effects in gene expression, but MNN can only analyze two batches at a time, and it becomes computationally infeasible when the number of batches is large. Here, we present CarDEC, a joint deep learning model that simultaneously clusters and denoises scRNA-seq data while correcting batch effects both in the embedding and the gene expression space. Comprehensive evaluations spanning different species and tissues showed that CarDEC outperforms Scanorama, DCA + Combat, scVI, and MNN. With CarDEC denoising, non-highly variable genes offer as much signal for clustering as the highly variable genes (HVGs), suggesting that CarDEC substantially boosted information content in scRNA-seq. We also showed that trajectory analysis using CarDEC's denoised and batch-corrected expression as input revealed marker genes and transcription factors that are otherwise obscured in the presence of batch effects. CarDEC is computationally fast, making it a desirable tool for large-scale scRNA-seq studies.

The emotional and mental health impact of the murder of George Floyd on the US population.

On May 25, 2020, George Floyd, an unarmed Black American male, was killed by a White police officer. Footage of the murder was widely shared. We examined the psychological impact of Floyd's death using two population surveys that collected data before and after his death; one from Gallup (117,568 responses from n = 47,355) and one from the US Census (409,652 responses from n = 319,471). According to the Gallup data, in the week following Floyd's death, anger and sadness increased to unprecedented levels in the US population. During this period, more than a third of the US population reported these emotions. These increases were more pronounced for Black Americans, nearly half of whom reported these emotions. According to the US Census Household Pulse data, in the week following Floyd's death, depression and anxiety severity increased among Black Americans at significantly higher rates than that of White Americans. Our estimates suggest that this increase corresponds to an additional 900,000 Black Americans who would have screened positive for depression, associated with a burden of roughly 2.7 million to 6.3 million mentally unhealthy days.

A daily diary study into the effects on mental health of COVID-19 pandemic-related behaviors.

BACKGROUND: Recommendations for promoting mental health during the COVID-19 pandemic include maintaining social contact, through virtual rather than physical contact, moderating substance/alcohol use, and limiting news and media exposure. We seek to understand if these pandemic-related behaviors impact subsequent mental health. METHODS: Daily online survey data were collected on adults during May/June 2020. Measures were of daily physical and virtual (online) contact with others; substance and media use; and indices of psychological striving, struggling and COVID-related worry. Using random-intercept cross-lagged panel analysis, dynamic within-person cross-lagged effects were separated from more static individual differences. RESULTS: In total, 1148 participants completed daily surveys [657 (57.2%) females, 484 (42.1%) males; mean age 40.6 (s.d. 12.4) years]. Daily increases in news consumed increased COVID-related worrying the next day [cross-lagged estimate = 0.034 (95% CI 0.018-0.049), FDR-adjusted p = 0.00005] and vice versa [0.03 (0.012-0.048), FDR-adjusted p = 0.0017]. Increased media consumption also exacerbated subsequent psychological struggling [0.064 (0.03-0.098), FDR-adjusted p = 0.0005]. There were no significant cross-lagged effects of daily changes in social distancing or virtual contact on later mental health. CONCLUSIONS: We delineate a cycle wherein a daily increase in media consumption results in a subsequent increase in COVID-related worries, which in turn increases daily media consumption. Moreover, the adverse impact of news extended to broader measures of psychological struggling. A similar dynamic did not unfold between the daily amount of physical or virtual contact and subsequent mental health. Findings are consistent with current recommendations to moderate news and media consumption in order to promote mental health.

Mining for equitable health: Assessing the impact of missing data in electronic health records.

Electronic health records (EHR) are collected as a routine part of healthcare delivery, and have great potential to be utilized to improve patient health outcomes. They contain multiple years of health information to be leveraged for risk prediction, disease detection, and treatment evaluation. However, they do not have a consistent, standardized format across institutions, particularly in the United States, and can present significant analytical challenges- they contain multi-scale data from heterogeneous domains and include both structured and unstructured data. Data for individual patients are collected at irregular time intervals and with varying frequencies. In addition to the analytical challenges, EHR can reflect inequity- patients belonging to different groups will have differing amounts of data in their health records. Many of these issues can contribute to biased data collection. The consequence is that the data for under-served groups may be less informative partly due to more fragmented care, which can be viewed as a type of missing data problem. For EHR data in this complex form, there is currently no framework for introducing realistic missing values. There has also been little to no work in assessing the impact of missing data in EHR. In this work, we first introduce a terminology to define three levels of EHR data and then propose a novel framework for simulating realistic missing data scenarios in EHR to adequately assess their impact on predictive modeling. We incorporate the use of a medical knowledge graph to capture dependencies between medical events to create a more realistic missing data framework. In an intensive care unit setting, we found that missing data have greater negative impact on the performance of disease prediction models in groups that tend to have less access to healthcare, or seek less healthcare. We also found that the impact of missing data on disease prediction models is stronger when using the knowledge graph framework to introduce realistic missing values as opposed to random event removal.

Estimating geographic subjective well-being from Twitter: A comparison of dictionary and data-driven language methods.

Researchers and policy makers worldwide are interested in measuring the subjective well-being of populations. When users post on social media, they leave behind digital traces that reflect their thoughts and feelings. Aggregation of such digital traces may make it possible to monitor well-being at large scale. However, social media-based methods need to be robust to regional effects if they are to produce reliable estimates. Using a sample of 1.53 billion geotagged English tweets, we provide a systematic evaluation of word-level and data-driven methods for text analysis for generating well-being estimates for 1,208 US counties. We compared Twitter-based county-level estimates with well-being measurements provided by the Gallup-Sharecare Well-Being Index survey through 1.73 million phone surveys. We find that word-level methods (e.g., Linguistic Inquiry and Word Count [LIWC] 2015 and Language Assessment by Mechanical Turk [LabMT]) yielded inconsistent county-level well-being measurements due to regional, cultural, and socioeconomic differences in language use. However, removing as few as three of the most frequent words led to notable improvements in well-being prediction. Data-driven methods provided robust estimates, approximating the Gallup data at up to r = 0.64. We show that the findings generalized to county socioeconomic and health outcomes and were robust when poststratifying the samples to be more representative of the general US population. Regional well-being estimation from social media data seems to be robust when supervised data-driven methods are used.

Expert-augmented machine learning.

Machine learning is proving invaluable across disciplines. However, its success is often limited by the quality and quantity of available data, while its adoption is limited by the level of trust afforded by given models. Human vs. machine performance is commonly compared empirically to decide whether a certain task should be performed by a computer or an expert. In reality, the optimal learning strategy may involve combining the complementary strengths of humans and machines. Here, we present expert-augmented machine learning (EAML), an automated method that guides the extraction of expert knowledge and its integration into machine-learned models. We used a large dataset of intensive-care patient data to derive 126 decision rules that predict hospital mortality. Using an online platform, we asked 15 clinicians to assess the relative risk of the subpopulation defined by each rule compared to the total sample. We compared the clinician-assessed risk to the empirical risk and found that, while clinicians agreed with the data in most cases, there were notable exceptions where they overestimated or underestimated the true risk. Studying the rules with greatest disagreement, we identified problems with the training data, including one miscoded variable and one hidden confounder. Filtering the rules based on the extent of disagreement between clinician-assessed risk and empirical risk, we improved performance on out-of-sample data and were able to train with less data. EAML provides a platform for automated creation of problem-specific priors, which help build robust and dependable machine-learning models in critical applications.

Using Facebook language to predict and describe excessive alcohol use.

BACKGROUND: Assessing risk for excessive alcohol use is important for applications ranging from recruitment into research studies to targeted public health messaging. Social media language provides an ecologically embedded source of information for assessing individuals who may be at risk for harmful drinking. METHODS: Using data collected on 3664 respondents from the general population, we examine how accurately language used on social media classifies individuals as at-risk for alcohol problems based on Alcohol Use Disorder Identification Test-Consumption score benchmarks. RESULTS: We find that social media language is moderately accurate (area under the curve = 0.75) at identifying individuals at risk for alcohol problems (i.e., hazardous drinking/alcohol use disorders) when used with models based on contextual word embeddings. High-risk alcohol use was predicted by individuals' usage of words related to alcohol, partying, informal expressions, swearing, and anger. Low-risk alcohol use was predicted by individuals' usage of social, affiliative, and faith-based words. CONCLUSIONS: The use of social media data to study drinking behavior in the general public is promising and could eventually support primary and secondary prevention efforts among Americans whose at-risk drinking may have otherwise gone "under the radar."

Prospective associations of text-message-based sentiment with symptoms of depression, generalized anxiety, and social anxiety.

OBJECTIVE: Language patterns may elucidate mechanisms of mental health conditions. To inform underlying theory and risk models, we evaluated prospective associations between in vivo text messaging language and differential symptoms of depression, generalized anxiety, and social anxiety. METHODS: Over 16 weeks, we collected outgoing text messages from 335 adults. Using Linguistic Inquiry and Word Count (LIWC), NRC Emotion Lexicon, and previously established depression and stress dictionaries, we evaluated the degree to which language features predict symptoms of depression, generalized anxiety, or social anxiety the following week using hierarchical linear models. To isolate the specificity of language effects, we also controlled for the effects of the two other symptom types. RESULTS: We found significant relationships of language features, including personal pronouns, negative emotion, cognitive and biological processes, and informal language, with common mental health conditions, including depression, generalized anxiety, and social anxiety (ps < .05). There was substantial overlap between language features and the three mental health outcomes. However, after controlling for other symptoms in the models, depressive symptoms were uniquely negatively associated with language about anticipation, trust, social processes, and affiliation (ßs: -.10 to -.09, ps < .05), whereas generalized anxiety symptoms were positively linked with these same language features (ßs: .12-.13, ps < .001). Social anxiety symptoms were uniquely associated with anger, sexual language, and swearing (ßs: .12-.13, ps < .05). CONCLUSION: Language that confers both common (e.g., personal pronouns and negative emotion) and specific (e.g., affiliation, anticipation, trust, and anger) risk for affective disorders is perceptible in prior week text messages, holding promise for understanding cognitive-behavioral mechanisms and tailoring digital interventions.

Word embeddings trained on published case reports are lightweight, effective for clinical tasks, and free of protected health information.

OBJECTIVE: Quantify tradeoffs in performance, reproducibility, and resource demands across several strategies for developing clinically relevant word embeddings. MATERIALS AND METHODS: We trained separate embeddings on all full-text manuscripts in the Pubmed Central (PMC) Open Access subset, case reports therein, the English Wikipedia corpus, the Medical Information Mart for Intensive Care (MIMIC) III dataset, and all notes in the University of Pennsylvania Health System (UPHS) electronic health record. We tested embeddings in six clinically relevant tasks including mortality prediction and de-identification, and assessed performance using the scaled Brier score (SBS) and the proportion of notes successfully de-identified, respectively. RESULTS: Embeddings from UPHS notes best predicted mortality (SBS 0.30, 95% CI 0.15 to 0.45) while Wikipedia embeddings performed worst (SBS 0.12, 95% CI -0.05 to 0.28). Wikipedia embeddings most consistently (78% of notes) and the full PMC corpus embeddings least consistently (48%) de-identified notes. Across all six tasks, the full PMC corpus demonstrated the most consistent performance, and the Wikipedia corpus the least. Corpus size ranged from 49 million tokens (PMC case reports) to 10 billion (UPHS). DISCUSSION: Embeddings trained on published case reports performed as least as well as embeddings trained on other corpora in most tasks, and clinical corpora consistently outperformed non-clinical corpora. No single corpus produced a strictly dominant set of embeddings across all tasks and so the optimal training corpus depends on intended use. CONCLUSION: Embeddings trained on published case reports performed comparably on most clinical tasks to embeddings trained on larger corpora. Open access corpora allow training of clinically relevant, effective, and reproducible embeddings.

The development of grit and growth mindset in Chinese children.

In a recent longitudinal study of U.S. adolescents, grit predicted rank-order increases in growth mindset and, to a lesser degree, growth mindset predicted rank-order increases in grit. The current investigation replicated and extended these findings in a younger non-Western, educated, industrialized, rich, and democratic (non-WEIRD) population. Two large samples totaling more than 5000 elementary school children in China completed self-report questionnaires assessing grit and growth mindset five times over 2 years. As in Park et al. (2020, Journal of Experimental Child Psychology, 198, 1048892020), we found reciprocal relations between grit and growth mindset. Grit systematically predicted rank-order increases in growth mindset at each subsequent 6-month interval. Growth mindset also predicted small rank-order increases in grit over the same period. These findings suggest that, over time, behavior may exert as much an influence on beliefs as the reverse-a dynamic possibly observable as early as in elementary school and not just in WEIRD cultures.

Life under stay-at-home orders: a panel study of change in social interaction and emotional wellbeing among older Americans during COVID-19 pandemic.

BACKGROUND: Recent research has shown the mental health consequence of social distancing during the COVID-19 pandemic, but longitudinal data are relatively scarce. It is unclear whether the pattern of isolation and elevated stress seen at the beginning of the pandemic persists over time. This study evaluates change in social interaction over six months and its impact on emotional wellbeing among older adults. METHODS: We drew data from a panel study with six repeated assessments of social interaction and emotional wellbeing conducted monthly May through October 2020. The sample included a total of 380 White, Black and Hispanic participants aged 50 and over, of whom 33% had low income, who residing in fourteen U.S. states with active stay-at-home orders in May 2020. The analysis examined how change in living arrangement, in-person interaction outside the household, quality of relationship with family and friends, and perceived social support affected trajectories of isolation stress, COVID worry and sadness. RESULTS: While their living arrangements (Odds Ratio [OR] = 0.95, 95% Confidence Interval [CI] = 0.87, 1.03) and relationship quality (OR = 0.94, 95% CI = 0.82, 1.01) remained stable, older adults experienced fluctuations in perceived social support (linear Slope b = -1.42, s.e. = 0.16, p < .001, quadratic slope b = 0.50, s.e. = 0.08, p < .001, cubic slope b = -0.04, s.e. = 0.01, p < .001) and increases in in-person conversations outside the household (OR = 1.19, 95% CI = 1.09, 1.29). Living with a spouse/partner stabilized isolation stress (change in linear slope b = 1.16, s.e. = 0.48, p < .05, in quadratic slope b = -0.62, s.e. = 0.26, p < .05, and in cubic slope = 0.09, s.e. = 0.04, p < .05) and COVID worry (change in quadratic slope b = -0.66, s.e. = 0.32, p < .05 and in cubic slope = 0.09, s.e. = 0.04, p < .05) over time. Individuals with better relationship quality with friends had decreased sadness over time (OR = 0.90, 95% CI = 0.82, 0.99). Changes in social support were associated with greater fluctuations in isolation stress and COVID worry. CONCLUSIONS: During the pandemic, social interactions are protective and lack of stability in feeling supported makes older adults vulnerable to stress. Efforts should focus on (re)building and maintaining companionship and support to mitigate the pandemic's negative impact.

Regional personality assessment through social media language.

OBJECTIVE: We explore the personality of counties as assessed through linguistic patterns on social media. Such studies were previously limited by the cost and feasibility of large-scale surveys; however, language-based computational models applied to large social media datasets now allow for large-scale personality assessment. METHOD: We applied a language-based assessment of the five factor model of personality to 6,064,267 U.S. Twitter users. We aggregated the Twitter-based personality scores to 2,041 counties and compared to political, economic, social, and health outcomes measured through surveys and by government agencies. RESULTS: There was significant personality variation across counties. Openness to experience was higher on the coasts, conscientiousness was uniformly spread, extraversion was higher in southern states, agreeableness was higher in western states, and emotional stability was highest in the south. Across 13 outcomes, language-based personality estimates replicated patterns that have been observed in individual-level and geographic studies. This includes higher Republican vote share in less agreeable counties and increased life satisfaction in more conscientious counties. CONCLUSIONS: Results suggest that regions vary in their personality and that these differences can be studied through computational linguistic analysis of social media. Furthermore, these methods may be used to explore other psychological constructs across geographies.

Building more accurate decision trees with the additive tree.

The expansion of machine learning to high-stakes application domains such as medicine, finance, and criminal justice, where making informed decisions requires clear understanding of the model, has increased the interest in interpretable machine learning. The widely used Classification and Regression Trees (CART) have played a major role in health sciences, due to their simple and intuitive explanation of predictions. Ensemble methods like gradient boosting can improve the accuracy of decision trees, but at the expense of the interpretability of the generated model. Additive models, such as those produced by gradient boosting, and full interaction models, such as CART, have been investigated largely in isolation. We show that these models exist along a spectrum, revealing previously unseen connections between these approaches. This paper introduces a rigorous formalization for the additive tree, an empirically validated learning technique for creating a single decision tree, and shows that this method can produce models equivalent to CART or gradient boosted stumps at the extremes by varying a single parameter. Although the additive tree is designed primarily to provide both the model interpretability and predictive performance needed for high-stakes applications like medicine, it also can produce decision trees represented by hybrid models between CART and boosted stumps that can outperform either of these approaches.

Linguistic predictors from Facebook postings of substance use disorder treatment retention versus discontinuation.

Background: Early indicators of who will remain in - or leave - treatment for substance use disorder (SUD) can drive targeted interventions to support long-term recovery.Objectives: To conduct a comprehensive study of linguistic markers of SUD treatment outcomes, the current study integrated features produced by machine learning models known to have social-psychology relevance.Methods: We extracted and analyzed linguistic features from participants' Facebook posts (N = 206, 39.32% female; 55,415 postings) over the two years before they entered a SUD treatment program. Exploratory features produced by both Linguistic Inquiry and Word Count (LIWC) and Latent Dirichlet Allocation (LDA) topic modeling and the features from theoretical domains of religiosity, affect, and temporal orientation via established AI-based linguistic models were utilized.Results: Patients who stayed in the SUD treatment for over 90 days used more words associated with religion, positive emotions, family, affiliations, and the present, and used more first-person singular pronouns (Cohen's d values: [-0.39, -0.57]). Patients who discontinued their treatment before 90 days discussed more diverse topics, focused on the past, and used more articles (Cohen's d values: [0.44, 0.57]). All ps < .05 with Benjamini-Hochberg False Discovery Rate correction.Conclusions: We confirmed the literature on protective and risk social-psychological factors linking to SUD treatment in language analysis, showing that Facebook language before treatment entry could be used to identify the markers of SUD treatment outcomes. This reflects the importance of taking these linguistic features and markers into consideration when designing and recommending SUD treatment plans.

Natural Language Processing in Surgery: A Systematic Review and Meta-analysis.

OBJECTIVE: The aim of this study was to systematically assess the application and potential benefits of natural language processing (NLP) in surgical outcomes research. SUMMARY BACKGROUND DATA: Widespread implementation of electronic health records (EHRs) has generated a massive patient data source. Traditional methods of data capture, such as billing codes and/or manual review of free-text narratives in EHRs, are highly labor-intensive, costly, subjective, and potentially prone to bias. METHODS: A literature search of PubMed, MEDLINE, Web of Science, and Embase identified all articles published starting in 2000 that used NLP models to assess perioperative surgical outcomes. Evaluation metrics of NLP systems were assessed by means of pooled analysis and meta-analysis. Qualitative synthesis was carried out to assess the results and risk of bias on outcomes. RESULTS: The present study included 29 articles, with over half (n = 15) published after 2018. The most common outcome identified using NLP was postoperative complications (n = 14). Compared to traditional non-NLP models, NLP models identified postoperative complications with higher sensitivity [0.92 (0.87-0.95) vs 0.58 (0.33-0.79), P < 0.001]. The specificities were comparable at 0.99 (0.96-1.00) and 0.98 (0.95-0.99), respectively. Using summary of likelihood ratio matrices, traditional non-NLP models have clinical utility for confirming documentation of outcomes/diagnoses, whereas NLP models may be reliably utilized for both confirming and ruling out documentation of outcomes/diagnoses. CONCLUSIONS: NLP usage to extract a range of surgical outcomes, particularly postoperative complications, is accelerating across disciplines and areas of clinical outcomes research. NLP and traditional non-NLP approaches demonstrate similar performance measures, but NLP is superior in ruling out documentation of surgical outcomes.

A Simulated Prospective Evaluation of a Deep Learning Model for Real-Time Prediction of Clinical Deterioration Among Ward Patients.

OBJECTIVES: The National Early Warning Score, Modified Early Warning Score, and quick Sepsis-related Organ Failure Assessment can predict clinical deterioration. These scores exhibit only moderate performance and are often evaluated using aggregated measures over time. A simulated prospective validation strategy that assesses multiple predictions per patient-day would provide the best pragmatic evaluation. We developed a deep recurrent neural network deterioration model and conducted a simulated prospective evaluation. DESIGN: Retrospective cohort study. SETTING: Four hospitals in Pennsylvania. PATIENTS: Inpatient adults discharged between July 1, 2017, and June 30, 2019. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: We trained a deep recurrent neural network and logistic regression model using data from electronic health records to predict hourly the 24-hour composite outcome of transfer to ICU or death. We analyzed 146,446 hospitalizations with 16.75 million patient-hours. The hourly event rate was 1.6% (12,842 transfers or deaths, corresponding to 260,295 patient-hours within the predictive horizon). On a hold-out dataset, the deep recurrent neural network achieved an area under the precision-recall curve of 0.042 (95% CI, 0.04-0.043), comparable with logistic regression model (0.043; 95% CI 0.041 to 0.045), and outperformed National Early Warning Score (0.034; 95% CI, 0.032-0.035), Modified Early Warning Score (0.028; 95% CI, 0.027- 0.03), and quick Sepsis-related Organ Failure Assessment (0.021; 95% CI, 0.021-0.022). For a fixed sensitivity of 50%, the deep recurrent neural network achieved a positive predictive value of 3.4% (95% CI, 3.4-3.5) and outperformed logistic regression model (3.1%; 95% CI 3.1-3.2), National Early Warning Score (2.0%; 95% CI, 2.0-2.0), Modified Early Warning Score (1.5%; 95% CI, 1.5-1.5), and quick Sepsis-related Organ Failure Assessment (1.5%; 95% CI, 1.5-1.5). CONCLUSIONS: Commonly used early warning scores for clinical decompensation, along with a logistic regression model and a deep recurrent neural network model, show very poor performance characteristics when assessed using a simulated prospective validation. None of these models may be suitable for real-time deployment.

Machine Learning and Surgical Outcomes Prediction: A Systematic Review.

BACKGROUND: Machine learning (ML) has garnered increasing attention as a means to quantitatively analyze the growing and complex medical data to improve individualized patient care. We herein aim to critically examine the current state of ML in predicting surgical outcomes, evaluate the quality of currently available research, and propose areas of improvement for future uses of ML in surgery. METHODS: A systematic review was conducted in accordance with the Preferred Reporting Items for a Systematic Review and Meta-Analysis (PRISMA) checklist. PubMed, MEDLINE, and Embase databases were reviewed under search syntax "machine learning" and "surgery" for papers published between 2015 and 2020. RESULTS: Of the initial 2677 studies, 45 papers met inclusion and exclusion criteria. Fourteen different subspecialties were represented with neurosurgery being most common. The most frequently used ML algorithms were random forest (n = 19), artificial neural network (n = 17), and logistic regression (n = 17). Common outcomes included postoperative mortality, complications, patient reported quality of life and pain improvement. All studies which compared ML algorithms to conventional studies which used area under the curve (AUC) to measure accuracy found improved outcome prediction with ML models. CONCLUSIONS: While still in its early stages, ML models offer surgeons an opportunity to capitalize on the myriad of clinical data available and improve individualized patient care. Limitations included heterogeneous outcome and imperfect quality of some of the papers. We therefore urge future research to agree upon methods of outcome reporting and require basic quality standards.