ABSTRACT
BACKGROUND: Data drift can negatively impact the performance of machine learning algorithms (MLAs) that were trained on historical data. As such, MLAs should be continuously monitored and tuned to overcome the systematic changes that occur in the distribution of data. In this paper, we study the extent of data drift and provide insights about its characteristics for sepsis onset prediction. This study will help elucidate the nature of data drift for prediction of sepsis and similar diseases. This may aid with the development of more effective patient monitoring systems that can stratify risk for dynamic disease states in hospitals. METHODS: We devise a series of simulations that measure the effects of data drift in patients with sepsis, using electronic health records (EHR). We simulate multiple scenarios in which data drift may occur, namely the change in the distribution of the predictor variables (covariate shift), the change in the statistical relationship between the predictors and the target (concept shift), and the occurrence of a major healthcare event (major event) such as the COVID-19 pandemic. We measure the impact of data drift on model performances, identify the circumstances that necessitate model retraining, and compare the effects of different retraining methodologies and model architecture on the outcomes. We present the results for two different MLAs, eXtreme Gradient Boosting (XGB) and Recurrent Neural Network (RNN). RESULTS: Our results show that the properly retrained XGB models outperform the baseline models in all simulation scenarios, hence signifying the existence of data drift. In the major event scenario, the area under the receiver operating characteristic curve (AUROC) at the end of the simulation period is 0.811 for the baseline XGB model and 0.868 for the retrained XGB model. In the covariate shift scenario, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.853 and 0.874 respectively. In the concept shift scenario and under the mixed labeling method, the retrained XGB models perform worse than the baseline model for most simulation steps. However, under the full relabeling method, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.852 and 0.877 respectively. The results for the RNN models were mixed, suggesting that retraining based on a fixed network architecture may be inadequate for an RNN. We also present the results in the form of other performance metrics such as the ratio of observed to expected probabilities (calibration) and the normalized rate of positive predictive values (PPV) by prevalence, referred to as lift, at a sensitivity of 0.8. CONCLUSION: Our simulations reveal that retraining periods of a couple of months or using several thousand patients are likely to be adequate to monitor machine learning models that predict sepsis. This indicates that a machine learning system for sepsis prediction will probably need less infrastructure for performance monitoring and retraining compared to other applications in which data drift is more frequent and continuous. Our results also show that in the event of a concept shift, a full overhaul of the sepsis prediction model may be necessary because it indicates a discrete change in the definition of sepsis labels, and mixing the labels for the sake of incremental training may not produce the desired results.
Subject(s)
COVID-19 , Communicable Diseases , Sepsis , Humans , Pandemics , COVID-19/diagnosis , Sepsis/diagnosis , Machine LearningABSTRACT
BACKGROUND: Acute respiratory distress syndrome (ARDS) is a condition that is often considered to have broad and subjective diagnostic criteria and is associated with significant mortality and morbidity. Early and accurate prediction of ARDS and related conditions such as hypoxemia and sepsis could allow timely administration of therapies, leading to improved patient outcomes. OBJECTIVE: The aim of this study is to perform an exploration of how multilabel classification in the clinical setting can take advantage of the underlying dependencies between ARDS and related conditions to improve early prediction of ARDS in patients. METHODS: The electronic health record data set included 40,703 patient encounters from 7 hospitals from April 20, 2018, to March 17, 2021. A recurrent neural network (RNN) was trained using data from 5 hospitals, and external validation was conducted on data from 2 hospitals. In addition to ARDS, 12 target labels for related conditions such as sepsis, hypoxemia, and COVID-19 were used to train the model to classify a total of 13 outputs. As a comparator, XGBoost models were developed for each of the 13 target labels. Model performance was assessed using the area under the receiver operating characteristic curve. Heat maps to visualize attention scores were generated to provide interpretability to the neural networks. Finally, cluster analysis was performed to identify potential phenotypic subgroups of patients with ARDS. RESULTS: The single RNN model trained to classify 13 outputs outperformed the individual XGBoost models for ARDS prediction, achieving an area under the receiver operating characteristic curve of 0.842 on the external test sets. Models trained on an increasing number of tasks resulted in improved performance. Earlier prediction of ARDS nearly doubled the rate of in-hospital survival. Cluster analysis revealed distinct ARDS subgroups, some of which had similar mortality rates but different clinical presentations. CONCLUSIONS: The RNN model presented in this paper can be used as an early warning system to stratify patients who are at risk of developing one of the multiple risk outcomes, hence providing practitioners with the means to take early action.
ABSTRACT
BACKGROUND: Pulmonary embolism (PE) is a life-threatening condition associated with ~10% of deaths of hospitalized patients. Machine learning algorithms (MLAs) which predict the onset of pulmonary embolism (PE) could enable earlier treatment and improve patient outcomes. However, the extent to which they generalize to broader patient populations impacts their clinical utility. OBJECTIVE: To conduct the first large-scale external validation of a machine learning-based PE prediction model which uses EHR data from the first three hours of a patient's hospital stay to predict the occurrence of PE within the next 10 days of the inpatient stay. METHODS: This retrospective study included approximately two million adult hospital admissions across 44 medical institutions in the US from 2011 to 2017. Demographics, vital signs, and lab tests from adult inpatients at 12 institutions (n = 331,268; 3.3% PE positive) were used for training an XGBoost model. External validation of the model was conducted on patient populations from each of 32 medical institutions (total n = 1,660,715; 3.7% PE positive) without retraining. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC). Backward elimination regression was used to identify correlations between characteristics of the external validation sets and AUROC. RESULTS: The model performed well (AUROC = 0.87) on the 20% hold-out subset of the training set. Despite demographic differences between the 32 external validation populations (percent PE positive: min = 1.54%, max = 6.47%), without retraining, the model had excellent discrimination, with a mean AUROC of 0.88 (min = 0.79, max = 0.93). Fixing sensitivity at 0.80, the model had a mean specificity of 0.85 (min = 0.64, max = 0.93). Backward elimination regression identified a negative association (ß = -0.015, p < 0.001) between the percentage of PE positive encounters and AUROC. CONCLUSIONS: A PE prediction model performed remarkably well across 32 different external patient populations without retraining and despite significant differences in demographic characteristics, demonstrating its generalizability and potential as a clinical decision support tool to aid PE detection and improve patient outcomes in a clinical setting.
Subject(s)
Machine Learning , Pulmonary Embolism , Adult , Algorithms , Humans , Pulmonary Embolism/diagnosis , ROC Curve , Retrospective StudiesABSTRACT
During embryogenesis, the initial vascular network forms by the process of vasculogenesis, or the specification of vascular progenitors de novo. In contrast, the majority of later-forming vessels arise by angiogenesis from the already established vasculature. Here, we show that new vascular progenitors in zebrafish embryos emerge from a distinct site along the yolk extension, or secondary vascular field (SVF), incorporate into the posterior cardinal vein, and contribute to subintestinal vasculature even after blood circulation has been initiated. We further demonstrate that SVF cells participate in vascular recovery after chemical ablation of vascular endothelial cells. Inducible inhibition of the function of vascular progenitor marker etv2/etsrp prevented SVF cell differentiation and resulted in the defective formation of subintestinal vasculature. Similar late-forming etv2+ progenitors were also observed in mouse embryos, suggesting that SVF cells are evolutionarily conserved. Our results characterize a distinct mechanism by which new vascular progenitors incorporate into established vasculature.
Subject(s)
Zebrafish Proteins , Zebrafish , Animals , Blood Vessels/metabolism , Embryo, Nonmammalian/metabolism , Endothelial Cells/metabolism , Gene Expression Regulation, Developmental , Mice , Neovascularization, Physiologic , Zebrafish/metabolism , Zebrafish Proteins/metabolismABSTRACT
During embryonic development, cells differentiate into a variety of distinct cell types and subtypes with diverse transcriptional profiles. To date, transcriptomic signatures of different cell lineages that arise during development have been only partially characterized. Here we used single-cell RNA-seq to perform transcriptomic analysis of over 20,000 cells disaggregated from the trunk region of zebrafish embryos at the 30 hpf stage. Transcriptional signatures of 27 different cell types and subtypes were identified and annotated during this analysis. This dataset will be a useful resource for many researchers in the fields of developmental and cellular biology and facilitate the understanding of molecular mechanisms that regulate cell lineage choices during development.
Subject(s)
Embryo, Nonmammalian/metabolism , Gene Expression Profiling , Single-Cell Analysis , Torso/embryology , Zebrafish/embryology , Zebrafish/genetics , Animals , Cell Lineage/genetics , Ectoderm/cytology , Ectoderm/embryology , Endoderm/cytology , Endoderm/embryology , Endothelium, Vascular/cytology , Endothelium, Vascular/embryology , Erythrocytes/metabolism , Fibroblasts/cytology , Gene Expression Regulation, Developmental , Mesoderm/cytology , Mesoderm/embryology , Muscle, Skeletal/embryology , Muscle, Skeletal/metabolism , Zebrafish Proteins/genetics , Zebrafish Proteins/metabolismABSTRACT
ETS transcription factors play an important role in the specification and differentiation of endothelial cells during vascular development. Despite previous studies, the role of the founding member of the ETS family, Ets1, in vascular development in vivo is only partially understood. Here, we generated a zebrafish ets1 mutant by TALEN genome editing and tested functional redundancy between Ets1 and a related ETS factor Etv2/Etsrp/ER71. While zebrafish ets1-/- mutants have a normal functional vascular system, etv2-/-;ets1-/embryos had more severe angiogenic defects and lower expression levels of kdr and kdrl, the two zebrafish homologs of the mammalian Vascular Endothelial Growth Factor Receptor 2 VEGFR2/Flk1, than etv2-/-embryos. Expression of constitutively active Mitogen-Activated Protein Kinase1 (MAP2K1) within endothelial cells partially rescued this angiogenic defect. Interestingly, ets1-/- embryos displayed extensive apoptosis within the trunk vasculature despite exhibiting normal vascular patterning. Loss of Ets1 combined with a partial knockdown of Etv2 function resulted in a decrease in endothelial cell numbers in the axial vasculature, which argues for a role of Ets1 in promoting vasculogenesis. We also demonstrate that although both Ets1 and Etv2 can induce ectopic vascular marker expression in zebrafish embryos, Ets1 activity is dependent on MAPK-mediated phosphorylation of its Thr30 and Ser33 residues, while Etv2 activity is not. Together, our results identify a novel function of Ets1 in regulating endothelial cell survival during vasculogenesis in vivo. Based on these findings, we propose a revised model of how Ets1 and Etv2 play unique and partially redundant roles to promote vascular development.
Subject(s)
Embryo, Nonmammalian/embryology , Neovascularization, Physiologic , Proto-Oncogene Protein c-ets-1/metabolism , Zebrafish Proteins/metabolism , Zebrafish/embryology , Animals , Apoptosis , Gene Knockout Techniques , MAP Kinase Kinase 1/genetics , MAP Kinase Kinase 1/metabolism , Proto-Oncogene Protein c-ets-1/genetics , Vascular Endothelial Growth Factor Receptor-2/genetics , Vascular Endothelial Growth Factor Receptor-2/metabolism , Zebrafish/genetics , Zebrafish Proteins/geneticsABSTRACT
Cell fate decisions involved in vascular and hematopoietic embryonic development are still poorly understood. An ETS transcription factor Etv2 functions as an evolutionarily conserved master regulator of vasculogenesis. Here we report a single-cell transcriptomic analysis of hematovascular development in wild-type and etv2 mutant zebrafish embryos. Distinct transcriptional signatures of different types of hematopoietic and vascular progenitors are identified using an etv2ci32Gt gene trap line, in which the Gal4 transcriptional activator is integrated into the etv2 gene locus. We observe a cell population with a skeletal muscle signature in etv2-deficient embryos. We demonstrate that multiple etv2ci32Gt; UAS:GFP cells differentiate as skeletal muscle cells instead of contributing to vasculature in etv2-deficient embryos. Wnt and FGF signaling promote the differentiation of these putative multipotent etv2 progenitor cells into skeletal muscle cells. We conclude that etv2 actively represses muscle differentiation in vascular progenitors, thus restricting these cells to a vascular endothelial fate.
Subject(s)
Blood Vessels/cytology , Gene Expression Profiling , Muscle, Skeletal/cytology , Single-Cell Analysis , Stem Cells/metabolism , Zebrafish Proteins/deficiency , Zebrafish/genetics , Animals , Animals, Genetically Modified , Biomarkers/metabolism , Cell Differentiation/genetics , Cell Movement , Embryo, Nonmammalian/metabolism , Fibroblast Growth Factors/metabolism , Green Fluorescent Proteins/metabolism , Heat-Shock Response , Models, Biological , Mutation/genetics , Somites/metabolism , Transcription, Genetic , Wnt Signaling Pathway , Zebrafish/embryology , Zebrafish Proteins/genetics , Zebrafish Proteins/metabolismABSTRACT
Vasculogenesis involves the differentiation of vascular endothelial progenitors de novo from undifferentiated mesoderm, their migration and coalescence to form the major embryonic vessels and the acquisition of arterial or venous identity. Vascular Endothelial Growth Factor (Vegf) signaling plays multiple roles during vascular development. However, its function during embryonic vasculogenesis has been controversial. Previous studies have implicated Vegf signaling in either regulating arteriovenous specification or overall vascular endothelial differentiation. To clarify the role of Vegf in embryonic vasculogenesis and identify its downstream targets, we used chemical inhibitors of Vegf receptor (Vegfr) signaling in zebrafish embryos as well as zebrafish genetic mutants. A high level of chemical inhibition of Vegfr signaling resulted in the reduction of overall vascular endothelial marker gene expression, including downregulation of both arterial and venous markers, ultimately leading to the apoptosis of vascular endothelial cells. In contrast, a low level of Vegfr inhibition specifically blocked arterial specification while the expression of venous markers appeared largely unaffected or increased. Inhibition of Vegfr signaling prior to the initiation of vasculogenesis reduced overall vascular endothelial differentiation, while inhibition of Vegfr signaling starting at mid-somitogenesis stages largely inhibited arterial specification. Conversely, Vegf overexpression resulted in the expansion of both arterial and pan-endothelial markers, while the expression of several venous-specific markers was downregulated. We further show that Vegf signaling affects overall endothelial differentiation by modulating the expression of the ETS transcription factor etv2/ etsrp. etv2 expression was downregulated in Vegfr- inhibited embryos, and expanded in Vegfaa-overexpressing embryos. Furthermore, vascular-specific overexpression of etv2 in Vegfr-inhibited embryos rescued defects in vascular endothelial differentiation. Similarly, vegfaa genetic mutants displayed a combination of the two phenotypes observed with chemical Vegfr inhibition: the expression of arterial and pan-endothelial markers including etv2 was downregulated while the expression of most venous markers was either expanded or unchanged. Based on these results we propose a revised model which explains the different phenotypes observed upon inhibition of Vegf signaling: low levels of Vegf signaling promote overall vascular endothelial differentiation and cell survival by upregulating etv2 expression, while high levels of Vegf signaling promote arterial and inhibit venous specification.