Development and external validation of deep learning clinical prediction models using variable-length time series data.

OBJECTIVES: To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection). MATERIALS AND METHODS: This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory/gated recurrent unit [LSTM/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC). RESULTS: The study comprised 373 825 admissions for training and 256 128 admissions for testing. LSTM/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM/GRU achieved the highest AUPRC in all tasks. DISCUSSION: When externally validated in 3 clinical tasks, the LSTM/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC. CONCLUSION: The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.

Automated stratification of trauma injury severity across multiple body regions using multi-modal, multi-class machine learning models.

OBJECTIVE: The timely stratification of trauma injury severity can enhance the quality of trauma care but it requires intense manual annotation from certified trauma coders. The objective of this study is to develop machine learning models for the stratification of trauma injury severity across various body regions using clinical text and structured electronic health records (EHRs) data. MATERIALS AND METHODS: Our study utilized clinical documents and structured EHR variables linked with the trauma registry data to create 2 machine learning models with different approaches to representing text. The first one fuses concept unique identifiers (CUIs) extracted from free text with structured EHR variables, while the second one integrates free text with structured EHR variables. Temporal validation was undertaken to ensure the models' temporal generalizability. Additionally, analyses to assess the variable importance were conducted. RESULTS: Both models demonstrated impressive performance in categorizing leg injuries, achieving high accuracy with macro-F1 scores of over 0.8. Additionally, they showed considerable accuracy, with macro-F1 scores exceeding or near 0.7, in assessing injuries in the areas of the chest and head. We showed in our variable importance analysis that the most important features in the model have strong face validity in determining clinically relevant trauma injuries. DISCUSSION: The CUI-based model achieves comparable performance, if not higher, compared to the free-text-based model, with reduced complexity. Furthermore, integrating structured EHR data improves performance, particularly when the text modalities are insufficiently indicative. CONCLUSIONS: Our multi-modal, multiclass models can provide accurate stratification of trauma injury severity and clinically relevant interpretations.

Development of a Human Evaluation Framework and Correlation with Automated Metrics for Natural Language Generation of Medical Diagnoses.

In the evolving landscape of clinical Natural Language Generation (NLG), assessing abstractive text quality remains challenging, as existing methods often overlook generative task complexities. This work aimed to examine the current state of automated evaluation metrics in NLG in healthcare. To have a robust and well-validated baseline with which to examine the alignment of these metrics, we created a comprehensive human evaluation framework. Employing ChatGPT-3.5-turbo generative output, we correlated human judgments with each metric. None of the metrics demonstrated high alignment; however, the SapBERT score-a Unified Medical Language System (UMLS)- showed the best results. This underscores the importance of incorporating domain-specific knowledge into evaluation efforts. Our work reveals the deficiency in quality evaluations for generated text and introduces our comprehensive human evaluation framework as a baseline. Future efforts should prioritize integrating medical knowledge databases to enhance the alignment of automated metrics, particularly focusing on refining the SapBERT score for improved assessments.

Alcohol Withdrawal Severity Measures for Identifying Patients Requiring High-Intensity Care.

OBJECTIVES: Alcohol withdrawal syndrome (AWS) may progress to require high-intensity care. Approaches to identify hospitalized patients with AWS who received higher level of care have not been previously examined. This study aimed to examine the utility of Clinical Institute Withdrawal Assessment Alcohol Revised (CIWA-Ar) for alcohol scale scores and medication doses for alcohol withdrawal management in identifying patients who received high-intensity care. DESIGN: A multicenter observational cohort study of hospitalized adults with alcohol withdrawal. SETTING: University of Chicago Medical Center and University of Wisconsin Hospital. PATIENTS: Inpatient encounters between November 2008 and February 2022 with a CIWA-Ar score greater than 0 and benzodiazepine or barbiturate administered within the first 24 hours. The primary composite outcome was patients who progressed to high-intensity care (intermediate care or ICU). INTERVENTIONS: None. MAIN RESULTS: Among the 8742 patients included in the study, 37.5% (n = 3280) progressed to high-intensity care. The odds ratio for the composite outcome increased above 1.0 when the CIWA-Ar score was 24. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) at this threshold were 0.12 (95% CI, 0.11-0.13), 0.95 (95% CI, 0.94-0.95), 0.58 (95% CI, 0.54-0.61), and 0.64 (95% CI, 0.63-0.65), respectively. The OR increased above 1.0 at a 24-hour lorazepam milligram equivalent dose cutoff of 15 mg. The sensitivity, specificity, PPV, and NPV at this threshold were 0.16 (95% CI, 0.14-0.17), 0.96 (95% CI, 0.95-0.96), 0.68 (95% CI, 0.65-0.72), and 0.65 (95% CI, 0.64-0.66), respectively. CONCLUSIONS: Neither CIWA-Ar scores nor medication dose cutoff points were effective measures for identifying patients with alcohol withdrawal who received high-intensity care. Research studies for examining outcomes in patients who deteriorate with AWS will require better methods for cohort identification.

Development and Validation of a Risk Score Predicting Death Without Transplant in Adult Heart Transplant Candidates.

Importance: The US heart allocation system prioritizes medically urgent candidates with a high risk of dying without transplant. The current therapy-based 6-status system is susceptible to manipulation and has limited rank ordering ability. Objective: To develop and validate a candidate risk score that incorporates current clinical, laboratory, and hemodynamic data. Design, Setting, and Participants: A registry-based observational study of adult heart transplant candidates (aged ≥18 years) from the US heart allocation system listed between January 1, 2019, and December 31, 2022, split by center into training (70%) and test (30%) datasets. Adult candidates were listed between January 1, 2019, and December 31, 2022. Main Outcomes and Measures: A US candidate risk score (US-CRS) model was developed by adding a predefined set of predictors to the current French Candidate Risk Score (French-CRS) model. Sensitivity analyses were performed, which included intra-aortic balloon pumps (IABP) and percutaneous ventricular assist devices (VAD) in the definition of short-term mechanical circulatory support (MCS) for the US-CRS. Performance of the US-CRS model, French-CRS model, and 6-status model in the test dataset was evaluated by time-dependent area under the receiver operating characteristic curve (AUC) for death without transplant within 6 weeks and overall survival concordance (c-index) with integrated AUC. Results: A total of 16 905 adult heart transplant candidates were listed (mean [SD] age, 53 [13] years; 73% male; 58% White); 796 patients (4.7%) died without a transplant. The final US-CRS contained time-varying short-term MCS (ventricular assist-extracorporeal membrane oxygenation or temporary surgical VAD), the log of bilirubin, estimated glomerular filtration rate, the log of B-type natriuretic peptide, albumin, sodium, and durable left ventricular assist device. In the test dataset, the AUC for death within 6 weeks of listing for the US-CRS model was 0.79 (95% CI, 0.75-0.83), for the French-CRS model was 0.72 (95% CI, 0.67-0.76), and 6-status model was 0.68 (95% CI, 0.62-0.73). Overall c-index for the US-CRS model was 0.76 (95% CI, 0.73-0.80), for the French-CRS model was 0.69 (95% CI, 0.65-0.73), and 6-status model was 0.67 (95% CI, 0.63-0.71). Classifying IABP and percutaneous VAD as short-term MCS reduced the effect size by 54%. Conclusions and Relevance: In this registry-based study of US heart transplant candidates, a continuous multivariable allocation score outperformed the 6-status system in rank ordering heart transplant candidates by medical urgency and may be useful for the medical urgency component of heart allocation.

Predicting transfers to intensive care in children using CEWT and other early warning systems.

Background and Objective: The Children's Early Warning Tool (CEWT), developed in Australia, is widely used in many countries to monitor the risk of deterioration in hospitalized children. Our objective was to compare CEWT prediction performance against a version of the Bedside Pediatric Early Warning Score (Bedside PEWS), Between the Flags (BTF), and the pediatric Calculated Assessment of Risk and Triage (pCART). Methods: We conducted a retrospective observational study of all patient admissions to the Comer Children's Hospital at the University of Chicago between 2009-2019. We compared performance for predicting the primary outcome of a direct ward-to-intensive care unit (ICU) transfer within the next 12 h using the area under the receiver operating characteristic curve (AUC). Alert rates at various score thresholds were also compared. Results: Of 50,815 ward admissions, 1,874 (3.7%) experienced the primary outcome. Among patients in Cohort 1 (years 2009-2017, on which the machine learning-based pCART was trained), CEWT performed slightly worse than Bedside PEWS but better than BTF (CEWT AUC 0.74 vs. Bedside PEWS 0.76, P < 0.001; vs. BTF 0.66, P < 0.001), while pCART performed best for patients in Cohort 2 (years 2018-2019, pCART AUC 0.84 vs. CEWT AUC 0.79, P < 0.001; vs. BTF AUC 0.67, P < 0.001; vs. Bedside PEWS 0.80, P < 0.001). Sensitivity, specificity, and positive predictive values varied across all four tools at the examined thresholds for alerts. Conclusion: CEWT has good discrimination for predicting which patients will likely be transferred to the ICU, while pCART performed the best.

The Pediatric Data Science and Analytics Subgroup of the Pediatric Acute Lung Injury and Sepsis Investigators Network: Use of Supervised Machine Learning Applications in Pediatric Critical Care Medicine Research.

OBJECTIVE: Perform a scoping review of supervised machine learning in pediatric critical care to identify published applications, methodologies, and implementation frequency to inform best practices for the development, validation, and reporting of predictive models in pediatric critical care. DESIGN: Scoping review and expert opinion. SETTING: We queried CINAHL Plus with Full Text (EBSCO), Cochrane Library (Wiley), Embase (Elsevier), Ovid Medline, and PubMed for articles published between 2000 and 2022 related to machine learning concepts and pediatric critical illness. Articles were excluded if the majority of patients were adults or neonates, if unsupervised machine learning was the primary methodology, or if information related to the development, validation, and/or implementation of the model was not reported. Article selection and data extraction were performed using dual review in the Covidence tool, with discrepancies resolved by consensus. SUBJECTS: Articles reporting on the development, validation, or implementation of supervised machine learning models in the field of pediatric critical care medicine. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Of 5075 identified studies, 141 articles were included. Studies were primarily (57%) performed at a single site. The majority took place in the United States (70%). Most were retrospective observational cohort studies. More than three-quarters of the articles were published between 2018 and 2022. The most common algorithms included logistic regression and random forest. Predicted events were most commonly death, transfer to ICU, and sepsis. Only 14% of articles reported external validation, and only a single model was implemented at publication. Reporting of validation methods, performance assessments, and implementation varied widely. Follow-up with authors suggests that implementation remains uncommon after model publication. CONCLUSIONS: Publication of supervised machine learning models to address clinical challenges in pediatric critical care medicine has increased dramatically in the last 5 years. While these approaches have the potential to benefit children with critical illness, the literature demonstrates incomplete reporting, absence of external validation, and infrequent clinical implementation.

Development and external validation of multimodal postoperative acute kidney injury risk machine learning models.

Objectives: To develop and externally validate machine learning models using structured and unstructured electronic health record data to predict postoperative acute kidney injury (AKI) across inpatient settings. Materials and Methods: Data for adult postoperative admissions to the Loyola University Medical Center (2009-2017) were used for model development and admissions to the University of Wisconsin-Madison (2009-2020) were used for validation. Structured features included demographics, vital signs, laboratory results, and nurse-documented scores. Unstructured text from clinical notes were converted into concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. The primary outcome was the development of Kidney Disease Improvement Global Outcomes stage 2 AKI within 7 days after leaving the operating room. We derived unimodal extreme gradient boosting machines (XGBoost) and elastic net logistic regression (GLMNET) models using structured-only data and multimodal models combining structured data with CUI features. Model comparison was performed using the receiver operating characteristic curve (AUROC), with Delong's test for statistical differences. Results: The study cohort included 138 389 adult patient admissions (mean [SD] age 58 [16] years; 11 506 [8%] African-American; and 70 826 [51%] female) across the 2 sites. Of those, 2959 (2.1%) developed stage 2 AKI or higher. Across all data types, XGBoost outperformed GLMNET (mean AUROC 0.81 [95% confidence interval (CI), 0.80-0.82] vs 0.78 [95% CI, 0.77-0.79]). The multimodal XGBoost model incorporating CUIs parameterized as term frequency-inverse document frequency (TF-IDF) showed the highest discrimination performance (AUROC 0.82 [95% CI, 0.81-0.83]) over unimodal models (AUROC 0.79 [95% CI, 0.78-0.80]). Discussion: A multimodality approach with structured data and TF-IDF weighting of CUIs increased model performance over structured data-only models. Conclusion: These findings highlight the predictive power of CUIs when merged with structured data for clinical prediction models, which may improve the detection of postoperative AKI.

Creation of a data commons for substance misuse related health research through privacy-preserving patient record linkage between hospitals and state agencies.

Objectives: Substance misuse is a complex and heterogeneous set of conditions associated with high mortality and regional/demographic variations. Existing data systems are siloed and have been ineffective in curtailing the substance misuse epidemic. Therefore, we aimed to build a novel informatics platform, the Substance Misuse Data Commons (SMDC), by integrating multiple data modalities to provide a unified record of information crucial to improving outcomes in substance misuse patients. Materials and Methods: The SMDC was created by linking electronic health record (EHR) data from adult cases of substance (alcohol, opioid, nonopioid drug) misuse at the University of Wisconsin hospitals to socioeconomic and state agency data. To ensure private and secure data exchange, Privacy-Preserving Record Linkage (PPRL) and Honest Broker services were utilized. The overlap in mortality reporting among the EHR, state Vital Statistics, and a commercial national data source was assessed. Results: The SMDC included data from 36 522 patients experiencing 62 594 healthcare encounters. Over half of patients were linked to the statewide ambulance database and prescription drug monitoring program. Chronic diseases accounted for most underlying causes of death, while drug-related overdoses constituted 8%. Our analysis of mortality revealed a 49.1% overlap across the 3 data sources. Nonoverlapping deaths were associated with poor socioeconomic indicators. Discussion: Through PPRL, the SMDC enabled the longitudinal integration of multimodal data. Combining death data from local, state, and national sources enhanced mortality tracking and exposed disparities. Conclusion: The SMDC provides a comprehensive resource for clinical providers and policymakers to inform interventions targeting substance misuse-related hospitalizations, overdoses, and death.

Hypotensive Events in Pediatric Patients Receiving Dexmedetomidine for MRI.

PURPOSE: Dexmedetomidine, the preferred pediatric sedating agent for magnetic resonance imaging (MRI), has the side effect of hypotension. Newer recommendations for reporting adverse events in pediatric procedural sedation include using a two-pronged definition. Our aim was to describe the incidence of hypotension in patients undergoing sedated MRI and to identify demographic and clinical factors associated with hypotension, applying a two-pronged definition, where a numerical threshold/clinical criterion must be met as well as at least one clinical intervention performed. DESIGN: An observational cohort study. METHODS: Medical record data were extracted for outpatients less than 18 years of age sedated primarily with dexmedetomidine for MRI in a single center for over a seven-year period. Patients who received propofol as an adjunct were also included. Hypotension was defined using a two-pronged approach, as a 20% reduction in systolic blood pressure from baseline lasting ≥10 minutes, coupled with a fluid bolus. Analysis included descriptive statistics, t tests and logistic regression using discrete-time survival analysis. FINDINGS: Of the 1,590 patient encounters, 90 (5.7%) experienced hypotension. Males were significantly more likely to have hypotension. Patients with hypotension had overall longer appointment times, including longer sedation times and recovery time. Greater blood pressure (BP) variability in the preceding 20 minutes also increased the risk of hypotension. CONCLUSIONS: Our lower incidence of hypotension is likely related to the two-pronged intervention-based definition used, as it likely more accurately reflects clinically meaningful hypotension. To our knowledge, this is the first study using this approach with this population. Research further examining the relationship between prolonged sedation, blood pressure variability, gender, hypotension, and recovery time is needed. Understanding these relationships will help interdisciplinary teams, including nurses in pediatric procedural areas, to reduce the incidence of hypotension, potentially maximize patient safety, and optimize throughput.

Automated Matching of Patients to Clinical Trials: A Patient-Centric Natural Language Processing Approach for Pediatric Leukemia.

PURPOSE: Matching patients to clinical trials is cumbersome and costly. Attempts have been made to automate the matching process; however, most have used a trial-centric approach, which focuses on a single trial. In this study, we developed a patient-centric matching tool that matches patient-specific demographic and clinical information with free-text clinical trial inclusion and exclusion criteria extracted using natural language processing to return a list of relevant clinical trials ordered by the patient's likelihood of eligibility. MATERIALS AND METHODS: Records from pediatric leukemia clinical trials were downloaded from ClinicalTrials.gov. Regular expressions were used to discretize and extract individual trial criteria. A multilabel support vector machine (SVM) was trained to classify sentence embeddings of criteria into relevant clinical categories. Labeled criteria were parsed using regular expressions to extract numbers, comparators, and relationships. In the validation phase, a patient-trial match score was generated for each trial and returned in the form of a ranked list for each patient. RESULTS: In total, 5,251 discretized criteria were extracted from 216 protocols. The most frequent criterion was previous chemotherapy/biologics (17%). The multilabel SVM demonstrated a pooled accuracy of 75%. The text processing pipeline was able to automatically extract 68% of eligibility criteria rules, as compared with 80% in a manual version of the tool. Automated matching was accomplished in approximately 4 seconds, as compared with several hours using manual derivation. CONCLUSION: To our knowledge, this project represents the first open-source attempt to generate a patient-centric clinical trial matching tool. The tool demonstrated acceptable performance when compared with a manual version, and it has potential to save time and money when matching patients to trials.

Association between mortality and critical events within 48 hours of transfer to the pediatric intensive care unit.

Introduction: Critical deterioration in hospitalized children, defined as ward to pediatric intensive care unit (PICU) transfer followed by mechanical ventilation (MV) or vasoactive infusion (VI) within 12 h, has been used as a primary metric to evaluate the effectiveness of clinical interventions or quality improvement initiatives. We explore the association between critical events (CEs), i.e., MV or VI events, within the first 48 h of PICU transfer from the ward or emergency department (ED) and in-hospital mortality. Methods: We conducted a retrospective study of a cohort of PICU transfers from the ward or the ED at two tertiary-care academic hospitals. We determined the association between mortality and occurrence of CEs within 48 h of PICU transfer after adjusting for age, gender, hospital, and prior comorbidities. Results: Experiencing a CE within 48 h of PICU transfer was associated with an increased risk of mortality [OR 12.40 (95% CI: 8.12-19.23, P < 0.05)]. The increased risk of mortality was highest in the first 12 h [OR 11.32 (95% CI: 7.51-17.15, P < 0.05)] but persisted in the 12-48 h time interval [OR 2.84 (95% CI: 1.40-5.22, P < 0.05)]. Varying levels of risk were observed when considering ED or ward transfers only, when considering different age groups, and when considering individual 12-h time intervals. Discussion: We demonstrate that occurrence of a CE within 48 h of PICU transfer was associated with mortality after adjusting for confounders. Studies focusing on the impact of quality improvement efforts may benefit from using CEs within 48 h of PICU transfer as an additional evaluation metric, provided these events could have been influenced by the initiative.

Prediction of Intracranial Hypertension and Brain Tissue Hypoxia Utilizing High-Resolution Data from the BOOST-II Clinical Trial.

The current approach to intracranial hypertension and brain tissue hypoxia is reactive, based on fixed thresholds. We used statistical machine learning on high-frequency intracranial pressure (ICP) and partial brain tissue oxygen tension (PbtO2) data obtained from the BOOST-II trial with the goal of constructing robust quantitative models to predict ICP/PbtO2 crises. We derived the following machine learning models: logistic regression (LR), elastic net, and random forest. We split the data set into 70-30% for training and testing and utilized a discrete-time survival analysis framework and 5-fold hyperparameter optimization strategy for all models. We compared model performances on discrimination between events and non-events of increased ICP or low PbtO2 with the area under the receiver operating characteristic (AUROC) curve. We further analyzed clinical utility through a decision curve analysis (DCA). When considering discrimination, the number of features, and interpretability, we identified the RF model that combined the most recent ICP reading, episode number, and longitudinal trends over the preceding 30 min as the best performing for predicting ICP crisis events within the next 30 min (AUC 0.78). For PbtO2, the LR model utilizing the most recent reading, episode number, and longitudinal trends over the preceding 30 min was the best performing (AUC, 0.84). The DCA showed clinical usefulness for wide risk of thresholds for both ICP and PbtO2 predictions. Acceptable alerting thresholds could range from 20% to 80% depending on a patient-specific assessment of the benefit-risk ratio of a given intervention in response to the alert.

Association Between Critical Events in the PICU and Outcomes in Neighboring Patients.

PICU patients who experience critical illness events, such as intubation, are at high risk for morbidity and mortality. Little is known about the impact of these events, which require significant resources, on outcomes in other patients. Therefore, we aimed to assess the association between critical events in PICU patients and the risk of similar events in neighboring patients over the next 6 hours. DESIGN: Retrospective observational cohort study. SETTING: Quaternary care PICU at the University of Chicago. PATIENTS: All children admitted to the PICU between 2012 and 2019. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: The primary outcome was a critical event defined as the initiation of invasive ventilation, initiating vasoactive medications, cardiac arrest, or death. The exposure was the occurrence of a critical event among other patients in the PICU within the preceding 6 hours. Discrete-time survival analysis using fixed 6-hour blocks beginning at the time of PICU admission was used to model the risk of experiencing a critical event in the PICU when an event occurred in the prior 6 hours. There were 13,628 admissions, of which 1,886 (14%) had a critical event. The initiation of mechanical ventilation was the most frequent event (n = 1585; 59%). In the fully adjusted analysis, there was a decreased risk of critical events (odds ratio, 0.82; 95% CI, 0.70-0.96) in the 6 hours following exposure to a critical event. This association was not present when considering longer intervals and was more pronounced in patients younger than 6 years old when compared with patients 7 years and older. CONCLUSION: Critical events in PICU patients are associated with decreased risk of similar events in neighboring patients. Further studies targeted toward exploring the mechanism behind this effect as well as identification of other nonpatient factors that adversely affect outcomes in children are warranted.

Early Diagnosis of Primary Immunodeficiency Disease Using Clinical Data and Machine Learning.

BACKGROUND: Primary immunodeficiency diseases (PIDD) are a group of immune-related disorders that have a current median delay of diagnosis between 6 and 9 years. Early diagnosis and treatment of PIDD has been associated with improved patient outcomes. OBJECTIVE: To develop a machine learning model using elements within the electronic health record data that are related to prior symptomatic treatment to predict PIDD. METHODS: We conducted a retrospective study of patients with PIDD identified using inclusion criteria of PIDD-related diagnoses, immunodeficiency-specific medications, and low immunoglobulin levels. We constructed a control group of age-, sex-, and race-matched patients with asthma. The primary outcome was the diagnosis of PIDD. We considered comorbidities, laboratory tests, medications, and radiological orders as features, all before diagnosis and indicative of symptom-related treatment. Features were presented sequentially to logistic regression, elastic net, and random forest classifiers, which were trained using a nested cross-validation approach. RESULTS: Our cohort consisted of 6422 patients, of whom 247 (4%) were diagnosed with PIDD. Our logistic regression model with comorbidities demonstrated good discrimination between patients with PIDD and those with asthma (c-statistic: 0.62 [0.58-0.65]). Adding laboratory results, medications, and radiological orders improved discrimination (c-statistic: 0.70 vs 0.62, P < .001), sensitivity, and specificity. Extending to the advanced machine learning models did not improve performance. CONCLUSIONS: We developed a prediction model for early diagnosis of PIDD using historical data that are related to symptomatic care, which has potential to fill an important need in reducing the time to diagnose PIDD, leading to better outcomes for immunodeficient patients.

PICU Survivorship: Factors Affecting Feasibility and Cohort Retention in a Long-Term Outcomes Study.

Our understanding of longitudinal outcomes of Pediatric Intensive Care Unit (PICU) survivors is limited by the heterogeneity of follow-up intervals, populations, and outcomes assessed. We sought to demonstrate (1) the feasibility of longitudinal multidimensional outcome assessment and (2) methods to promote cohort retention. The objective of this presented study was to provide details of follow-up methodology in a PICU survivor cohort and not to present the outcomes at long-term follow-up for this cohort. We enrolled 152 children aged 0 to 17 years admitted to the PICU in a prospective longitudinal cohort study. We examined resource utilization, family impact of critical illness, and neurodevelopment using the PICU Outcomes Portfolio (POP) Survey which included a study-specific survey and validated tools: 1. Functional Status Scale, 2. Pediatric Evaluation of Disability Inventory Computer Adaptive Test, 3. Pediatric Quality of Life Inventory, 4. Strengths and Difficulties Questionnaire, and 5. Vanderbilt Assessment Scales for Attention Deficit-Hyperactivity Disorder. POP Survey completion rates were 89%, 78%, and 84% at 1, 3, and 6 months. Follow-up rates at 1, 2, and 3 years were 80%, 55%, and 43%. Implementing a longitudinal multidimensional outcome portfolio for PICU survivors is feasible within an urban, tertiary-care, academic hospital. Our attrition after one year demonstrates the long-term follow-up challenges in this population. Our findings inform ongoing efforts to implement core outcome sets after pediatric critical illness.

Predicting neurological outcomes after in-hospital cardiac arrests for patients with Coronavirus Disease 2019.

BACKGROUND: Machine learning models are more accurate than standard tools for predicting neurological outcomes in patients resuscitated after cardiac arrest. However, their accuracy in patients with Coronavirus Disease 2019 (COVID-19) is unknown. Therefore, we compared their performance in a cohort of cardiac arrest patients with COVID-19. METHODS: We conducted a retrospective analysis of resuscitation survivors in the Get With The Guidelines®-Resuscitation (GWTG-R) COVID-19 registry between February 2020 and May 2021. The primary outcome was a favorable neurological outcome, indicated by a discharge Cerebral Performance Category score ≤ 2. Pre- and peri-arrest variables were used as predictors. We applied our published logistic regression, neural network, and gradient boosted machine models developed in patients without COVID-19 to the COVID-19 cohort. We also updated the neural network model using transfer learning. Performance was compared between models and the Cardiac Arrest Survival Post-Resuscitation In-Hospital (CASPRI) score. RESULTS: Among the 4,125 patients with COVID-19 included in the analysis, 484 (12 %) patients survived with favorable neurological outcomes. The gradient boosted machine, trained on non-COVID-19 patients was the best performing model for predicting neurological outcomes in COVID-19 patients, significantly better than the CASPRI score (c-statistic: 0.75 vs 0.67, P < 0.001). While calibration improved for the neural network with transfer learning, it did not surpass the gradient boosted machine in terms of discrimination. CONCLUSION: Our gradient boosted machine model developed in non-COVID patients had high discrimination and adequate calibration in COVID-19 resuscitation survivors and may provide clinicians with important information for these patients.

Identifying infected patients using semi-supervised and transfer learning.

OBJECTIVES: Early identification of infection improves outcomes, but developing models for early identification requires determining infection status with manual chart review, limiting sample size. Therefore, we aimed to compare semi-supervised and transfer learning algorithms with algorithms based solely on manual chart review for identifying infection in hospitalized patients. MATERIALS AND METHODS: This multicenter retrospective study of admissions to 6 hospitals included "gold-standard" labels of infection from manual chart review and "silver-standard" labels from nonchart-reviewed patients using the Sepsis-3 infection criteria based on antibiotic and culture orders. "Gold-standard" labeled admissions were randomly allocated to training (70%) and testing (30%) datasets. Using patient characteristics, vital signs, and laboratory data from the first 24 hours of admission, we derived deep learning and non-deep learning models using transfer learning and semi-supervised methods. Performance was compared in the gold-standard test set using discrimination and calibration metrics. RESULTS: The study comprised 432 965 admissions, of which 2724 underwent chart review. In the test set, deep learning and non-deep learning approaches had similar discrimination (area under the receiver operating characteristic curve of 0.82). Semi-supervised and transfer learning approaches did not improve discrimination over models fit using only silver- or gold-standard data. Transfer learning had the best calibration (unreliability index P value: .997, Brier score: 0.173), followed by self-learning gradient boosted machine (P value: .67, Brier score: 0.170). DISCUSSION: Deep learning and non-deep learning models performed similarly for identifying infection, as did models developed using Sepsis-3 and manual chart review labels. CONCLUSION: In a multicenter study of almost 3000 chart-reviewed patients, semi-supervised and transfer learning models showed similar performance for model discrimination as baseline XGBoost, while transfer learning improved calibration.

A prediction model for bacteremia and transfer to intensive care in pediatric and adolescent cancer patients with febrile neutropenia.

Febrile neutropenia (FN) is a common condition in children receiving chemotherapy. Our goal in this study was to develop a model for predicting blood stream infection (BSI) and transfer to intensive care (TIC) at time of presentation in pediatric cancer patients with FN. We conducted an observational cohort analysis of pediatric and adolescent cancer patients younger than 24 years admitted for fever and chemotherapy-induced neutropenia over a 7-year period. We excluded stem cell transplant recipients who developed FN after transplant and febrile non-neutropenic episodes. The primary outcome was onset of BSI, as determined by positive blood culture within 7 days of onset of FN. The secondary outcome was transfer to intensive care (TIC) within 14 days of FN onset. Predictor variables include demographics, clinical, and laboratory measures on initial presentation for FN. Data were divided into independent derivation (2009-2014) and prospective validation (2015-2016) cohorts. Prediction models were built for both outcomes using logistic regression and random forest and compared with Hakim model. Performance was assessed using area under the receiver operating characteristic curve (AUC) metrics. A total of 505 FN episodes (FNEs) were identified in 230 patients. BSI was diagnosed in 106 (21%) and TIC occurred in 56 (10.6%) episodes. The most common oncologic diagnosis with FN was acute lymphoblastic leukemia (ALL), and the highest rate of BSI was in patients with AML. Patients who had BSI had higher maximum temperature, higher rates of prior BSI and higher incidence of hypotension at time of presentation compared with patients who did not have BSI. FN patients who were transferred to the intensive care (TIC) had higher temperature and higher incidence of hypotension at presentation compared to FN patients who didn't have TIC. We compared 3 models: (1) random forest (2) logistic regression and (3) Hakim model. The areas under the curve for BSI prediction were (0.79, 0.65, and 0.64, P < 0.05) for models 1, 2, and 3, respectively. And for TIC prediction were (0.88, 0.76, and 0.65, P < 0.05) respectively. The random forest model demonstrated higher accuracy in predicting BSI and TIC and showed a negative predictive value (NPV) of 0.91 and 0.97 for BSI and TIC respectively at the best cutoff point as determined by Youden's Index. Likelihood ratios (LRs) (post-test probability) for RF model have potential utility of identifying low risk for BSI and TIC (0.24 and 0.12) and high-risk patients (3.5 and 6.8) respectively. Our prediction model has a very good diagnostic performance in clinical practices for both BSI and TIC in FN patients at the time of presentation. The model can be used to identify a group of individuals at low risk for BSI who may benefit from early discharge and reduced length of stay, also it can identify FN patients at high risk of complications who might benefit from more intensive therapies at presentation.

Multiple Organ Dysfunction Interactions in Critically Ill Children.

Introduction: Multiple organ dysfunction (MOD) is a common pathway to morbidity and death in critically ill children. Defining organ dysfunction is challenging, as we lack a complete understanding of the complex pathobiology. Current pediatric organ dysfunction criteria assign the same diagnostic value-the same "weight"- to each organ system. While each organ dysfunction in isolation contributes to the outcome, there are likely complex interactions between multiple failing organs that are not simply additive. Objective: Determine whether certain combinations of organ system dysfunctions have a significant interaction associated with higher risk of morbidity or mortality in critically ill children. Methods: We conducted a retrospective observational cohort study of critically ill children at two large academic medical centers from 2010 and 2018. Patients were included in the study if they had at least two organ dysfunctions by day 3 of PICU admission based on the Pediatric Organ Dysfunction Information Update Mandate (PODIUM) criteria. Mortality was described as absolute number of deaths and mortality rate. Combinations of two pediatric organ dysfunctions were analyzed with interaction terms as independent variables and mortality or persistent MOD as the dependent variable in logistic regression models. Results: Overall, 7,897 patients met inclusion criteria and 446 patients (5.6%) died. The organ dysfunction interactions that were significantly associated with the highest absolute number of deaths were cardiovascular + endocrinologic, cardiovascular + neurologic, and cardiovascular + respiratory. Additionally, the interactions associated with the highest mortality rates were liver + cardiovascular, respiratory + hematologic, and respiratory + renal. Among patients with persistent MOD, the most common organ dysfunctions with significant interaction terms were neurologic + respiratory, hematologic + immunologic, and endocrinologic + respiratory. Further analysis using classification and regression trees (CART) demonstrated that the absence of respiratory and liver dysfunction was associated with the lowest likelihood of mortality. Implications and Future Directions: Certain combinations of organ dysfunctions are associated with a higher risk of persistent MOD or death. Notably, the three most common organ dysfunction interactions were associated with 75% of the mortality in our cohort. Critically ill children with MOD presenting with these combinations of organ dysfunctions warrant further study.