Plasma Microbial Cell-Free DNA Sequencing in Immunocompromised Patients With Pneumonia: A Prospective Observational Study.

BACKGROUND: Pneumonia is a common cause of morbidity and mortality, yet a causative pathogen is identified in a minority of cases. Plasma microbial cell-free DNA sequencing may improve diagnostic yield in immunocompromised patients with pneumonia. METHODS: In this prospective, multicenter, observational study of immunocompromised adults undergoing bronchoscopy to establish a pneumonia etiology, plasma microbial cell-free DNA sequencing was compared to standardized usual care testing. Pneumonia etiology was adjudicated by a blinded independent committee. The primary outcome, additive diagnostic value, was assessed in the Per Protocol population (patients with complete testing results and no major protocol deviations) and defined as the percent of patients with an etiology of pneumonia exclusively identified by plasma microbial cell-free DNA sequencing. Clinical additive diagnostic value was assessed in the Per Protocol subgroup with negative usual care testing. RESULTS: Of 257 patients, 173 met Per Protocol criteria. A pneumonia etiology was identified by usual care in 52/173 (30.1%), plasma microbial cell-free DNA sequencing in 49/173 (28.3%) and the combination of both in 73/173 (42.2%) patients. Plasma microbial cell-free DNA sequencing exclusively identified an etiology of pneumonia in 21/173 patients (additive diagnostic value 12.1%, 95% confidence interval [CI], 7.7% to 18.0%, P < .001). In the Per Protocol subgroup with negative usual care testing, plasma microbial cell-free DNA sequencing identified a pneumonia etiology in 21/121 patients (clinical additive diagnostic value 17.4%, 95% CI, 11.1% to 25.3%). CONCLUSIONS: Non-invasive plasma microbial cell-free DNA sequencing significantly increased diagnostic yield in immunocompromised patients with pneumonia undergoing bronchoscopy and extensive microbiologic and molecular testing. CLINICAL TRIALS REGISTRATION: NCT04047719.

Incorporating informatively collected laboratory data from EHR in clinical prediction models.

BACKGROUND: Electronic Health Records (EHR) are widely used to develop clinical prediction models (CPMs). However, one of the challenges is that there is often a degree of informative missing data. For example, laboratory measures are typically taken when a clinician is concerned that there is a need. When data are the so-called Not Missing at Random (NMAR), analytic strategies based on other missingness mechanisms are inappropriate. In this work, we seek to compare the impact of different strategies for handling missing data on CPMs performance. METHODS: We considered a predictive model for rapid inpatient deterioration as an exemplar implementation. This model incorporated twelve laboratory measures with varying levels of missingness. Five labs had missingness rate levels around 50%, and the other seven had missingness levels around 90%. We included them based on the belief that their missingness status can be highly informational for the prediction. In our study, we explicitly compared the various missing data strategies: mean imputation, normal-value imputation, conditional imputation, categorical encoding, and missingness embeddings. Some of these were also combined with the last observation carried forward (LOCF). We implemented logistic LASSO regression, multilayer perceptron (MLP), and long short-term memory (LSTM) models as the downstream classifiers. We compared the AUROC of testing data and used bootstrapping to construct 95% confidence intervals. RESULTS: We had 105,198 inpatient encounters, with 4.7% having experienced the deterioration outcome of interest. LSTM models generally outperformed other cross-sectional models, where embedding approaches and categorical encoding yielded the best results. For the cross-sectional models, normal-value imputation with LOCF generated the best results. CONCLUSION: Strategies that accounted for the possibility of NMAR missing data yielded better model performance than those did not. The embedding method had an advantage as it did not require prior clinical knowledge. Using LOCF could enhance the performance of cross-sectional models but have countereffects in LSTM models.

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study.

BACKGROUND: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. OBJECTIVE: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. METHODS: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. RESULTS: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. CONCLUSIONS: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration.

OBJECTIVES: Previous studies have looked at National Early Warning Score performance in predicting in-hospital deterioration and death, but data are lacking with respect to patient outcomes following implementation of National Early Warning Score. We sought to determine the effectiveness of National Early Warning Score implementation on predicting and preventing patient deterioration in a clinical setting. DESIGN: Retrospective cohort study. SETTING: Tertiary care academic facility and a community hospital. PATIENTS: Patients 18 years old or older hospitalized from March 1, 2014, to February 28, 2015, during preimplementation of National Early Warning Score to August 1, 2015, to July 31, 2016, after National Early Warning Score was implemented. INTERVENTIONS: Implementation of National Early Warning Score within the electronic health record and associated best practice alert. MEASUREMENTS AND MAIN RESULTS: In this study of 85,322 patients (42,402 patients pre-National Early Warning Score and 42,920 patients post-National Early Warning Score implementation), the primary outcome of rate of ICU transfer or death did not change after National Early Warning Score implementation, with adjusted hazard ratio of 0.94 (0.84-1.05) and 0.90 (0.77-1.05) at our academic and community hospital, respectively. In total, 175,357 best practice advisories fired during the study period, with the best practice advisory performing better at the community hospital than the academic at predicting an event within 12 hours 7.4% versus 2.2% of the time, respectively. Retraining National Early Warning Score with newly generated hospital-specific coefficients improved model performance. CONCLUSIONS: At both our academic and community hospital, National Early Warning Score had poor performance characteristics and was generally ignored by frontline nursing staff. As a result, National Early Warning Score implementation had no appreciable impact on defined clinical outcomes. Refitting of the model using site-specific data improved performance and supports validating predictive models on local data.

Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Clinical Utility of Plasma Microbial Cell-Free DNA Sequencing Among Immunocompromised Patients With Pneumonia.

Background: Plasma microbial cell-free DNA (mcfDNA) sequencing can establish the etiology of multiple infectious syndromes by identifying microbial DNA in plasma. However, data are needed to define the clinical scenarios where this tool offers the highest clinical benefit. Methods: We conducted a prospective multicenter observational study that evaluated the impact of plasma mcfDNA sequencing compared with usual care testing among adults with hematologic malignancies. This is a secondary analysis of an expanded cohort that evaluated the clinical utility of plasma mcfDNA sequencing across prespecified and adjudicated outcomes. We examined the percentage of participants for whom plasma mcfDNA sequencing identified a probable cause of pneumonia or clinically relevant nonpneumonia infection. We then assessed potential changes in antimicrobial therapy based on plasma mcfDNA sequencing results and the potential for early mcfDNA testing to avoid bronchoscopy and its associated adverse events. Results: Of 223 participants, at least 1 microbial detection by plasma mcfDNA sequencing was adjudicated as a probable cause of pneumonia in 57 (25.6%) and a clinically relevant nonpneumonia infection in 88 (39.5%). A probable cause of pneumonia was exclusively identified by plasma mcfDNA sequencing in 23 (10.3%) participants. Antimicrobial therapy would have changed for 41 (18.4%) participants had plasma mcfDNA results been available in real time. Among the 57 participants with a probable cause of pneumonia identified by plasma mcfDNA sequencing, bronchoscopy identified no additional probable cause of pneumonia in 52 (91.2%). Conclusions: Plasma mcfDNA sequencing could improve management of both pneumonia and other concurrent infections in immunocompromised patients with suspected pneumonia.

The Eyes Have It-for Idiopathic Pulmonary Fibrosis: a Preliminary Observation.

INTRODUCTION: The disease origins of idiopathic pulmonary fibrosis (IPF), which occurs at higher rates in certain races/ethnicities, are not understood. The highest rates occur in white persons of European descent, particularly those with light skin, who are also susceptible to lysosomal organelle dysfunction of the skin leading to fibroproliferative disease . We had observed clinically that the vast majority of patients with IPF had light-colored eyes, suggesting a phenotypic characteristic. METHODS: We pursued this observation through a research database from the USA Veterans Administration, a population that has a high occurrence of IPF due to predominance of elderly male smokers. Using this medical records database, which included facial photos, we compared the frequency of light (blue, green, hazel) and dark (light brown, brown) eyes among white patients diagnosed with IPF compared with a control group of lung granuloma only (no other radiologic evidence of interstitial lung disease). RESULTS: Light eye color was significantly more prevalent in patients with IPF than in the control group with lung granuloma [114/147 (77.6%) versus 129/263 (49.0%], p < 0.001), indicating that light-colored eyes are a phenotype associated with IPF . CONCLUSION: We provide evidence that light eye color is predominant among white persons with IPF.

A framework for the oversight and local deployment of safe and high-quality prediction models.

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.

Unanticipated Respiratory Compromise and Unplanned Intubations on General Medical and Surgical Floors.

BACKGROUND: Unanticipated respiratory compromise that lead to unplanned intubations is a known phenomenon in hospitalized patients. Most events occur in patients at high risk in well-monitored units; less is known about the incidence, risk factors, and trajectory of patients thought at low risk on lightly monitored general care wards. The aims of our study were to quantify demographic and clinical characteristics associated with unplanned intubations on general care floors and to analyze the medications administered, monitoring strategies, and vital-sign trajectories before the event. METHODS: We performed a multicenter retrospective cohort study of hospitalized subjects on the general floor who had unanticipated, unplanned intubations on general care floors from August 2014 to February 2018. RESULTS: We identified 448 unplanned intubations. The incidence rate was 0.420 per 1,000 bed-days (95% CI 0.374-0.470) in the academic hospital and was 0.430 (95% CI 0.352-0.520) and 0.394 per 1,000 bed-days (95% CI 0.301-0.506) at our community hospitals. Extrapolating these rates to total hospital admissions in the United States, we estimate 64,000 events annually. The mortality rate was 49.1%. Within 12 h preceding the event, 35.3% of the subjects received opiates. All received vital-sign assessments. Most were monitored with pulse oximetry. In contrast, 2.5% were on cardiac telemetry, and only 4 subjects used capnography; 53.7% showed significant vital-sign changes in the 24 h before the event. However, 46.3% had no significant change in any vital signs. CONCLUSIONS: Our study showed unanticipated respiratory compromise that required an unplanned intubation of subjects on the general care floor, although not common, carried a high mortality. Besides pulse oximetry and routine vital-sign assessments, very little monitoring was in use. A significant portion of the subjects had no vital-sign abnormalities leading up to the event. Further research is needed to determine the phenotype of the different etiologies of unexpected acute respiratory failure to identify better risk stratification and monitoring strategies.

Development, Implementation, and Evaluation of an In-Hospital Optimized Early Warning Score for Patient Deterioration.

Background. Identification of patients at risk of deteriorating during their hospitalization is an important concern. However, many off-shelf scores have poor in-center performance. In this article, we report our experience developing, implementing, and evaluating an in-hospital score for deterioration. Methods. We abstracted 3 years of data (2014-2016) and identified patients on medical wards that died or were transferred to the intensive care unit. We developed a time-varying risk model and then implemented the model over a 10-week period to assess prospective predictive performance. We compared performance to our currently used tool, National Early Warning Score. In order to aid clinical decision making, we transformed the quantitative score into a three-level clinical decision support tool. Results. The developed risk score had an average area under the curve of 0.814 (95% confidence interval = 0.79-0.83) versus 0.740 (95% confidence interval = 0.72-0.76) for the National Early Warning Score. We found the proposed score was able to respond to acute clinical changes in patients' clinical status. Upon implementing the score, we were able to achieve the desired positive predictive value but needed to retune the thresholds to get the desired sensitivity. Discussion. This work illustrates the potential for academic medical centers to build, refine, and implement risk models that are targeted to their patient population and work flow.

Machine learning for early detection of sepsis: an internal and temporal validation study.

OBJECTIVE: Determine if deep learning detects sepsis earlier and more accurately than other models. To evaluate model performance using implementation-oriented metrics that simulate clinical practice. MATERIALS AND METHODS: We trained internally and temporally validated a deep learning model (multi-output Gaussian process and recurrent neural network [MGP-RNN]) to detect sepsis using encounters from adult hospitalized patients at a large tertiary academic center. Sepsis was defined as the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria, a blood culture order, and at least one element of end-organ failure. The training dataset included demographics, comorbidities, vital signs, medication administrations, and labs from October 1, 2014 to December 1, 2015, while the temporal validation dataset was from March 1, 2018 to August 31, 2018. Comparisons were made to 3 machine learning methods, random forest (RF), Cox regression (CR), and penalized logistic regression (PLR), and 3 clinical scores used to detect sepsis, SIRS, quick Sequential Organ Failure Assessment (qSOFA), and National Early Warning Score (NEWS). Traditional discrimination statistics such as the C-statistic as well as metrics aligned with operational implementation were assessed. RESULTS: The training set and internal validation included 42 979 encounters, while the temporal validation set included 39 786 encounters. The C-statistic for predicting sepsis within 4 h of onset was 0.88 for the MGP-RNN compared to 0.836 for RF, 0.849 for CR, 0.822 for PLR, 0.756 for SIRS, 0.619 for NEWS, and 0.481 for qSOFA. MGP-RNN detected sepsis a median of 5 h in advance. Temporal validation assessment continued to show the MGP-RNN outperform all 7 clinical risk score and machine learning comparisons. CONCLUSIONS: We developed and validated a novel deep learning model to detect sepsis. Using our data elements and feature set, our modeling approach outperformed other machine learning methods and clinical scores.