Clinical Utility of Plasma Microbial Cell-Free DNA Sequencing Among Immunocompromised Patients With Pneumonia.

Background: Plasma microbial cell-free DNA (mcfDNA) sequencing can establish the etiology of multiple infectious syndromes by identifying microbial DNA in plasma. However, data are needed to define the clinical scenarios where this tool offers the highest clinical benefit. Methods: We conducted a prospective multicenter observational study that evaluated the impact of plasma mcfDNA sequencing compared with usual care testing among adults with hematologic malignancies. This is a secondary analysis of an expanded cohort that evaluated the clinical utility of plasma mcfDNA sequencing across prespecified and adjudicated outcomes. We examined the percentage of participants for whom plasma mcfDNA sequencing identified a probable cause of pneumonia or clinically relevant nonpneumonia infection. We then assessed potential changes in antimicrobial therapy based on plasma mcfDNA sequencing results and the potential for early mcfDNA testing to avoid bronchoscopy and its associated adverse events. Results: Of 223 participants, at least 1 microbial detection by plasma mcfDNA sequencing was adjudicated as a probable cause of pneumonia in 57 (25.6%) and a clinically relevant nonpneumonia infection in 88 (39.5%). A probable cause of pneumonia was exclusively identified by plasma mcfDNA sequencing in 23 (10.3%) participants. Antimicrobial therapy would have changed for 41 (18.4%) participants had plasma mcfDNA results been available in real time. Among the 57 participants with a probable cause of pneumonia identified by plasma mcfDNA sequencing, bronchoscopy identified no additional probable cause of pneumonia in 52 (91.2%). Conclusions: Plasma mcfDNA sequencing could improve management of both pneumonia and other concurrent infections in immunocompromised patients with suspected pneumonia.

Incorporating informatively collected laboratory data from EHR in clinical prediction models.

BACKGROUND: Electronic Health Records (EHR) are widely used to develop clinical prediction models (CPMs). However, one of the challenges is that there is often a degree of informative missing data. For example, laboratory measures are typically taken when a clinician is concerned that there is a need. When data are the so-called Not Missing at Random (NMAR), analytic strategies based on other missingness mechanisms are inappropriate. In this work, we seek to compare the impact of different strategies for handling missing data on CPMs performance. METHODS: We considered a predictive model for rapid inpatient deterioration as an exemplar implementation. This model incorporated twelve laboratory measures with varying levels of missingness. Five labs had missingness rate levels around 50%, and the other seven had missingness levels around 90%. We included them based on the belief that their missingness status can be highly informational for the prediction. In our study, we explicitly compared the various missing data strategies: mean imputation, normal-value imputation, conditional imputation, categorical encoding, and missingness embeddings. Some of these were also combined with the last observation carried forward (LOCF). We implemented logistic LASSO regression, multilayer perceptron (MLP), and long short-term memory (LSTM) models as the downstream classifiers. We compared the AUROC of testing data and used bootstrapping to construct 95% confidence intervals. RESULTS: We had 105,198 inpatient encounters, with 4.7% having experienced the deterioration outcome of interest. LSTM models generally outperformed other cross-sectional models, where embedding approaches and categorical encoding yielded the best results. For the cross-sectional models, normal-value imputation with LOCF generated the best results. CONCLUSION: Strategies that accounted for the possibility of NMAR missing data yielded better model performance than those did not. The embedding method had an advantage as it did not require prior clinical knowledge. Using LOCF could enhance the performance of cross-sectional models but have countereffects in LSTM models.

Use of artificial intelligence in critical care: opportunities and obstacles.

BACKGROUND: Perhaps nowhere else in the healthcare system than in the intensive care unit environment are the challenges to create useful models with direct time-critical clinical applications more relevant and the obstacles to achieving those goals more massive. Machine learning-based artificial intelligence (AI) techniques to define states and predict future events are commonplace activities of modern life. However, their penetration into acute care medicine has been slow, stuttering and uneven. Major obstacles to widespread effective application of AI approaches to the real-time care of the critically ill patient exist and need to be addressed. MAIN BODY: Clinical decision support systems (CDSSs) in acute and critical care environments support clinicians, not replace them at the bedside. As will be discussed in this review, the reasons are many and include the immaturity of AI-based systems to have situational awareness, the fundamental bias in many large databases that do not reflect the target population of patient being treated making fairness an important issue to address and technical barriers to the timely access to valid data and its display in a fashion useful for clinical workflow. The inherent "black-box" nature of many predictive algorithms and CDSS makes trustworthiness and acceptance by the medical community difficult. Logistically, collating and curating in real-time multidimensional data streams of various sources needed to inform the algorithms and ultimately display relevant clinical decisions support format that adapt to individual patient responses and signatures represent the efferent limb of these systems and is often ignored during initial validation efforts. Similarly, legal and commercial barriers to the access to many existing clinical databases limit studies to address fairness and generalizability of predictive models and management tools. CONCLUSIONS: AI-based CDSS are evolving and are here to stay. It is our obligation to be good shepherds of their use and further development.

Plasma Microbial Cell-Free DNA Sequencing in Immunocompromised Patients With Pneumonia: A Prospective Observational Study.

BACKGROUND: Pneumonia is a common cause of morbidity and mortality, yet a causative pathogen is identified in a minority of cases. Plasma microbial cell-free DNA sequencing may improve diagnostic yield in immunocompromised patients with pneumonia. METHODS: In this prospective, multicenter, observational study of immunocompromised adults undergoing bronchoscopy to establish a pneumonia etiology, plasma microbial cell-free DNA sequencing was compared to standardized usual care testing. Pneumonia etiology was adjudicated by a blinded independent committee. The primary outcome, additive diagnostic value, was assessed in the Per Protocol population (patients with complete testing results and no major protocol deviations) and defined as the percent of patients with an etiology of pneumonia exclusively identified by plasma microbial cell-free DNA sequencing. Clinical additive diagnostic value was assessed in the Per Protocol subgroup with negative usual care testing. RESULTS: Of 257 patients, 173 met Per Protocol criteria. A pneumonia etiology was identified by usual care in 52/173 (30.1%), plasma microbial cell-free DNA sequencing in 49/173 (28.3%) and the combination of both in 73/173 (42.2%) patients. Plasma microbial cell-free DNA sequencing exclusively identified an etiology of pneumonia in 21/173 patients (additive diagnostic value 12.1%, 95% confidence interval [CI], 7.7% to 18.0%, P < .001). In the Per Protocol subgroup with negative usual care testing, plasma microbial cell-free DNA sequencing identified a pneumonia etiology in 21/121 patients (clinical additive diagnostic value 17.4%, 95% CI, 11.1% to 25.3%). CONCLUSIONS: Non-invasive plasma microbial cell-free DNA sequencing significantly increased diagnostic yield in immunocompromised patients with pneumonia undergoing bronchoscopy and extensive microbiologic and molecular testing. CLINICAL TRIALS REGISTRATION: NCT04047719.

Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Development of an interactive dashboard for gun violence pattern analysis and intervention design at the local level.

Introduction: Gun violence remains a concerning and persistent issue in our country. Novel dashboards may integrate and summarize important clinical and non-clinical data that can inform targeted interventions to address the underlying causes of gun violence. Methods: Data from various clinical and non-clinical sources were sourced, cleaned, and integrated into a customizable dashboard that summarizes and provides insight into the underlying factors that impact local gun violence episodes. Results: The dashboards contained data from 7786 encounters and 1152 distinct patients from our Emergency Department's Trauma Registry with various patterns noted by the team. A multidisciplinary executive team, including subject matter experts in community-based interventions, epidemiology, and social sciences, was formed to design targeted interventions based on these observations. Conclusion: Targeted interventions to reduce gun violence require a multimodal data sourcing and standardization approach, the inclusion of neighborhood-level data, and a dedicated multidisciplinary team to act on the generated insights.

The Eyes Have It-for Idiopathic Pulmonary Fibrosis: a Preliminary Observation.

INTRODUCTION: The disease origins of idiopathic pulmonary fibrosis (IPF), which occurs at higher rates in certain races/ethnicities, are not understood. The highest rates occur in white persons of European descent, particularly those with light skin, who are also susceptible to lysosomal organelle dysfunction of the skin leading to fibroproliferative disease . We had observed clinically that the vast majority of patients with IPF had light-colored eyes, suggesting a phenotypic characteristic. METHODS: We pursued this observation through a research database from the USA Veterans Administration, a population that has a high occurrence of IPF due to predominance of elderly male smokers. Using this medical records database, which included facial photos, we compared the frequency of light (blue, green, hazel) and dark (light brown, brown) eyes among white patients diagnosed with IPF compared with a control group of lung granuloma only (no other radiologic evidence of interstitial lung disease). RESULTS: Light eye color was significantly more prevalent in patients with IPF than in the control group with lung granuloma [114/147 (77.6%) versus 129/263 (49.0%], p < 0.001), indicating that light-colored eyes are a phenotype associated with IPF . CONCLUSION: We provide evidence that light eye color is predominant among white persons with IPF.

A framework for the oversight and local deployment of safe and high-quality prediction models.

Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.

Application of Machine Learning in Pulmonary Function Assessment Where Are We Now and Where Are We Going?

Analysis of pulmonary function tests (PFTs) is an area where machine learning (ML) may benefit clinicians, researchers, and the patients. PFT measures spirometry, lung volumes, and carbon monoxide diffusion capacity of the lung (DLCO). The results are usually interpreted by the clinicians using discrete numeric data according to published guidelines. PFT interpretations by clinicians, however, are known to have inter-rater variability and the inaccuracy can impact patient care. This variability may be caused by unfamiliarity of the guidelines, lack of training, inadequate understanding of lung physiology, or simply mental lapses. A rules-based automated interpretation system can recapitulate expert's pattern recognition capability and decrease errors. ML can also be used to analyze continuous data or the graphics, including the flow-volume loop, the DLCO and the nitrogen washout curves. These analyses can discover novel physiological biomarkers. In the era of wearables and telehealth, particularly with the COVID-19 pandemic restricting PFTs to be done in the clinical laboratories, ML can also be used to combine mobile spirometry results with an individual's clinical profile to deliver precision medicine. There are, however, hurdles in the development and commercialization of the ML-assisted PFT interpretation programs, including the need for high quality representative data, the existence of different formats for data acquisition and sharing in PFT software by different vendors, and the need for collaboration amongst clinicians, biomedical engineers, and information technologists. Hurdles notwithstanding, the new developments would represent significant advances that could be the future of PFT, the oldest test still in use in clinical medicine.

Interstitial lung disease in a veterans affairs regional network; a retrospective cohort study.

BACKGROUND: The epidemiology of Interstitial Lung Diseases (ILD) in the Veterans Health Administration (VHA) is presently unknown. RESEARCH QUESTION: Describe the incidence/prevalence, clinical characteristics, and outcomes of ILD patients within the Veteran's Administration Mid-Atlantic Health Care Network (VISN6). STUDY DESIGN AND METHODS: A multi-center retrospective cohort study was performed of veterans receiving hospital or outpatient ILD care from January 1, 2008 to December 31st, 2015 in six VISN6 facilities. Patients were identified by at least one visit encounter with a 515, 516, or other ILD ICD-9 code. Demographic and clinical characteristics were summarized using median, 25th and 75th percentile for continuous variables and count/percentage for categorical variables. Characteristics and incidence/prevalence rates were summarized, and stratified by ILD ICD-9 code. Kaplan Meier curves were generated to define overall survival. RESULTS: 3293 subjects met the inclusion criteria. 879 subjects (26%) had no evidence of ILD following manual medical record review. Overall estimated prevalence in verified ILD subjects was 256 per 100,000 people with a mean incidence across the years of 70 per 100,000 person-years (0.07%). The prevalence and mean incidence when focusing on people with an ILD diagnostic code who had a HRCT scan or a bronchoscopic or surgical lung biopsy was 237 per 100,000 people (0.237%) and 63 per 100,000 person-years respectively (0.063%). The median survival was 76.9 months for 515 codes, 103.4 months for 516 codes, and 83.6 months for 516.31. INTERPRETATION: This retrospective cohort study defines high ILD incidence/prevalence within the VA. Therefore, ILD is an important VA health concern.

Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study.

BACKGROUND: Machine learning models have the potential to improve diagnostic accuracy and management of acute conditions. Despite growing efforts to evaluate and validate such models, little is known about how to best translate and implement these products as part of routine clinical care. OBJECTIVE: This study aims to explore the factors influencing the integration of a machine learning sepsis early warning system (Sepsis Watch) into clinical workflows. METHODS: We conducted semistructured interviews with 15 frontline emergency department physicians and rapid response team nurses who participated in the Sepsis Watch quality improvement initiative. Interviews were audio recorded and transcribed. We used a modified grounded theory approach to identify key themes and analyze qualitative data. RESULTS: A total of 3 dominant themes emerged: perceived utility and trust, implementation of Sepsis Watch processes, and workforce considerations. Participants described their unfamiliarity with machine learning models. As a result, clinician trust was influenced by the perceived accuracy and utility of the model from personal program experience. Implementation of Sepsis Watch was facilitated by the easy-to-use tablet application and communication strategies that were developed by nurses to share model outputs with physicians. Barriers included the flow of information among clinicians and gaps in knowledge about the model itself and broader workflow processes. CONCLUSIONS: This study generated insights into how frontline clinicians perceived machine learning models and the barriers to integrating them into clinical workflows. These findings can inform future efforts to implement machine learning interventions in real-world settings and maximize the adoption of these interventions.

Evaluation of ML-Based Clinical Decision Support Tool to Replace an Existing Tool in an Academic Health System: Lessons Learned.

There is increasing application of machine learning tools to problems in healthcare, with an ultimate goal to improve patient safety and health outcomes. When applied appropriately, machine learning tools can augment clinical care provided to patients. However, even if a model has impressive performance characteristics, prospectively evaluating and effectively implementing models into clinical care remains difficult. The primary objective of this paper is to recount our experiences and challenges in comparing a novel machine learning-based clinical decision support tool to legacy, non-machine learning tools addressing potential safety events in the hospitals and to summarize the obstacles which prevented evaluation of clinical efficacy of tools prior to widespread institutional use. We collected and compared safety events data, specifically patient falls and pressure injuries, between the standard of care approach and machine learning (ML)-based clinical decision support (CDS). Our assessment was limited to performance of the model rather than the workflow due to challenges in directly comparing both approaches. We did note a modest improvement in falls with ML-based CDS; however, it was not possible to determine that overall improvement was due to model characteristics.

Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?

There is little known about how academic medical centers (AMCs) in the US develop, implement, and maintain predictive modeling and machine learning (PM and ML) models. We conducted semi-structured interviews with leaders from AMCs to assess their use of PM and ML in clinical care, understand associated challenges, and determine recommended best practices. Each transcribed interview was iteratively coded and reconciled by a minimum of 2 investigators to identify key barriers to and facilitators of PM and ML adoption and implementation in clinical care. Interviews were conducted with 33 individuals from 19 AMCs nationally. AMCs varied greatly in the use of PM and ML within clinical care, from some just beginning to explore their utility to others with multiple models integrated into clinical care. Informants identified 5 key barriers to the adoption and implementation of PM and ML in clinical care: (1) culture and personnel, (2) clinical utility of the PM and ML tool, (3) financing, (4) technology, and (5) data. Recommendation to the informatics community to overcome these barriers included: (1) development of robust evaluation methodologies, (2) partnership with vendors, and (3) development and dissemination of best practices. For institutions developing clinical PM and ML applications, they are advised to: (1) develop appropriate governance, (2) strengthen data access, integrity, and provenance, and (3) adhere to the 5 rights of clinical decision support. This article highlights key challenges of implementing PM and ML in clinical care at AMCs and suggests best practices for development, implementation, and maintenance at these institutions.

Machine learning for early detection of sepsis: an internal and temporal validation study.

OBJECTIVE: Determine if deep learning detects sepsis earlier and more accurately than other models. To evaluate model performance using implementation-oriented metrics that simulate clinical practice. MATERIALS AND METHODS: We trained internally and temporally validated a deep learning model (multi-output Gaussian process and recurrent neural network [MGP-RNN]) to detect sepsis using encounters from adult hospitalized patients at a large tertiary academic center. Sepsis was defined as the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria, a blood culture order, and at least one element of end-organ failure. The training dataset included demographics, comorbidities, vital signs, medication administrations, and labs from October 1, 2014 to December 1, 2015, while the temporal validation dataset was from March 1, 2018 to August 31, 2018. Comparisons were made to 3 machine learning methods, random forest (RF), Cox regression (CR), and penalized logistic regression (PLR), and 3 clinical scores used to detect sepsis, SIRS, quick Sequential Organ Failure Assessment (qSOFA), and National Early Warning Score (NEWS). Traditional discrimination statistics such as the C-statistic as well as metrics aligned with operational implementation were assessed. RESULTS: The training set and internal validation included 42 979 encounters, while the temporal validation set included 39 786 encounters. The C-statistic for predicting sepsis within 4 h of onset was 0.88 for the MGP-RNN compared to 0.836 for RF, 0.849 for CR, 0.822 for PLR, 0.756 for SIRS, 0.619 for NEWS, and 0.481 for qSOFA. MGP-RNN detected sepsis a median of 5 h in advance. Temporal validation assessment continued to show the MGP-RNN outperform all 7 clinical risk score and machine learning comparisons. CONCLUSIONS: We developed and validated a novel deep learning model to detect sepsis. Using our data elements and feature set, our modeling approach outperformed other machine learning methods and clinical scores.

Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study.

BACKGROUND: Successful integrations of machine learning into routine clinical care are exceedingly rare, and barriers to its adoption are poorly characterized in the literature. OBJECTIVE: This study aims to report a quality improvement effort to integrate a deep learning sepsis detection and management platform, Sepsis Watch, into routine clinical care. METHODS: In 2016, a multidisciplinary team consisting of statisticians, data scientists, data engineers, and clinicians was assembled by the leadership of an academic health system to radically improve the detection and treatment of sepsis. This report of the quality improvement effort follows the learning health system framework to describe the problem assessment, design, development, implementation, and evaluation plan of Sepsis Watch. RESULTS: Sepsis Watch was successfully integrated into routine clinical care and reshaped how local machine learning projects are executed. Frontline clinical staff were highly engaged in the design and development of the workflow, machine learning model, and application. Novel machine learning methods were developed to detect sepsis early, and implementation of the model required robust infrastructure. Significant investment was required to align stakeholders, develop trusting relationships, define roles and responsibilities, and to train frontline staff, leading to the establishment of 3 partnerships with internal and external research groups to evaluate Sepsis Watch. CONCLUSIONS: Machine learning models are commonly developed to enhance clinical decision making, but successful integrations of machine learning into routine clinical care are rare. Although there is no playbook for integrating deep learning into clinical care, learnings from the Sepsis Watch integration can inform efforts to develop machine learning technologies at other health care delivery systems.

Unanticipated Respiratory Compromise and Unplanned Intubations on General Medical and Surgical Floors.

BACKGROUND: Unanticipated respiratory compromise that lead to unplanned intubations is a known phenomenon in hospitalized patients. Most events occur in patients at high risk in well-monitored units; less is known about the incidence, risk factors, and trajectory of patients thought at low risk on lightly monitored general care wards. The aims of our study were to quantify demographic and clinical characteristics associated with unplanned intubations on general care floors and to analyze the medications administered, monitoring strategies, and vital-sign trajectories before the event. METHODS: We performed a multicenter retrospective cohort study of hospitalized subjects on the general floor who had unanticipated, unplanned intubations on general care floors from August 2014 to February 2018. RESULTS: We identified 448 unplanned intubations. The incidence rate was 0.420 per 1,000 bed-days (95% CI 0.374-0.470) in the academic hospital and was 0.430 (95% CI 0.352-0.520) and 0.394 per 1,000 bed-days (95% CI 0.301-0.506) at our community hospitals. Extrapolating these rates to total hospital admissions in the United States, we estimate 64,000 events annually. The mortality rate was 49.1%. Within 12 h preceding the event, 35.3% of the subjects received opiates. All received vital-sign assessments. Most were monitored with pulse oximetry. In contrast, 2.5% were on cardiac telemetry, and only 4 subjects used capnography; 53.7% showed significant vital-sign changes in the 24 h before the event. However, 46.3% had no significant change in any vital signs. CONCLUSIONS: Our study showed unanticipated respiratory compromise that required an unplanned intubation of subjects on the general care floor, although not common, carried a high mortality. Besides pulse oximetry and routine vital-sign assessments, very little monitoring was in use. A significant portion of the subjects had no vital-sign abnormalities leading up to the event. Further research is needed to determine the phenotype of the different etiologies of unexpected acute respiratory failure to identify better risk stratification and monitoring strategies.