Appropriate use of blood cultures in the emergency department through machine learning (ABC): study protocol for a randomised controlled non-inferiority trial.

INTRODUCTION: The liberal use of blood cultures in emergency departments (EDs) leads to low yields and high numbers of false-positive results. False-positive, contaminated cultures are associated with prolonged hospital stays, increased antibiotic usage and even higher hospital mortality rates. This trial aims to investigate whether a recently developed and validated machine learning model for predicting blood culture outcomes can safely and effectively guide clinicians in withholding unnecessary blood culture analysis. METHODS AND ANALYSIS: A randomised controlled, non-inferiority trial comparing current practice with a machine learning-guided approach. The primary objective is to determine whether the machine learning based approach is non-inferior to standard practice based on 30-day mortality. Secondary outcomes include hospital length-of stay and hospital admission rates. Other outcomes include model performance and antibiotic usage. Participants will be recruited in the EDs of multiple hospitals in the Netherlands. A total of 7584 participants will be included. ETHICS AND DISSEMINATION: Possible participants will receive verbal information and a paper information brochure regarding the trial. They will be given at least 1 hour consideration time before providing informed consent. Research results will be published in peer-reviewed journals. This study has been approved by the Amsterdam University Medical Centers' local medical ethics review committee (No 22.0567). The study will be conducted in concordance with the principles of the Declaration of Helsinki and in accordance with the Medical Research Involving Human Subjects Act, General Data Privacy Regulation and Medical Device Regulation. TRIAL REGISTRATION NUMBER: NCT06163781.

Impact of Blood Culture Contamination on Antibiotic Use, Resource Utilization, and Clinical Outcomes: A Retrospective Cohort Study in Dutch and US Hospitals.

Background: Blood culture contamination (BCC) has been associated with prolonged antibiotic use (AU) and increased health care utilization; however, this has not been widely reevaluated in the era of increased attention to antibiotic stewardship. We evaluated the impact of BCC on AU, resource utilization, and length of stay in Dutch and US patients. Methods: This retrospective observational study examined adults admitted to 2 hospitals in the Netherlands and 5 hospitals in the United States undergoing ≥2 blood culture (BC) sets. Exclusion criteria included neutropenia, no hospital admission, or death within 48 hours of hospitalization. The impact of BCC on clinical outcomes-overall inpatient days of antibiotic therapy, test utilization, length of stay, and mortality-was determined via a multivariable regression model. Results: An overall 22 927 patient admissions were evaluated: 650 (4.1%) and 339 (4.8%) with BCC and 11 437 (71.8%) and 4648 (66.3%) with negative BC results from the Netherlands and the United States, respectively. Dutch and US patients with BCC had a mean ± SE 1.74 ± 0.27 (P < .001) and 1.58 ± 0.45 (P < .001) more days of antibiotic therapy than patients with negative BC results. They also had 0.6 ± 0.1 (P < .001) more BCs drawn. Dutch but not US patients with BCC had longer hospital stays (3.36 days; P < .001). There was no difference in mortality between groups in either cohort. AU remained higher in US but not Dutch patients with BCC in a subanalysis limited to BC obtained within the first 24 hours of admission. Conclusions: BCC remains associated with higher inpatient AU and health care utilization as compared with patients with negative BC results, although the impact on these outcomes differs by country.

Detecting changes in the performance of a clinical machine learning tool over time.

BACKGROUND: Excessive use of blood cultures (BCs) in Emergency Departments (EDs) results in low yields and high contamination rates, associated with increased antibiotic use and unnecessary diagnostics. Our team previously developed and validated a machine learning model to predict BC outcomes and enhance diagnostic stewardship. While the model showed promising initial results, concerns over performance drift due to evolving patient demographics, clinical practices, and outcome rates warrant continual monitoring and evaluation of such models. METHODS: A real-time evaluation of the model's performance was conducted between October 2021 and September 2022. The model was integrated into Amsterdam UMC's Electronic Health Record system, predicting BC outcomes for all adult patients with BC draws in real time. The model's performance was assessed monthly using metrics including the Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores. Statistical Process Control (SPC) charts were used to monitor variation over time. FINDINGS: Across 3.035 unique adult patient visits, the model achieved an average AUC of 0.78, AUPRC of 0.41, and a Brier score of 0.10 for predicting the outcome of BCs drawn in the ED. While specific population characteristics changed over time, no statistical points outside the statistical control range were detected in the AUC, AUPRC, and Brier scores, indicating stable model performance. The average BC positivity rate during the study period was 13.4%. INTERPRETATION: Despite significant changes in clinical practice, our BC stewardship tool exhibited stable performance, suggesting its robustness to changing environments. Using SPC charts for various metrics enables simple and effective monitoring of potential performance drift. The assessment of the variation of outcome rates and population changes may guide the specific interventions, such as intercept correction or recalibration, that may be needed to maintain a stable model performance over time. This study suggested no need to recalibrate or correct our BC stewardship tool. FUNDING: No funding to disclose.

Embracing cohort heterogeneity in clinical machine learning development: a step toward generalizable models.

This study is a simple illustration of the benefit of averaging over cohorts, rather than developing a prediction model from a single cohort. We show that models trained on data from multiple cohorts can perform significantly better in new settings than models based on the same amount of training data but from just a single cohort. Although this concept seems simple and obvious, no current prediction model development guidelines recommend such an approach.

Leaving the hospital on time: hospital bed utilization and reasons for discharge delay in the Netherlands.

Inappropriate bed occupancy due to delayed hospital discharge affects both physical and psychological well-being in patients and can disrupt patient flow. The Dutch healthcare system is facing ongoing pressure, especially during the current coronavirus disease pandemic, intensifying the need for optimal use of hospital beds. The aim of this study was to quantify inappropriate patient stays and describe the underlying reasons for the delays in discharge. The Day of Care Survey (DoCS) is a validated tool used to gain information about appropriate and inappropriate bed occupancy in hospitals. Between February 2019 and January 2021, the DoCS was performed five times in three different hospitals within the region of Amsterdam, the Netherlands. All inpatients were screened, using standardized criteria, for their need for in-hospital care at the time of survey and reasons for discharge delay. A total of 782 inpatients were surveyed. Of these patients, 94 (12%) were planned for definite discharge that day. Of all other patients, 145 (21%, ranging from 14% to 35%) were without the need for acute in-hospital care. In 74% (107/145) of patients, the reason for discharge delay was due to issues outside the hospital; most frequently due to a shortage of available places in care homes (26%, 37/145). The most frequent reason for discharge delay inside the hospital was patients awaiting a decision or review by the treating physician (14%, 20/145). Patients who did not meet the criteria for hospital stay were, in general, older [median 75, interquartile range (IQR) 65-84 years, and 67, IQR 55-75 years, respectively, P < .001] and had spent more days in hospital (7, IQR 5-14 days, and 3, IQR 1-8 days respectively, P < .001). Approximately one in five admitted patients occupying hospital beds did not meet the criteria for acute in-hospital stay or care at the time of the survey. Most delays were related to issues outside the immediate control of the hospital. Improvement programmes working with stakeholders focusing on the transfer from hospital to outside areas of care need to be further developed and may offer potential for the greatest gain. The DoCS can be a tool to periodically monitor changes and improvements in patient flow.

The usefulness of implementing minimum retest intervals in reducing inappropriate laboratory test requests in a Dutch hospital.

OBJECTIVES: Inappropriate use of laboratory testing remains a challenging problem worldwide. Minimum retest intervals (MRI) are used to reduce inappropriate laboratory testing. However, their effectiveness and the usefulness in reducing inappropriate laboratory testing is still a matter of debate. The aim of this study was to evaluate the effectiveness of broadly implemented MRIs as a means of reducing inappropriate laboratory test requests. METHODS: We performed a retrospective study in a general care and teaching hospital in the Netherlands, where MRI alerts have been implemented as standard care since June 7th 2017. Clinical chemistry test orders in adult internal medicine patients placed between July 13th 2017 and December 31st 2019 were included. The primary outcome was the effectiveness of MRIs, expressed as percentages of tests ordered and barred as a result of MRIs. RESULTS: Of a total of 218,511 test requests, 4,159 (1.90%) got an MRI alert. These MRIs were overruled by physicians in 21.76% of the cases. As a result of implementing MRIs, 3,254 (1.49%) tests were barred. The financial savings for the department of internal medicine directly related to the included barred laboratory tests during this period were 11,880 euros on a total amount of 636,598 euros for all performed tests. CONCLUSIONS: Only a small proportion of laboratory tests are barred after implementation of MRIs, with a limited impact on the annual costs. However, MRIs provide a continuous reminder to focus on appropriate testing and the effectiveness of MRIs is potentially higher than described in this study.

Diagnostic stewardship for blood cultures in the emergency department: A multicenter validation and prospective evaluation of a machine learning prediction tool.

BACKGROUND: Overuse of blood cultures (BCs) in emergency departments (EDs) leads to low yields and high numbers of contaminated cultures, accompanied by increased diagnostics, antibiotic usage, prolonged hospitalization, and mortality. We aimed to simplify and validate a recently developed machine learning model to help safely withhold BC testing in low-risk patients. METHODS: We extracted data from the electronic health records (EHR) for 44.123 unique ED visits with BC sampling in the Amsterdam UMC (locations VUMC and AMC; the Netherlands), Zaans Medical Center (ZMC; the Netherlands), and Beth Israel Deaconess Medical Center (BIDMC; United States) in periods between 2011 and 2021. We trained a machine learning model on the VUMC data to predict blood culture outcomes and validated it in the AMC, ZMC, and BIDMC with subsequent real-time prospective evaluation in the VUMC. FINDINGS: The model had an Area Under the Receiver Operating Characteristics curve (AUROC) of 0.81 (95%-CI = 0.78-0.83) in the VUMC test set. The most important predictors were temperature, creatinine, and C-reactive protein. The AUROCs in the validation cohorts were 0.80 (AMC; 0.78-0.82), 0.76 (ZMC; 0.74-0.78), and 0.75 (BIDMC; 0.74-0.76). During real-time prospective evaluation in the EHR of the VUMC, it reached an AUROC of 0.76 (0.71-0.81) among 590 patients with BC draws in the ED. The prospective evaluation showed that the model can be used to safely withhold blood culture analyses in at least 30% of patients in the ED. INTERPRETATION: We developed a machine learning model to predict blood culture outcomes in the ED, which retained its performance during external validation and real-time prospective evaluation. Our model can identify patients at low risk of having a positive blood culture. Using the model in practice can significantly reduce the number of blood culture analyses and thus avoid the hidden costs of false-positive culture results. FUNDING: This research project was funded by the Amsterdam Public Health - Quality of Care program and the Dutch "Doen of Laten" project (project number: 839205002).

Using machine learning to predict blood culture outcomes in the emergency department: a single-centre, retrospective, observational study.

OBJECTIVES: To develop predictive models for blood culture (BC) outcomes in an emergency department (ED) setting. DESIGN: Retrospective observational study. SETTING: ED of a large teaching hospital in the Netherlands between 1 September 2018 and 24 June 2020. PARTICIPANTS: Adult patients from whom BCs were collected in the ED. Data of demographic information, vital signs, administered medications in the ED and laboratory and radiology results were extracted from the electronic health record, if available at the end of the ED visits. MAIN OUTCOME MEASURES: The primary outcome was the performance of two models (logistic regression and gradient boosted trees) to predict bacteraemia in ED patients, defined as at least one true positive BC collected at the ED. RESULTS: In 4885 out of 51 399 ED visits (9.5%), BCs were collected. In 598/4885 (12.2%) visits, at least one of the BCs was true positive. Both a gradient boosted tree model and a logistic regression model showed good performance in predicting BC results with area under curve of the receiver operating characteristics of 0.77 (95% CI 0.73 to 0.82) and 0.78 (95% CI 0.73 to 0.82) in the test sets, respectively. In the gradient boosted tree model, the optimal threshold would predict 69% of BCs in the test set to be negative, with a negative predictive value of over 94%. CONCLUSIONS: Both models can accurately identify patients with low risk of bacteraemia at the ED in this single-centre setting and may be useful to reduce unnecessary BCs and associated healthcare costs. Further studies are necessary for validation and to investigate the potential clinical benefits and possible risks after implementation.

Association of a Multifaceted Intervention With Ordering of Unnecessary Laboratory Tests Among Caregivers in Internal Medicine Departments.

Importance: Inappropriate use of laboratory testing is a challenging problem. Estimated overuse rates of approximately 20% have been reported. Effective, sustainable solutions to stimulate optimal use are needed. Objective: To determine the association of a multifaceted intervention with laboratory test volume. Design, Setting, and Participants: A before-after quality improvement study was performed between August 1, 2016, and April 30, 2018, in the internal medicine departments of 4 teaching hospitals in the Netherlands. Data on laboratory order volumes from 19 comparable hospitals were used as controls. The participants were clinicians ordering laboratory tests. Interventions: The intervention included creating awareness through education and feedback, intensified supervision of residents, and changes in order entry systems. Interventions were performed by local project teams and guided by a central project team during a 6-month period. Sustainability was investigated during an 8-month follow-up period. Main Outcomes and Measures: The primary outcome was the change in slope for laboratory test volume. Secondary outcomes were change in slope for laboratory expenditure, order volumes and expenditure for other diagnostic procedures, and clinical outcomes. Data were collected on duration of hospital stay, rate of repeated outpatient visits, 30-day readmission rate, and rate of unexpected prolonged duration of hospital stay for patients admitted for pneumonia. Results: The numbers of internists and residents ordering tests in hospitals 1 to 4 were 16 and 30, 18 and 20, 13 and 17, and 21 and 60, respectively. Statistically significant changes in slope for laboratory test volume per patient contact were found at hospital 1 (change in slope, -1.55; 95% CI, -1.98 to -1.11; P < .001), hospital 3 (change in slope, -0.74; 95% CI, -1.42 to -0.07; P = .03), and hospital 4 (change in slope, -2.18; 95% CI, -3.27 to -1.08; P < .001). At hospital 2, the change in slope was not statistically significant (-0.34; 95% CI, -2.27 to 1.58; P = .73). Laboratory test volume per patient contact decreased by 11.4%, whereas the volume increased by 2.4% in 19 comparable hospitals. Statistically significant changes in slopes for laboratory costs and volumes and costs for other diagnostic procedures were also observed. Clinical outcomes were not associated with negative changes. Important facilitators were education, continuous attention for overuse, feedback, and residents' involvement. Important barriers were difficulties in data retrieval, difficulty in incorporation of principles in daily practice, and high resident turnover. Conclusions and relevance: A set of interventions aimed at changing caregivers' mindset was associated with a reduction in the laboratory test volume in all departments, whereas the volume increased in comparable hospitals in the Netherlands. This study provides a framework for nationwide implementation of interventions to reduce unnecessary laboratory testing.

Persistent high IgG phase I antibody levels against Coxiella burnetii among veterinarians compared to patients previously diagnosed with acute Q fever after three years of follow-up.

BACKGROUND: Little is known about the development of chronic Q fever in occupational risk groups. The aim of this study was to perform long-term follow-up of Coxiella burnetii seropositive veterinarians and investigate the course of IgG phase I and phase II antibodies against C. burnetii antigens and to compare this course with that in patients previously diagnosed with acute Q fever. METHODS: Veterinarians with IgG phase I ≥ 1:256 (immunofluorescence assay) that participated in a previous seroprevalence study were asked to provide a second blood sample three years later. IgG antibody profiles were compared to a group of acute Q fever patients who had IgG phase I ≥ 1:256 twelve months after diagnosis. RESULTS: IgG phase I was detected in all veterinarians (n = 76) and in 85% of Q fever patients (n = 98) after three years (p<0.001). IgG phase I ≥ 1:1,024, indicating possible chronic Q fever, was found in 36% of veterinarians and 12% of patients (OR 3.95, 95% CI: 1.84-8.49). CONCLUSIONS: IgG phase I persists among veterinarians presumably because of continuous exposure to C. burnetii during their work. Serological and clinical follow-up of occupationally exposed risk groups should be considered.