Development and evaluation of trigger tools to identify pediatric blood management errors.

BACKGROUND: Pediatric patient blood management (PBM) programs require continuous surveillance of errors and near misses. However, most PBM programs rely on passive surveillance methods. Our objective was to develop and evaluate a set of automated trigger tools for active surveillance of pediatric PBM errors. MATERIALS AND METHODS: We used the Rand-UCLA method with an expert panel of pediatric transfusion medicine specialists to identify and prioritize candidate trigger tools for all transfused blood products. We then iteratively developed automated queries of electronic health record (EHR) data for the highest priority triggers. Two physicians manually reviewed a subset of cases meeting trigger tool criteria and estimated each trigger tool's positive predictive value (PPV). We then estimated the rate of PBM errors, whether they reached the patient, and adverse events for each trigger tool across four years in a single pediatric health system. RESULTS: We identified 28 potential triggers for pediatric PBM errors and developed 5 automated trigger tools (positive patient identification, missing irradiation, unwashed products despite prior anaphylaxis, transfusion lasting >4 hours, over-transfusion by volume). The PPV for ordering errors ranged from 38-100%. The most frequently detected near miss event reaching patients was first transfusions without positive patient identification (estimate 303, 95% CI: 288-318 per year). The only adverse events detected were from over-transfusions by volume, including 4 adverse events detected on manual review that had not been reported in passive surveillance systems. DISCUSSION: It is feasible to automatically detect pediatric PBM errors using existing data captured in the EHR that enable active surveillance systems. Over-transfusions may be one of the most frequent causes of harm in the pediatric environment.

Crowdsourcing with Enhanced Data Quality Assurance: An Efficient Approach to Mitigate Resource Scarcity Challenges in Training Large Language Models for Healthcare.

Large Language Models (LLMs) have demonstrated immense potential in artificial intelligence across various domains, including healthcare. However, their efficacy is hindered by the need for high-quality labeled data, which is often expensive and time-consuming to create, particularly in low-resource domains like healthcare. To address these challenges, we propose a crowdsourcing (CS) framework enriched with quality control measures at the pre-, real-time-, and post-data gathering stages. Our study evaluated the effectiveness of enhancing data quality through its impact on LLMs (Bio-BERT) for predicting autism-related symptoms. The results show that real-time quality control improves data quality by 19% compared to pre-quality control. Fine-tuning Bio-BERT using crowdsourced data generally increased recall compared to the Bio-BERT baseline but lowered precision. Our findings highlighted the potential of crowdsourcing and quality control in resource-constrained environments and offered insights into optimizing healthcare LLMs for informed decision-making and improved patient care.

Transparent deep learning to identify autism spectrum disorders (ASD) in EHR using clinical notes.

OBJECTIVE: Machine learning (ML) is increasingly employed to diagnose medical conditions, with algorithms trained to assign a single label using a black-box approach. We created an ML approach using deep learning that generates outcomes that are transparent and in line with clinical, diagnostic rules. We demonstrate our approach for autism spectrum disorders (ASD), a neurodevelopmental condition with increasing prevalence. METHODS: We use unstructured data from the Centers for Disease Control and Prevention (CDC) surveillance records labeled by a CDC-trained clinician with ASD A1-3 and B1-4 criterion labels per sentence and with ASD cases labels per record using Diagnostic and Statistical Manual of Mental Disorders (DSM5) rules. One rule-based and three deep ML algorithms and six ensembles were compared and evaluated using a test set with 6773 sentences (N = 35 cases) set aside in advance. Criterion and case labeling were evaluated for each ML algorithm and ensemble. Case labeling outcomes were compared also with seven traditional tests. RESULTS: Performance for criterion labeling was highest for the hybrid BiLSTM ML model. The best case labeling was achieved by an ensemble of two BiLSTM ML models using a majority vote. It achieved 100% precision (or PPV), 83% recall (or sensitivity), 100% specificity, 91% accuracy, and 0.91 F-measure. A comparison with existing diagnostic tests shows that our best ensemble was more accurate overall. CONCLUSIONS: Transparent ML is achievable even with small datasets. By focusing on intermediate steps, deep ML can provide transparent decisions. By leveraging data redundancies, ML errors at the intermediate level have a low impact on final outcomes.

Leveraging explainable artificial intelligence to optimize clinical decision support.

OBJECTIVE: To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. METHODS: We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. RESULTS: The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. CONCLUSION: We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.