Exploring the role of Large Language Models in haematology: A focused review of applications, benefits and limitations.

Large language models (LLMs) have significantly impacted various fields with their ability to understand and generate human-like text. This study explores the potential benefits and limitations of integrating LLMs, such as ChatGPT, into haematology practices. Utilizing systematic review methodologies, we analysed studies published after 1 December 2022, from databases like PubMed, Web of Science and Scopus, and assessing each for bias with the QUADAS-2 tool. We reviewed 10 studies that applied LLMs in various haematology contexts. These models demonstrated proficiency in specific tasks, such as achieving 76% diagnostic accuracy for haemoglobinopathies. However, the research highlighted inconsistencies in performance and reference accuracy, indicating variability in reliability across different uses. Additionally, the limited scope of these studies and constraints on datasets could potentially limit the generalizability of our findings. The findings suggest that, while LLMs provide notable advantages in enhancing diagnostic processes and educational resources within haematology, their integration into clinical practice requires careful consideration. Before implementing them in haematology, rigorous testing and specific adaptation are essential. This involves validating their accuracy and reliability across different scenarios. Given the field's complexity, it is also critical to continuously monitor these models and adapt them responsively.

Derivation, external and clinical validation of a deep learning approach for detecting intracranial hypertension.

Increased intracranial pressure (ICP) ≥15 mmHg is associated with adverse neurological outcomes, but needs invasive intracranial monitoring. Using the publicly available MIMIC-III Waveform Database (2000-2013) from Boston, we developed an artificial intelligence-derived biomarker for elevated ICP (aICP) for adult patients. aICP uses routinely collected extracranial waveform data as input, reducing the need for invasive monitoring. We externally validated aICP with an independent dataset from the Mount Sinai Hospital (2020-2022) in New York City. The AUROC, accuracy, sensitivity, and specificity on the external validation dataset were 0.80 (95% CI, 0.80-0.80), 73.8% (95% CI, 72.0-75.6%), 73.5% (95% CI 72.5-74.5%), and 73.0% (95% CI, 72.0-74.0%), respectively. We also present an exploratory analysis showing aICP predictions are associated with clinical phenotypes. A ten-percentile increment was associated with brain malignancy (OR = 1.68; 95% CI, 1.09-2.60), intracerebral hemorrhage (OR = 1.18; 95% CI, 1.07-1.32), and craniotomy (OR = 1.43; 95% CI, 1.12-1.84; P < 0.05 for all).

A foundation model for clinician-centered drug repurposing.

Drug repurposing-identifying new therapeutic uses for approved drugs-is often a serendipitous and opportunistic endeavour to expand the use of drugs for new diseases. The clinical utility of drug-repurposing artificial intelligence (AI) models remains limited because these models focus narrowly on diseases for which some drugs already exist. Here we introduce TxGNN, a graph foundation model for zero-shot drug repurposing, identifying therapeutic candidates even for diseases with limited treatment options or no existing drugs. Trained on a medical knowledge graph, TxGNN uses a graph neural network and metric learning module to rank drugs as potential indications and contraindications for 17,080 diseases. When benchmarked against 8 methods, TxGNN improves prediction accuracy for indications by 49.2% and contraindications by 35.1% under stringent zero-shot evaluation. To facilitate model interpretation, TxGNN's Explainer module offers transparent insights into multi-hop medical knowledge paths that form TxGNN's predictive rationales. Human evaluation of TxGNN's Explainer showed that TxGNN's predictions and explanations perform encouragingly on multiple axes of performance beyond accuracy. Many of TxGNN's new predictions align well with off-label prescriptions that clinicians previously made in a large healthcare system. TxGNN's drug-repurposing predictions are accurate, consistent with off-label drug use, and can be investigated by human experts through multi-hop interpretable rationales.

Deep Learning-Based Electrocardiogram Analysis Predicts Biventricular Dysfunction and Dilation in Congenital Heart Disease.

BACKGROUND: Artificial intelligence-enhanced electrocardiogram (AI-ECG) analysis shows promise to detect biventricular pathophysiology. However, AI-ECG analysis remains underexplored in congenital heart disease (CHD). OBJECTIVES: The purpose of this study was to develop and externally validate an AI-ECG model to predict cardiovascular magnetic resonance (CMR)-defined biventricular dysfunction/dilation in patients with CHD. METHODS: We trained (80%) and tested (20%) a convolutional neural network on paired ECG-CMRs (≤30 days apart) from patients with and without CHD to detect left ventricular (LV) dysfunction (ejection fraction ≤40%), RV dysfunction (ejection fraction ≤35%), and LV and RV dilation (end-diastolic volume z-score ≥4). Performance was assessed during internal testing and external validation on an outside health care system using area under receiver-operating curve (AUROC) and area under precision recall curve. RESULTS: The internal and external cohorts comprised 8,584 ECG-CMR pairs (n = 4,941; median CMR age 20.7 years) and 909 ECG-CMR pairs (n = 746; median CMR age 25.4 years), respectively. Model performance was similar for internal testing (AUROC: LV dysfunction 0.87; LV dilation 0.86; RV dysfunction 0.88; RV dilation 0.81) and external validation (AUROC: LV dysfunction 0.89; LV dilation 0.83; RV dysfunction 0.82; RV dilation 0.80). Model performance was lowest in functionally single ventricle patients. Tetralogy of Fallot patients predicted to be at high risk of ventricular dysfunction had lower survival (P < 0.001). Model explainability via saliency mapping revealed that lateral precordial leads influence all outcome predictions, with high-risk features including QRS widening and T-wave inversions for RV dysfunction/dilation. CONCLUSIONS: AI-ECG shows promise to predict biventricular dysfunction/dilation, which may help inform CMR timing in CHD.

A foundational transformer leveraging full night, multichannel sleep study data accurately classifies sleep stages.

Study Objectives: To investigate whether a foundational transformer model using 8-hour, multichannel data from polysomnograms can outperform existing artificial intelligence (AI) methods for sleep stage classification. Methods: We utilized the Sleep Heart Health Study (SHHS) visits 1 and 2 for training and validation and the Multi-Ethnic Study of Atherosclerosis (MESA) for testing of our model. We trained a self-supervised foundational transformer (called PFTSleep) that encodes 8-hour long sleep studies at 125 Hz with 7 signals including brain, movement, cardiac, oxygen, and respiratory channels. These encodings are used as input for training of an additional model to classify sleep stages, without adjusting the weights of the foundational transformer. We compared our results to existing AI methods that did not utilize 8-hour data or the full set of signals but did report evaluation metrics for the SHHS dataset. Results: We trained and validated a model with 8,444 sleep studies with 7 signals including brain, movement, cardiac, oxygen, and respiratory channels and tested on an additional 2,055 studies. In total, we trained and tested 587,944 hours of sleep study signal data. Area under the precision recall curve (AUPRC) scores were 0.82, 0.40, 0.53, 0.75, and 0.82 and area under the receiving operating characteristics curve (AUROC) scores were 0.99, 0.95, 0.96, 0.98, and 0.99 for wake, N1, N2, N3, and REM, respectively, on the SHHS validation set. For MESA, the AUPRC scores were 0.56, 0.16, 0.40, 0.45, and 0.65 and AUROC scores were 0.94, 0.77, 0.87, 0.91, and 0.96, respectively. Our model was compared to the longest context window state-of-the-art model and showed increases in macro evaluation scores, notably sensitivity (3.7% increase) and multi-class REM (3.39% increase) and wake (0.97% increase) F1 scores. Conclusions: Utilizing full night, multi-channel PSG data encodings derived from a foundational transformer improve sleep stage classification over existing methods.

Assessing GPT-4 multimodal performance in radiological image analysis.

OBJECTIVES: This study aims to assess the performance of a multimodal artificial intelligence (AI) model capable of analyzing both images and textual data (GPT-4V), in interpreting radiological images. It focuses on a range of modalities, anatomical regions, and pathologies to explore the potential of zero-shot generative AI in enhancing diagnostic processes in radiology. METHODS: We analyzed 230 anonymized emergency room diagnostic images, consecutively collected over 1 week, using GPT-4V. Modalities included ultrasound (US), computerized tomography (CT), and X-ray images. The interpretations provided by GPT-4V were then compared with those of senior radiologists. This comparison aimed to evaluate the accuracy of GPT-4V in recognizing the imaging modality, anatomical region, and pathology present in the images. RESULTS: GPT-4V identified the imaging modality correctly in 100% of cases (221/221), the anatomical region in 87.1% (189/217), and the pathology in 35.2% (76/216). However, the model's performance varied significantly across different modalities, with anatomical region identification accuracy ranging from 60.9% (39/64) in US images to 97% (98/101) and 100% (52/52) in CT and X-ray images (p < 0.001). Similarly, pathology identification ranged from 9.1% (6/66) in US images to 36.4% (36/99) in CT and 66.7% (34/51) in X-ray images (p < 0.001). These variations indicate inconsistencies in GPT-4V's ability to interpret radiological images accurately. CONCLUSION: While the integration of AI in radiology, exemplified by multimodal GPT-4, offers promising avenues for diagnostic enhancement, the current capabilities of GPT-4V are not yet reliable for interpreting radiological images. This study underscores the necessity for ongoing development to achieve dependable performance in radiology diagnostics. CLINICAL RELEVANCE STATEMENT: Although GPT-4V shows promise in radiological image interpretation, its high diagnostic hallucination rate (> 40%) indicates it cannot be trusted for clinical use as a standalone tool. Improvements are necessary to enhance its reliability and ensure patient safety. KEY POINTS: GPT-4V's capability in analyzing images offers new clinical possibilities in radiology. GPT-4V excels in identifying imaging modalities but demonstrates inconsistent anatomy and pathology detection. Ongoing AI advancements are necessary to enhance diagnostic reliability in radiological applications.

A foundation model for clinician-centered drug repurposing.

Drug repurposing - identifying new therapeutic uses for approved drugs - is often serendipitous and opportunistic, expanding the use of drugs for new diseases. The clinical utility of drug repurposing AI models remains limited because the models focus narrowly on diseases for which some drugs already exist. Here, we introduce TXGNN, a graph foundation model for zero-shot drug repurposing, identifying therapeutic candidates even for diseases with limited treatment options or no existing drugs. Trained on a medical knowledge graph, TXGNN utilizes a graph neural network and metric-learning module to rank drugs as potential indications and contraindications across 17,080 diseases. When benchmarked against eight methods, TXGNN improves prediction accuracy for indications by 49.2% and contraindications by 35.1% under stringent zero-shot evaluation. To facilitate model interpretation, TXGNN's Explainer module offers transparent insights into multi-hop medical knowledge paths that form TXGNN's predictive rationales. Human evaluation of TXGNN's Explainer showed that TXGNN's predictions and explanations perform encouragingly on multiple axes of performance beyond accuracy. Many of TxGNN's novel predictions align with off-label prescriptions clinicians make in a large healthcare system. TXGNN's drug repurposing predictions are accurate, consistent with off-label drug use, and can be investigated by human experts through multi-hop interpretable rationales.

Comparison of blood-based liver fibrosis scores in the Mount Sinai Health System, MASLD Registry, and NHANES 2017-2020 study.

BACKGROUND: Liver fibrosis is a critical public health concern, necessitating early detection to prevent progression. This study evaluates the recently developed LiverRisk score and steatosis-associated Fibrosis Estimator (SAFE) score against established indices for prognostication and/or fibrosis prediction in 4diverse cohorts, including participants with metabolic dysfunction-associated steatotic liver disease (MASLD). METHODS: We used data from the Mount Sinai Data Warehouse (32,828 participants without liver disease diagnoses), the Mount Sinai MASLD/MASH Longitudinal Registry (422 participants with MASLD), and National Health and Nutrition Examination Survey 2017-2020 (4133 participants representing the general population) to compare LiverRisk score, FIB-4 index, APRI, and SAFE score. Analyses included Cox proportional hazards regressions, Kaplan-Meier estimates, and classification metrics to evaluate performance in prognostication and fibrosis prediction. RESULTS: In Mount Sinai Data Warehouse, LiverRisk score was significantly associated with future liver-related outcomes but did not significantly outperform FIB-4 or APRI for predicting any of the outcomes. In the general population, LiverRisk score and SAFE score outperformed FIB-4 and APRI in identifying fibrosis, but LiverRisk score underperformed among participants who were non-White or had type 2 diabetes. Among participants with MASLD, SAFE score outperformed FIB-4 and APRI in 1 of 2 cohorts, but there were generally few significant performance differences between all 4 scores. CONCLUSIONS: LiverRisk score does not consistently outperform existing predictors in diverse populations, and further validation is needed before adoption in settings with significant differences from the original derivation cohorts. It remains necessary to replicate the ability of these scores to predict liver-specific mortality, as well as to develop diagnostic tools for liver fibrosis that are accessible and substantially better than current scores, especially among patients with MASLD and other chronic liver conditions.

Development and Validation of a Policy Tree Approach for Optimizing Intravenous Fluids in Critically Ill Patients with Sepsis and Acute Kidney Injury.

Purpose: Intravenous fluids are mainstay of management of acute kidney injury (AKI) after sepsis but can cause fluid overload. Recent literature shows that restrictive fluid strategy may be beneficial in some patients with AKI, however, identifying these patients is challenging. We aimed to develop and validate a machine learning algorithm to identify patients who would benefit from a restrictive fluid strategy. Methods: We included patients with sepsis who developed AKI within 48 hours of ICU admission and defined restrictive fluid strategy as receiving <500mL fluids within 24 hours after AKI. Our primary outcome was early AKI reversal within 48 hours of AKI onset, and secondary outcomes included sustained AKI reversal and major adverse kidney events (MAKE) at discharge. We used a causal forest, a machine learning algorithm to estimate individual treatment effects and policy tree algorithm to identify patients who would benefit by restrictive fluid strategy. We developed the algorithm in MIMIC-IV and validated it in eICU database. Results: Among 2,091 patients in the external validation cohort, policy tree recommended restrictive fluids for 88.2%. Among these, patients who received restrictive fluids demonstrated significantly higher rate of early AKI reversal (48.2% vs 39.6%, p<0.001), sustained AKI reversal (36.7% vs 27.4%, p<0.001) and lower rates of MAKE by discharge (29.3% vs 35.1%, p=0.019). These results were consistent in adjusted analysis. Conclusion: Policy tree based on causal machine learning can identify septic patients with AKI who benefit from a restrictive fluid strategy. This approach needs to be validated in prospective trials.

Distinguishing neonatal culture-negative sepsis from rule-out sepsis with artificial intelligence-derived graphs.

IMPACT: Novel artificial intelligence methods can aide in identification of cases of conditions using only unstructured electronic health record data. This graph-based method compares comprehensive electronic health records among neonates using temporal data. This provides a scalable solution to distinguish culture negative sepsis from rule out sepsis using a data-driven method.

Multimodal fusion learning for long QT syndrome pathogenic genotypes in a racially diverse population.

Congenital long QT syndrome (LQTS) diagnosis is complicated by limited genetic testing at scale, low prevalence, and normal QT corrected interval in patients with high-risk genotypes. We developed a deep learning approach combining electrocardiogram (ECG) waveform and electronic health record data to assess whether patients had pathogenic variants causing LQTS. We defined patients with high-risk genotypes as having ≥1 pathogenic variant in one of the LQTS-susceptibility genes. We trained the model using data from United Kingdom Biobank (UKBB) and then fine-tuned in a racially/ethnically diverse cohort using Mount Sinai BioMe Biobank. Following group-stratified 5-fold splitting, the fine-tuned model achieved area under the precision-recall curve of 0.29 (95% confidence interval [CI] 0.28-0.29) and area under the receiver operating curve of 0.83 (0.82-0.83) on independent testing data from BioMe. Multimodal fusion learning has promise to identify individuals with pathogenic genetic mutations to enable patient prioritization for further work up.

Advancing radiology practice and research: harnessing the potential of large language models amidst imperfections.

Large language models (LLMs) are transforming the field of natural language processing (NLP). These models offer opportunities for radiologists to make a meaningful impact in their field. NLP is a part of artificial intelligence (AI) that uses computer algorithms to study and understand text data. Recent advances in NLP include the Attention mechanism and the Transformer architecture. Transformer-based LLMs, such as GPT-4 and Gemini, are trained on massive amounts of data and generate human-like text. They are ideal for analysing large text data in academic research and clinical practice in radiology. Despite their promise, LLMs have limitations, including their dependency on the diversity and quality of their training data and the potential for false outputs. Albeit these limitations, the use of LLMs in radiology holds promise and is gaining momentum. By embracing the potential of LLMs, radiologists can gain valuable insights and improve the efficiency of their work. This can ultimately lead to improved patient care.

Role of artificial intelligence in critical care nutrition support and research.

Nutrition plays a key role in the comprehensive care of critically ill patients. Determining optimal nutrition strategy, however, remains a subject of intense debate. Artificial intelligence (AI) applications are becoming increasingly common in medicine, and specifically in critical care, driven by the data-rich environment of intensive care units. In this review, we will examine the evidence regarding the application of AI in critical care nutrition. As of now, the use of AI in critical care nutrition is relatively limited, with its primary emphasis on malnutrition screening and tolerance of enteral nutrition. Despite the current scarcity of evidence, the potential for AI for more personalized nutrition management for critically ill patients is substantial. This stems from the ability of AI to integrate multiple data streams reflecting patients' changing needs while addressing inherent heterogeneity. The application of AI in critical care nutrition holds promise for optimizing patient outcomes through tailored and adaptive nutrition interventions. A successful implementation of AI, however, necessitates a multidisciplinary approach, coupled with careful consideration of challenges related to data management, financial aspects, and patient privacy.

Evaluating prompt engineering on GPT-3.5's performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4.

This study was designed to assess how different prompt engineering techniques, specifically direct prompts, Chain of Thought (CoT), and a modified CoT approach, influence the ability of GPT-3.5 to answer clinical and calculation-based medical questions, particularly those styled like the USMLE Step 1 exams. To achieve this, we analyzed the responses of GPT-3.5 to two distinct sets of questions: a batch of 1000 questions generated by GPT-4, and another set comprising 95 real USMLE Step 1 questions. These questions spanned a range of medical calculations and clinical scenarios across various fields and difficulty levels. Our analysis revealed that there were no significant differences in the accuracy of GPT-3.5's responses when using direct prompts, CoT, or modified CoT methods. For instance, in the USMLE sample, the success rates were 61.7% for direct prompts, 62.8% for CoT, and 57.4% for modified CoT, with a p-value of 0.734. Similar trends were observed in the responses to GPT-4 generated questions, both clinical and calculation-based, with p-values above 0.05 indicating no significant difference between the prompt types. The conclusion drawn from this study is that the use of CoT prompt engineering does not significantly alter GPT-3.5's effectiveness in handling medical calculations or clinical scenario questions styled like those in USMLE exams. This finding is crucial as it suggests that performance of ChatGPT remains consistent regardless of whether a CoT technique is used instead of direct prompts. This consistency could be instrumental in simplifying the integration of AI tools like ChatGPT into medical education, enabling healthcare professionals to utilize these tools with ease, without the necessity for complex prompt engineering.

Applications of large language models in psychiatry: a systematic review.

Background: With their unmatched ability to interpret and engage with human language and context, large language models (LLMs) hint at the potential to bridge AI and human cognitive processes. This review explores the current application of LLMs, such as ChatGPT, in the field of psychiatry. Methods: We followed PRISMA guidelines and searched through PubMed, Embase, Web of Science, and Scopus, up until March 2024. Results: From 771 retrieved articles, we included 16 that directly examine LLMs' use in psychiatry. LLMs, particularly ChatGPT and GPT-4, showed diverse applications in clinical reasoning, social media, and education within psychiatry. They can assist in diagnosing mental health issues, managing depression, evaluating suicide risk, and supporting education in the field. However, our review also points out their limitations, such as difficulties with complex cases and potential underestimation of suicide risks. Conclusion: Early research in psychiatry reveals LLMs' versatile applications, from diagnostic support to educational roles. Given the rapid pace of advancement, future investigations are poised to explore the extent to which these models might redefine traditional roles in mental health care.

Assessing calibration and bias of a deployed machine learning malnutrition prediction model within a large healthcare system.

Malnutrition is a frequently underdiagnosed condition leading to increased morbidity, mortality, and healthcare costs. The Mount Sinai Health System (MSHS) deployed a machine learning model (MUST-Plus) to detect malnutrition upon hospital admission. However, in diverse patient groups, a poorly calibrated model may lead to misdiagnosis, exacerbating health care disparities. We explored the model's calibration across different variables and methods to improve calibration. Data from adult patients admitted to five MSHS hospitals from January 1, 2021 - December 31, 2022, were analyzed. We compared MUST-Plus prediction to the registered dietitian's formal assessment. Hierarchical calibration was assessed and compared between the recalibration sample (N = 49,562) of patients admitted between January 1, 2021 - December 31, 2022, and the hold-out sample (N = 17,278) of patients admitted between January 1, 2023 - September 30, 2023. Statistical differences in calibration metrics were tested using bootstrapping with replacement. Before recalibration, the overall model calibration intercept was -1.17 (95% CI: -1.20, -1.14), slope was 1.37 (95% CI: 1.34, 1.40), and Brier score was 0.26 (95% CI: 0.25, 0.26). Both weak and moderate measures of calibration were significantly different between White and Black patients and between male and female patients. Logistic recalibration significantly improved calibration of the model across race and gender in the hold-out sample. The original MUST-Plus model showed significant differences in calibration between White vs. Black patients. It also overestimated malnutrition in females compared to males. Logistic recalibration effectively reduced miscalibration across all patient subgroups. Continual monitoring and timely recalibration can improve model accuracy.

Transforming Cardiovascular Care With Artificial Intelligence: From Discovery to Practice: JACC State-of-the-Art Review.

Artificial intelligence (AI) has the potential to transform every facet of cardiovascular practice and research. The exponential rise in technology powered by AI is defining new frontiers in cardiovascular care, with innovations that span novel diagnostic modalities, new digital native biomarkers of disease, and high-performing tools evaluating care quality and prognosticating clinical outcomes. These digital innovations promise expanded access to cardiovascular screening and monitoring, especially among those without access to high-quality, specialized care historically. Moreover, AI is propelling biological and clinical discoveries that will make future cardiovascular care more personalized, precise, and effective. The review brings together these diverse AI innovations, highlighting developments in multimodal cardiovascular AI across clinical practice and biomedical discovery, and envisioning this new future backed by contemporary science and emerging discoveries. Finally, we define the critical path and the safeguards essential to realizing this AI-enabled future that helps achieve optimal cardiovascular health and outcomes for all.