Retrieval-Augmented Generation for Extracting CHA2DS2-VASc Risk Factors from Unstructured Clinical Notes in Patients with Atrial Fibrillation.

Background: Assessment of stroke risk in patients with atrial fibrillation (AF) is crucial for guiding anticoagulation therapy. CHA2DS2-VASc is a widely used score for defining this risk, but current assessments rely on manual calculation by clinicians or approximations from structured EHR data elements. Unstructured clinical notes contain rich information that could enhance risk assessment. We developed and validated a Retrieval-Augmented Generation (RAG) approach to extract CHA2DS2-VASc risk factors from unstructured notes in patients with AF. Methods: We employed a RAG architecture paired with the large language model, Llama3.1, to extract features relevant to CHA2DS2-VASc scores from unstructured notes. The model was deployed on a random set of 1,000 clinical notes (934 AF patients) from Yale New Haven Health System (YNHHS). To establish a gold standard, 2 clinicians manually reviewed and labeled CHA2DS2-VASc risk factors in a random subset of 200 notes. The CHA2DS2-VASc scores were calculated for each patient using structured data alone and by incorporating risk factors identified with RAG. We assessed performance across risk factors using macro-averaged area under the receiver operating characteristic (AUROC). For external validation, we utilized 100 manually labeled clinical notes from the MIMIC-IV database. Results: The RAG model demonstrated robust performance in extracting risk factors from clinical notes. In the 1000 clinical notes, RAG identified several risk factors more frequently than structured elements, including hypertension (82.4% vs 26.2%), stroke/TIA (62.9% vs 45.5%), vascular disease (83.4% vs 56.6%), and diabetes (84.1% vs 47.2%). In the 200 expert-annotated notes, the RAG approach achieved high performance for various risk factors, with AUROCs ranging from 0.96 to 0.98 for hypertension, diabetes, and age ≥75 years. Incorporating risk factors identified by RAG increased CHA2DS2-VASc scores compared with using structured data alone. Conclusion: An LLM-optimized RAG can accurately extract CHA2DS2-VASc risk factors from unstructured clinical notes in AF patients. This approach can enable computable risk assessment and guide appropriate anticoagulation therapy.

Automated Transformation of Unstructured Cardiovascular Diagnostic Reports into Structured Datasets Using Sequentially Deployed Large Language Models.

Background: Rich data in cardiovascular diagnostic testing are often sequestered in unstructured reports, with the necessity of manual abstraction limiting their use in real-time applications in patient care and research. Methods: We developed a two-step process that sequentially deploys generative and interpretative large language models (LLMs; Llama2 70b and Llama2 13b). Using a Llama2 70b model, we generated varying formats of transthoracic echocardiogram (TTE) reports from 3,000 real-world echo reports with paired structured elements, leveraging temporal changes in reporting formats to define the variations. Subsequently, we fine-tuned Llama2 13b using sequentially larger batches of generated echo reports as inputs, to extract data from free-text narratives across 18 clinically relevant echocardiographic fields. This was set up as a prompt-based supervised training task. We evaluated the fine-tuned Llama2 13b model, HeartDx-LM, on several distinct echocardiographic datasets: (i) reports across the different time periods and formats at Yale New Haven Health System (YNHHS), (ii) the Medical Information Mart for Intensive Care (MIMIC) III dataset, and (iii) the MIMIC IV dataset. We used the accuracy of extracted fields and Cohen's Kappa as the metrics and have publicly released the HeartDX-LM model. Results: The HeartDX-LM model was trained on randomly selected 2,000 synthetic echo reports with varying formats and paired structured labels, with a wide range of clinical findings. We identified a lower threshold of 500 annotated reports required for fine-tuning Llama2 13b to achieve stable and consistent performance. At YNHHS, the HeartDx-LM model accurately extracted 69,144 out of 70,032 values (98.7%) across 18 clinical fields from unstructured reports in the test set from contemporary records where paired structured data were also available. In older echo reports where only unstructured reports were available, the model achieved 87.1% accuracy against expert annotations for the same 18 fields for a random sample of 100 reports. Similarly, in expert-annotated external validation sets from MIMIC-IV and MIMIC-III, HeartDx-LM correctly extracted 201 out of 220 available values (91.3%) and 615 out of 707 available values (87.9%), respectively, from 100 randomly chosen and expert annotated echo reports from each set. Conclusion: We developed a novel method using paired large and moderate-sized LLMs to automate the extraction of unstructured echocardiographic reports into tabular datasets. Our approach represents a scalable strategy that transforms unstructured reports into computable elements that can be leveraged to improve cardiovascular care quality and enable research.

A Deep Learning Approach for Automated Extraction of Functional Status and New York Heart Association Class for Heart Failure Patients During Clinical Encounters.

Introduction: Serial functional status assessments are critical to heart failure (HF) management but are often described narratively in documentation, limiting their use in quality improvement or patient selection for clinical trials. We developed and validated a deep learning-based natural language processing (NLP) strategy to extract functional status assessments from unstructured clinical notes. Methods: We identified 26,577 HF patients across outpatient services at Yale New Haven Hospital (YNHH), Greenwich Hospital (GH), and Northeast Medical Group (NMG) (mean age 76.1 years; 52.0% women). We used expert annotated notes from YNHH for model development/internal testing and from GH and NMG for external validation. The primary outcomes were NLP models to detect (a) explicit New York Heart Association (NYHA) classification, (b) HF symptoms during activity or rest, and (c) functional status assessment frequency. Results: Among 3,000 expert-annotated notes, 13.6% mentioned NYHA class, and 26.5% described HF symptoms. The model to detect NYHA classes achieved a class-weighted AUROC of 0.99 (95% CI: 0.98-1.00) at YNHH, 0.98 (0.96-1.00) at NMG, and 0.98 (0.92-1.00) at GH. The activity-related HF symptom model achieved an AUROC of 0.94 (0.89-0.98) at YNHH, 0.94 (0.91-0.97) at NMG, and 0.95 (0.92-0.99) at GH. Deploying the NYHA model among 166,655 unannotated notes from YNHH identified 21,528 (12.9%) with NYHA mentions and 17,642 encounters (10.5%) classifiable into functional status groups based on activity-related symptoms. Conclusions: We developed and validated an NLP approach to extract NYHA classification and activity-related HF symptoms from clinical notes, enhancing the ability to track optimal care and identify trial-eligible patients.

Automated Identification of Heart Failure with Reduced Ejection Fraction using Deep Learning-based Natural Language Processing.

Background: The lack of automated tools for measuring care quality has limited the implementation of a national program to assess and improve guideline-directed care in heart failure with reduced ejection fraction (HFrEF). A key challenge for constructing such a tool has been an accurate, accessible approach for identifying patients with HFrEF at hospital discharge, an opportunity to evaluate and improve the quality of care. Methods: We developed a novel deep learning-based language model for identifying patients with HFrEF from discharge summaries using a semi-supervised learning framework. For this purpose, hospitalizations with heart failure at Yale New Haven Hospital (YNHH) between 2015 to 2019 were labeled as HFrEF if the left ventricular ejection fraction was under 40% on antecedent echocardiography. The model was internally validated with model-based net reclassification improvement (NRI) assessed against chart-based diagnosis codes. We externally validated the model on discharge summaries from hospitalizations with heart failure at Northwestern Medicine, community hospitals of Yale New Haven Health in Connecticut and Rhode Island, and the publicly accessible MIMIC-III database, confirmed with chart abstraction. Results: A total of 13,251 notes from 5,392 unique individuals (mean age 73 ± 14 years, 48% female), including 2,487 patients with HFrEF (46.1%), were used for model development (train/held-out test: 70/30%). The deep learning model achieved an area under receiving operating characteristic (AUROC) of 0.97 and an area under precision-recall curve (AUPRC) of 0.97 in detecting HFrEF on the held-out set. In external validation, the model had high performance in identifying HFrEF from discharge summaries with AUROC 0.94 and AUPRC 0.91 on 19,242 notes from Northwestern Medicine, AUROC 0.95 and AUPRC 0.96 on 139 manually abstracted notes from Yale community hospitals, and AUROC 0.91 and AUPRC 0.92 on 146 manually reviewed notes at MIMIC-III. Model-based prediction of HFrEF corresponded to an overall NRI of 60.2 ± 1.9% compared with the chart diagnosis codes (p-value < 0.001) and an increase in AUROC from 0.61 [95% CI: 060-0.63] to 0.91 [95% CI 0.90-0.92]. Conclusions: We developed and externally validated a deep learning language model that automatically identifies HFrEF from clinical notes with high precision and accuracy, representing a key element in automating quality assessment and improvement for individuals with HFrEF.

Characteristics of patients referred to a cardiovascular disease clinic for post-acute sequelae of SARS-CoV-2 infection.

Introduction: There is limited literature on cardiovascular manifestations of post-acute sequelae of SARS-CoV-2 infection (PASC). Methods: This observational study aimed to describe the characteristics, diagnostic evaluations, and new cardiac diagnoses in patients referred to a cardiovascular disease clinic designed for patients with PASC, and to identify factors associated with cardiovascular symptoms with no identifiable cardiac pathology. Results: Of 126 patients, average age was 46 years, and 34 % were male. Patients presented on average five months after COVID-19 diagnosis. The most common symptoms were dyspnea (52 %), chest pain/pressure (48 %), palpitations (44 %), and fatigue (42 %), commonly associated with exertion or exercise intolerance. New cardiovascular diseases were present in 23 % of cases. The remainder exhibited common symptoms which we termed "cardiovascular PASC syndrome." Discussion: We found that only one in four patients had a new cardiovascular diagnosis, but most displayed a pattern of symptoms associated with exercise intolerance.