Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores.

Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Regulation of RNA editing by intracellular acidification.

The hydrolytic deamination of adenosine-to-inosine (A-to-I) by RNA editing is a widespread post-transcriptional modification catalyzed by the adenosine deaminase acting on RNA (ADAR) family of proteins. ADAR-mediated RNA editing modulates cellular pathways involved in innate immunity, RNA splicing, RNA interference, and protein recoding, and has been investigated as a strategy for therapeutic intervention of genetic disorders. Despite advances in basic and translational research, the mechanisms regulating RNA editing are poorly understood. Though several trans-acting regulators of editing have been shown to modulate ADAR protein expression, previous studies have not identified factors that modulate ADAR catalytic activity. Here, we show that RNA editing increases upon intracellular acidification, and that these effects are predominantly explained by both enhanced ADAR base-flipping and deamination rate at acidic pH. We also show that the extent of RNA editing increases with the reduction in pH associated with conditions of cellular hypoxia.

Xbp1s-FoxO1 axis governs lipid accumulation and contractile performance in heart failure with preserved ejection fraction.

Heart failure with preserved ejection fraction (HFpEF) is now the dominant form of heart failure and one for which no efficacious therapies exist. Obesity and lipid mishandling greatly contribute to HFpEF. However, molecular mechanism(s) governing metabolic alterations and perturbations in lipid homeostasis in HFpEF are largely unknown. Here, we report that cardiomyocyte steatosis in HFpEF is coupled with increases in the activity of the transcription factor FoxO1 (Forkhead box protein O1). FoxO1 depletion, as well as over-expression of the Xbp1s (spliced form of the X-box-binding protein 1) arm of the UPR (unfolded protein response) in cardiomyocytes each ameliorates the HFpEF phenotype in mice and reduces myocardial lipid accumulation. Mechanistically, forced expression of Xbp1s in cardiomyocytes triggers ubiquitination and proteasomal degradation of FoxO1 which occurs, in large part, through activation of the E3 ubiquitin ligase STUB1 (STIP1 homology and U-box-containing protein 1) a novel and direct transcriptional target of Xbp1s. Our findings uncover the Xbp1s-FoxO1 axis as a pivotal mechanism in the pathogenesis of cardiometabolic HFpEF and unveil previously unrecognized mechanisms whereby the UPR governs metabolic alterations in cardiomyocytes.

Nitrosative stress drives heart failure with preserved ejection fraction.

Heart failure with preserved ejection fraction (HFpEF) is a common syndrome with high morbidity and mortality for which there are no evidence-based therapies. Here we report that concomitant metabolic and hypertensive stress in mice-elicited by a combination of high-fat diet and inhibition of constitutive nitric oxide synthase using Nω-nitro-L-arginine methyl ester (L-NAME)-recapitulates the numerous systemic and cardiovascular features of HFpEF in humans. Expression of one of the unfolded protein response effectors, the spliced form of X-box-binding protein 1 (XBP1s), was reduced in the myocardium of our rodent model and in humans with HFpEF. Mechanistically, the decrease in XBP1s resulted from increased activity of inducible nitric oxide synthase (iNOS) and S-nitrosylation of the endonuclease inositol-requiring protein 1α (IRE1α), culminating in defective XBP1 splicing. Pharmacological or genetic suppression of iNOS, or cardiomyocyte-restricted overexpression of XBP1s, each ameliorated the HFpEF phenotype. We report that iNOS-driven dysregulation of the IRE1α-XBP1 pathway is a crucial mechanism of cardiomyocyte dysfunction in HFpEF.