Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores.

Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Cell-Specific Transcriptional Responses to Heat Shock in the Mouse Utricle Epithelium.

Sensory epithelia of the inner ear contain mechanosensory hair cells (HCs) and glia-like supporting cells (SCs), both of which are required for hearing and balance functions. Each of these cell types has unique responses to ototoxic and cytoprotective stimuli. Non-lethal heat stress in the mammalian utricle induces heat shock proteins (HSPs) and protects against ototoxic drug-induced hair cell death. Induction of HSPs in the utricle demonstrates cell-type specificity at the protein level, with HSP70 induction occurring primarily in SCs, while HSP32 (also known as heme oxygenase 1, HMOX1) is induced primarily in resident macrophages. Neither of these HSPs are robustly induced in HCs, suggesting that HCs may have little capacity for induction of stress-induced protective responses. To determine the transcriptional responses to heat shock of these different cell types, we performed cell-type-specific transcriptional profiling using the RiboTag method, which allows for immunoprecipitation (IP) of actively translating mRNAs from specific cell types. RNA-Seq differential gene expression analyses demonstrated that the RiboTag method identified known cell type-specific markers as well as new markers for HCs and SCs. Gene expression differences suggest that HCs and SCs exhibit differential transcriptional heat shock responses. The chaperonin family member Cct8 was significantly enriched only in heat-shocked HCs, while Hspa1l (HSP70 family), and Hspb1 and Cryab (HSP27 and HSP20 families, respectively) were enriched only in SCs. Together our data indicate that HCs exhibit a limited but unique heat shock response, and SCs exhibit a broader and more robust transcriptional response to protective heat stress.

A Pliable Mediator Acts as a Functional Rather Than an Architectural Bridge between Promoters and Enhancers.

While Mediator plays a key role in eukaryotic transcription, little is known about its mechanism of action. This study combines CRISPR-Cas9 genetic screens, degron assays, Hi-C, and cryoelectron microscopy (cryo-EM) to dissect the function and structure of mammalian Mediator (mMED). Deletion analyses in B, T, and embryonic stem cells (ESC) identified a core of essential subunits required for Pol II recruitment genome-wide. Conversely, loss of non-essential subunits mostly affects promoters linked to multiple enhancers. Contrary to current models, however, mMED and Pol II are dispensable to physically tether regulatory DNA, a topological activity requiring architectural proteins. Cryo-EM analysis revealed a conserved core, with non-essential subunits increasing structural complexity of the tail module, a primary transcription factor target. Changes in tail structure markedly increase Pol II and kinase module interactions. We propose that Mediator's structural pliability enables it to integrate and transmit regulatory signals and act as a functional, rather than an architectural bridge, between promoters and enhancers.