RESUMO
Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.
RESUMO
The aminopeptidase activity (AP) of the leukotriene A4 hydrolase (LTA4H) enzyme has emerged as a therapeutic target to modulate host immunity. Initial reports focused on the benefits of augmenting the LTA4H AP activity and clearing its putative pro-inflammatory substrate Pro-Gly-Pro (PGP). However, recent reports have introduced substantial complexity disconnecting the LTA4H modulator 4-methoxydiphenylmethane (4MDM) from PGP as follows: (1) 4MDM inhibits PGP hydrolysis and subsequently inhibition of LTA4H AP activity, and (2) 4MDM activates the same enzyme target in the presence of alternative substrates. Differential modulation of LTA4H by 4MDM was probed in a murine model of acute lung inflammation, which showed that 4MDM modulates the host neutrophilic response independent of clearing PGP. X-ray crystallography showed that 4MDM and PGP bind at the zinc binding pocket and no allosteric binding was observed. We then determined that 4MDM modulation is not dependent on the allosteric binding of the ligand, but on the N-terminal side chain of the peptide. In conclusion, our study revealed that a peptidase therapeutic target can interact with its substrate and ligand in complex biochemical mechanisms. This raises an important consideration when ligands are designed to explain some of the unpredictable outcomes observed in therapeutic discovery targeting LTA4H.