Validation and clinical discovery demonstration of breast cancer data from a real-world data extraction platform.

Objective: To validate and demonstrate the clinical discovery utility of a novel patient-mediated, medical record collection and data extraction platform developed to improve access and utilization of real-world clinical data. Materials and Methods: Clinical variables were extracted from the medical records of 1011 consented patients with breast cancer. To validate the extracted data, case report forms completed using the structured data output of the platform were compared to manual chart review for 50 randomly-selected patients with metastatic breast cancer. To demonstrate the platform's clinical discovery utility, we identified 194 patients with early-stage clinical data who went on to develop distant metastases and utilized the platform-extracted data to assess associations between time to distant metastasis (TDM) and early-stage tumor histology, molecular type, and germline BRCA status. Results: The platform-extracted data for the validation cohort had 97.6% precision (91.98%-100% by variable type) and 81.48% recall (58.15%-95.00% by variable type) compared to manual chart review. In our discovery cohort, the shortest TDM was significantly associated with metaplastic (739.0 days) and inflammatory histologies (1005.8 days), HR-/HER2- molecular types (1187.4 days), and positive BRCA status (1042.5 days) as compared to other histologies, molecular types, and negative BRCA status, respectively. Multivariable analyses did not produce statistically significant results. Discussion: The precision and recall of platform-extracted clinical data are reported, although specificity could not be assessed. The data can generate clinically-relevant insights. Conclusion: The structured real-world data produced by a novel patient-mediated, medical record-extraction platform are reliable and can power clinical discovery.

Evaluating Large Language Models in Extracting Cognitive Exam Dates and Scores.

Importance: Large language models (LLMs) are crucial for medical tasks. Ensuring their reliability is vital to avoid false results. Our study assesses two state-of-the-art LLMs (ChatGPT and LlaMA-2) for extracting clinical information, focusing on cognitive tests like MMSE and CDR. Objective: Evaluate ChatGPT and LlaMA-2 performance in extracting MMSE and CDR scores, including their associated dates. Methods: Our data consisted of 135,307 clinical notes (Jan 12th, 2010 to May 24th, 2023) mentioning MMSE, CDR, or MoCA. After applying inclusion criteria 34,465 notes remained, of which 765 underwent ChatGPT (GPT-4) and LlaMA-2, and 22 experts reviewed the responses. ChatGPT successfully extracted MMSE and CDR instances with dates from 742 notes. We used 20 notes for fine-tuning and training the reviewers. The remaining 722 were assigned to reviewers, with 309 each assigned to two reviewers simultaneously. Inter-rater-agreement (Fleiss' Kappa), precision, recall, true/false negative rates, and accuracy were calculated. Our study follows TRIPOD reporting guidelines for model validation. Results: For MMSE information extraction, ChatGPT (vs. LlaMA-2) achieved accuracy of 83% (vs. 66.4%), sensitivity of 89.7% (vs. 69.9%), true-negative rates of 96% (vs 60.0%), and precision of 82.7% (vs 62.2%). For CDR the results were lower overall, with accuracy of 87.1% (vs. 74.5%), sensitivity of 84.3% (vs. 39.7%), true-negative rates of 99.8% (98.4%), and precision of 48.3% (vs. 16.1%). We qualitatively evaluated the MMSE errors of ChatGPT and LlaMA-2 on double-reviewed notes. LlaMA-2 errors included 27 cases of total hallucination, 19 cases of reporting other scores instead of MMSE, 25 missed scores, and 23 cases of reporting only the wrong date. In comparison, ChatGPT's errors included only 3 cases of total hallucination, 17 cases of wrong test reported instead of MMSE, and 19 cases of reporting a wrong date. Conclusions: In this diagnostic/prognostic study of ChatGPT and LlaMA-2 for extracting cognitive exam dates and scores from clinical notes, ChatGPT exhibited high accuracy, with better performance compared to LlaMA-2. The use of LLMs could benefit dementia research and clinical care, by identifying eligible patients for treatments initialization or clinical trial enrollments. Rigorous evaluation of LLMs is crucial to understanding their capabilities and limitations.

Exploration of HER2 (ERBB2) immunohistochemistry in non-small cell lung cancer: correlation with ERBB2 mutational status: experimental research.

ERBB2 (HER2) is a gene in humans that encodes the ERBB2 protein, a member of the epidermal growth factor receptor family. Non-small cell lung carcinomas do not commonly harbour ERBB2 mutations, with clinical trials conducted to assess for targeted response and progression-free survival. We retrieved cases of lung adenocarcinoma with next-generation sequencing proven ERBB2 point mutations (n=8) or amplifications (n=11) and assessed the concordance of commercially available ERBB2 (HER2) immunohistochemical antibodies with the next-generation sequencing result. At present, no commercially available ERBB2 clone can accurately detect ERBB2 mutations consistently in non-small cell lung carcinoma specimens, but amplifications can be detected with reasonable diagnostic accuracy.

A Multiplex Microsphere IgG Assay for SARS-CoV-2 Using ACE2-Mediated Inhibition as a Surrogate for Neutralization.

The coronavirus disease 2019 (COVID-19) pandemic has highlighted the challenges inherent to the serological detection of a novel pathogen such as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Serological tests can be used diagnostically and for surveillance, but their usefulness depends on their throughput, sensitivity, and specificity. Here, we describe a multiplex fluorescent microsphere-based assay, 3Flex, that can detect antibodies to three major SARS-CoV-2 antigens-spike (S) protein, the spike ACE2 receptor-binding domain (RBD), and nucleocapsid (NP). Specificity was assessed using 213 prepandemic samples. Sensitivity was measured and compared to that of the Abbott Architect SARS-CoV-2 IgG assay using serum samples from 125 unique patients equally binned (n = 25) into 5 time intervals (≤5, 6 to 10, 11 to 15, 16 to 20, and ≥21 days from symptom onset). With samples obtained at ≤5 days from symptom onset, the 3Flex assay was more sensitive (48.0% versus 32.0%), but the two assays performed comparably using serum obtained ≥21 days from symptom onset. A larger collection (n = 534) of discarded sera was profiled from patients (n = 140) whose COVID-19 course was characterized through chart review. This revealed the relative rise, peak (S, 23.8; RBD, 23.6; NP, 16.7 [in days from symptom onset]), and decline of the antibody response. Considerable interperson variation was observed with a subset of extensively sampled intensive care unit (ICU) patients. Using soluble ACE2, inhibition of antibody binding was demonstrated for S and RBD, and not for NP. Taking the data together, this study described the performance of an assay built on a flexible and high-throughput serological platform that proved adaptable to the emergence of a novel infectious agent.

A multiplex microsphere IgG assay for SARS-CoV-2 using ACE2-mediated inhibition as a surrogate for neutralization.

The COVID-19 pandemic has highlighted challenges inherent to serological detection of a novel pathogen like SARS-CoV-2. Serological tests can be used diagnostically and for surveillance, but their usefulness depends on throughput, sensitivity and specificity. Here, we describe a multiplex fluorescent microsphere-based assay, 3Flex, that can detect antibodies to three SARS-CoV-2 antigens-spike (S) protein, the spike ACE2 receptor-binding domain (RBD), and nucleocapsid (NP). Specificity was assessed using 213 pre-pandemic samples. Sensitivity was measured and compared to the Abbott™ ARCHITECT™ SARS-CoV-2 IgG assay using serum from 125 unique patients equally binned ( n = 25) into 5 time intervals (≤5, 6 to 10, 11 to 15, 16 to 20, and ≥21 days from symptom onset). With samples obtained at ≤5 days from symptom onset, the 3Flex assay was more sensitive (48.0% vs. 32.0%), but the two assays performed comparably using serum obtained ≥21 days from symptom onset. A larger collection ( n = 534) of discarded sera was profiled from patients ( n = 140) whose COVID-19 course was characterized through chart review. This revealed the relative rise, peak (S, 23.8; RBD, 23.6; NP, 16.7; in days from symptom onset), and decline of the antibody response. Considerable interperson variation was observed with a subset of extensively sampled ICU patients. Using soluble ACE2, inhibition of antibody binding was demonstrated for S and RBD, and not for NP. Taken together, this study described the performance of an assay built on a flexible and high-throughput serological platform that proved adaptable to the emergence of a novel infectious agent.