Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department.

Importance: The introduction of large language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4; OpenAI), has generated significant interest in health care, yet studies evaluating their performance in a clinical setting are lacking. Determination of clinical acuity, a measure of a patient's illness severity and level of required medical attention, is one of the foundational elements of medical reasoning in emergency medicine. Objective: To determine whether an LLM can accurately assess clinical acuity in the emergency department (ED). Design, Setting, and Participants: This cross-sectional study identified all adult ED visits from January 1, 2012, to January 17, 2023, at the University of California, San Francisco, with a documented Emergency Severity Index (ESI) acuity level (immediate, emergent, urgent, less urgent, or nonurgent) and with a corresponding ED physician note. A sample of 10 000 pairs of ED visits with nonequivalent ESI scores, balanced for each of the 10 possible pairs of 5 ESI scores, was selected at random. Exposure: The potential of the LLM to classify acuity levels of patients in the ED based on the ESI across 10 000 patient pairs. Using deidentified clinical text, the LLM was queried to identify the patient with a higher-acuity presentation within each pair based on the patients' clinical history. An earlier LLM was queried to allow comparison with this model. Main Outcomes and Measures: Accuracy score was calculated to evaluate the performance of both LLMs across the 10 000-pair sample. A 500-pair subsample was manually classified by a physician reviewer to compare performance between the LLMs and human classification. Results: From a total of 251 401 adult ED visits, a balanced sample of 10 000 patient pairs was created wherein each pair comprised patients with disparate ESI acuity scores. Across this sample, the LLM correctly inferred the patient with higher acuity for 8940 of 10 000 pairs (accuracy, 0.89 [95% CI, 0.89-0.90]). Performance of the comparator LLM (accuracy, 0.84 [95% CI, 0.83-0.84]) was below that of its successor. Among the 500-pair subsample that was also manually classified, LLM performance (accuracy, 0.88 [95% CI, 0.86-0.91]) was comparable with that of the physician reviewer (accuracy, 0.86 [95% CI, 0.83-0.89]). Conclusions and Relevance: In this cross-sectional study of 10 000 pairs of ED visits, the LLM accurately identified the patient with higher acuity when given pairs of presenting histories extracted from patients' first ED documentation. These findings suggest that the integration of an LLM into ED workflows could enhance triage processes while maintaining triage quality and warrants further investigation.

Evaluating Large Language Models for Drafting Emergency Department Discharge Summaries.

Importance: Large language models (LLMs) possess a range of capabilities which may be applied to the clinical domain, including text summarization. As ambient artificial intelligence scribes and other LLM-based tools begin to be deployed within healthcare settings, rigorous evaluations of the accuracy of these technologies are urgently needed. Objective: To investigate the performance of GPT-4 and GPT-3.5-turbo in generating Emergency Department (ED) discharge summaries and evaluate the prevalence and type of errors across each section of the discharge summary. Design: Cross-sectional study. Setting: University of California, San Francisco ED. Participants: We identified all adult ED visits from 2012 to 2023 with an ED clinician note. We randomly selected a sample of 100 ED visits for GPT-summarization. Exposure: We investigate the potential of two state-of-the-art LLMs, GPT-4 and GPT-3.5-turbo, to summarize the full ED clinician note into a discharge summary. Main Outcomes and Measures: GPT-3.5-turbo and GPT-4-generated discharge summaries were evaluated by two independent Emergency Medicine physician reviewers across three evaluation criteria: 1) Inaccuracy of GPT-summarized information; 2) Hallucination of information; 3) Omission of relevant clinical information. On identifying each error, reviewers were additionally asked to provide a brief explanation for their reasoning, which was manually classified into subgroups of errors. Results: From 202,059 eligible ED visits, we randomly sampled 100 for GPT-generated summarization and then expert-driven evaluation. In total, 33% of summaries generated by GPT-4 and 10% of those generated by GPT-3.5-turbo were entirely error-free across all evaluated domains. Summaries generated by GPT-4 were mostly accurate, with inaccuracies found in only 10% of cases, however, 42% of the summaries exhibited hallucinations and 47% omitted clinically relevant information. Inaccuracies and hallucinations were most commonly found in the Plan sections of GPT-generated summaries, while clinical omissions were concentrated in text describing patients' Physical Examination findings or History of Presenting Complaint. Conclusions and Relevance: In this cross-sectional study of 100 ED encounters, we found that LLMs could generate accurate discharge summaries, but were liable to hallucination and omission of clinically relevant information. A comprehensive understanding of the location and type of errors found in GPT-generated clinical text is important to facilitate clinician review of such content and prevent patient harm.

Characterisation of digital therapeutic clinical trials: a systematic review with natural language processing.

Digital therapeutics (DTx) are a somewhat novel class of US Food and Drug Administration-regulated software that help patients prevent, manage, or treat disease. Here, we use natural language processing to characterise registered DTx clinical trials and provide insights into the clinical development landscape for these novel therapeutics. We identified 449 DTx clinical trials, initiated or expected to be initiated between 2010 and 2030, from ClinicalTrials.gov using 27 search terms, and available data were analysed, including trial durations, locations, MeSH categories, enrolment, and sponsor types. Topic modelling of eligibility criteria, done with BERTopic, showed that DTx trials frequently exclude patients on the basis of age, comorbidities, pregnancy, language barriers, and digital determinants of health, including smartphone or data plan access. Our comprehensive overview of the DTx development landscape highlights challenges in designing inclusive DTx clinical trials and presents opportunities for clinicians and researchers to address these challenges. Finally, we provide an interactive dashboard for readers to conduct their own analyses.

Bottom-up and top-down paradigms of artificial intelligence research approaches to healthcare data science using growing real-world big data.

OBJECTIVES: As the real-world electronic health record (EHR) data continue to grow exponentially, novel methodologies involving artificial intelligence (AI) are becoming increasingly applied to enable efficient data-driven learning and, ultimately, to advance healthcare. Our objective is to provide readers with an understanding of evolving computational methods and help in deciding on methods to pursue. TARGET AUDIENCE: The sheer diversity of existing methods presents a challenge for health scientists who are beginning to apply computational methods to their research. Therefore, this tutorial is aimed at scientists working with EHR data who are early entrants into the field of applying AI methodologies. SCOPE: This manuscript describes the diverse and growing AI research approaches in healthcare data science and categorizes them into 2 distinct paradigms, the bottom-up and top-down paradigms to provide health scientists venturing into artificial intelligent research with an understanding of the evolving computational methods and help in deciding on methods to pursue through the lens of real-world healthcare data.

Autoantibodies to Perilipin-1 Define a Subset of Acquired Generalized Lipodystrophy.

Acquired lipodystrophy is often characterized as an idiopathic subtype of lipodystrophy. Despite suspicion of an immune-mediated pathology, biomarkers such as autoantibodies are generally lacking. Here, we used an unbiased proteome-wide screening approach to identify autoantibodies to the adipocyte-specific lipid droplet protein perilipin 1 (PLIN1) in a murine model of autoimmune polyendocrine syndrome type 1 (APS1). We then tested for PLIN1 autoantibodies in human subjects with acquired lipodystrophy with two independent severe breaks in immune tolerance (including APS1) along with control subjects using a specific radioligand binding assay and indirect immunofluorescence on fat tissue. We identified autoantibodies to PLIN1 in these two cases, including the first reported case of APS1 with acquired lipodystrophy and a second patient who acquired lipodystrophy as an immune-related adverse event following cancer immunotherapy. Lastly, we also found PLIN1 autoantibodies to be specifically enriched in a subset of patients with acquired generalized lipodystrophy (17 of 46 [37%]), particularly those with panniculitis and other features of autoimmunity. These data lend additional support to new literature that suggests that PLIN1 autoantibodies represent a marker of acquired autoimmune lipodystrophies and further link them to a break in immune tolerance.

Autoantibody discovery across monogenic, acquired, and COVID-19-associated autoimmunity with scalable PhIP-seq.

Phage immunoprecipitation sequencing (PhIP-seq) allows for unbiased, proteome-wide autoantibody discovery across a variety of disease settings, with identification of disease-specific autoantigens providing new insight into previously poorly understood forms of immune dysregulation. Despite several successful implementations of PhIP-seq for autoantigen discovery, including our previous work (Vazquez et al., 2020), current protocols are inherently difficult to scale to accommodate large cohorts of cases and importantly, healthy controls. Here, we develop and validate a high throughput extension of PhIP-seq in various etiologies of autoimmune and inflammatory diseases, including APS1, IPEX, RAG1/2 deficiency, Kawasaki disease (KD), multisystem inflammatory syndrome in children (MIS-C), and finally, mild and severe forms of COVID-19. We demonstrate that these scaled datasets enable machine-learning approaches that result in robust prediction of disease status, as well as the ability to detect both known and novel autoantigens, such as prodynorphin (PDYN) in APS1 patients, and intestinally expressed proteins BEST4 and BTNL8 in IPEX patients. Remarkably, BEST4 antibodies were also found in two patients with RAG1/2 deficiency, one of whom had very early onset IBD. Scaled PhIP-seq examination of both MIS-C and KD demonstrated rare, overlapping antigens, including CGNL1, as well as several strongly enriched putative pneumonia-associated antigens in severe COVID-19, including the endosomal protein EEA1. Together, scaled PhIP-seq provides a valuable tool for broadly assessing both rare and common autoantigen overlap between autoimmune diseases of varying origins and etiologies.

Autoantibody discovery across monogenic, acquired, and COVID19-associated autoimmunity with scalable PhIP-Seq.

Phage Immunoprecipitation-Sequencing (PhIP-Seq) allows for unbiased, proteome-wide autoantibody discovery across a variety of disease settings, with identification of disease-specific autoantigens providing new insight into previously poorly understood forms of immune dysregulation. Despite several successful implementations of PhIP-Seq for autoantigen discovery, including our previous work (Vazquez et al. 2020), current protocols are inherently difficult to scale to accommodate large cohorts of cases and importantly, healthy controls. Here, we develop and validate a high throughput extension of PhIP-seq in various etiologies of autoimmune and inflammatory diseases, including APS1, IPEX, RAG1/2 deficiency, Kawasaki Disease (KD), Multisystem Inflammatory Syndrome in Children (MIS-C), and finally, mild and severe forms of COVID19. We demonstrate that these scaled datasets enable machine-learning approaches that result in robust prediction of disease status, as well as the ability to detect both known and novel autoantigens, such as PDYN in APS1 patients, and intestinally expressed proteins BEST4 and BTNL8 in IPEX patients. Remarkably, BEST4 antibodies were also found in 2 patients with RAG1/2 deficiency, one of whom had very early onset IBD. Scaled PhIP-Seq examination of both MIS-C and KD demonstrated rare, overlapping antigens, including CGNL1, as well as several strongly enriched putative pneumonia-associated antigens in severe COVID19, including the endosomal protein EEA1. Together, scaled PhIP-Seq provides a valuable tool for broadly assessing both rare and common autoantigen overlap between autoimmune diseases of varying origins and etiologies.

Immunodietica: interrogating the role of diet in autoimmune disease.

Diet is an environmental factor in autoimmune disorders, where the immune system erroneously destroys one's own tissues. Yet, interactions between diet and autoimmunity remain largely unexplored, particularly the impact of immunogenetics, one's human leukocyte antigen (HLA) allele make-up, in this interplay. Here, we interrogated animals and plants for the presence of epitopes implicated in human autoimmune diseases. We mapped autoimmune epitope distribution across organisms and determined their tissue expression pattern. Interestingly, diet-derived epitopes implicated in a disease were more likely to bind to HLA alleles associated with that disease than to protective alleles, with visible differences between organisms with similar autoimmune epitope content. We then analyzed an individual's HLA haplotype, generating a personalized heatmap of potential dietary autoimmune triggers. Our work uncovered differences in autoimmunogenic potential across food sources and revealed differential binding of diet-derived epitopes to autoimmune disease-associated HLA alleles, shedding light on the impact of diet on autoimmunity.

Identification of novel, clinically correlated autoantigens in the monogenic autoimmune syndrome APS1 by proteome-wide PhIP-Seq.

The identification of autoantigens remains a critical challenge for understanding and treating autoimmune diseases. Autoimmune polyendocrine syndrome type 1 (APS1), a rare monogenic form of autoimmunity, presents as widespread autoimmunity with T and B cell responses to multiple organs. Importantly, autoantibody discovery in APS1 can illuminate fundamental disease pathogenesis, and many of the antigens found in APS1 extend to more common autoimmune diseases. Here, we performed proteome-wide programmable phage-display (PhIP-Seq) on sera from a cohort of people with APS1 and discovered multiple common antibody targets. These novel APS1 autoantigens exhibit tissue-restricted expression, including expression in enteroendocrine cells, pineal gland, and dental enamel. Using detailed clinical phenotyping, we find novel associations between autoantibodies and organ-restricted autoimmunity, including a link between anti-KHDC3L autoantibodies and premature ovarian insufficiency, and between anti-RFX6 autoantibodies and diarrheal-type intestinal dysfunction. Our study highlights the utility of PhIP-Seq for extensively interrogating antigenic repertoires in human autoimmunity and the importance of antigen discovery for improved understanding of disease mechanisms.

The immune system uses antibodies to fight microbes that cause disease. White blood cells pump antibodies into the bloodstream, and these antibodies latch onto bacteria and viruses, targeting them for destruction. But sometimes, the immune system gets it wrong. In autoimmune diseases, white blood cells mistakenly make antibodies that target the body's own tissues. Detecting these 'autoantibodies' in the blood can help doctors to diagnose autoimmune diseases. But the identities and targets of many autoantibodies remain unknown. In one rare disease, called autoimmune polyendocrine syndrome type 1 (APS-1), a faulty gene makes the immune system much more likely to make autoantibodies. People with this disease can develop an autoimmune response against many different healthy organs. Although APS-1 is rare, some of the autoantibodies made by individuals with the disease are the same as the ones in more common autoimmune diseases, like type 1 diabetes. Therefore, investigating the other autoantibodies produced by individuals with APS-1 could reveal the autoantibodies driving other autoimmune diseases. Autoantibodies bind to specific regions of healthy proteins, and one way to identify them is to use hundreds of thousands of tiny viruses in a technique called proteome-wide programmable phage-display, or PhIP-Seq. Each phage carries one type of protein segment. When mixed with blood serum from a patient, the autoantibodies stick to the phages that carry the target proteins for that autoantibody. These complexes can be isolated using biochemical techniques. Sequencing the genes of these phages then reveals the identity of the autoantibodies' targets. Using this technique, Vazquez et al successfully pulled 23 known autoantibodies from the serum of patients with APS-1. Then, experiments to search for new targets began. This revealed many new autoantibodies, targeting proteins found only in specific tissues. They included one that targets a protein found on cells in the gut, and another that targets a protein found on egg cells in the ovaries. Matching the PhIP-Seq data to patient symptoms confirmed that these new antibodies correlate with the features of specific autoimmune diseases. For example, patients with antibodies that targeted the gut protein were more likely to have gut symptoms, while patients with antibodies that targeted the egg cell protein were more likely to have problems with their ovaries. Further investigations using PhIP-Seq could reveal the identities of even more autoantibodies. This might pave the way for new antibody tests to diagnose autoimmune diseases and identify tissues at risk of damage. This could be useful not only for people with APS-1, but also for more common autoimmune diseases that target the same organs.