Comparative Evaluation of LLMs in Clinical Oncology.

BACKGROUND: As artificial intelligence (AI) tools become widely accessible, more patients and medical professionals will turn to them for medical information. Large language models (LLMs), a subset of AI, excel in natural language processing tasks and hold considerable promise for clinical use. Fields such as oncology, in which clinical decisions are highly dependent on a continuous influx of new clinical trial data and evolving guidelines, stand to gain immensely from such advancements. It is therefore of critical importance to benchmark these models and describe their performance characteristics to guide their safe application to clinical oncology. Accordingly, the primary objectives of this work were to conduct comprehensive evaluations of LLMs in the field of oncology and to identify and characterize strategies that medical professionals can use to bolster their confidence in a model's response. METHODS: This study tested five publicly available LLMs (LLaMA 1, PaLM 2, Claude-v1, generative pretrained transformer 3.5 [GPT-3.5], and GPT-4) on a comprehensive battery of 2044 oncology questions, including topics from medical oncology, surgical oncology, radiation oncology, medical statistics, medical physics, and cancer biology. Model prompts were presented independently of each other, and each prompt was repeated three times to assess output consistency. For each response, models were instructed to provide a self-appraised confidence score (from 1 to 4). Model performance was also evaluated against a novel validation set comprising 50 oncology questions curated to eliminate any risk of overlap with the data used to train the LLMs. RESULTS: There was significant heterogeneity in performance between models (analysis of variance, P<0.001). Relative to a human benchmark (2013 and 2014 examination results), GPT-4 was the only model to perform above the 50th percentile. Overall, model performance varied as a function of subject area across all models, with worse performance observed in clinical oncology subcategories compared with foundational topics (medical statistics, medical physics, and cancer biology). Within the clinical oncology subdomain, worse performance was observed in female-predominant malignancies. A combination of model selection, prompt repetition, and confidence self-appraisal allowed for the identification of high-performing subgroups of questions with observed accuracies of 81.7 and 81.1% in the Claude-v1 and GPT-4 models, respectively. Evaluation of the novel validation question set produced similar trends in model performance while also highlighting improved performance in newer, centrally hosted models (GPT-4 Turbo and Gemini 1.0 Ultra) and local models (Mixtral 8×7B and LLaMA 2). CONCLUSIONS: Of the models tested on a standardized set of oncology questions, GPT-4 was observed to have the highest performance. Although this performance is impressive, all LLMs continue to have clinically significant error rates, including examples of overconfidence and consistent inaccuracies. Given the enthusiasm to integrate these new implementations of AI into clinical practice, continued standardized evaluations of the strengths and limitations of these products will be critical to guide both patients and medical professionals. (Funded by the National Institutes of Health Clinical Center for Research and the Intramural Research Program of the National Institutes of Health; Z99 CA999999.).

Inferring characteristics of the tumor-immune microenvironment of HNSCC patients from single-cell transcriptomics of peripheral blood.

In this study, we explore the possibility of inferring characteristics of the tumor-immune microenvironment (TIME) from the blood. Specifically, we investigate two datasets of head and neck squamous cell carcinoma (HNSCC) patients with matched scRNA-Seq from peripheral blood mononuclear cells (PBMCs) and tumor tissues. Our analysis shows that the immune cell fractions and gene expression profiles of various immune cells within the tumor microenvironment can be inferred from the matched PBMC scRNA-Seq data. We find that the established exhausted T-cell signature can be predicted from the blood and serve as a valuable prognostic blood biomarker of immunotherapy response. Additionally, our study reveals that the inferred ratio between tumor memory B and regulatory T cell fractions is predictive of immunotherapy response and is superior to the well-established cytolytic and exhausted T-cell signatures. These results highlight the promising potential of PBMC scRNA-Seq in cancer immunotherapy and warrant, and will hopefully facilitate, further investigations on a larger scale. The code for predicting tumor immune microenvironment from PBMC scRNA-Seq, TIMEP, is provided, offering other researchers the opportunity to investigate its prospective applications in various other indications.

Identification of intracellular bacteria from multiple single-cell RNA-seq platforms using CSI-Microbes.

The study of the tumor microbiome has been garnering increased attention. We developed a computational pipeline (CSI-Microbes) for identifying microbial reads from single-cell RNA sequencing (scRNA-seq) data and for analyzing differential abundance of taxa. Using a series of controlled experiments and analyses, we performed the first systematic evaluation of the efficacy of recovering microbial unique molecular identifiers by multiple scRNA-seq technologies, which identified the newer 10x chemistries (3' v3 and 5') as the best suited approach. We analyzed patient esophageal and colorectal carcinomas and found that reads from distinct genera tend to co-occur in the same host cells, testifying to possible intracellular polymicrobial interactions. Microbial reads are disproportionately abundant within myeloid cells that up-regulate proinflammatory cytokines like IL1Β and CXCL8, while infected tumor cells up-regulate antigen processing and presentation pathways. These results show that myeloid cells with bacteria engulfed are a major source of bacterial RNA within the tumor microenvironment (TME) and may inflame the TME and influence immunotherapy response.

Loss of tumor suppressors promotes inflammatory tumor microenvironment and enhances LAG3+T cell mediated immune suppression.

Low response rate, treatment relapse, and resistance remain key challenges for cancer treatment with immune checkpoint blockade (ICB). Here we report that loss of specific tumor suppressors (TS) induces an inflammatory response and promotes an immune suppressive tumor microenvironment. Importantly, low expression of these TSs is associated with a higher expression of immune checkpoint inhibitory mediators. Here we identify, by using in vivo CRISPR/Cas9 based loss-of-function screening, that NF1, TSC1, and TGF-ß RII as TSs regulating immune composition. Loss of each of these three TSs leads to alterations in chromatin accessibility and enhances IL6-JAK3-STAT3/6 inflammatory pathways. This results in an immune suppressive landscape, characterized by increased numbers of LAG3+ CD8 and CD4 T cells. ICB targeting LAG3 and PD-L1 simultaneously inhibits metastatic progression in preclinical triple negative breast cancer (TNBC) mouse models of NF1-, TSC1- or TGF-ß RII- deficient tumors. Our study thus reveals a role of TSs in regulating metastasis via non-cell-autonomous modulation of the immune compartment and provides proof-of-principle for ICB targeting LAG3 for patients with NF1-, TSC1- or TGF-ß RII-inactivated cancers.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics.

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

Chromosome 7 gain compensates for chromosome 10 loss in glioma.

The co-occurrence of chromosome 10 loss and chromosome 7 gain in gliomas is the most frequent loss-gain co-aneuploidy pair in human cancers. This phenomenon has been investigated since the late 1980s without resolution. Expanding beyond previous gene-centric studies, we investigated the co-occurrence in a genome-wide manner taking an evolutionary perspective. Mining of large-scale tumor aneuploidy data confirmed the previous finding of a small-scale longitudinal study that the most likely order is chromosome 10 loss followed by chromosome 7 gain. Extensive analysis of genomic and transcriptomic data from both patients and cell lines revealed that this co-occurrence can be explained by functional rescue interactions that are highly enriched on chromosome 7, which could potentially compensate for any detrimental consequences arising from the loss of chromosome 10. Transcriptomic data from various normal, non-cancerous human brain tissues were analyzed to assess which tissues may be most predisposed to tolerate compensation of chromosome 10 loss by chromosome 7 gain. The analysis indicated that the pre-existing transcriptomic states in the cortex and frontal cortex, where gliomas arise, are more favorable than other brain regions for compensation by rescuer genes that are active on chromosome 7. Collectively, these findings suggest that the phenomenon of chromosome 10 loss and chromosome 7 gain in gliomas is orchestrated by a complex interaction of many genes residing within these two chromosomes and provide a plausible reason why this co-occurrence happens preferentially in cancers originating in certain regions of the brain.

Decoupling the correlation between cytotoxic and exhausted T lymphocyte states enhances melanoma immunotherapy response prediction.

Cytotoxic T lymphocyte (CTL) and terminal exhausted T lymphocyte (ETL) activities crucially influence immune checkpoint inhibitor (ICI) response. Despite this, the efficacy of ETL and CTL transcriptomic signatures for response prediction remains limited. Investigating this across the TCGA and publicly available single-cell cohorts, we find a strong positive correlation between ETL and CTL expression signatures in most cancers. We hence posited that their limited predictability arises due to their mutually canceling effects on ICI response. Thus, we developed DETACH, a computational method to identify a gene set whose expression pinpoints to a subset of melanoma patients where the CTL and ETL correlation is low. DETACH enhances CTL's prediction accuracy, outperforming existing signatures. DETACH signature genes activity also demonstrates a positive correlation with lymphocyte infiltration and the prevalence of reactive T cells in the tumor microenvironment (TME), advancing our understanding of the CTL cell state within the TME.

ASS1 metabolically contributes to the nuclear and cytosolic p53-mediated DNA damage response.

Downregulation of the urea cycle enzyme argininosuccinate synthase (ASS1) in multiple tumors is associated with a poor prognosis partly because of the metabolic diversion of cytosolic aspartate for pyrimidine synthesis, supporting proliferation and mutagenesis owing to nucleotide imbalance. Here, we find that prolonged loss of ASS1 promotes DNA damage in colon cancer cells and fibroblasts from subjects with citrullinemia type I. Following acute induction of DNA damage with doxorubicin, ASS1 expression is elevated in the cytosol and the nucleus with at least a partial dependency on p53; ASS1 metabolically restrains cell cycle progression in the cytosol by restricting nucleotide synthesis. In the nucleus, ASS1 and ASL generate fumarate for the succination of SMARCC1, destabilizing the chromatin-remodeling complex SMARCC1-SNF5 to decrease gene transcription, specifically in a subset of the p53-regulated cell cycle genes. Thus, following DNA damage, ASS1 is part of the p53 network that pauses cell cycle progression, enabling genome maintenance and survival. Loss of ASS1 contributes to DNA damage and promotes cell cycle progression, likely contributing to cancer mutagenesis and, hence, adaptability potential.

LORIS robustly predicts patient outcomes with immune checkpoint blockade therapy using common clinical, pathologic and genomic features.

Despite the revolutionary impact of immune checkpoint blockade (ICB) in cancer treatment, accurately predicting patient responses remains challenging. Here, we analyzed a large dataset of 2,881 ICB-treated and 841 non-ICB-treated patients across 18 solid tumor types, encompassing a wide range of clinical, pathologic and genomic features. We developed a clinical score called LORIS (logistic regression-based immunotherapy-response score) using a six-feature logistic regression model. LORIS outperforms previous signatures in predicting ICB response and identifying responsive patients even with low tumor mutational burden or programmed cell death 1 ligand 1 expression. LORIS consistently predicts patient objective response and short-term and long-term survival across most cancer types. Moreover, LORIS showcases a near-monotonic relationship with ICB response probability and patient survival, enabling precise patient stratification. As an accurate, interpretable method using a few readily measurable features, LORIS may help improve clinical decision-making in precision medicine to maximize patient benefit. LORIS is available as an online tool at https://loris.ccr.cancer.gov/ .

Prediction of DNA methylation-based tumor types from histopathology in central nervous system tumors with deep learning.

Precision in the diagnosis of diverse central nervous system (CNS) tumor types is crucial for optimal treatment. DNA methylation profiles, which capture the methylation status of thousands of individual CpG sites, are state-of-the-art data-driven means to enhance diagnostic accuracy but are also time consuming and not widely available. Here, to address these limitations, we developed Deep lEarning from histoPathoLOgy and methYlation (DEPLOY), a deep learning model that classifies CNS tumors to ten major categories from histopathology. DEPLOY integrates three distinct components: the first classifies CNS tumors directly from slide images ('direct model'), the second initially generates predictions for DNA methylation beta values, which are subsequently used for tumor classification ('indirect model'), and the third classifies tumor types directly from routinely available patient demographics. First, we find that DEPLOY accurately predicts beta values from histopathology images. Second, using a ten-class model trained on an internal dataset of 1,796 patients, we predict the tumor categories in three independent external test datasets including 2,156 patients, achieving an overall accuracy of 95% and balanced accuracy of 91% on samples that are predicted with high confidence. These results showcase the potential future use of DEPLOY to assist pathologists in diagnosing CNS tumors within a clinically relevant short time frame.

PERCEPTION predicts patient response and resistance to treatment using single-cell transcriptomics of their tumors.

Tailoring optimal treatment for individual cancer patients remains a significant challenge. To address this issue, we developed PERCEPTION (PERsonalized Single-Cell Expression-Based Planning for Treatments In ONcology), a precision oncology computational pipeline. Our approach uses publicly available matched bulk and single-cell (sc) expression profiles from large-scale cell-line drug screens. These profiles help build treatment response models based on patients' sc-tumor transcriptomics. PERCEPTION demonstrates success in predicting responses to targeted therapies in cultured and patient-tumor-derived primary cells, as well as in two clinical trials for multiple myeloma and breast cancer. It also captures the resistance development in patients with lung cancer treated with tyrosine kinase inhibitors. PERCEPTION outperforms published state-of-the-art sc-based and bulk-based predictors in all clinical cohorts. PERCEPTION is accessible at https://github.com/ruppinlab/PERCEPTION . Our work, showcasing patient stratification using sc-expression profiles of their tumors, will encourage the adoption of sc-omics profiling in clinical settings, enhancing precision oncology tools based on sc-omics.

Outcome differences by sex in oncology clinical trials.

Identifying sex differences in outcomes and toxicity between males and females in oncology clinical trials is important and has also been mandated by National Institutes of Health policies. Here we analyze the Trialtrove database, finding that, strikingly, only 472/89,221 oncology clinical trials (0.5%) had curated post-treatment sex comparisons. Among 288 trials with comparisons of survival, outcome, or response, 16% report males having statistically significant better survival outcome or response, while 42% reported significantly better survival outcome or response for females. The strongest differences are in trials of EGFR inhibitors in lung cancer and rituximab in non-Hodgkin's lymphoma (both favoring females). Among 44 trials with side effect comparisons, more trials report significantly lesser side effects in males (N = 22) than in females (N = 13). Thus, while statistical comparisons between sexes in oncology trials are rarely reported, important differences in outcome and toxicity exist. These considerable outcome and toxicity differences highlight the need for reporting sex differences more thoroughly going forward.

Programming a Ferroptosis-to-Apoptosis Transition Landscape Revealed Ferroptosis Biomarkers and Repressors for Cancer Therapy.

Ferroptosis and apoptosis are key cell-death pathways implicated in several human diseases including cancer. Ferroptosis is driven by iron-dependent lipid peroxidation and currently has no characteristic biomarkers or gene signatures. Here a continuous phenotypic gradient between ferroptosis and apoptosis coupled to transcriptomic and metabolomic landscapes is established. The gradual ferroptosis-to-apoptosis transcriptomic landscape is used to generate a unique, unbiased transcriptomic predictor, the Gradient Gene Set (GGS), which classified ferroptosis and apoptosis with high accuracy. Further GGS optimization using multiple ferroptotic and apoptotic datasets revealed highly specific ferroptosis biomarkers, which are robustly validated in vitro and in vivo. A subset of the GGS is associated with poor prognosis in breast cancer patients and PDXs and contains different ferroptosis repressors. Depletion of one representative, PDGFA-assaociated protein 1(PDAP1), is found to suppress basal-like breast tumor growth in a mouse model. Omics and mechanistic studies revealed that ferroptosis is associated with enhanced lysosomal function, glutaminolysis, and the tricarboxylic acid (TCA) cycle, while its transition into apoptosis is attributed to enhanced endoplasmic reticulum(ER)-stress and phosphatidylethanolamine (PE)-to-phosphatidylcholine (PC) metabolic shift. Collectively, this study highlights molecular mechanisms underlying ferroptosis execution, identified a highly predictive ferroptosis gene signature with prognostic value, ferroptosis versus apoptosis biomarkers, and ferroptosis repressors for breast cancer therapy.

Transcriptomic Profiling of Plasma Extracellular Vesicles Enables Reliable Annotation of the Cancer-Specific Transcriptome and Molecular Subtype.

Longitudinal monitoring of patients with advanced cancers is crucial to evaluate both disease burden and treatment response. Current liquid biopsy approaches mostly rely on the detection of DNA-based biomarkers. However, plasma RNA analysis can unleash tremendous opportunities for tumor state interrogation and molecular subtyping. Through the application of deep learning algorithms to the deconvolved transcriptomes of RNA within plasma extracellular vesicles (evRNA), we successfully predicted consensus molecular subtypes in patients with metastatic colorectal cancer. Analysis of plasma evRNA also enabled monitoring of changes in transcriptomic subtype under treatment selection pressure and identification of molecular pathways associated with recurrence. This approach also revealed expressed gene fusions and neoepitopes from evRNA. These results demonstrate the feasibility of using transcriptomic-based liquid biopsy platforms for precision oncology approaches, spanning from the longitudinal monitoring of tumor subtype changes to the identification of expressed fusions and neoantigens as cancer-specific therapeutic targets, sans the need for tissue-based sampling. SIGNIFICANCE: The development of an approach to interrogate molecular subtypes, cancer-associated pathways, and differentially expressed genes through RNA sequencing of plasma extracellular vesicles lays the foundation for liquid biopsy-based longitudinal monitoring of patient tumor transcriptomes.

DNA and RNA base editors can correct the majority of pathogenic single nucleotide variants.

The majority of human genetic diseases are caused by single nucleotide variants (SNVs) in the genome sequence. Excitingly, new genomic techniques known as base editing have opened efficient pathways to correct erroneous nucleotides. Due to reliance on deaminases, which have the capability to convert A to I(G) and C to U, the direct applicability of base editing might seem constrained in terms of the range of mutations that can be reverted. In this evaluation, we assess the potential of DNA and RNA base editing methods for treating human genetic diseases. Our findings indicate that 62% of pathogenic SNVs found within genes can be amended by base editing; 30% are G>A and T>C SNVs that can be corrected by DNA base editing, and most of them by RNA base editing as well, and 29% are C>T and A>G SNVs that can be corrected by DNA base editing directed to the complementary strand. For each, we also present several factors that affect applicability such as bystander and off-target occurrences. For cases where editing the mismatched nucleotide is not feasible, we introduce an approach that calculates the optimal substitution of the deleterious amino acid with a new amino acid, further expanding the scope of applicability. As personalized therapy is rapidly advancing, our demonstration that most SNVs can be treated by base editing is of high importance. The data provided will serve as a comprehensive resource for those seeking to design therapeutic base editors and study their potential in curing genetic diseases.

Chromosome 7 to the rescue: overcoming chromosome 10 loss in gliomas.

The co-occurrence of chromosome 10 loss and chromosome 7 gain in gliomas is the most frequent loss-gain co-aneuploidy pair in human cancers, a phenomenon that has been investigated without resolution since the late 1980s. Expanding beyond previous gene-centric studies, we investigate the co-occurrence in a genome-wide manner taking an evolutionary perspective. First, by mining large tumor aneuploidy data, we predict that the more likely order is 10 loss followed by 7 gain. Second, by analyzing extensive genomic and transcriptomic data from both patients and cell lines, we find that this co-occurrence can be explained by functional rescue interactions that are highly enriched on 7, which can possibly compensate for any detrimental consequences arising from the loss of 10. Finally, by analyzing transcriptomic data from normal, non-cancerous, human brain tissues, we provide a plausible reason why this co-occurrence happens preferentially in cancers originating in certain regions of the brain.

A systematic analysis of the landscape of synthetic lethality-driven precision oncology.

BACKGROUND: Synthetic lethality (SL) denotes a genetic interaction between two genes whose co-inactivation is detrimental to cells. Because more than 25 years have passed since SL was proposed as a promising way to selectively target cancer vulnerabilities, it is timely to comprehensively assess its impact so far and discuss its future. METHODS: We systematically analyzed the literature and clinical trial data from the PubMed and Trialtrove databases to portray the preclinical and clinical landscape of SL oncology. FINDINGS: We identified 235 preclinically validated SL pairs and found 1,207 pertinent clinical trials, and the number keeps increasing over time. About one-third of these SL clinical trials go beyond the typically studied DNA damage response (DDR) pathway, testifying to the recently broadening scope of SL applications in clinical oncology. We find that SL oncology trials have a greater success rate than non-SL-based trials. However, about 75% of the preclinically validated SL interactions have not yet been tested in clinical trials. CONCLUSIONS: Dissecting the recent efforts harnessing SL to identify predictive biomarkers, novel therapeutic targets, and effective combination therapy, our systematic analysis reinforces the hope that SL may serve as a key driver of precision oncology going forward. FUNDING: Funded by the Samsung Research Funding & Incubation Center of Samsung Electronics, the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Republic of Korea government (MSIT), the Kwanjeong Educational Foundation, the Intramural Research Program of the National Institutes of Health (NIH), National Cancer Institute (NCI), and Center for Cancer Research (CCR).