Comparative Evaluation of LLMs in Clinical Oncology.

BACKGROUND: As artificial intelligence (AI) tools become widely accessible, more patients and medical professionals will turn to them for medical information. Large language models (LLMs), a subset of AI, excel in natural language processing tasks and hold considerable promise for clinical use. Fields such as oncology, in which clinical decisions are highly dependent on a continuous influx of new clinical trial data and evolving guidelines, stand to gain immensely from such advancements. It is therefore of critical importance to benchmark these models and describe their performance characteristics to guide their safe application to clinical oncology. Accordingly, the primary objectives of this work were to conduct comprehensive evaluations of LLMs in the field of oncology and to identify and characterize strategies that medical professionals can use to bolster their confidence in a model's response. METHODS: This study tested five publicly available LLMs (LLaMA 1, PaLM 2, Claude-v1, generative pretrained transformer 3.5 [GPT-3.5], and GPT-4) on a comprehensive battery of 2044 oncology questions, including topics from medical oncology, surgical oncology, radiation oncology, medical statistics, medical physics, and cancer biology. Model prompts were presented independently of each other, and each prompt was repeated three times to assess output consistency. For each response, models were instructed to provide a self-appraised confidence score (from 1 to 4). Model performance was also evaluated against a novel validation set comprising 50 oncology questions curated to eliminate any risk of overlap with the data used to train the LLMs. RESULTS: There was significant heterogeneity in performance between models (analysis of variance, P<0.001). Relative to a human benchmark (2013 and 2014 examination results), GPT-4 was the only model to perform above the 50th percentile. Overall, model performance varied as a function of subject area across all models, with worse performance observed in clinical oncology subcategories compared with foundational topics (medical statistics, medical physics, and cancer biology). Within the clinical oncology subdomain, worse performance was observed in female-predominant malignancies. A combination of model selection, prompt repetition, and confidence self-appraisal allowed for the identification of high-performing subgroups of questions with observed accuracies of 81.7 and 81.1% in the Claude-v1 and GPT-4 models, respectively. Evaluation of the novel validation question set produced similar trends in model performance while also highlighting improved performance in newer, centrally hosted models (GPT-4 Turbo and Gemini 1.0 Ultra) and local models (Mixtral 8×7B and LLaMA 2). CONCLUSIONS: Of the models tested on a standardized set of oncology questions, GPT-4 was observed to have the highest performance. Although this performance is impressive, all LLMs continue to have clinically significant error rates, including examples of overconfidence and consistent inaccuracies. Given the enthusiasm to integrate these new implementations of AI into clinical practice, continued standardized evaluations of the strengths and limitations of these products will be critical to guide both patients and medical professionals. (Funded by the National Institutes of Health Clinical Center for Research and the Intramural Research Program of the National Institutes of Health; Z99 CA999999.).

Integrated analyses highlight interactions between the three-dimensional genome and DNA, RNA and epigenomic alterations in metastatic prostate cancer.

The impact of variations in the three-dimensional structure of the genome has been recognized, but solid cancer tissue studies are limited. Here, we performed integrated deep Hi-C sequencing with matched whole-genome sequencing, whole-genome bisulfite sequencing, 5-hydroxymethylcytosine (5hmC) sequencing and RNA sequencing across a cohort of 80 biopsy samples from patients with metastatic castration-resistant prostate cancer. Dramatic differences were present in gene expression, 5-methylcytosine/5hmC methylation and in structural variation versus mutation rate between A and B (open and closed) chromatin compartments. A subset of tumors exhibited depleted regional chromatin contacts at the AR locus, linked to extrachromosomal circular DNA (ecDNA) and worse response to AR signaling inhibitors. We also identified topological subtypes associated with stark differences in methylation structure, gene expression and prognosis. Our data suggested that DNA interactions may predispose to structural variant formation, exemplified by the recurrent TMPRSS2-ERG fusion. This comprehensive integrated sequencing effort represents a unique clinical tumor resource.

A prostate cancer gastrointestinal transcriptional phenotype may be associated with diminished response to AR-targeted therapy.

Background: Prostate cancer is a heterogenous disease, but once it becomes metastatic it eventually becomes treatment resistant. One mechanism of resistance to AR-targeting therapy is lineage plasticity, where the tumor undergoes a transformation to an AR-indifferent phenotype, most studied in the context of neuroendocrine prostate cancer (NEPC). However, activation of additional de- or trans-differentiation programs, including a gastrointestinal (GI) gene expression program, has been suggested as an alternative method of resistance. In this study, we explored the previously identified GI prostate cancer phenotype (PCa-GI) in a large cohort of metastatic castration-resistant prostate cancer (mCRPC) patient biopsy samples. Methods: We analyzed a dataset of 634 mCRPC samples with batch effect corrected gene expression data from the West Coast Dream Team (WCDT), the East Coast Dream Team (ECDT), the Fred Hutchinson Cancer Research Center (FHCRC) and the Weill Cornell Medical center (WCM). Survival data was available from the WCDT and ECDT cohorts. We calculated a gene expression GI score using the sum of z-scores of genes from a published set of PCa-GI-defining genes (N=38). Survival analysis was performed using the Kaplan-Meier method and Cox proportional hazards regression with endpoint overall survival from time of biopsy to death of any cause. Results: We found that the PCa-GI score had a bimodal distribution, identifying a distinct set of tumors with an activated GI expression pattern. Approximately 35% of samples were classified as PCa-GI high, which was concordant with prior reports. Liver metastases had the highest median score but after excluding liver samples, 29% of the remaining samples were still classified as PCa-GI high, suggesting a distinct phenotype not exclusive to liver metastases. No correlation was observed between GI score and proliferation, AR signaling, or NEPC scores. Furthermore, the PCa-GI score was not associated with genomic alterations in AR, FOXA1, RB1, TP53 or PTEN. However, tumors with MYC amplifications showed significantly higher GI scores (p=0.0001). Patients with PCa-GI tumors had a shorter survival (HR=1.5 [1.1-2.1], p=0.02), but this result was not significant after adjusting for the liver as metastatic site (HR=1.2 [0.82-1.7], p=0.35). Patients with PCa-GI low samples had a better outcome after androgen receptor signaling inhibitors (ASI, abiraterone or enzalutamide) than other therapies (HR=0.37 [0.22-0.61], p=0.0001) while the benefit of ASI was smaller and non-significant for PCa-GI high samples (HR=0.55 [0.29-1.1], p=0.07). A differential pathway analysis identified FOXA2 signaling to be upregulated PCa-GI high tumors (FDR = 3.7 × 10-13). Conclusions: The PCa-GI phenotype is prevalent in clinical mCRPC samples and may represent a distinct biological entity. PCa-GI tumors may respond less to ASI and could offer a strategy to study novel therapeutic targets.

Mechanistic Characterization of Cancer-associated Fibroblast Depletion via an Antibody-Drug Conjugate Targeting Fibroblast Activation Protein.

Cancer-associated fibroblasts (CAF) are a prominent cell type within the tumor microenvironment (TME) where they are known to promote cancer cell growth and survival, angiogenesis, drug resistance, and immunosuppression. The transmembrane prolyl protease fibroblast activation protein (FAP) is expressed on the surface of highly protumorigenic CAFs found in the stroma of nearly every cancer of epithelial origin. The widespread expression of FAP has made it an attractive therapeutic target based on the underlying hypothesis that eliminating protumorigenic CAFs will disrupt the cross-talk between components of TME resulting in cancer cell death and immune infiltration. This hypothesis, however, has never been directly proven. To eliminate FAP-expressing CAFs, we developed an antibody-drug conjugate using our anti-FAP antibody, huB12, coupled to a monomethyl auristatin E (huB12-MMAE) payload. After determining that huB12 was an effective targeting vector, we found that huB12-MMAE potently eliminated FAP-expressing cells as monocultures in vitro and significantly prolonged survival in vivo using a xenograft engineered to overexpress FAP. We investigated the effects of selectively eliminating CAFs using a layered, open microfluidic cell coculture platform, known as the Stacks. Analysis of mRNA and protein expression found that treatment with huB12-MMAE resulted in the increased secretion of the proinflammatory cytokines IL6 and IL8 by CAFs and an associated increase in expression of proinflammatory genes in cancer cells. We also detected increased secretion of CSF1, a cytokine involved in myeloid recruitment and differentiation. Our findings suggest that the mechanism of FAP-targeted therapies is through effects on the immune microenvironment and antitumor immune response. SIGNIFICANCE: The direct elimination of FAP-expressing CAFs disrupts the cross-talk with cancer cells leading to a proinflammatory response and alterations in the immune microenvironment and antitumor immune response.

A platform-independent AI tumor lineage and site (ATLAS) classifier.

Histopathologic diagnosis and classification of cancer plays a critical role in guiding treatment. Advances in next-generation sequencing have ushered in new complementary molecular frameworks. However, existing approaches do not independently assess both site-of-origin (e.g. prostate) and lineage (e.g. adenocarcinoma) and have minimal validation in metastatic disease, where classification is more difficult. Utilizing gradient-boosted machine learning, we developed ATLAS, a pair of separate AI Tumor Lineage and Site-of-origin models from RNA expression data on 8249 tumor samples. We assessed performance independently in 10,376 total tumor samples, including 1490 metastatic samples, achieving an accuracy of 91.4% for cancer site-of-origin and 97.1% for cancer lineage. High confidence predictions (encompassing the majority of cases) were accurate 98-99% of the time in both localized and remarkably even in metastatic samples. We also identified emergent properties of our lineage scores for tumor types on which the model was never trained (zero-shot learning). Adenocarcinoma/sarcoma lineage scores differentiated epithelioid from biphasic/sarcomatoid mesothelioma. Also, predicted lineage de-differentiation identified neuroendocrine/small cell tumors and was associated with poor outcomes across tumor types. Our platform-independent single-sample approach can be easily translated to existing RNA-seq platforms. ATLAS can complement and guide traditional histopathologic assessment in challenging situations and tumors of unknown primary.