Evaluation of the Diagnostic Accuracy of GPT-4 in Five Thousand Rare Disease Cases.

Large language models (LLM) have shown great promise in supporting differential diagnosis, but 23 available published studies on the diagnostic accuracy evaluated small cohorts (number of cases, 30-422, mean 104) and have evaluated LLM responses subjectively by manual curation (23/23 studies). The performance of LLMs for rare disease diagnosis has not been evaluated systematically. Here, we perform a rigorous and large-scale analysis of the performance of a GPT-4 in prioritizing candidate diagnoses, using the largest-ever cohort of rare disease patients. Our computational study used 5267 computational case reports from previously published data. Each case was formatted as a Global Alliance for Genomics and Health (GA4GH) phenopacket, in which clinical anomalies were represented as Human Phenotype Ontology (HPO) terms. We developed software to generate prompts from each phenopacket. Prompts were sent to Generative Pre-trained Transformer 4 (GPT-4), and the rank of the correct diagnosis, if present in the response, was recorded. The mean reciprocal rank of the correct diagnosis was 0.24 (with the reciprocal of the MRR corresponding to a rank of 4.2), and the correct diagnosis was placed in rank 1 in 19.2% of the cases, in the first 3 ranks in 28.6%, and in the first 10 ranks in 32.5%. Our study is the largest to be reported to date and provides a realistic estimate of the performance of GPT-4 in rare disease medicine.

A corpus of GA4GH Phenopackets: case-level phenotyping for genomic diagnostics and discovery.

The Global Alliance for Genomics and Health (GA4GH) Phenopacket Schema was released in 2022 and approved by ISO as a standard for sharing clinical and genomic information about an individual, including phenotypic descriptions, numerical measurements, genetic information, diagnoses, and treatments. A phenopacket can be used as an input file for software that supports phenotype-driven genomic diagnostics and for algorithms that facilitate patient classification and stratification for identifying new diseases and treatments. There has been a great need for a collection of phenopackets to test software pipelines and algorithms. Here, we present phenopacket-store. Version 0.1.12 of phenopacket-store includes 4916 phenopackets representing 277 Mendelian and chromosomal diseases associated with 236 genes, and 2872 unique pathogenic alleles curated from 605 different publications. This represents the first large-scale collection of case-level, standardized phenotypic information derived from case reports in the literature with detailed descriptions of the clinical data and will be useful for many purposes, including the development and testing of software for prioritizing genes and diseases in diagnostic genomics, machine learning analysis of clinical phenotype data, patient stratification, and genotype-phenotype correlations. This corpus also provides best-practice examples for curating literature-derived data using the GA4GH Phenopacket Schema.

Sarcopenia and adipose tissue evaluation by artificial intelligence predicts the overall survival after TAVI.

Sarcopenia is a serious systemic disease that reduces overall survival. TAVI is selectively performed in patients with severe aortic stenosis who are not indicated for open cardiac surgery due to severe polymorbidity. Artificial intelligence-assisted body composition assessment from available CT scans appears to be a simple tool to stratify these patients into low and high risk based on future estimates of all-cause mortality. Within our study, the segmentation of preprocedural CT scans at the level of the lumbar third vertebra in patients undergoing TAVI was performed using a neural network (AutoMATiCA). The obtained parameters (area and density of skeletal muscles and intramuscular, visceral, and subcutaneous adipose tissue) were analyzed using Cox univariate and multivariable models for continuous and categorical variables to assess the relation of selected variables with all-cause mortality. 866 patients were included (median(interquartile range)): age 79.7 (74.9-83.3) years; BMI 28.9 (25.9-32.6) kg/m2. Survival analysis was performed on all automatically obtained parameters of muscle and fat density and area. Skeletal muscle index (SMI in cm2/m2), visceral (VAT in HU) and subcutaneous adipose tissue (SAT in HU) density predicted the all-cause mortality in patients after TAVI expressed as hazard ratio (HR) with 95% confidence interval (CI): SMI HR 0.986, 95% CI (0.975-0.996); VAT 1.015 (1.002-1.028) and SAT 1.014 (1.004-1.023), all p < 0.05. Automatic body composition assessment can estimate higher all-cause mortality risk in patients after TAVI, which may be useful in preoperative clinical reasoning and stratification of patients.

Predicting the impact of rare variants on RNA splicing in CAGI6.

Variants which disrupt splicing are a frequent cause of rare disease that have been under-ascertained clinically. Accurate and efficient methods to predict a variant's impact on splicing are needed to interpret the growing number of variants of unknown significance (VUS) identified by exome and genome sequencing. Here, we present the results of the CAGI6 Splicing VUS challenge, which invited predictions of the splicing impact of 56 variants ascertained clinically and functionally validated to determine splicing impact. The performance of 12 prediction methods, along with SpliceAI and CADD, was compared on the 56 functionally validated variants. The maximum accuracy achieved was 82% from two different approaches, one weighting SpliceAI scores by minor allele frequency, and one applying the recently published Splicing Prediction Pipeline (SPiP). SPiP performed optimally in terms of sensitivity, while an ensemble method combining multiple prediction tools and information from databases exceeded all others for specificity. Several challenge methods equalled or exceeded the performance of SpliceAI, with ultimate choice of prediction method likely to depend on experimental or clinical aims. One quarter of the variants were incorrectly predicted by at least 50% of the methods, highlighting the need for further improvements to splicing prediction methods for successful clinical application.