Clustering of clinical symptoms using large language models reveals low diagnostic specificity of proposed alternatives to consensus mast cell activation syndrome criteria.

BACKGROUND: The rate of diagnosis of mast cell activation syndrome (MCAS) has increased since the disorder's original description as a mastocytosis-like phenotype. While a set of consortium MCAS criteria is well described and widely accepted, this increase occurs in the setting of a broader set of proposed alternative MCAS criteria. OBJECTIVE: Effective diagnostic criteria must minimize the range of unrelated diagnoses that can be erroneously classified as the condition of interest. We sought to determine if the symptoms associated with alternative MCAS criteria result in less concise or consistent diagnostic alternatives, reducing diagnostic specificity. METHODS: We used multiple large language models, including ChatGPT, Claude, and Gemini, to bootstrap the probabilities of diagnoses that are compatible with consortium or alternative MCAS criteria. We utilized diversity and network analysis to quantify diagnostic precision and specificity compared to control diagnostic criteria including systemic lupus erythematosus (SLE), Kawasaki disease, and migraines. RESULTS: Compared to consortium MCAS criteria, alternative MCAS criteria are associated with more variable (Shannon diversity 5.8 vs. 4.6, respectively; p-value=0.004) and less precise (mean Bray-Curtis similarity 0.07 vs 0.19, respectively; p-value=0.004) diagnoses. The diagnosis networks derived from consortium and alternative MCAS criteria had lower between-network similarity compared to the similarity between diagnosis networks derived from two distinct SLE criteria (cosine similarity 0.55 vs. 0.86, respectively; p-value=0.0022). CONCLUSION: Alternative MCAS criteria are associated with a distinct set of diagnoses compared to consortium MCAS criteria and have lower diagnostic consistency. This lack of specificity is pronounced in relation to multiple control criteria, raising the concern that alternative criteria could disproportionately contribute to MCAS overdiagnosis, to the exclusion of more appropriate diagnoses.

Evaluating large language models on medical, lay-language, and self-reported descriptions of genetic conditions.

Large language models (LLMs) are generating interest in medical settings. For example, LLMs can respond coherently to medical queries by providing plausible differential diagnoses based on clinical notes. However, there are many questions to explore, such as evaluating differences between open- and closed-source LLMs as well as LLM performance on queries from both medical and non-medical users. In this study, we assessed multiple LLMs, including Llama-2-chat, Vicuna, Medllama2, Bard/Gemini, Claude, ChatGPT3.5, and ChatGPT-4, as well as non-LLM approaches (Google search and Phenomizer) regarding their ability to identify genetic conditions from textbook-like clinician questions and their corresponding layperson translations related to 63 genetic conditions. For open-source LLMs, larger models were more accurate than smaller LLMs: 7b, 13b, and larger than 33b parameter models obtained accuracy ranges from 21%-49%, 41%-51%, and 54%-68%, respectively. Closed-source LLMs outperformed open-source LLMs, with ChatGPT-4 performing best (89%-90%). Three of 11 LLMs and Google search had significant performance gaps between clinician and layperson prompts. We also evaluated how in-context prompting and keyword removal affected open-source LLM performance. Models were provided with 2 types of in-context prompts: list-type prompts, which improved LLM performance, and definition-type prompts, which did not. We further analyzed removal of rare terms from descriptions, which decreased accuracy for 5 of 7 evaluated LLMs. Finally, we observed much lower performance with real individuals' descriptions; LLMs answered these questions with a maximum 21% accuracy.

Expanding the genetic and phenotypic landscape of replication factor C complex-related disorders: RFC4 deficiency is linked to a multisystemic disorder.

The precise regulation of DNA replication is vital for cellular division and genomic integrity. Central to this process is the replication factor C (RFC) complex, encompassing five subunits, which loads proliferating cell nuclear antigen onto DNA to facilitate the recruitment of replication and repair proteins and enhance DNA polymerase processivity. While RFC1's role in cerebellar ataxia, neuropathy, and vestibular areflexia syndrome (CANVAS) is known, the contributions of RFC2-5 subunits on human Mendelian disorders is largely unexplored. Our research links bi-allelic variants in RFC4, encoding a core RFC complex subunit, to an undiagnosed disorder characterized by incoordination and muscle weakness, hearing impairment, and decreased body weight. We discovered across nine affected individuals rare, conserved, predicted pathogenic variants in RFC4, all likely to disrupt the C-terminal domain indispensable for RFC complex formation. Analysis of a previously determined cryo-EM structure of RFC bound to proliferating cell nuclear antigen suggested that the variants disrupt interactions within RFC4 and/or destabilize the RFC complex. Cellular studies using RFC4-deficient HeLa cells and primary fibroblasts demonstrated decreased RFC4 protein, compromised stability of the other RFC complex subunits, and perturbed RFC complex formation. Additionally, functional studies of the RFC4 variants affirmed diminished RFC complex formation, and cell cycle studies suggested perturbation of DNA replication and cell cycle progression. Our integrated approach of combining in silico, structural, cellular, and functional analyses establishes compelling evidence that bi-allelic loss-of-function RFC4 variants contribute to the pathogenesis of this multisystemic disorder. These insights broaden our understanding of the RFC complex and its role in human health and disease.

Approximating facial expression effects on diagnostic accuracy via generative AI in medical genetics.

Artificial intelligence (AI) is increasingly used in genomics research and practice, and generative AI has garnered significant recent attention. In clinical applications of generative AI, aspects of the underlying datasets can impact results, and confounders should be studied and mitigated. One example involves the facial expressions of people with genetic conditions. Stereotypically, Williams (WS) and Angelman (AS) syndromes are associated with a "happy" demeanor, including a smiling expression. Clinical geneticists may be more likely to identify these conditions in images of smiling individuals. To study the impact of facial expression, we analyzed publicly available facial images of approximately 3500 individuals with genetic conditions. Using a deep learning (DL) image classifier, we found that WS and AS images with non-smiling expressions had significantly lower prediction probabilities for the correct syndrome labels than those with smiling expressions. This was not seen for 22q11.2 deletion and Noonan syndromes, which are not associated with a smiling expression. To further explore the effect of facial expressions, we computationally altered the facial expressions for these images. We trained HyperStyle, a GAN-inversion technique compatible with StyleGAN2, to determine the vector representations of our images. Then, following the concept of InterfaceGAN, we edited these vectors to recreate the original images in a phenotypically accurate way but with a different facial expression. Through online surveys and an eye-tracking experiment, we examined how altered facial expressions affect the performance of human experts. We overall found that facial expression is associated with diagnostic accuracy variably in different genetic conditions.

Recognition of Genetic Conditions After Learning With Images Created Using Generative Artificial Intelligence.

Importance: The lack of standardized genetics training in pediatrics residencies, along with a shortage of medical geneticists, necessitates innovative educational approaches. Objective: To compare pediatric resident recognition of Kabuki syndrome (KS) and Noonan syndrome (NS) after 1 of 4 educational interventions, including generative artificial intelligence (AI) methods. Design, Setting, and Participants: This comparative effectiveness study used generative AI to create images of children with KS and NS. From October 1, 2022, to February 28, 2023, US pediatric residents were provided images through a web-based survey to assess whether these images helped them recognize genetic conditions. Interventions: Participants categorized 20 images after exposure to 1 of 4 educational interventions (text-only descriptions, real images, and 2 types of images created by generative AI). Main Outcomes and Measures: Associations between educational interventions with accuracy and self-reported confidence. Results: Of 2515 contacted pediatric residents, 106 and 102 completed the KS and NS surveys, respectively. For KS, the sensitivity of text description was 48.5% (128 of 264), which was not significantly different from random guessing (odds ratio [OR], 0.94; 95% CI, 0.69-1.29; P = .71). Sensitivity was thus compared for real images vs random guessing (60.3% [188 of 312]; OR, 1.52; 95% CI, 1.15-2.00; P = .003) and 2 types of generative AI images vs random guessing (57.0% [212 of 372]; OR, 1.32; 95% CI, 1.04-1.69; P = .02 and 59.6% [193 of 324]; OR, 1.47; 95% CI, 1.12-1.94; P = .006) (denominators differ according to survey responses). The sensitivity of the NS text-only description was 65.3% (196 of 300). Compared with text-only, the sensitivity of the real images was 74.3% (205 of 276; OR, 1.53; 95% CI, 1.08-2.18; P = .02), and the sensitivity of the 2 types of images created by generative AI was 68.0% (204 of 300; OR, 1.13; 95% CI, 0.77-1.66; P = .54) and 71.0% (247 of 328; OR, 1.30; 95% CI, 0.92-1.83; P = .14). For specificity, no intervention was statistically different from text only. After the interventions, the number of participants who reported being unsure about important diagnostic facial features decreased from 56 (52.8%) to 5 (7.6%) for KS (P < .001) and 25 (24.5%) to 4 (4.7%) for NS (P < .001). There was a significant association between confidence level and sensitivity for real and generated images. Conclusions and Relevance: In this study, real and generated images helped participants recognize KS and NS; real images appeared most helpful. Generated images were noninferior to real images and could serve an adjunctive role, particularly for rare conditions.

Comparison of clinical geneticist and computer visual attention in assessing genetic conditions.

Artificial intelligence (AI) for facial diagnostics is increasingly used in the genetics clinic to evaluate patients with potential genetic conditions. Current approaches focus on one type of AI called Deep Learning (DL). While DL- based facial diagnostic platforms have a high accuracy rate for many conditions, less is understood about how this technology assesses and classifies (categorizes) images, and how this compares to humans. To compare human and computer attention, we performed eye-tracking analyses of geneticist clinicians (n = 22) and non-clinicians (n = 22) who viewed images of people with 10 different genetic conditions, as well as images of unaffected individuals. We calculated the Intersection-over-Union (IoU) and Kullback-Leibler divergence (KL) to compare the visual attentions of the two participant groups, and then the clinician group against the saliency maps of our deep learning classifier. We found that human visual attention differs greatly from DL model's saliency results. Averaging over all the test images, IoU and KL metric for the successful (accurate) clinician visual attentions versus the saliency maps were 0.15 and 11.15, respectively. Individuals also tend to have a specific pattern of image inspection, and clinicians demonstrate different visual attention patterns than non-clinicians (IoU and KL of clinicians versus non-clinicians were 0.47 and 2.73, respectively). This study shows that humans (at different levels of expertise) and a computer vision model examine images differently. Understanding these differences can improve the design and use of AI tools, and lead to more meaningful interactions between clinicians and AI technologies.

Analysis of large-language model versus human performance for genetics questions.

Large-language models like ChatGPT have recently received a great deal of attention. One area of interest pertains to how these models could be used in biomedical contexts, including related to human genetics. To assess one facet of this, we compared the performance of ChatGPT versus human respondents (13,642 human responses) in answering 85 multiple-choice questions about aspects of human genetics. Overall, ChatGPT did not perform significantly differently (p = 0.8327) than human respondents; ChatGPT was 68.2% accurate, compared to 66.6% accuracy for human respondents. Both ChatGPT and humans performed better on memorization-type questions versus critical thinking questions (p < 0.0001). When asked the same question multiple times, ChatGPT frequently provided different answers (16% of initial responses), including for both initially correct and incorrect answers, and gave plausible explanations for both correct and incorrect answers. ChatGPT's performance was impressive, but currently demonstrates significant shortcomings for clinical or other high-stakes use. Addressing these limitations will be important to guide adoption in real-life situations.

Generative Methods for Pediatric Genetics Education.

Artificial intelligence (AI) is used in an increasing number of areas, with recent interest in generative AI, such as using ChatGPT to generate programming code or DALL-E to make illustrations. We describe the use of generative AI in medical education. Specifically, we sought to determine whether generative AI could help train pediatric residents to better recognize genetic conditions. From publicly available images of individuals with genetic conditions, we used generative AI methods to create new images, which were checked for accuracy with an external classifier. We selected two conditions for study, Kabuki (KS) and Noonan (NS) syndromes, which are clinically important conditions that pediatricians may encounter. In this study, pediatric residents completed 208 surveys, where they each classified 20 images following exposure to one of 4 possible educational interventions, including with and without generative AI methods. Overall, we find that generative images perform similarly but appear to be slightly less helpful than real images. Most participants reported that images were useful, although real images were felt to be more helpful. We conclude that generative AI images may serve as an adjunctive educational tool, particularly for less familiar conditions, such as KS.

Human and computer attention in assessing genetic conditions.

Deep learning (DL) and other types of artificial intelligence (AI) are increasingly used in many biomedical areas, including genetics. One frequent use in medical genetics involves evaluating images of people with potential genetic conditions to help with diagnosis. A central question involves better understanding how AI classifiers assess images compared to humans. To explore this, we performed eye-tracking analyses of geneticist clinicians and non-clinicians. We compared results to DL-based saliency maps. We found that human visual attention when assessing images differs greatly from the parts of images weighted by the DL model. Further, individuals tend to have a specific pattern of image inspection, and clinicians demonstrate different visual attention patterns than non-clinicians.

Artificial intelligence and the impact on medical genetics.

Virtually all areas of biomedicine will be increasingly affected by applications of artificial intelligence (AI). We discuss how AI may affect fields of medical genetics, including both clinicians and laboratorians. In addition to reviewing the anticipated impact, we provide recommendations for ways in which these groups may want to evolve in light of the influence of AI. We also briefly discuss how educational and training programs can play a key role in preparing the future workforce given these anticipated changes.

Prediction of HLA genotypes from single-cell transcriptome data.

The human leukocyte antigen (HLA) locus plays a central role in adaptive immune function and has significant clinical implications for tissue transplant compatibility and allelic disease associations. Studies using bulk-cell RNA sequencing have demonstrated that HLA transcription may be regulated in an allele-specific manner and single-cell RNA sequencing (scRNA-seq) has the potential to better characterize these expression patterns. However, quantification of allele-specific expression (ASE) for HLA loci requires sample-specific reference genotyping due to extensive polymorphism. While genotype prediction from bulk RNA sequencing is well described, the feasibility of predicting HLA genotypes directly from single-cell data is unknown. Here we evaluate and expand upon several computational HLA genotyping tools by comparing predictions from human single-cell data to gold-standard, molecular genotyping. The highest 2-field accuracy averaged across all loci was 76% by arcasHLA and increased to 86% using a composite model of multiple genotyping tools. We also developed a highly accurate model (AUC 0.93) for predicting HLA-DRB345 copy number in order to improve genotyping accuracy of the HLA-DRB locus. Genotyping accuracy improved with read depth and was reproducible at repeat sampling. Using a metanalytic approach, we also show that HLA genotypes from PHLAT and OptiType can generate ASE ratios that are highly correlated (R2 = 0.8 and 0.94, respectively) with those derived from gold-standard genotyping.

The future of commercial genetic testing.

PURPOSE OF REVIEW: There are thousands of different clinical genetic tests currently available. Genetic testing and its applications continue to change rapidly for multiple reasons. These reasons include technological advances, accruing evidence about the impact and effects of testing, and many complex financial and regulatory factors. RECENT FINDINGS: This article considers a number of key issues and axes related to the current and future state of clinical genetic testing, including targeted versus broad testing, simple/Mendelian versus polygenic and multifactorial testing models, genetic testing for individuals with high suspicion of genetic conditions versus ascertainment through population screening, the rise of artificial intelligence in multiple aspects of the genetic testing process, and how developments such as rapid genetic testing and the growing availability of new therapies for genetic conditions may affect the field. SUMMARY: Genetic testing is expanding and evolving, including into new clinical applications. Developments in the field of genetics will likely result in genetic testing becoming increasingly in the purview of a very broad range of clinicians, including general paediatricians as well as paediatric subspecialists.

Age-related survey of clinical genetics literature and related resources.

Genetic conditions affect people throughout their entire lifespan; however, many clinical geneticists focus on the care of pediatric individuals. We analyzed the medical literature and related resources to help assess to what extent adults with genetic diseases were represented. This included general literature searches of PubMed (from 2001 through 2022), specific databases (the FDA orphan drug list and the Clinical Genomic Database) related to management and direct treatment of genetic conditions, and textbooks and morphology guides relevant to the diagnosis of genetic conditions. In the field of genetics/genomics in general, we overall detected a statistically significant emphasis on pediatric populations in the medical literature compared to select other disciplines and compared with the global population distribution. Clinical genetics articles about adults tended to focus on younger adult ages. In clinical genetics, management and treatments, as well as illustrations in several educational/diagnostic resources tended to focus on pediatric populations.

Whither social media and clinical genetics?

Social media has become ubiquitous in daily life, and increasingly impacts medical and scientific fields, including related to clinical genetics. Recent events have led to questions about the use of certain social media platforms, as well as social media more generally. We discuss these considerations, including alternative and emerging platforms that can offer forums for the clinical genetics and related communities.

Analysis of large-language model versus human performance for genetics questions.

Large-language models like ChatGPT have recently received a great deal of attention. To assess ChatGPT in the field of genetics, we compared its performance to human respondents in answering genetics questions (involving 13,636 responses) that had been posted on social media platforms starting in 2021. Overall, ChatGPT did not perform significantly differently than human respondents, but did significantly better on memorization-type questions versus critical thinking questions, frequently provided different answers when asked questions multiple times, and provided plausible explanations for both correct and incorrect answers.

Perspectives on the future of dysmorphology.

The field of clinical genetics and genomics continues to evolve. In the past few decades, milestones like the initial sequencing of the human genome, dramatic changes in sequencing technologies, and the introduction of artificial intelligence, have upended the field and offered fascinating new insights. Though difficult to predict the precise paths the field will follow, rapid change may continue to be inevitable. Within genetics, the practice of dysmorphology, as defined by pioneering geneticist David W. Smith in the 1960s as "the study of, or general subject of abnormal development of tissue form" has also been affected by technological advances as well as more general trends in biomedicine. To address possibilities, potential, and perils regarding the future of dysmorphology, a group of clinical geneticists, representing different career stages, areas of focus, and geographic regions, have contributed to this piece by providing insights about how the practice of dysmorphology will develop over the next several decades.

Investigating Determinants and Evaluating Deep Learning Training Approaches for Visual Acuity in Foveal Hypoplasia.

Purpose: To describe the relationships between foveal structure and visual function in a cohort of individuals with foveal hypoplasia (FH) and to estimate FH grade and visual acuity using a deep learning classifier. Design: Retrospective cohort study and experimental study. Participants: A total of 201 patients with FH were evaluated at the National Eye Institute from 2004 to 2018. Methods: Structural components of foveal OCT scans and corresponding clinical data were analyzed to assess their contributions to visual acuity. To automate FH scoring and visual acuity correlations, we evaluated the following 3 inputs for training a neural network predictor: (1) OCT scans, (2) OCT scans and metadata, and (3) real OCT scans and fake OCT scans created from a generative adversarial network. Main Outcome Measures: The relationships between visual acuity outcomes and determinants, such as foveal morphology, nystagmus, and refractive error. Results: The mean subject age was 24.4 years (range, 1-73 years; standard deviation = 18.25 years) at the time of OCT imaging. The mean best-corrected visual acuity (n = 398 eyes) was equivalent to a logarithm of the minimal angle of resolution (LogMAR) value of 0.75 (Snellen 20/115). Spherical equivalent refractive error (SER) ranged from -20.25 diopters (D) to +13.63 D with a median of +0.50 D. The presence of nystagmus and a high-LogMAR value showed a statistically significant relationship (P < 0.0001). The participants whose SER values were farther from plano demonstrated higher LogMAR values (n = 382 eyes). The proportion of patients with nystagmus increased with a higher FH grade. Variability in SER with grade 4 (range, -20.25 D to +13.00 D) compared with grade 1 (range, -8.88 D to +8.50 D) was statistically significant (P < 0.0001). Our neural network predictors reliably estimated the FH grading and visual acuity (correlation to true value > 0.85 and > 0.70, respectively) for a test cohort of 37 individuals (98 OCT scans). Training the predictor on real OCT scans with metadata and fake OCT scans improved the accuracy over the model trained on real OCT scans alone. Conclusions: Nystagmus and foveal anatomy impact visual outcomes in patients with FH, and computational algorithms reliably estimate FH grading and visual acuity.