Critical assessment of variant prioritization methods for rare disease diagnosis within the rare genomes project.

BACKGROUND: A major obstacle faced by families with rare diseases is obtaining a genetic diagnosis. The average "diagnostic odyssey" lasts over five years and causal variants are identified in under 50%, even when capturing variants genome-wide. To aid in the interpretation and prioritization of the vast number of variants detected, computational methods are proliferating. Knowing which tools are most effective remains unclear. To evaluate the performance of computational methods, and to encourage innovation in method development, we designed a Critical Assessment of Genome Interpretation (CAGI) community challenge to place variant prioritization models head-to-head in a real-life clinical diagnostic setting. METHODS: We utilized genome sequencing (GS) data from families sequenced in the Rare Genomes Project (RGP), a direct-to-participant research study on the utility of GS for rare disease diagnosis and gene discovery. Challenge predictors were provided with a dataset of variant calls and phenotype terms from 175 RGP individuals (65 families), including 35 solved training set families with causal variants specified, and 30 unlabeled test set families (14 solved, 16 unsolved). We tasked teams to identify causal variants in as many families as possible. Predictors submitted variant predictions with estimated probability of causal relationship (EPCR) values. Model performance was determined by two metrics, a weighted score based on the rank position of causal variants, and the maximum F-measure, based on precision and recall of causal variants across all EPCR values. RESULTS: Sixteen teams submitted predictions from 52 models, some with manual review incorporated. Top performers recalled causal variants in up to 13 of 14 solved families within the top 5 ranked variants. Newly discovered diagnostic variants were returned to two previously unsolved families following confirmatory RNA sequencing, and two novel disease gene candidates were entered into Matchmaker Exchange. In one example, RNA sequencing demonstrated aberrant splicing due to a deep intronic indel in ASNS, identified in trans with a frameshift variant in an unsolved proband with phenotypes consistent with asparagine synthetase deficiency. CONCLUSIONS: Model methodology and performance was highly variable. Models weighing call quality, allele frequency, predicted deleteriousness, segregation, and phenotype were effective in identifying causal variants, and models open to phenotype expansion and non-coding variants were able to capture more difficult diagnoses and discover new diagnoses. Overall, computational models can significantly aid variant prioritization. For use in diagnostics, detailed review and conservative assessment of prioritized variants against established criteria is needed.

Critical assessment of variant prioritization methods for rare disease diagnosis within the Rare Genomes Project.

Background: A major obstacle faced by rare disease families is obtaining a genetic diagnosis. The average "diagnostic odyssey" lasts over five years, and causal variants are identified in under 50%. The Rare Genomes Project (RGP) is a direct-to-participant research study on the utility of genome sequencing (GS) for diagnosis and gene discovery. Families are consented for sharing of sequence and phenotype data with researchers, allowing development of a Critical Assessment of Genome Interpretation (CAGI) community challenge, placing variant prioritization models head-to-head in a real-life clinical diagnostic setting. Methods: Predictors were provided a dataset of phenotype terms and variant calls from GS of 175 RGP individuals (65 families), including 35 solved training set families, with causal variants specified, and 30 test set families (14 solved, 16 unsolved). The challenge tasked teams with identifying the causal variants in as many test set families as possible. Ranked variant predictions were submitted with estimated probability of causal relationship (EPCR) values. Model performance was determined by two metrics, a weighted score based on rank position of true positive causal variants and maximum F-measure, based on precision and recall of causal variants across EPCR thresholds. Results: Sixteen teams submitted predictions from 52 models, some with manual review incorporated. Top performing teams recalled the causal variants in up to 13 of 14 solved families by prioritizing high quality variant calls that were rare, predicted deleterious, segregating correctly, and consistent with reported phenotype. In unsolved families, newly discovered diagnostic variants were returned to two families following confirmatory RNA sequencing, and two prioritized novel disease gene candidates were entered into Matchmaker Exchange. In one example, RNA sequencing demonstrated aberrant splicing due to a deep intronic indel in ASNS, identified in trans with a frameshift variant, in an unsolved proband with phenotype overlap with asparagine synthetase deficiency. Conclusions: By objective assessment of variant predictions, we provide insights into current state-of-the-art algorithms and platforms for genome sequencing analysis for rare disease diagnosis and explore areas for future optimization. Identification of diagnostic variants in unsolved families promotes synergy between researchers with clinical and computational expertise as a means of advancing the field of clinical genome interpretation.

CAGI SickKids challenges: Assessment of phenotype and variant predictions derived from clinical and genomic data of children with undiagnosed diseases.

Whole-genome sequencing (WGS) holds great potential as a diagnostic test. However, the majority of patients currently undergoing WGS lack a molecular diagnosis, largely due to the vast number of undiscovered disease genes and our inability to assess the pathogenicity of most genomic variants. The CAGI SickKids challenges attempted to address this knowledge gap by assessing state-of-the-art methods for clinical phenotype prediction from genomes. CAGI4 and CAGI5 participants were provided with WGS data and clinical descriptions of 25 and 24 undiagnosed patients from the SickKids Genome Clinic Project, respectively. Predictors were asked to identify primary and secondary causal variants. In addition, for CAGI5, groups had to match each genome to one of three disorder categories (neurologic, ophthalmologic, and connective), and separately to each patient. The performance of matching genomes to categories was no better than random but two groups performed significantly better than chance in matching genomes to patients. Two of the ten variants proposed by two groups in CAGI4 were deemed to be diagnostic, and several proposed pathogenic variants in CAGI5 are good candidates for phenotype expansion. We discuss implications for improving in silico assessment of genomic variants and identifying new disease genes.

Assessment of Validity of a Blood-Based 3-Gene Signature Score for Progression and Diagnosis of Tuberculosis, Disease Severity, and Treatment Response.

Importance: The World Health Organization identified the need for a non-sputum-based triage test to identify those in need of further tuberculosis (TB) testing. Objective: To determine whether the 3-gene TB score can be a diagnostic tool throughout the course of TB disease, from latency to diagnosis to treatment response, and posttreatment residual inflammation. Design, Setting, and Participants: This nested case-control study analyzed the 3-gene TB score in 3 cohorts, each focusing on a different stage of TB disease: (1) the Adolescent Cohort Study profiled whole-blood samples from adolescents with latent Mycobacterium tuberculosis infection, some of which progressed to active TB (ATB), using RNA sequencing; (2) the Brazil Active Screen Study collected whole blood from an actively screened case-control cohort of adult inmates from 2 prisons in Mato Grosso do Sul, Brazil, for ATB from January 2016 to February 2016; and (3) the Catalysis Treatment Response Cohort (CTRC) identified culture-positive adults in primary health care clinics in Cape Town, South Africa, from 2005 to 2007 and collected whole blood for RNA sequencing from patients with ATB at diagnosis and weeks 1, 4, and 24. The CTRC patients also had positron emission tomography-computed tomography scans at diagnosis, week 4, and week 24. Analyses were performed from September 2017 to June 2018. Main Outcomes and Measures: A 3-gene messenger RNA expression score, measured by quantitative polymerase chain reaction or RNA sequencing, was evaluated for distinguishing the following: individuals who progressed to ATB from those who did not, individuals with ATB from those without, and individuals with slower treatment response during TB therapy. Results: Patients evaluated in this study included 144 adolescents from the Adolescent Cohort Study (aged 12-18 years; 96 female and 48 male), 81 adult prison inmates from the Brazil Active Screen Study (aged 20-72 years; 81 male), and 138 adult community members from the CTRC (aged 17-64 years; 81 female and 57 male). The 3-gene TB score identified progression from latent M tuberculosis infection to ATB 6 months prior to sputum conversion with 86% sensitivity and 84% specificity (area under the curve [AUC], 0.86; 95% CI, 0.77-0.96) and patients with ATB in the Brazil Active Screen Study cohort (AUC, 0.87; 95% CI, 0.78-0.95) and CTRC (AUC, 0.94; 95% CI, 0.88-0.99). It also identified CTRC patients with failed treatment at the end of treatment (AUC, 0.93; 95% CI, 0.83-1.00). Collectively, across all cohorts, the 3-gene TB score identified patients with ATB with 90% sensitivity, 70% specificity, and 99.3% negative predictive value at 4% prevalence. Conclusions and Relevance: Across 3 independent prospective cohorts, the 3-gene TB score approaches the World Health Organization target product profile benchmarks for non-sputum-based triage test with high negative predictive value. This gene expression diagnostic approach should be considered for further validation and future implementation.