Clinical Validation of Digitally Acquired Clinical Data and Machine Learning Models for Remote Measurement of Psoriasis and Psoriatic Arthritis: A Proof-of-Concept Study.

OBJECTIVE: Psoriatic disease remains underdiagnosed and undertreated. We developed and validated a suite of novel, sensor-based smartphone assessments (Psorcast app) that can be self-administered to measure cutaneous and musculoskeletal signs and symptoms of psoriatic disease. METHODS: Participants with psoriasis (PsO) or psoriatic arthritis (PsA) and healthy controls were recruited between June 5, 2019, and November 10, 2021, at 2 academic medical centers. Concordance and accuracy of digital measures and image-based machine learning models were compared to their analogous clinical measures from trained rheumatologists and dermatologists. RESULTS: Of 104 study participants, 51 (49%) were female and 53 (51%) were male, with a mean age of 42.3 years (SD 12.6). Seventy-nine (76%) participants had PsA, 16 (15.4%) had PsO, and 9 (8.7%) were healthy controls. Digital patient assessment of percent body surface area (BSA) affected with PsO demonstrated very strong concordance (Lin concordance correlation coefficient [CCC] 0.94 [95% CI 0.91-0.96]) with physician-assessed BSA. The in-clinic and remote target plaque physician global assessments showed fair-to-moderate concordance (CCCerythema 0.72 [0.59-0.85]; CCCinduration 0.72 [0.62-0.82]; CCCscaling 0.60 [0.48-0.72]). Machine learning models of hand photos taken by patients accurately identified clinically diagnosed nail PsO with an accuracy of 0.76. The Digital Jar Open assessment categorized physician-assessed upper extremity involvement, considering joint tenderness or enthesitis (AUROC 0.68 [0.47-0.85]). CONCLUSION: The Psorcast digital assessments achieved significant clinical validity, although they require further validation in larger cohorts before use in evidence-based medicine or clinical trial settings. The smartphone software and analysis pipelines from the Psorcast suite are open source and freely available.

Combining accurate tumor genome simulation with crowdsourcing to benchmark somatic structural variant detection.

BACKGROUND: The phenotypes of cancer cells are driven in part by somatic structural variants. Structural variants can initiate tumors, enhance their aggressiveness, and provide unique therapeutic opportunities. Whole-genome sequencing of tumors can allow exhaustive identification of the specific structural variants present in an individual cancer, facilitating both clinical diagnostics and the discovery of novel mutagenic mechanisms. A plethora of somatic structural variant detection algorithms have been created to enable these discoveries; however, there are no systematic benchmarks of them. Rigorous performance evaluation of somatic structural variant detection methods has been challenged by the lack of gold standards, extensive resource requirements, and difficulties arising from the need to share personal genomic information. RESULTS: To facilitate structural variant detection algorithm evaluations, we create a robust simulation framework for somatic structural variants by extending the BAMSurgeon algorithm. We then organize and enable a crowdsourced benchmarking within the ICGC-TCGA DREAM Somatic Mutation Calling Challenge (SMC-DNA). We report here the results of structural variant benchmarking on three different tumors, comprising 204 submissions from 15 teams. In addition to ranking methods, we identify characteristic error profiles of individual algorithms and general trends across them. Surprisingly, we find that ensembles of analysis pipelines do not always outperform the best individual method, indicating a need for new ways to aggregate somatic structural variant detection approaches. CONCLUSIONS: The synthetic tumors and somatic structural variant detection leaderboards remain available as a community benchmarking resource, and BAMSurgeon is available at https://github.com/adamewing/bamsurgeon .

Inferring causal molecular networks: empirical assessment through a community-based effort.

It remains unclear whether causal, rather than merely correlational, relationships in molecular networks can be inferred in complex biological settings. Here we describe the HPN-DREAM network inference challenge, which focused on learning causal influences in signaling networks. We used phosphoprotein data from cancer cell lines as well as in silico data from a nonlinear dynamical model. Using the phosphoprotein data, we scored more than 2,000 networks submitted by challenge participants. The networks spanned 32 biological contexts and were scored in terms of causal validity with respect to unseen interventional data. A number of approaches were effective, and incorporating known biology was generally advantageous. Additional sub-challenges considered time-course prediction and visualization. Our results suggest that learning causal relationships may be feasible in complex settings such as disease states. Furthermore, our scoring approach provides a practical way to empirically assess inferred molecular networks in a causal sense.

Combining tumor genome simulation with crowdsourcing to benchmark somatic single-nucleotide-variant detection.

The detection of somatic mutations from cancer genome sequences is key to understanding the genetic basis of disease progression, patient survival and response to therapy. Benchmarking is needed for tool assessment and improvement but is complicated by a lack of gold standards, by extensive resource requirements and by difficulties in sharing personal genomic information. To resolve these issues, we launched the ICGC-TCGA DREAM Somatic Mutation Calling Challenge, a crowdsourced benchmark of somatic mutation detection algorithms. Here we report the BAMSurgeon tool for simulating cancer genomes and the results of 248 analyses of three in silico tumors created with it. Different algorithms exhibit characteristic error profiles, and, intriguingly, false positives show a trinucleotide profile very similar to one found in human tumors. Although the three simulated tumors differ in sequence contamination (deviation from normal cell sequence) and in subclonality, an ensemble of pipelines outperforms the best individual pipeline in all cases. BAMSurgeon is available at https://github.com/adamewing/bamsurgeon/.

Enabling transparent and collaborative computational analysis of 12 tumor types within The Cancer Genome Atlas.

The Cancer Genome Atlas Pan-Cancer Analysis Working Group collaborated on the Synapse software platform to share and evolve data, results and methodologies while performing integrative analysis of molecular profiling data from 12 tumor types. The group's work serves as a pilot case study that provides (i) a template for future large collaborative studies; (ii) a system to support collaborative projects; and (iii) a public resource of highly curated data, results and automated systems for the evaluation of community-developed models.

Systematic analysis of challenge-driven improvements in molecular prognostic models for breast cancer.

Although molecular prognostics in breast cancer are among the most successful examples of translating genomic analysis to clinical applications, optimal approaches to breast cancer clinical risk prediction remain controversial. The Sage Bionetworks-DREAM Breast Cancer Prognosis Challenge (BCC) is a crowdsourced research study for breast cancer prognostic modeling using genome-scale data. The BCC provided a community of data analysts with a common platform for data access and blinded evaluation of model accuracy in predicting breast cancer survival on the basis of gene expression data, copy number data, and clinical covariates. This approach offered the opportunity to assess whether a crowdsourced community Challenge would generate models of breast cancer prognosis commensurate with or exceeding current best-in-class approaches. The BCC comprised multiple rounds of blinded evaluations on held-out portions of data on 1981 patients, resulting in more than 1400 models submitted as open source code. Participants then retrained their models on the full data set of 1981 samples and submitted up to five models for validation in a newly generated data set of 184 breast cancer patients. Analysis of the BCC results suggests that the best-performing modeling strategy outperformed previously reported methods in blinded evaluations; model performance was consistent across several independent evaluations; and aggregating community-developed models achieved performance on par with the best-performing individual models.