RESUMO
Background: Computational approaches to support rare disease diagnosis are challenging to build, requiring the integration of complex data types such as ontologies, gene-to-phenotype associations, and cross-species data into variant and gene prioritisation algorithms (VGPAs). However, the performance of VGPAs has been difficult to measure and is impacted by many factors, for example, ontology structure, annotation completeness or changes to the underlying algorithm. Assertions of the capabilities of VGPAs are often not reproducible, in part because there is no standardised, empirical framework and openly available patient data to assess the efficacy of VGPAs - ultimately hindering the development of effective prioritisation tools. Results: In this paper, we present our benchmarking tool, PhEval, which aims to provide a standardised and empirical framework to evaluate phenotype-driven VGPAs. The inclusion of standardised test corpora and test corpus generation tools in the PhEval suite of tools allows open benchmarking and comparison of methods on standardised data sets. Conclusions: PhEval and the standardised test corpora solve the issues of patient data availability and experimental tooling configuration when benchmarking and comparing rare disease VGPAs. By providing standardised data on patient cohorts from real-world case-reports and controlling the configuration of evaluated VGPAs, PhEval enables transparent, portable, comparable and reproducible benchmarking of VGPAs. As these tools are often a key component of many rare disease diagnostic pipelines, a thorough and standardised method of assessment is essential for improving patient diagnosis and care.
RESUMO
Respiratory viruses' detection is vitally important in coping with pandemics such as COVID-19. Conventional methods typically require laboratory-based, high-cost equipment. An emerging alternative method is Near-Infrared (NIR) spectroscopy, especially a portable one of the type that has the benefits of low cost, portability, rapidity, ease of use, and mass deployability in both clinical and field settings. One obstacle to its effective application lies in its common limitations, which include relatively low specificity and general quality. Characteristically, the spectra curves show an interweaving feature for the virus-present and virus-absent samples. This then provokes the idea of using machine learning methods to overcome the difficulty. While a subsequent obstacle coincides with the fact that a direct deployment of the machine learning approaches leads to inadequate accuracy of the modelling results. This paper presents a data-driven study on the detection of two common respiratory viruses, the respiratory syncytial virus (RSV) and the Sendai virus (SEV), using a portable NIR spectrometer supported by a machine learning solution enhanced by an algorithm of variable selection via the Variable Importance in Projection (VIP) scores and its Quantile value, along with variable truncation processing, to overcome the obstacles to a certain extent. We conducted extensive experiments with the aid of the specifically developed algorithm of variable selection, using a total of four datasets, achieving classification accuracy of: (1) 0.88, 0.94, and 0.93 for RSV, SEV, and RSV + SEV, respectively, averaged over multiple runs, for the neural network modelling of taking in turn 3 sessions of data for training and the remaining one session of an 'unknown' dataset for testing. (2) the average accuracy of 0.94 (RSV), 0.97 (SEV), and 0.97 (RSV + SEV) for model validation and 0.90 (RSV), 0.93 (SEV), and 0.91 (RSV + SEV) for model testing, using two of the datasets for model training, one for model validation and the other for model testing. These results demonstrate the feasibility of using portable NIR spectroscopy coupled with machine learning to detect respiratory viruses with good accuracy, and the approach could be a viable solution for population screening.