Bayesian Networks in the Management of Hospital Admissions: A Comparison between Explainable AI and Black Box AI during the Pandemic.

Artificial Intelligence (AI) and Machine Learning (ML) approaches that could learn from large data sources have been identified as useful tools to support clinicians in their decisional process; AI and ML implementations have had a rapid acceleration during the recent COVID-19 pandemic. However, many ML classifiers are "black box" to the final user, since their underlying reasoning process is often obscure. Additionally, the performance of such models suffers from poor generalization ability in the presence of dataset shifts. Here, we present a comparison between an explainable-by-design ("white box") model (Bayesian Network (BN)) versus a black box model (Random Forest), both studied with the aim of supporting clinicians of Policlinico San Matteo University Hospital in Pavia (Italy) during the triage of COVID-19 patients. Our aim is to evaluate whether the BN predictive performances are comparable with those of a widely used but less explainable ML model such as Random Forest and to test the generalization ability of the ML models across different waves of the pandemic.

Critical assessment of variant prioritization methods for rare disease diagnosis within the rare genomes project.

BACKGROUND: A major obstacle faced by families with rare diseases is obtaining a genetic diagnosis. The average "diagnostic odyssey" lasts over five years and causal variants are identified in under 50%, even when capturing variants genome-wide. To aid in the interpretation and prioritization of the vast number of variants detected, computational methods are proliferating. Knowing which tools are most effective remains unclear. To evaluate the performance of computational methods, and to encourage innovation in method development, we designed a Critical Assessment of Genome Interpretation (CAGI) community challenge to place variant prioritization models head-to-head in a real-life clinical diagnostic setting. METHODS: We utilized genome sequencing (GS) data from families sequenced in the Rare Genomes Project (RGP), a direct-to-participant research study on the utility of GS for rare disease diagnosis and gene discovery. Challenge predictors were provided with a dataset of variant calls and phenotype terms from 175 RGP individuals (65 families), including 35 solved training set families with causal variants specified, and 30 unlabeled test set families (14 solved, 16 unsolved). We tasked teams to identify causal variants in as many families as possible. Predictors submitted variant predictions with estimated probability of causal relationship (EPCR) values. Model performance was determined by two metrics, a weighted score based on the rank position of causal variants, and the maximum F-measure, based on precision and recall of causal variants across all EPCR values. RESULTS: Sixteen teams submitted predictions from 52 models, some with manual review incorporated. Top performers recalled causal variants in up to 13 of 14 solved families within the top 5 ranked variants. Newly discovered diagnostic variants were returned to two previously unsolved families following confirmatory RNA sequencing, and two novel disease gene candidates were entered into Matchmaker Exchange. In one example, RNA sequencing demonstrated aberrant splicing due to a deep intronic indel in ASNS, identified in trans with a frameshift variant in an unsolved proband with phenotypes consistent with asparagine synthetase deficiency. CONCLUSIONS: Model methodology and performance was highly variable. Models weighing call quality, allele frequency, predicted deleteriousness, segregation, and phenotype were effective in identifying causal variants, and models open to phenotype expansion and non-coding variants were able to capture more difficult diagnoses and discover new diagnoses. Overall, computational models can significantly aid variant prioritization. For use in diagnostics, detailed review and conservative assessment of prioritized variants against established criteria is needed.

Forecasting dominance of SARS-CoV-2 lineages by anomaly detection using deep AutoEncoders.

The coronavirus disease of 2019 (COVID-19) pandemic is characterized by sequential emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variants and lineages outcompeting previously circulating ones because of, among other factors, increased transmissibility and immune escape1-3. We devised an unsupervised deep learning AutoEncoder for viral genomes anomaly detection to predict future dominant lineages (FDLs), i.e., lineages or sublineages comprising ≥10% of viral sequences added to the GISAID database on a given week4. The algorithm was trained and validated by assembling global and country-specific data sets from 16,187,950 Spike protein sequences sampled between December 24th, 2019, and November 8th, 2023. The AutoEncoder flags low frequency FDLs (0.01% - 3%), with median lead times of 4-16 weeks. Over time, positive predictive values oscillate, decreasing linearly with the number of unique sequences per data set, showing average performance up to 30 times better than baseline approaches. The B.1.617.2 vaccine reference strain was flagged as FDL when its frequency was only 0.01%, more than one year earlier of being considered for an updated COVID-19 vaccine. Our AutoEncoder, applicable in principle to any pathogen, also pinpoints specific mutations potentially linked to increased fitness, and may provide significant insights for the optimization of public health pre-emptive intervention strategies.

Critical assessment of variant prioritization methods for rare disease diagnosis within the Rare Genomes Project.

Background: A major obstacle faced by rare disease families is obtaining a genetic diagnosis. The average "diagnostic odyssey" lasts over five years, and causal variants are identified in under 50%. The Rare Genomes Project (RGP) is a direct-to-participant research study on the utility of genome sequencing (GS) for diagnosis and gene discovery. Families are consented for sharing of sequence and phenotype data with researchers, allowing development of a Critical Assessment of Genome Interpretation (CAGI) community challenge, placing variant prioritization models head-to-head in a real-life clinical diagnostic setting. Methods: Predictors were provided a dataset of phenotype terms and variant calls from GS of 175 RGP individuals (65 families), including 35 solved training set families, with causal variants specified, and 30 test set families (14 solved, 16 unsolved). The challenge tasked teams with identifying the causal variants in as many test set families as possible. Ranked variant predictions were submitted with estimated probability of causal relationship (EPCR) values. Model performance was determined by two metrics, a weighted score based on rank position of true positive causal variants and maximum F-measure, based on precision and recall of causal variants across EPCR thresholds. Results: Sixteen teams submitted predictions from 52 models, some with manual review incorporated. Top performing teams recalled the causal variants in up to 13 of 14 solved families by prioritizing high quality variant calls that were rare, predicted deleterious, segregating correctly, and consistent with reported phenotype. In unsolved families, newly discovered diagnostic variants were returned to two families following confirmatory RNA sequencing, and two prioritized novel disease gene candidates were entered into Matchmaker Exchange. In one example, RNA sequencing demonstrated aberrant splicing due to a deep intronic indel in ASNS, identified in trans with a frameshift variant, in an unsolved proband with phenotype overlap with asparagine synthetase deficiency. Conclusions: By objective assessment of variant predictions, we provide insights into current state-of-the-art algorithms and platforms for genome sequencing analysis for rare disease diagnosis and explore areas for future optimization. Identification of diagnostic variants in unsolved families promotes synergy between researchers with clinical and computational expertise as a means of advancing the field of clinical genome interpretation.

Performance of an AI algorithm during the different phases of the COVID pandemics: what can we learn from the AI and vice versa.

Background: Artificial intelligence (AI) has proved to be of great value in diagnosing and managing Sars-Cov-2 infection. ALFABETO (ALL-FAster-BEtter-TOgether) is a tool created to support healthcare professionals in the triage, mainly in optimizing hospital admissions. Methods: The AI was trained during the pandemic's "first wave" (February-April 2020). Our aim was to assess the performance during the "third wave" of the pandemics (February-April 2021) and evaluate its evolution. The neural network proposed behavior (hospitalization vs home care) was compared with what was actually done. If there were discrepancies between ALFABETO's predictions and clinicians' decisions, the disease's progression was monitored. Clinical course was defined as "favorable/mild" if patients could be managed at home or in spoke centers and "unfavorable/severe" if patients need to be managed in a hub center. Results: ALFABETO showed accuracy of 76%, AUROC of 83%; specificity was 78% and recall 74%. ALFABETO also showed high precision (88%). 81 hospitalized patients were incorrectly predicted to be in "home care" class. Among those "home-cared" by the AI and "hospitalized" by the clinicians, 3 out of 4 misclassified patients (76.5%) showed a favorable/mild clinical course. ALFABETO's performance matched the reports in literature. Conclusions: The discrepancies mostly occurred when the AI predicted patients could stay at home but clinicians hospitalized them; these cases could be handled in spoke centers rather than hubs, and the discrepancies may aid clinicians in patient selection. The interaction between AI and human experience has the potential to improve both AI performance and our comprehension of pandemic management.

A synthetic dataset of liver disorder patients.

The data in this article include 10,000 synthetic patients with liver disorders, characterized by 70 different variables, including clinical features, and patient outcomes, such as hospital admission or surgery. Patient data are generated, simulating as close as possible real patient data, using a publicly available Bayesian network describing a casual model for liver disorders. By varying the network parameters, we also generated an additional set of 500 patients with characteristics that deviated from the initial patient population. We provide an overview of the synthetic data generation process and the associated scripts for generating the cohorts. This dataset can be useful for the machine learning models training and validation, especially under the effect of dataset shift between training and testing sets.

MALDI mass spectrometry imaging shows a gradual change in the proteome landscape during mouse ovarian folliculogenesis.

Our knowledge regarding the role proteins play in the mutual relationship among oocytes, surrounding follicle cells, stroma, and the vascular network inside the ovary is still poor and obtaining insights into this context would significantly aid our understanding of folliculogenesis. Here, we describe a spatial proteomics approach to characterize the proteome of individual follicles at different growth stages in a whole prepubertal 25-day-old mouse ovary. A total of 401 proteins were identified by nano-scale liquid chromatography-electrospray ionization-tandem mass spectrometry (nLC-ESI-MS/MS), 69 with a known function in ovary biology, as demonstrated by earlier proteomics studies. Enrichment analysis highlighted significant KEGG and Reactome pathways, with apoptosis, developmental biology, PI3K-Akt, epigenetic regulation of gene expression, and extracellular matrix organization being well represented. Then, correlating these data with the spatial information provided by matrix-assisted laser desorption/ionization mass spectrometry imaging (MALDI-MSI) on 276 follicles enabled the protein profiles of single follicle types to be mapped within their native context, highlighting 94 proteins that were detected throughout the secondary to the pre-ovulatory transition. Statistical analyses identified a group of 37 proteins that showed a gradual quantitative change during follicle differentiation, comprising 10 with a known role in follicle growth (NUMA1, TPM2), oocyte germinal vesicle-to-metaphase II transition (SFPQ, ACTBL, MARCS, NUCL), ovulation (GELS, CO1A2), and preimplantation development (TIF1B, KHDC3). The proteome landscape identified includes molecules of known function in the ovary, but also those whose specific role is emerging. Altogether, this work demonstrates the utility of performing spatial proteomics in the context of the ovary and offers sound bases for more in-depth investigations that aim to further unravel its spatial proteome.

Why did AI get this one wrong? - Tree-based explanations of machine learning model predictions.

Increasingly complex learning methods such as boosting, bagging and deep learning have made ML models more accurate, but harder to interpret and explain, culminating in black-box machine learning models. Model developers and users alike are often presented with a trade-off between performance and intelligibility, especially in high-stakes applications like medicine. In the present article we propose a novel methodological approach for generating explanations for the predictions of a generic machine learning model, given a specific instance for which the prediction has been made. The method, named AraucanaXAI, is based on surrogate, locally-fitted classification and regression trees that are used to provide post-hoc explanations of the prediction of a generic machine learning model. Advantages of the proposed XAI approach include superior fidelity to the original model, ability to deal with non-linear decision boundaries, and native support to both classification and regression problems. We provide a packaged, open-source implementation of the AraucanaXAI method and evaluate its behaviour in a number of different settings that are commonly encountered in medical applications of AI. These include potential disagreement between the model prediction and physician's expert opinion and low reliability of the prediction due to data scarcity.

Correlation between PD-L1 Expression of Non-Small Cell Lung Cancer and Data from IVIM-DWI Acquired during Magnetic Resonance of the Thorax: Preliminary Results.

This study aims to investigate the correlation between intravoxel incoherent motion diffusion-weighted imaging (IVIM-DWI) parameters in magnetic resonance imaging (MRI) and programmed death-ligand 1 (PD-L1) expression in non-small cell lung cancer (NSCLC). Twenty-one patients diagnosed with stage III NSCLC from April 2021 to April 2022 were included. The tumors were distinguished into two groups: no PD-L1 expression (<1%), and positive PD-L1 expression (≥1%). Conventional MRI and IVIM-DWI sequences were acquired with a 1.5-T system. Both fixed-size ROIs and freehand segmentations of the tumors were evaluated, and the data were analyzed through a software using four different algorithms. The diffusion (D), pseudodiffusion (D*), and perfusion fraction (pf) were obtained. The correlation between IVIM parameters and PD-L1 expression was studied with Pearson correlation coefficient. The Wilcoxon−Mann−Whitney test was used to study IVIM parameter distributions in the two groups. Twelve patients (57%) had PD-L1 ≥1%, and 9 (43%) <1%. There was a statistically significant correlation between D* values and PD-L1 expression in images analyzed with algorithm 0, for fixed-size ROIs (189.2 ± 65.709 µm²/s × 104 in no PD-L1 expression vs. 122.0 ± 31.306 µm²/s × 104 in positive PD-L1 expression, p = 0.008). The values obtained with algorithms 1, 2, and 3 were not significantly different between the groups. The IVIM-DWI MRI parameter D* can reflect PD-L1 expression in NSCLC.

Dynamic Prediction of Non-Neutral SARS-Cov-2 Variants Using Incremental Machine Learning.

In this work we show that Incremental Machine Learning can be used to predict the classification of emerging SARS-CoV-2 lineages, dynamically distinguishing between neutral variants and non-neutral ones, i.e. variants of interest or variants of concerns. Starting from the Spike protein primary sequences collected in the GISAID db, we have derived a set of k-mers features, i.e., aminoacid subsequences with fixed length k. We have then implemented a Logistic Regression Incremental Learner that was monthly tested on the variants collected since February 2020 until October 2021. The average value of balanced accuracy of the classifier is 0.72 ± 0.2, which increased to 0.78 ± 0.16 in the last 12 months. The alpha, beta, gamma, eta, kappa and delta variants were recognized as non-neutral variants with mean recall ∼90%. In summary, incremental learning proved to be a useful instrument for pandemic surveillance, given its capability to update the model on new data over time.

A machine learning approach based on ACMG/AMP guidelines for genomic variant classification and prioritization.

Genomic variant interpretation is a critical step of the diagnostic procedure, often supported by the application of tools that may predict the damaging impact of each variant or provide a guidelines-based classification. We propose the application of Machine Learning methodologies, in particular Penalized Logistic Regression, to support variant classification and prioritization. Our approach combines ACMG/AMP guidelines for germline variant interpretation as well as variant annotation features and provides a probabilistic score of pathogenicity, thus supporting the prioritization and classification of variants that would be interpreted as uncertain by the ACMG/AMP guidelines. We compared different approaches in terms of variant prioritization and classification on different datasets, showing that our data-driven approach is able to solve more variant of uncertain significance (VUS) cases in comparison with guidelines-based approaches and in silico prediction tools.

Evaluating pointwise reliability of machine learning prediction.

Interest in Machine Learning applications to tackle clinical and biological problems is increasing. This is driven by promising results reported in many research papers, the increasing number of AI-based software products, and by the general interest in Artificial Intelligence to solve complex problems. It is therefore of importance to improve the quality of machine learning output and add safeguards to support their adoption. In addition to regulatory and logistical strategies, a crucial aspect is to detect when a Machine Learning model is not able to generalize to new unseen instances, which may originate from a population distant to that of the training population or from an under-represented subpopulation. As a result, the prediction of the machine learning model for these instances may be often wrong, given that the model is applied outside its "reliable" space of work, leading to a decreasing trust of the final users, such as clinicians. For this reason, when a model is deployed in practice, it would be important to advise users when the model's predictions may be unreliable, especially in high-stakes applications, including those in healthcare. Yet, reliability assessment of each machine learning prediction is still poorly addressed. Here, we review approaches that can support the identification of unreliable predictions, we harmonize the notation and terminology of relevant concepts, and we highlight and extend possible interrelationships and overlap among concepts. We then demonstrate, on simulated and real data for ICU in-hospital death prediction, a possible integrative framework for the identification of reliable and unreliable predictions. To do so, our proposed approach implements two complementary principles, namely the density principle and the local fit principle. The density principle verifies that the instance we want to evaluate is similar to the training set. The local fit principle verifies that the trained model performs well on training subsets that are more similar to the instance under evaluation. Our work can contribute to consolidating work in machine learning especially in medicine.

Predicting emerging SARS-CoV-2 variants of concern through a One Class dynamic anomaly detection algorithm.

OBJECTIVES: The objective of this study is the implementation of an automatic procedure to weekly detect new SARS-CoV-2 variants and non-neutral variants (variants of concern (VOC) and variants of interest (VOI)). METHODS: We downloaded spike protein primary sequences from the public resource GISAID and we represented each sequence as k-mer counts. For each week since 1 July 2020, we evaluate if each sequence represents an anomaly based on a One Class support vector machine (SVM) classification algorithm trained on neutral protein sequences collected from February to June 2020. RESULTS: We assess the ability of the One Class classifier to detect known VOC and VOI, such as Alpha, Delta or Omicron, ahead of their official classification by health authorities. In median, the classifier predicts a non-neutral variant as outlier 10 weeks before the official date of designation as VOC/VOI. DISCUSSION: The identification of non-neutral variants during a pandemic usually relies on indicators available during time, such as changing population size of a variant. Automatic variant surveillance systems based on protein sequences can enhance the fast identification of variants of potential concern. CONCLUSION: Machine learning, and in particular One Class SVM classification, can support the detection of potentially VOC/VOI variants during an evolving pandemics.

Cytoplasmic movements of the early human embryo: imaging and artificial intelligence to predict blastocyst development.

RESEARCH QUESTION: Can artificial intelligence and advanced image analysis extract and harness novel information derived from cytoplasmic movements of the early human embryo to predict development to blastocyst? DESIGN: In a proof-of-principle study, 230 human preimplantation embryos were retrospectively assessed using an artificial neural network. After intracytoplasmic sperm injection, embryos underwent time-lapse monitoring for 44 h. For comparison, standard embryo assessment of each embryo by a single embryologist was carried out to predict development to blastocyst stage based on a single picture frame taken at 42 h of development. In the experimental approach, in embryos that developed to blastocyst or destined to arrest, cytoplasm movement velocity was recorded by time-lapse monitoring during the first 44 h of culture and analysed with a Particle Image Velocimetry algorithm to extract quantitative information. Three main artificial intelligence approaches, the k-Nearest Neighbour, the Long-Short Term Memory Neural Network and the hybrid ensemble classifier were used to classify the embryos. RESULTS: Blind operator assessment classified each embryo in terms of ability to develop to blastocyst, with 75.4% accuracy, 76.5% sensitivity, 74.3% specificity, 74.3% precision and 75.4% F1 score. Integration of results from artificial intelligence models with the blind operator classification, resulted in 82.6% accuracy, 79.4% sensitivity, 85.7% specificity, 84.4% precision and 81.8% F1 score. CONCLUSIONS: The present study suggests the possibility of predicting human blastocyst development at early cleavage stages by detection of cytoplasm movement velocity and artificial intelligence analysis. This indicates the importance of the dynamics of the cytoplasm as a novel and valuable source of data to assess embryo viability.

Integrated Multi-Omics Analyses in Oncology: A Review of Machine Learning Methods and Tools.

In recent years, high-throughput sequencing technologies provide unprecedented opportunity to depict cancer samples at multiple molecular levels. The integration and analysis of these multi-omics datasets is a crucial and critical step to gain actionable knowledge in a precision medicine framework. This paper explores recent data-driven methodologies that have been developed and applied to respond major challenges of stratified medicine in oncology, including patients' phenotyping, biomarker discovery, and drug repurposing. We systematically retrieved peer-reviewed journals published from 2014 to 2019, select and thoroughly describe the tools presenting the most promising innovations regarding the integration of heterogeneous data, the machine learning methodologies that successfully tackled the complexity of multi-omics data, and the frameworks to deliver actionable results for clinical practice. The review is organized according to the applied methods: Deep learning, Network-based methods, Clustering, Features Extraction, and Transformation, Factorization. We provide an overview of the tools available in each methodological group and underline the relationship among the different categories. Our analysis revealed how multi-omics datasets could be exploited to drive precision oncology, but also current limitations in the development of multi-omics data integration.

A Reliable Machine Learning Approach applied to Single-Cell Classification in Acute Myeloid Leukemia.

Machine Learning research applied to the medical field is increasing. However, few of the proposed approaches are actually deployed in clinical settings. One reason is that current methods may not be able to generalize on new unseen instances which differ from the training population, thus providing unreliable classifications. Approaches to measure classification reliability could be useful to assess whether to trust prediction on new cases. Here, we propose a new reliability measure based on the similarity of a new instance to the training set. In particular, we evaluate whether this example would be selected as informative by an instance selection method, in comparison with the available training set. We show that this method distinguishes reliable examples, for which we can trust the classifier's prediction, from unreliable ones, both on simulated data and in a real-case scenario, to distinguish tumor and normal cells in Acute Myeloid Leukemia patients.

CardioVAI: An automatic implementation of ACMG-AMP variant interpretation guidelines in the diagnosis of cardiovascular diseases.

Variant interpretation for the diagnosis of genetic diseases is a complex process. The American College of Medical Genetics and Genomics, with the Association for Molecular Pathology, have proposed a set of evidence-based guidelines to support variant pathogenicity assessment and reporting in Mendelian diseases. Cardiovascular disorders are a field of application of these guidelines, but practical implementation is challenging due to the genetic disease heterogeneity and the complexity of information sources that need to be integrated. Decision support systems able to automate variant interpretation in the light of specific disease domains are demanded. We implemented CardioVAI (Cardio Variant Interpreter), an automated system for guidelines based variant classification in cardiovascular-related genes. Different omics-resources were integrated to assess pathogenicity of every genomic variant in 72 cardiovascular diseases related genes. We validated our method on benchmark datasets of high-confident assessed variants, reaching pathogenicity and benignity concordance up to 83 and 97.08%, respectively. We compared CardioVAI to similar methods and analyzed the main differences in terms of guidelines implementation. We finally made available CardioVAI as a web resource (http://cardiovai.engenome.com/) that allows users to further specialize guidelines recommendations.