Training calibration-based counterfactual explainers for deep learning models in medical image analysis.

The rapid adoption of artificial intelligence methods in healthcare is coupled with the critical need for techniques to rigorously introspect models and thereby ensure that they behave reliably. This has led to the design of explainable AI techniques that uncover the relationships between discernible data signatures and model predictions. In this context, counterfactual explanations that synthesize small, interpretable changes to a given query while producing desired changes in model predictions have become popular. This under-constrained, inverse problem is vulnerable to introducing irrelevant feature manipulations, particularly when the model's predictions are not well-calibrated. Hence, in this paper, we propose the TraCE (training calibration-based explainers) technique, which utilizes a novel uncertainty-based interval calibration strategy for reliably synthesizing counterfactuals. Given the wide-spread adoption of machine-learned solutions in radiology, our study focuses on deep models used for identifying anomalies in chest X-ray images. Using rigorous empirical studies, we demonstrate the superiority of TraCE explanations over several state-of-the-art baseline approaches, in terms of several widely adopted evaluation metrics. Our findings show that TraCE can be used to obtain a holistic understanding of deep models by enabling progressive exploration of decision boundaries, to detect shortcuts, and to infer relationships between patient attributes and disease severity.

MARGIN: Uncovering Deep Neural Networks Using Graph Signal Analysis.

Interpretability has emerged as a crucial aspect of building trust in machine learning systems, aimed at providing insights into the working of complex neural networks that are otherwise opaque to a user. There are a plethora of existing solutions addressing various aspects of interpretability ranging from identifying prototypical samples in a dataset to explaining image predictions or explaining mis-classifications. While all of these diverse techniques address seemingly different aspects of interpretability, we hypothesize that a large family of interepretability tasks are variants of the same central problem which is identifying relative change in a model's prediction. This paper introduces MARGIN, a simple yet general approach to address a large set of interpretability tasks MARGIN exploits ideas rooted in graph signal analysis to determine influential nodes in a graph, which are defined as those nodes that maximally describe a function defined on the graph. By carefully defining task-specific graphs and functions, we demonstrate that MARGIN outperforms existing approaches in a number of disparate interpretability challenges.

Preventing Failures by Dataset Shift Detection in Safety-Critical Graph Applications.

Dataset shift refers to the problem where the input data distribution may change over time (e.g., between training and test stages). Since this can be a critical bottleneck in several safety-critical applications such as healthcare, drug-discovery, etc., dataset shift detection has become an important research issue in machine learning. Though several existing efforts have focused on image/video data, applications with graph-structured data have not received sufficient attention. Therefore, in this paper, we investigate the problem of detecting shifts in graph structured data through the lens of statistical hypothesis testing. Specifically, we propose a practical two-sample test based approach for shift detection in large-scale graph structured data. Our approach is very flexible in that it is suitable for both undirected and directed graphs, and eliminates the need for equal sample sizes. Using empirical studies, we demonstrate the effectiveness of the proposed test in detecting dataset shifts. We also corroborate these findings using real-world datasets, characterized by directed graphs and a large number of nodes.

Critical analysis of the reporting quality of randomized trials within Endodontics using the Preferred Reporting Items for RAndomized Trials in Endodontics (PRIRATE) 2020 quality standard checklist.

AIM: To critically evaluate the reporting quality of a random sample of clinical trials published in Endodontics against the PRIRATE 2020 checklist and to analyse the association between the quality of reported trials and a variety of parameters. METHODOLOGY: Fifty randomized clinical trials relating to Endodontics were randomly selected from the PubMed database from 2015 to 2019 and evaluated by two independent reviewers. For each trial, a score of '1' was awarded when it fully reported each item in the PRIRATE guidelines whereas a score of '0' was awarded when an item was not reported; when the item was reported inadequately a score of '0.5' was awarded. For the items that were not relevant to the trial, 'Not Applicable (NA)' was given. Based on the interquartile range of the overall scores received, trials were categorized into 'Low' (0-58.4%), 'Moderate' (58.5-72.8%) and 'High' (72.9-100%) quality. The associations between characteristics and quality of clinical trials were investigated. Descriptive statistics, frequency analysis and percentage analyses were used to describe the data. To determine the significance of categorical data, the chi-square test was used. The probability value 0.05 was considered as the level of significance. RESULTS: Based on the overall scores, 13 (26%), 25(50%) and 12 (24%) of the reports of clinical trials were categorized as 'High', 'Moderate' and 'Low' quality, respectively. Three items (1b, 6d, 11e) were adequately reported in all manuscripts whilst two items (5k, 5m) were scored 'NA' in all the reports. The reports published from Europe had a significantly greater percentage of 'High'-quality scores, compared to Asia, Middle East, North America and South America (P = 0.0002). The 'High'-quality reports were published significantly more often in impact factor journals (P = 0.045). Reports of clinical trials published in journals that adhered to the CONSORT guidelines had significantly more 'High' scores compared to those that did not (P = 0.008). Clinical trials with protocols registered a priori had a significantly greater percentage of 'High' scores compared to the trials that were not registered in advance (P = 0.003). No significant difference occurred between the quality of clinical trials and the number of authors, journal (Endodontic specialty vs. Non-Endodontic specialty) or year of publication. CONCLUSIONS: Reports of randomized clinical trials published in the speciality of Endodontics had a substantial number of deficiencies. To create high-quality reports of clinical trials, authors should comply with the PRIRATE 2020 guidelines.

PRIASE 2021 guidelines for reporting animal studies in Endodontology: explanation and elaboration.

Laws and ethics require that before conducting human clinical trials, a new material, device or drug may have to undergo testing in animals in order to minimize health risks to humans, unless suitable supporting grandfather data already exist. The Preferred Reporting Items for Animal Studies in Endodontology (PRIASE) 2021 guidelines were developed exclusively for the specialty of Endodontology by integrating and adapting the ARRIVE (Animals in Research: Reporting In Vivo Experiments) guidelines and the Clinical and Laboratory Images in Publications (CLIP) principles using a validated consensus-based methodology. Implementation of the PRIASE 2021 guidelines will reduce potential sources of bias and thus improve the quality, accuracy, reproducibility, completeness and transparency of reports describing animal studies in Endodontology. The PRIASE 2021 guidelines consist of a checklist with 11 domains and 43 individual items and a flowchart. The aim of the current document is to provide an explanation for each item in the PRIASE 2021 checklist and flowchart and is supplemented with examples from the literature in order for readers to understand their significance and to provide usage guidance. A link to the PRIASE 2021 explanation and elaboration document and PRIASE 2021 checklist and flowchart is available on the Preferred Reporting Items for study Designs in Endodontology (PRIDE) website (http://pride-endodonticguidelines.org/priase/).

PRIASE 2021 guidelines for reporting animal studies in Endodontology: a consensus-based development.

Animal testing is crucial in situations when research on humans is not allowed because of unknown health risks and ethical concerns. The current project aims to develop reporting guidelines exclusively for animal studies in Endodontology, using an established consensus-based methodology. The guidelines have been named: Preferred Reporting Items for Animal Studies in Endodontology (PRIASE) 2021. Nine individuals (PD, VN, AK, PM, MN, JF, EP, JJ and SJ), including the project leaders (PD, VN) formed a steering committee. The steering committee developed a novel checklist by adapting and integrating their animal testing and peer review experience with the Animals in Research: Reporting In Vivo Experiments (ARRIVE) guidelines and also the Clinical and Laboratory Images in Publications (CLIP) principles. A PRIASE Delphi Group (PDG) and PRIASE Online Meeting Group (POMG) were also formed. Thirty-one PDG members participated in the online Delphi process and achieved consensus on the checklist items and flowchart that were used to formulate the PRIASE guidelines. The novel PRIASE 2021 guidelines were discussed with the POMG on 9 September 2020 via a Zoom online video call attended by 21 individuals from across the globe and seven steering committee members. Following the discussions, the guidelines were modified and then piloted by several authors whilst writing a manuscript involving research on animals. The PRIASE 2021 guidelines are a checklist consisting of 11 domains and 43 individual items together with a flowchart. The PRIASE 2021 guidelines are focused on improving the methodological principles, reproducibility and quality of animal studies in order to enhance their reliability as well as repeatability to estimate the effects of endodontic treatments and usefulness for guiding future clinical studies on humans.

Coverage-Based Designs Improve Sample Mining and Hyperparameter Optimization.

Sampling one or more effective solutions from large search spaces is a recurring idea in machine learning (ML), and sequential optimization has become a popular solution. Typical examples include data summarization, sample mining for predictive modeling, and hyperparameter optimization. Existing solutions attempt to adaptively trade off between global exploration and local exploitation, in which the initial exploratory sample is critical to their success. While discrepancy-based samples have become the de facto approach for exploration, results from computer graphics suggest that coverage-based designs, e.g., Poisson disk sampling, can be a superior alternative. In order to successfully adopt coverage-based sample designs to ML applications, which were originally developed for 2-D image analysis, we propose fundamental advances by constructing a parameterized family of designs with provably improved coverage characteristics and developing algorithms for effective sample synthesis. Using experiments in sample mining and hyperparameter optimization for supervised learning, we show that our approach consistently outperforms the existing exploratory sampling methods in both blind exploration and sequential search with Bayesian optimization.

Designing accurate emulators for scientific processes using calibration-driven deep models.

Predictive models that accurately emulate complex scientific processes can achieve speed-ups over numerical simulators or experiments and at the same time provide surrogates for improving the subsequent analysis. Consequently, there is a recent surge in utilizing modern machine learning methods to build data-driven emulators. In this work, we study an often overlooked, yet important, problem of choosing loss functions while designing such emulators. Popular choices such as the mean squared error or the mean absolute error are based on a symmetric noise assumption and can be unsuitable for heterogeneous data or asymmetric noise distributions. We propose Learn-by-Calibrating, a novel deep learning approach based on interval calibration for designing emulators that can effectively recover the inherent noise structure without any explicit priors. Using a large suite of use-cases, we demonstrate the efficacy of our approach in providing high-quality emulators, when compared to widely-adopted loss function choices, even in small-data regimes.

DDxNet: a deep learning model for automatic interpretation of electronic health records, electrocardiograms and electroencephalograms.

Effective patient care mandates rapid, yet accurate, diagnosis. With the abundance of non-invasive diagnostic measurements and electronic health records (EHR), manual interpretation for differential diagnosis has become time-consuming and challenging. This has led to wide-spread adoption of AI-powered tools, in pursuit of improving accuracy and efficiency of this process. While the unique challenges presented by each modality and clinical task demand customized tools, the cumbersome process of making problem-specific choices has triggered the critical need for a generic solution to enable rapid development of models in practice. In this spirit, we develop DDxNet, a deep architecture for time-varying clinical data, which we demonstrate to be well-suited for diagnostic tasks involving different modalities (ECG/EEG/EHR), required level of characterization (abnormality detection/phenotyping) and data fidelity (single-lead ECG/22-channel EEG). Using multiple benchmark problems, we show that DDxNet produces high-fidelity predictive models, and sometimes even provides significant performance gains over problem-specific solutions.

Preferred Reporting items for OBservational studies in Endodontics (PROBE) guidelines: a development protocol.

Observational studies have a significant role in establishing the prevalence and incidence of diseases in populations, as well as determining the benefits and risks associated with health-related interventions. Observational studies principally encompass cohort, case-control, case series and cross-sectional designs. Inadequate reporting of observational studies is likely to have a negative impact on decision-making in day-to-day clinical practice; however, no reporting guidelines have been published for observational studies in Endodontics. The aim of this project is to develop reporting guidelines for authors when creating manuscripts describing observational studies in the field of Endodontology in an attempt to improve the quality of publications. The new guidelines for observational studies will be named: 'Preferred Reporting items for OBservational studies in Endodontics (PROBE)'. A steering committee was formed by the project leaders (PD, VN) to develop the guidelines through a five-phase consensus process. The steering committee will review and adapt items from the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement and the Clinical and Laboratory Images in Publications (CLIP) principles, as well as identify new items that add value to Endodontics. The steering committee will create a PROBE Delphi Group (PDG), consisting of 30 members across the globe to review and refine the draft checklist items and flowchart. The items will be assessed by the PDG on a nine-point Likert scale for relevance and inclusion. The agreed items will then be discussed by a PROBE Face-to-Face meeting group (PFMG) made up of 20 individuals to further refine the guidelines. After receiving feedback from the PFMG, the steering committee will pilot and finalize the guidelines. The approved PROBE guidelines will be disseminated through publication in relevant journals, and be presented at national and international conferences. The PROBE checklist and flowchart will be available and downloadable from the Preferred Reporting Items for study Designs in Endodontics (PRIDE) website: www.pride-endodonticguidelines.org. The PROBE steering committee encourages clinicians, researchers, editors and peer reviewers to provide feedback on the PROBE guidelines to inform the steering group when the guidelines are updated.

Improved surrogates in inertial confinement fusion with manifold and cycle consistencies.

Neural networks have become the method of choice in surrogate modeling because of their ability to characterize arbitrary, high-dimensional functions in a data-driven fashion. This paper advocates for the training of surrogates that are 1) consistent with the physical manifold, resulting in physically meaningful predictions, and 2) cyclically consistent with a jointly trained inverse model; i.e., backmapping predictions through the inverse results in the original input parameters. We find that these two consistencies lead to surrogates that are superior in terms of predictive performance, are more resilient to sampling artifacts, and tend to be more data efficient. Using inertial confinement fusion (ICF) as a test-bed problem, we model a one-dimensional semianalytic numerical simulator and demonstrate the effectiveness of our approach.

PRIRATE 2020 guidelines for reporting randomized trials in Endodontics: explanation and elaboration.

Well-designed and properly conducted randomized clinical trials provide a true estimate of the effects of interventions and are acknowledged as the gold standard in terms of clinical study design. However, the quality of randomized clinical trials published in the field of Endodontics is suboptimal. The Preferred Reporting Items for RAndomized Trials in Endodontics (PRIRATE) 2020 guidelines were developed exclusively for Endodontics by integrating and adapting the CONsolidated Standards of Reporting Trials (CONSORT) statement and Clinical and Laboratory Images in Publications (CLIP) principles, through an accepted and well-documented consensus process. Full implementation of the PRIRATE 2020 guidelines will minimize potential sources of bias and thus enhance the standard of manuscripts submitted for publication, which will ultimately improve the reporting of randomized clinical trials in Endodontics. The aim of this document is to provide an explanation for each item in the PRIRATE 2020 checklist and flowchart with examples from the literature in order to help authors understand their rationale and significance. A link to this PRIRATE 2020 explanation and elaboration document is available on the Preferred Reporting Items for study Designs in Endodontology (PRIDE) website at http://www.pride-endodonticguidelines.org/prirate/.

PRIRATE 2020 guidelines for reporting randomized trials in Endodontics: a consensus-based development.

In evidence-based health care, randomized clinical trials provide the most accurate and reliable information on the effectiveness of an intervention. This project aimed to develop reporting guidelines, exclusively for randomized clinical trials in the dental specialty of Endodontology, using a well-documented, validated consensus-based methodology. The guidelines have been named Preferred Reporting Items for RAndomized Trials in Endodontics (PRIRATE) 2020. A total of eight individuals (PD, VN, HD, LB, TK, JJ, EP and SP), including the project leaders (PD and VN) formed a steering committee. The committee developed a checklist based on the items in the Consolidated Standards of Reporting Trials (CONSORT) guidelines and Clinical and Laboratory Images in Publications (CLIP) principles. A PRIRATE Delphi Group (PDG) and PRIRATE Face-to-Face Meeting group (PFMG) were also formed. Thirty PDG members participated in the online Delphi process and achieved consensus on the checklist items and flowchart that make up the PRIRATE guidelines. The guidelines were discussed at a meeting of the PFMG at the 19th European Society of Endodontology (ESE) Biennial congress, held on 13 September 2019 in Vienna, Austria. A total of 21 individuals from across the globe and four steering committee members (PD, VN, HD and LB) attended the meeting. As a consequence of the discussions, the guidelines were modified and then piloted by several authors whilst writing a manuscript. The PRIRATE 2020 guidelines contain a checklist consisting of 11 sections and 58 individual items as well as a flowchart, considered essential for authors to include when writing manuscripts for randomized clinical trials in Endodontics.

PRICE 2020 guidelines for reporting case reports in Endodontics: explanation and elaboration.

Case reports play a key role in showcasing new, unusual or rare disease(s), and the impact of newer therapeutic approaches or interventions. The Preferred Reporting Items for Case reports in Endodontics (PRICE) 2020 guidelines are being introduced exclusively for Endodontics by adapting and integrating the CAse REport (CARE) guidelines and Clinical and Laboratory Images in Publications principles. The PRICE 2020 guidelines have been developed to help authors improve the completeness, accuracy and transparency of case reports in Endodontics and thus enhance the standard of manuscripts submitted for publication. The aim of this document is to provide a comprehensive explanation for each item in the PRICE 2020 checklist along with examples from the literature that demonstrate compliance with these guidelines. This information will highlight the importance of each item and provide practical examples to help authors understand the necessity of providing comprehensive information when preparing case reports. A link to this PRICE 2020 explanation and elaboration document is available on the Preferred Reporting Items for study Designs in Endodontology website at http://www.pride-endodonticguidelines.org.

PRICE 2020 guidelines for reporting case reports in Endodontics: a consensus-based development.

Case reports can provide early information about new, unusual or rare disease(s), newer treatment strategies, improved therapeutic benefits and adverse effects of interventions or medications. This paper describes the process that led to the development of the Preferred Reporting Items for Case reports in Endodontics (PRICE) 2020 guidelines through a consensus-based methodology. A steering committee was formed with eight members (PD, VN, BC, PM, PS, EP, JJ and SP), including the project leaders (PD, VN). The steering committee developed an initial checklist by combining and modifying the items from the Case Report (CARE) guidelines and Clinical and Laboratory Images in Publications (CLIP) principles. A PRICE Delphi Group (PDG) and PRICE Face-to-Face Meeting Group (PFMG) were then formed. The members of the PDG were invited to participate in an online Delphi process to achieve consensus on the wording and utility of the checklist items and the accompanying flow chart that was created to complement the PRICE 2020 guidelines. The revised PRICE checklist and flow chart developed by the online Delphi process was discussed by the PFMG at a meeting held during the 19th European Society of Endodontology (ESE) Biennial Congress in Vienna, Austria, in September 2019. Following the meeting, the steering committee created a final version of the guidelines, which were piloted by several authors during the writing of a case report. In order to help improve the clarity, completeness and quality of case reports in Endodontics, we encourage authors to use the PRICE 2020 guidelines.

Scalable Topological Data Analysis and Visualization for Evaluating Data-Driven Models in Scientific Applications.

With the rapid adoption of machine learning techniques for large-scale applications in science and engineering comes the convergence of two grand challenges in visualization. First, the utilization of black box models (e.g., deep neural networks) calls for advanced techniques in exploring and interpreting model behaviors. Second, the rapid growth in computing has produced enormous datasets that require techniques that can handle millions or more samples. Although some solutions to these interpretability challenges have been proposed, they typically do not scale beyond thousands of samples, nor do they provide the high-level intuition scientists are looking for. Here, we present the first scalable solution to explore and analyze high-dimensional functions often encountered in the scientific data analysis pipeline. By combining a new streaming neighborhood graph construction, the corresponding topology computation, and a novel data aggregation scheme, namely topology aware datacubes, we enable interactive exploration of both the topological and the geometric aspect of high-dimensional data. Following two use cases from high-energy-density (HED) physics and computational biology, we demonstrate how these capabilities have led to crucial new insights in both applications.

GrAMME: Semisupervised Learning Using Multilayered Graph Attention Models.

Modern data analysis pipelines are becoming increasingly complex due to the presence of multiview information sources. While graphs are effective in modeling complex relationships, in many scenarios, a single graph is rarely sufficient to succinctly represent all interactions, and hence, multilayered graphs have become popular. Though this leads to richer representations, extending solutions from the single-graph case is not straightforward. Consequently, there is a strong need for novel solutions to solve classical problems, such as node classification, in the multilayered case. In this article, we consider the problem of semisupervised learning with multilayered graphs. Though deep network embeddings, e.g., DeepWalk, are widely adopted for community discovery, we argue that feature learning with random node attributes, using graph neural networks, can be more effective. To this end, we propose to use attention models for effective feature learning and develop two novel architectures, GrAMME-SG and GrAMME-Fusion, that exploit the interlayer dependences for building multilayered graph embeddings. Using empirical studies on several benchmark data sets, we evaluate the proposed approaches and demonstrate significant performance improvements in comparison with the state-of-the-art network embedding strategies. The results also show that using simple random features is an effective choice, even in cases where explicit node attributes are not available.

Preferred Reporting Items for Animal Studies in Endodontology: a development protocol.

The regulated use of animals in endodontic research is often necessary to investigate the biological mechanisms of endodontic diseases and to measure the preclinical efficacy, biocompatibility, toxicology and safety of new treatments, biomaterials, sealers, drugs, disinfectants, irrigants, devices and instruments. Animal testing is most crucial in situations when research on humans is not ethical, practical or has unknown health risks. Currently, there is a wide variability in the quality of manuscripts that report the results of animal studies. Towards the goal of improving the quality of publications, guidelines for preventing disability, pain, and suffering to animals, and enhanced reporting requirements for animal research have been developed. These guidelines are referred to as Animals in Research: Reporting In Vivo Experiments (ARRIVE). Henceforth, causing any form of animal suffering for research purposes is not acceptable and cannot be justified under any circumstances. The present report describes a protocol for the development of welfare and reporting guidelines for animal studies conducted in the specialty of Endodontology: the Preferred Reporting Items for Animal Studies in Endodontology (PRIASE) guidelines. The PRIASE guidelines will be developed by adapting and modifying the ARRIVE guidelines and the Clinical and Laboratory Images in Publication (CLIP) principles. The development of the new PRIASE guidelines will include a five-step consensus process. An initial draft of the PRIASE guidelines will be developed by a steering committee. Each item in the draft guidelines will then be evaluated by members of a PRIASE Delphi Group (PDG) for its clarity using a dichotomous scale (yes or no) and suitability for its inclusion using a 9-point Likert scale. The online surveys will continue until each item achieves this standard, and a set of items are agreed for further analysis by a PRIASE Face-to-face Consensus Meeting Group (PFCMG). Following the consensus meeting, the steering committee will finalize and confirm the PRIASE guidelines taking into account the responses and comments of the PFCMG. The PRIASE guidelines will be published and disseminated internationally and updated periodically based on feedback from stakeholders.

Quality of abstract of systematic reviews and meta-analyses in paediatric dentistry journals.

AIM: To systematically evaluate the reporting quality of the abstract of systematic reviews and meta-analyses in paediatric dentistry journals. MATERIALS AND METHODS: Systematic reviews with meta-analyses in paediatric dentistry were searched in PubMed and Scopus databases from inception to December 2017. Selection of studies by title and abstract screening followed by full-text assessment was independently done by two reviewers. The quality of abstracts was assessed by PRISMA-Abstract checklist comprising of 12 items; one each for title and objective, three items for methods, three items for results, two items for discussion and two items for others. PRISMA-A median scores were calculated and compared with the article characteristics. Statistical significance was set at p < 0.05 and multi-variate analysis was performed using Kruskal-Wallis test. RESULTS: A total of 24 studies were included in the analysis. The mean PRISMA-Abstract score was 7.46 ± 1.19. None of the studies were of high quality (score 10-12), 20 were of moderate (score 7-9), and 4 were of low quality (score 1-6). Journals that adhered to PRISMA guidelines showed significantly higher quality (p < 0.05). No association was found between the quality and the number of authors, country, journals, year of publication, word count and focus of study. CONCLUSION: Majority of abstracts of systematic reviews and meta-analyses in paediatric dentistry journals were of moderate quality. Adoption and adherence to PRISMA-Abstract checklist by the journal editors and authors will enhance the reporting quality of abstracts.