Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Community consensus on core open science practices to monitor in biomedicine.

The state of open science needs to be monitored to track changes over time and identify areas to create interventions to drive improvements. In order to monitor open science practices, they first need to be well defined and operationalized. To reach consensus on what open science practices to monitor at biomedical research institutions, we conducted a modified 3-round Delphi study. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices, and for round 3, we hosted two half-day virtual meetings to discuss and vote on items that had not reached consensus. Ultimately, participants reached consensus on 19 open science practices. This core set of open science practices will form the foundation for institutional dashboards and may also be of value for the development of policy, education, and interventions.

Epigenetic reprogramming of a distal developmental enhancer cluster drives SOX2 overexpression in breast and lung adenocarcinoma.

Enhancer reprogramming has been proposed as a key source of transcriptional dysregulation during tumorigenesis, but the molecular mechanisms underlying this process remain unclear. Here, we identify an enhancer cluster required for normal development that is aberrantly activated in breast and lung adenocarcinoma. Deletion of the SRR124-134 cluster disrupts expression of the SOX2 oncogene, dysregulates genome-wide transcription and chromatin accessibility and reduces the ability of cancer cells to form colonies in vitro. Analysis of primary tumors reveals a correlation between chromatin accessibility at this cluster and SOX2 overexpression in breast and lung cancer patients. We demonstrate that FOXA1 is an activator and NFIB is a repressor of SRR124-134 activity and SOX2 transcription in cancer cells, revealing a co-opting of the regulatory mechanisms involved in early development. Notably, we show that the conserved SRR124 and SRR134 regions are essential during mouse development, where homozygous deletion results in the lethal failure of esophageal-tracheal separation. These findings provide insights into how developmental enhancers can be reprogrammed during tumorigenesis and underscore the importance of understanding enhancer dynamics during development and disease.

The manuscript by Abatti et al. shows that epigenetic reactivation of a pair of distal enhancers that drive Sox2 expression during development (to permit separation of the esophagus and trachea) is responsible for the tumor-promoting re-expression of SOX2 in breast and lung tumors. Intriguingly, the same transcription factors that act on the enhancers during development to either activate or repress them (i.e. FOXA1 and NFIB, respectively) are also required for altering chromatin accessibility of the enhancers and SOX2 transcription in breast and lung cancer cells. With their work, the authors unravel the exact mechanism of how developmentally active enhancers become repurposed in a tumor context and show the relevance of this repurposing event for cancer.

Sensitive tumour detection and classification using plasma cell-free DNA methylomes.

The use of liquid biopsies for cancer detection and management is rapidly gaining prominence1. Current methods for the detection of circulating tumour DNA involve sequencing somatic mutations using cell-free DNA, but the sensitivity of these methods may be low among patients with early-stage cancer given the limited number of recurrent mutations2-5. By contrast, large-scale epigenetic alterations-which are tissue- and cancer-type specific-are not similarly constrained6 and therefore potentially have greater ability to detect and classify cancers in patients with early-stage disease. Here we develop a sensitive, immunoprecipitation-based protocol to analyse the methylome of small quantities of circulating cell-free DNA, and demonstrate the ability to detect large-scale DNA methylation changes that are enriched for tumour-specific patterns. We also demonstrate robust performance in cancer detection and classification across an extensive collection of plasma samples from several tumour types. This work sets the stage to establish biomarkers for the minimally invasive detection, interception and classification of early-stage cancers based on plasma cell-free DNA methylation patterns.

Assessing and assuring interoperability of a genomics file format.

MOTIVATION: Bioinformatics software tools operate largely through the use of specialized genomics file formats. Often these formats lack formal specification, making it difficult or impossible for the creators of these tools to robustly test them for correct handling of input and output. This causes problems in interoperability between different tools that, at best, wastes time and frustrates users. At worst, interoperability issues could lead to undetected errors in scientific results. RESULTS: We developed a new verification system, Acidbio, which tests for correct behavior in bioinformatics software packages. We crafted tests to unify correct behavior when tools encounter various edge cases-potentially unexpected inputs that exemplify the limits of the format. To analyze the performance of existing software, we tested the input validation of 80 Bioconda packages that parsed the Browser Extensible Data (BED) format. We also used a fuzzing approach to automatically perform additional testing. Of 80 software packages examined, 75 achieved less than 70% correctness on our test suite. We categorized multiple root causes for the poor performance of different types of software. Fuzzing detected other errors that the manually designed test suite could not. We also created a badge system that developers can use to indicate more precisely which BED variants their software accepts and to advertise the software's performance on the test suite. AVAILABILITY AND IMPLEMENTATION: Acidbio is available at https://github.com/hoffmangroup/acidbio. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Exploring the merits of research performance measures that comply with the San Francisco Declaration on Research Assessment and strategies to overcome barriers of adoption: qualitative interviews with administrators and researchers.

BACKGROUND: In prior research, we identified and prioritized ten measures to assess research performance that comply with the San Francisco Declaration on Research Assessment, a principle adopted worldwide that discourages metrics-based assessment. Given the shift away from assessment based on Journal Impact Factor, we explored potential barriers to implementing and adopting the prioritized measures. METHODS: We identified administrators and researchers across six research institutes, conducted telephone interviews with consenting participants, and used qualitative description and inductive content analysis to derive themes. RESULTS: We interviewed 18 participants: 6 administrators (research institute business managers and directors) and 12 researchers (7 on appointment committees) who varied by career stage (2 early, 5 mid, 5 late). Participants appreciated that the measures were similar to those currently in use, comprehensive, relevant across disciplines, and generated using a rigorous process. They also said the reporting template was easy to understand and use. In contrast, a few administrators thought the measures were not relevant across disciplines. A few participants said it would be time-consuming and difficult to prepare narratives when reporting the measures, and several thought that it would be difficult to objectively evaluate researchers from a different discipline without considerable effort to read their work. Strategies viewed as necessary to overcome barriers and support implementation of the measures included high-level endorsement of the measures, an official launch accompanied by a multi-pronged communication strategy, training for both researchers and evaluators, administrative support or automated reporting for researchers, guidance for evaluators, and sharing of approaches across research institutes. CONCLUSIONS: While participants identified many strengths of the measures, they also identified a few limitations and offered corresponding strategies to address the barriers that we will apply at our organization. Ongoing work is needed to develop a framework to help evaluators translate the measures into an overall assessment. Given little prior research that identified research assessment measures and strategies to support adoption of those measures, this research may be of interest to other organizations that assess the quality and impact of research.

Coronary angiography of the ex-situ beating donor heart in a portable organ care system.

OBJECTIVES: To determine safety and feasibility of ex-situ coronary angiography. BACKGROUND: To cater for the perpetually growing demand for heart donors, interest in donation following circulatory death (DCD) has been rekindled. Further pursuit of donor pool expansion has led to eligibility extension to "marginal" donors who are at higher risk of coronary artery disease (CAD). Excluding CAD in potentially eligible DCD donors, for whom ante-mortem angiography is commonly not permitted, is therefore challenging. Ex-situ coronary angiography serves as an ethical and feasible diagnostic tool to assess for preclusive CAD. METHODS: We undertook a systematic review of the published literature and institutional retrospective review of case experience with ex-situ coronary angiography of donor hearts, supported by a portable organ care system. RESULTS: Combined literature and institutional case review yielded nine total cases of ex-situ coronary angiography of donor human hearts plus one experimental porcine model. Of the eight cases of ex-situ coronary angiography performed at our institute, all were conducted without complication or injury to the allograft. Two thirds of reported human cases have proceeded to successful transplantation. CONCLUSIONS: Diagnostic coronary angiography of the ex-situ beating donor heart is safe, feasible, and demonstrates novel clinical utility in mitigating subsequent transplantation of unsuitable allografts. In the setting of suspected coronary atherosclerosis of the donor heart, which may preclude favorable transplantation outcomes, ex-situ coronary angiography should be considered at eligible transplant centers.

Segmentation and genome annotation algorithms for identifying chromatin state and other genomic patterns.

Segmentation and genome annotation (SAGA) algorithms are widely used to understand genome activity and gene regulation. These algorithms take as input epigenomic datasets, such as chromatin immunoprecipitation-sequencing (ChIP-seq) measurements of histone modifications or transcription factor binding. They partition the genome and assign a label to each segment such that positions with the same label exhibit similar patterns of input data. SAGA algorithms discover categories of activity such as promoters, enhancers, or parts of genes without prior knowledge of known genomic elements. In this sense, they generally act in an unsupervised fashion like clustering algorithms, but with the additional simultaneous function of segmenting the genome. Here, we review the common methodological framework that underlies these methods, review variants of and improvements upon this basic framework, and discuss the outlook for future work. This review is intended for those interested in applying SAGA methods and for computational researchers interested in improving upon them.

Development and Validation of a Parenting Stress Module for Parents of Children Using Cochlear Implants.

OBJECTIVES: The purpose of this study was to develop and validate a cochlear implant (CI)-specific parenting stress measure using the FDA Guidance on Patient-Reported Outcomes (2009). METHODS: The development and psychometric validation of the Parenting Stress-CI module for both the Early Childhood (EC; 0-5 years) and School-Age (SA; 6-12 years) versions are reported in this article. Instrument development consisted of qualitative interviews with parents of children with CIs (EC: N = 19; SA: N = 21), content analysis, item development, and cognitive testing of the instrument. Last, we conducted the psychometric validation (EC: N = 72; SA: N = 64), including analyses of internal consistency, test-retest reliability (∼2 weeks between administrations; N = 24), and convergent validity with the Parenting Stress Index-4 (PSI-4). RESULTS: The final EC version includes 15 questions, and the SA version includes 8 questions. Both the EC and SA versions had strong reliability (EC α = .88; SA α = .85), with all items significantly correlated with the overall module (r = .43-.80). Both versions also had strong test-retest reliability (r = .99, p < .001). Last, analyses of convergent validity demonstrated significant correlations with the PSI-4 Total Stress scale for both Parenting Stress-CI versions (EC r = .66, p < .00; SA r = .45, p < .001). CONCLUSIONS: The Parenting Stress-CI modules are reliable and valid condition-specific parenting stress instruments for parents of children with CIs ages 0-12 years, filling a significant gap in the literature. These fully validated instruments can be used to assess parental needs for support and guide the development of targeted, family centered interventions.

An evaluation of different approaches which use Google Street View imagery to ground truth land degradation assessments.

Member states of the United Nations Convention to Combat Desertification are required to report on the proportion of land that is degraded in their countries, a requirement that is also tied into the UN Sustainable Development Goals (SDGs). National land degradation assessments are often conducted with the use of remote sensing data which are not always ground truthed. Google Street View (GSV) provides high resolution, panoramic imagery across large parts of the world that has the potential to be used to ground truth land degradation assessments. We apply three different methodologies (visual interpretation of GSV images, GSV image classification and vegetation index extraction) to derive vegetation cover estimates from Google Street View imagery for the Hardeveld bioregion of the Succulent Karoo biome in South Africa. Visual estimates of cover best predict known habitat condition values (adjusted R2 = 0.86), whilst estimates derived from an unsupervised classification of GSV images also predict habitat condition relatively well (adjusted R2 = 0.52). These results show the potential for using GSV imagery, and other large collections of ground-level landscape photographs, as a rough ground-truthing tool, especially in instances where more traditional ground-truthing approaches are not possible.

Online Improvement of Condition-Baesd Maintenance Policy via Monte Carlo Tree Search.

Often in manufacturing systems, scenarios arise where the demand for maintenance exceeds the capacity of maintenance resources. This results in the problem of allocating the limited resources among machines competing for them. This maintenance scheduling problem can be formulated as a Markov decision process (MDP) with the goal of finding the optimal dynamic maintenance action given the current system state. However, as the system becomes more complex, solving an MDP suffers from the curse of dimensionality. To overcome this issue, we propose a two-stage approach that first optimizes a static condition-based maintenance (CBM) policy using a genetic algorithm (GA) and then improves the policy online via Monte Carlo tree search (MCTS). The static policy significantly reduces the state space of the online problem by allowing us to ignore machines that are not sufficiently degraded. Furthermore, we formulate MCTS to seek a maintenance schedule that maximizes the long-term production volume of the system to reconcile the conflict between maintenance and production objectives. We demonstrate that the resulting online policy is an improvement over the static CBM policy found by GA. Note to Practitioners­: This article proposes a method of scheduling maintenance in complex manufacturing systems in scenarios where there is frequent competition for maintenance resources. We use a condition-based maintenance policy that prescribes maintenance actions based on a machine's current health. However, when several machines are due for maintenance, a maintenance technician must choose between multiple competing jobs. While a common approach is to establish rules that dictate how maintenance jobs should be prioritized, such as the first-in, first-out rule, the goal of this work is to improve upon static policies in real time. We do this by strategically evaluating sequences of maintenance actions and playing out many "what-if" scenarios to see how the system will behave in the future. Implementation of the proposed method relies on the construction of a simulation model of the target system. This model is capable of retrieving the current state of the physical system, including the degradation state of machines, the availability of maintenance resources, and the distribution of parts throughout buffers in the system. We present several simulation experiments that demonstrate the improvement in system performance that our approach provides. Future work will aim to improve the efficiency of maintenance prioritization through online learning as well as more accurately identify manufacturing system configurations that will yield the greatest benefit of these methods.

Sex-Based Impact of Creatine Supplementation on Depressive Symptoms, Brain Serotonin and SSRI Efficacy in an Animal Model of Treatment-Resistant Depression.

BACKGROUND: Rates of major depressive disorder (MDD) increase with living at altitude. In our model, rats housed at moderate altitude (in hypobaric hypoxia) exhibit increased depression-like behavior, altered brain serotonin and a lack of antidepressant response to most selective serotonin reuptake inhibitors (SSRIs). A forebrain deficit in the bioenergetic marker creatine is noted in people living at altitude or with MDD. METHODS: Rats housed at 4500 ft were given dietary creatine monohydrate (CRMH, 4% w/w, 5 weeks) vs. un-supplemented diet, and impact on depression-like behavior, brain bioenergetics, serotonin and SSRI efficacy assessed. RESULTS: CRMH significantly improved brain creatine in a sex-based manner. At altitude, CRMH increased serotonin levels in the female prefrontal cortex and striatum but reduced male striatal and hippocampal serotonin. Dietary CRMH was antidepressant in the forced swim test and anti-anhedonic in the sucrose preference test in only females at altitude, with motor behavior unchanged. CRMH improved fluoxetine efficacy (20 mg/kg) in only males at altitude: CRMH + SSRI significantly improved male striatal creatine and serotonin vs. CRMH alone. CONCLUSIONS: Dietary CRMH exhibits sex-based efficacy in resolving altitude-related deficits in brain biomarkers, depression-like behavior and SSRI efficacy, and may be effective clinically for SSRI-resistant depression at altitude. This is the first study to link CRMH treatment to improving brain serotonin.

Top considerations for creating bioinformatics software documentation.

Investing in documenting your bioinformatics software well can increase its impact and save your time. To maximize the effectiveness of your documentation, we suggest following a few guidelines we propose here. We recommend providing multiple avenues for users to use your research software, including a navigable HTML interface with a quick start, useful help messages with detailed explanation and thorough examples for each feature of your software. By following these guidelines, you can assure that your hard work maximally benefits yourself and others.

Comparative analysis of metazoan chromatin organization.

Genome function is dynamically regulated in part by chromatin, which consists of the histones, non-histone proteins and RNA molecules that package DNA. Studies in Caenorhabditis elegans and Drosophila melanogaster have contributed substantially to our understanding of molecular mechanisms of genome function in humans, and have revealed conservation of chromatin components and mechanisms. Nevertheless, the three organisms have markedly different genome sizes, chromosome architecture and gene organization. On human and fly chromosomes, for example, pericentric heterochromatin flanks single centromeres, whereas worm chromosomes have dispersed heterochromatin-like regions enriched in the distal chromosomal 'arms', and centromeres distributed along their lengths. To systematically investigate chromatin organization and associated gene regulation across species, we generated and analysed a large collection of genome-wide chromatin data sets from cell lines and developmental stages in worm, fly and human. Here we present over 800 new data sets from our ENCODE and modENCODE consortia, bringing the total to over 1,400. Comparison of combinatorial patterns of histone modifications, nuclear lamina-associated domains, organization of large-scale topological domains, chromatin environment at promoters and enhancers, nucleosome positioning, and DNA replication patterns reveals many conserved features of chromatin organization among the three organisms. We also find notable differences in the composition and locations of repressive chromatin. These data sets and analyses provide a rich resource for comparative and species-specific investigations of chromatin composition, organization and function.

Fathers of Children With Congenital Heart Disease: Sources of Stress and Opportunities for Intervention.

OBJECTIVES: To examine sources of stress for fathers of children with congenital heart disease and opportunities for intervention to prevent or reduce paternal mental health problems. DESIGN: Qualitative study using online crowdsourcing, an innovative research methodology to create an online community to serve as a research sample. SETTING: Yammer, an online social networking site. SUBJECTS: Geographically diverse sample of 70 parents (25 fathers and 45 mothers) of young children with congenital heart disease. INTERVENTIONS: Participants joined a private group on Yammer and responded to 37 open-ended study questions over a 6-month period. Qualitative data were coded and analyzed using an iterative process, and themes regarding sources of stress for fathers of children with congenital heart disease and opportunities for intervention were identified. MEASUREMENTS AND MAIN RESULTS: Four broad themes regarding sources of stress for fathers of children with congenital heart disease emerged from the qualitative data from both mothers and fathers: societal expectations for fatherhood and standards of masculinity, balancing work and family responsibilities, feeling overlooked as a partner in care, and lack of father supports. To begin to address these sources of stress, participants recommended that care teams acknowledge and normalize the impact of congenital heart disease on fathers, provide support for balancing work and family responsibilities, recognize and promote father knowledge and engagement, and provide formal and informal supports for fathers of children with congential heart disease. CONCLUSIONS: Fathers of children with congenital heart disease experience unique sources of stress in the absence of targeted interventions to meet their needs. Care teams play an important role in acknowledging the experiences of fathers and including and engaging fathers in care.

Umap and Bismap: quantifying genome and methylome mappability.

Short-read sequencing enables assessment of genetic and biochemical traits of individual genomic regions, such as the location of genetic variation, protein binding and chemical modifications. Every region in a genome assembly has a property called 'mappability', which measures the extent to which it can be uniquely mapped by sequence reads. In regions of lower mappability, estimates of genomic and epigenomic characteristics from sequencing assays are less reliable. These regions have increased susceptibility to spurious mapping from reads from other regions of the genome with sequencing errors or unexpected genetic variation. Bisulfite sequencing approaches used to identify DNA methylation exacerbate these problems by introducing large numbers of reads that map to multiple regions. Both to correct assumptions of uniformity in downstream analysis and to identify regions where the analysis is less reliable, it is necessary to know the mappability of both ordinary and bisulfite-converted genomes. We introduce the Umap software for identifying uniquely mappable regions of any genome. Its Bismap extension identifies mappability of the bisulfite-converted genome. A Umap and Bismap track hub for human genome assemblies GRCh37/hg19 and GRCh38/hg38, and mouse assemblies GRCm37/mm9 and GRCm38/mm10 is available at https://bismap.hoffmanlab.org for use with genome browsers.

Inflammasome Antagonism by Human Parainfluenza Virus Type 3 C Protein.

Human parainfluenza virus type 3 (HPIV3) is a negative-sense single-stranded RNA virus belonging to the Paramyxoviridae family. HPIV3 is a lung-tropic virus causing airway diseases, including pneumonia, croup, and bronchiolitis, during infancy and childhood. The activation of the inflammasome by pathogens results in the production of proinflammatory cytokines such as interleukin-1ß (IL-1ß) during infection. Thus, the inflammasome-mediated proinflammatory response plays a critical role in regulating the immune response and virus clearance. The inflammasome is a multimeric protein complex triggering caspase-1 activation. Activated caspase-1 cleaves pro-IL-1ß into its mature (and active) secretory form. Our study revealed inflammasome activation in macrophages following HPIV3 infection. Specifically, the activation of the NLRP3/ASC inflammasome resulted in the production of mature IL-1ß from HPIV3-infected cells. Furthermore, Toll-like receptor 2 (TLR2) activation (first signal) and potassium efflux (second signal) constituted two cellular events mediating inflammasome activation following HPIV3 infection. During our studies, we surprisingly identified the HPIV3 C protein as an antagonist of inflammasome activation. The HPIV3 C protein is an accessory protein encoded by the open reading frame of the viral phosphoprotein (P) gene. The HPIV3 C protein interacted with the NLRP3 protein and blocked inflammasome activation by promoting the proteasomal degradation of the NLRP3 protein. Thus, our studies report NLRP3/ASC inflammasome activation by HPIV3 via TLR2 signaling and potassium efflux. Furthermore, we have identified HPIV3 C as a viral component involved in antagonizing inflammasome activation.IMPORTANCE Human parainfluenza virus type 3 (HPIV3) is a paramyxovirus that causes respiratory tract diseases during infancy and childhood. Currently, there is no effective vaccine or antiviral therapy for HPIV3. Therefore, in order to develop anti-HPIV3 agents (therapeutics and vaccines), it is important to study the HPIV3-host interaction during the immune response. Inflammasomes play an important role in the immune response. Inflammasome activation by HPIV3 has not been previously reported. Our studies demonstrated inflammasome activation by HPIV3 in macrophages. Specifically, HPIV3 activated the NLRP3/ASC inflammasome by TLR2 activation and potassium efflux. C proteins of paramyxoviruses are accessory proteins encoded by the viral phosphoprotein gene. The role of the C protein in inflammasome regulation was unknown. Surprisingly, our studies revealed that the HPIV3 C protein antagonizes inflammasome activation. In addition, we highlighted for the first time a mechanism utilized by paramyxovirus accessory proteins to block inflammasome activation. The HPIV3 C protein interacted with the NLRP3 protein to trigger the proteasomal degradation of the NLRP3 protein.