Challenges and best practices in omics benchmarking.

Technological advances enabling massively parallel measurement of biological features - such as microarrays, high-throughput sequencing and mass spectrometry - have ushered in the omics era, now in its third decade. The resulting complex landscape of analytical methods has naturally fostered the growth of an omics benchmarking industry. Benchmarking refers to the process of objectively comparing and evaluating the performance of different computational or analytical techniques when processing and analysing large-scale biological data sets, such as transcriptomics, proteomics and metabolomics. With thousands of omics benchmarking studies published over the past 25 years, the field has matured to the point where the foundations of benchmarking have been established and well described. However, generating meaningful benchmarking data and properly evaluating performance in this complex domain remains challenging. In this Review, we highlight some common oversights and pitfalls in omics benchmarking. We also establish a methodology to bring the issues that can be addressed into focus and to be transparent about those that cannot: this takes the form of a spreadsheet template of guidelines for comprehensive reporting, intended to accompany publications. In addition, a survey of recent developments in benchmarking is provided as well as specific guidance for commonly encountered difficulties.

BEERS2: RNA-Seq simulation through high fidelity in silico modeling.

Simulation of RNA-seq reads is critical in the assessment, comparison, benchmarking and development of bioinformatics tools. Yet the field of RNA-seq simulators has progressed little in the last decade. To address this need we have developed BEERS2, which combines a flexible and highly configurable design with detailed simulation of the entire library preparation and sequencing pipeline. BEERS2 takes input transcripts (typically fully length messenger RNA transcripts with polyA tails) from either customizable input or from CAMPAREE simulated RNA samples. It produces realistic reads of these transcripts as FASTQ, SAM or BAM formats with the SAM or BAM formats containing the true alignment to the reference genome. It also produces true transcript-level quantification values. BEERS2 combines a flexible and highly configurable design with detailed simulation of the entire library preparation and sequencing pipeline and is designed to include the effects of polyA selection and RiboZero for ribosomal depletion, hexamer priming sequence biases, GC-content biases in polymerase chain reaction (PCR) amplification, barcode read errors and errors during PCR amplification. These characteristics combine to make BEERS2 the most complete simulation of RNA-seq to date. Finally, we demonstrate the use of BEERS2 by measuring the effect of several settings on the popular Salmon pseudoalignment algorithm.

Comparative evaluation of full-length isoform quantification from RNA-Seq.

BACKGROUND: Full-length isoform quantification from RNA-Seq is a key goal in transcriptomics analyses and has been an area of active development since the beginning. The fundamental difficulty stems from the fact that RNA transcripts are long, while RNA-Seq reads are short. RESULTS: Here we use simulated benchmarking data that reflects many properties of real data, including polymorphisms, intron signal and non-uniform coverage, allowing for systematic comparative analyses of isoform quantification accuracy and its impact on differential expression analysis. Genome, transcriptome and pseudo alignment-based methods are included; and a simple approach is included as a baseline control. CONCLUSIONS: Salmon, kallisto, RSEM, and Cufflinks exhibit the highest accuracy on idealized data, while on more realistic data they do not perform dramatically better than the simple approach. We determine the structural parameters with the greatest impact on quantification accuracy to be length and sequence compression complexity and not so much the number of isoforms. The effect of incomplete annotation on performance is also investigated. Overall, the tested methods show sufficient divergence from the truth to suggest that full-length isoform quantification and isoform level DE should still be employed selectively.

CAMPAREE: a robust and configurable RNA expression simulator.

BACKGROUND: The accurate interpretation of RNA-Seq data presents a moving target as scientists continue to introduce new experimental techniques and analysis algorithms. Simulated datasets are an invaluable tool to accurately assess the performance of RNA-Seq analysis methods. However, existing RNA-Seq simulators focus on modeling the technical biases and artifacts of sequencing, rather than on simulating the original RNA samples. A first step in simulating RNA-Seq is to simulate RNA. RESULTS: To fill this need, we developed the Configurable And Modular Program Allowing RNA Expression Emulation (CAMPAREE), a simulator using empirical data to simulate diploid RNA samples at the level of individual molecules. We demonstrated CAMPAREE's use for generating idealized coverage plots from real data, and for adding the ability to generate allele-specific data to existing RNA-Seq simulators that do not natively support this feature. CONCLUSIONS: Separating input sample modeling from library preparation/sequencing offers added flexibility for both users and developers to mix-and-match different sample and sequencing simulators to suit their specific needs. Furthermore, the ability to maintain sample and sequencing simulators independently provides greater agility to incorporate new biological findings about transcriptomics and new developments in sequencing technologies. Additionally, by simulating at the level of individual molecules, CAMPAREE has the potential to model molecules transcribed from the same genes as a heterogeneous population of transcripts with different states of degradation and processing (splicing, editing, etc.). CAMPAREE was developed in Python, is open source, and freely available at https://github.com/itmat/CAMPAREE .

Prognostic utility of rhythmic components in 24-h ambulatory blood pressure monitoring for the risk stratification of chronic kidney disease patients with cardiovascular co-morbidity.

Chronic kidney disease (CKD) represents a significant global burden. Hypertension is a modifiable risk factor for rapid progression of CKD. We extend the risk stratification by introducing the non-parametric determination of rhythmic components in 24-h profiles of ambulatory blood pressure monitoring (ABPM) in the Chronic Renal Insufficiency Cohort (CRIC) and the African American Study for Kidney Disease and Hypertension (AASK) cohort using Cox proportional hazards models. We find that rhythmic profiling of BP through JTK_CYCLE analysis identifies subgroups of CRIC participants that were more likely to die due to cardiovascular causes. While our fully adjusted model shows a trend towards a significant association between absent cyclic components and cardiovascular death in the full CRIC cohort (HR: 1.71,95% CI: 0.99-2.97, p = 0.056), CRIC participants with a history of cardiovascular disease (CVD) and absent cyclic components in their BP profile had at any time a 3.4-times higher risk of cardiovascular death than CVD patients with cyclic components present in their BP profile (HR: 3.37, 95% CI: 1.45-7.87, p = 0.005). This increased risk was not explained by the dipping or non-dipping pattern in ABPM. Due to the large differences in patient characteristics, the results do not replicate in the AASK cohort. This study suggests rhythmic blood pressure components as a potential novel biomarker to unmask excess risk among CKD patients with prior cardiovascular disease.

Meta-analysis of Diurnal Transcriptomics in Mouse Liver Reveals Low Repeatability of Rhythm Analyses.

To assess the consistency of biological rhythms across studies, 57 public mouse liver tissue timeseries totaling 1096 RNA-seq samples were obtained and analyzed. Only the control groups of each study were included, to create comparable data. Technical factors in RNA-seq library preparation were the largest contributors to transcriptome-level differences, beyond biological or experiment-specific factors such as lighting conditions. Core clock genes were remarkably consistent in phase across all studies. Overlap of genes identified as rhythmic across studies was generally low, with no pair of studies having over 60% overlap. Distributions of phases of significant genes were remarkably inconsistent across studies, but the genes that consistently identified as rhythmic had acrophase clustering near ZT0 and ZT12. Despite the discrepancies between single-study analyses, cross-study analyses found substantial consistency. Running compareRhythms on each pair of studies identified a median of only 11% of the identified rhythmic genes as rhythmic in only 1 of the 2 studies. Data were integrated across studies in a joint and individual variance estimate (JIVE) analysis, which showed that the top 2 components of joint within-study variation are determined by time of day. A shape-invariant model with random effects was fit to the genes to identify the underlying shape of the rhythms, consistent across all studies, including identifying 72 genes with consistently multiple peaks.

BEERS2: RNA-Seq simulation through high fidelity in silico modeling.

Simulation of RNA-seq reads is critical in the assessment, comparison, benchmarking, and development of bioinformatics tools. Yet the field of RNA-seq simulators has progressed little in the last decade. To address this need we have developed BEERS2, which combines a flexible and highly configurable design with detailed simulation of the entire library preparation and sequencing pipeline. BEERS2 takes input transcripts (typically fully-length mRNA transcripts with polyA tails) from either customizable input or from CAMPAREE simulated RNA samples. It produces realistic reads of these transcripts as FASTQ, SAM, or BAM formats with the SAM or BAM formats containing the true alignment to the reference genome. It also produces true transcript-level quantification values. BEERS2 combines a flexible and highly configurable design with detailed simulation of the entire library preparation and sequencing pipeline and is designed to include the effects of polyA selection and RiboZero for ribosomal depletion, hexamer priming sequence biases, GC-content biases in PCR amplification, barcode read errors, and errors during PCR amplification. These characteristics combine to make BEERS2 the most complete simulation of RNA-seq to date. Finally, we demonstrate the use of BEERS2 by measuring the effect of several settings on the popular Salmon pseudoalignment algorithm.

Diurnal rhythms of wrist temperature are associated with future disease risk in the UK Biobank.

Many chronic disease symptomatologies involve desynchronized sleep-wake cycles, indicative of disrupted biorhythms. This can be interrogated using body temperature rhythms, which have circadian as well as sleep-wake behavior/environmental evoked components. Here, we investigated the association of wrist temperature amplitudes with a future onset of disease in the UK Biobank one year after actigraphy. Among 425 disease conditions (range n = 200-6728) compared to controls (range n = 62,107-91,134), a total of 73 (17%) disease phenotypes were significantly associated with decreased amplitudes of wrist temperature (Benjamini-Hochberg FDR q < 0.05) and 26 (6.1%) PheCODEs passed a more stringent significance level (Bonferroni-correction α < 0.05). A two-standard deviation (1.8° Celsius) lower wrist temperature amplitude corresponded to hazard ratios of 1.91 (1.58-2.31 95% CI) for NAFLD, 1.69 (1.53-1.88) for type 2 diabetes, 1.25 (1.14-1.37) for renal failure, 1.23 (1.17-1.3) for hypertension, and 1.22 (1.11-1.33) for pneumonia (phenome-wide atlas available at http://bioinf.itmat.upenn.edu/biorhythm_atlas/ ). This work suggests peripheral thermoregulation as a digital biomarker.

Sexual dimorphism in the response to chronic circadian misalignment on a high-fat diet.

Longitudinal studies associate shiftwork with cardiometabolic disorders but do not establish causation or elucidate mechanisms of disease. We developed a mouse model based on shiftwork schedules to study circadian misalignment in both sexes. Behavioral and transcriptional rhythmicity were preserved in female mice despite exposure to misalignment. Females were protected from the cardiometabolic impact of circadian misalignment on a high-fat diet seen in males. The liver transcriptome and proteome revealed discordant pathway perturbations between the sexes. Tissue-level changes were accompanied by gut microbiome dysbiosis only in male mice, biasing toward increased potential for diabetogenic branched chain amino acid production. Antibiotic ablation of the gut microbiota diminished the impact of misalignment. In the United Kingdom Biobank, females showed stronger circadian rhythmicity in activity and a lower incidence of metabolic syndrome than males among job-matched shiftworkers. Thus, we show that female mice are more resilient than males to chronic circadian misalignment and that these differences are conserved in humans.

Prognostic utility of rhythmic components in 24-hour ambulatory blood pressure monitoring for the risk stratification of chronic kidney disease patients with cardiovascular co-morbidity.

Background: Chronic kidney disease (CKD) represents a significant global burden. Hypertension is a modifiable risk factor for rapid progression of CKD. Methods: We extend the risk stratification by introducing the non-parametric determination of rhythmic components in 24-hour profiles of ambulatory blood pressure monitoring (ABPM) in the African American Study for Kidney Disease and Hypertension (AASK) cohort and the Chronic Renal Insufficiency Cohort (CRIC) using Cox proportional hazards models. Results: We find that rhythmic profiling of BP through JTK_Cycle analysis identifies subgroups of CRIC participants at advanced risk of cardiovascular death. CRIC participants with a history of cardiovascular disease (CVD) and absent cyclic components in their BP profile had at any time a 3.4-times higher risk of cardiovascular death than CVD patients with cyclic components present in their BP profile (HR: 3.38, 95% CI: 1.45-7.88, p=0.005). This substantially increased risk was independent of whether ABPM followed a dipping or non-dipping pattern whereby non-dipping or reverse dipping were not significantly associated with cardiovascular death in patients with prior CVD (p>0.1). In the AASK cohort, unadjusted models demonstrate a higher risk in reaching end stage renal disease among participants without rhythmic ABPM components (HR:1.80, 95% CI: 1.10-2.96); however, full adjustment abolished this association. Conclusions: This study proposes rhythmic blood pressure components as a novel biomarker to unmask excess risk among CKD patients with prior cardiovascular disease.

Deep Phenotyping of the Lipidomic Response in COVID and non-COVID Sepsis.

Lipids may influence cellular penetrance by pathogens and the immune response that they evoke. Here we find a broad based lipidomic storm driven predominantly by secretory (s) phospholipase A 2 (sPLA 2 ) dependent eicosanoid production occurs in patients with sepsis of viral and bacterial origin and relates to disease severity in COVID-19. Elevations in the cyclooxygenase (COX) products of arachidonic acid (AA), PGD 2 and PGI 2 , and the AA lipoxygenase (LOX) product, 12-HETE, and a reduction in the high abundance lipids, ChoE 18:3, LPC-O-16:0 and PC-O-30:0 exhibit relative specificity for COVID-19 amongst such patients, correlate with the inflammatory response and link to disease severity. Linoleic acid (LA) binds directly to SARS-CoV-2 and both LA and its di-HOME products reflect disease severity in COVID-19. AA and LA metabolites and LPC-O-16:0 linked variably to the immune response. These studies yield prognostic biomarkers and therapeutic targets for patients with sepsis, including COVID-19. An interactive purpose built interactive network analysis tool was developed, allowing the community to interrogate connections across these multiomic data and generate novel hypotheses.

Deep phenotyping of the lipidomic response in COVID-19 and non-COVID-19 sepsis.

BACKGROUND: Lipids may influence cellular penetrance by viral pathogens and the immune response that they evoke. We deeply phenotyped the lipidomic response to SARs-CoV-2 and compared that with infection with other pathogens in patients admitted with acute respiratory distress syndrome to an intensive care unit (ICU). METHODS: Mass spectrometry was used to characterise lipids and relate them to proteins, peripheral cell immunotypes and disease severity. RESULTS: Circulating phospholipases (sPLA2, cPLA2 (PLA2G4A) and PLA2G2D) were elevated on admission in all ICU groups. Cyclooxygenase, lipoxygenase and epoxygenase products of arachidonic acid (AA) were elevated in all ICU groups compared with controls. sPLA2 predicted severity in COVID-19 and correlated with TxA2, LTE4 and the isoprostane, iPF2α-III, while PLA2G2D correlated with LTE4. The elevation in PGD2, like PGI2 and 12-HETE, exhibited relative specificity for COVID-19 and correlated with sPLA2 and the interleukin-13 receptor to drive lymphopenia, a marker of disease severity. Pro-inflammatory eicosanoids remained correlated with severity in COVID-19 28 days after admission. Amongst non-COVID ICU patients, elevations in 5- and 15-HETE and 9- and 13-HODE reflected viral rather than bacterial disease. Linoleic acid (LA) binds directly to SARS-CoV-2 and both LA and its di-HOME products reflected disease severity in COVID-19. In healthy marines, these lipids rose with seroconversion. Eicosanoids linked variably to the peripheral cellular immune response. PGE2, TxA2 and LTE4 correlated with T cell activation, as did PGD2 with non-B non-T cell activation. In COVID-19, LPS stimulated peripheral blood mononuclear cell PGF2α correlated with memory T cells, dendritic and NK cells while LA and DiHOMEs correlated with exhausted T cells. Three high abundance lipids - ChoE 18:3, LPC-O-16:0 and PC-O-30:0 - were altered specifically in COVID. LPC-O-16:0 was strongly correlated with T helper follicular cell activation and all three negatively correlated with multi-omic inflammatory pathways and disease severity. CONCLUSIONS: A broad based lipidomic storm is a predictor of poor prognosis in ARDS. Alterations in sPLA2, PGD2 and 12-HETE and the high abundance lipids, ChoE 18:3, LPC-O-16:0 and PC-O-30:0 exhibit relative specificity for COVID-19 amongst such patients and correlate with the inflammatory response to link to disease severity.

Circadian regulation of lung repair and regeneration.

Optimal lung repair and regeneration are essential for recovery from viral infections, including influenza A virus (IAV). We have previously demonstrated that acute inflammation and mortality induced by IAV is under circadian control. However, it is not known whether the influence of the circadian clock persists beyond the acute outcomes. Here, we utilize the UK Biobank to demonstrate an association between poor circadian rhythms and morbidity from lower respiratory tract infections, including the need for hospitalization and mortality after discharge; this persists even after adjusting for common confounding factors. Furthermore, we use a combination of lung organoid assays, single-cell RNA sequencing, and IAV infection in different models of clock disruption to investigate the role of the circadian clock in lung repair and regeneration. We show that lung organoids have a functional circadian clock and the disruption of this clock impairs regenerative capacity. Finally, we find that the circadian clock acts through distinct pathways in mediating lung regeneration - in tracheal cells via the Wnt/ß-catenin pathway and through IL-1ß in alveolar epithelial cells. We speculate that adding a circadian dimension to the critical process of lung repair and regeneration will lead to novel therapies and improve outcomes.

Nitecap: An Exploratory Circadian Analysis Web Application.

Circadian omics analyses present investigators with large amounts of data to consider and many choices for methods of analysis. Visualization is crucial as rhythmicity can take many forms and p-values offer an incomplete picture. Yet statically viewing the entirety of high-throughput datasets is impractical, and there is often limited ability to assess the impact of choices, such as significance threshold cutoffs. Nitecap provides an intuitive and unified web-based solution to these problems. Through highly responsive visualizations, Nitecap enables investigators to see dataset-wide behavior. It supports deep analyses, including comparisons of two conditions. Moreover, it focuses upon ease-of-use and enables collaboration through dataset sharing. As an application, we investigated cross talk between peripheral clocks in adipose and liver tissues and determined that adipocyte clock disruption does not substantially modulate the transcriptional rhythmicity of liver but does advance the phase of core clock gene Bmal1 (Arntl) expression in the liver. Nitecap is available at nitecap.org and is free-to-use.

Accounting for Time: Circadian Rhythms in the Time of COVID-19.

The COVID-19 pandemic has necessitated novel approaches and collaborative efforts across multiple disciplines. It is known that various aspects of our physiology and response to pathogens are under tight clock control. However, the assimilation of circadian biology into our clinical and research practices is still evolving. Using a focused review of the literature and original analyses of the UK Biobank, we discuss how circadian biology may inform our diagnostic and therapeutic strategies in this pandemic.

Loss of circadian protection against influenza infection in adult mice exposed to hyperoxia as neonates.

Adverse early-life exposures have a lasting negative impact on health. Neonatal hyperoxia that is a risk factor for bronchopulmonary dysplasia confers susceptibility to influenza A virus (IAV) infection later in life. Given our previous findings that the circadian clock protects against IAV, we asked if the long-term impact of neonatal hyperoxia vis-à-vis IAV infection includes circadian disruption. Here, we show that neonatal hyperoxia abolishes the clock-mediated time of day protection from IAV in mice, independent of viral burden through host tolerance pathways. We discovered that the lung intrinsic clock (and not the central or immune clocks) mediated this dysregulation. Loss of circadian protein, Bmal1, in alveolar type 2 (AT2) cells recapitulates the increased mortality, loss of temporal gating, and other key features of hyperoxia-exposed animals. Our data suggest a novel role for the circadian clock in AT2 cells in mediating long-term effects of early-life exposures to the lungs.

Targeting MRTF/SRF in CAP2-dependent dilated cardiomyopathy delays disease onset.

About one-third of dilated cardiomyopathy (DCM) cases are caused by mutations in sarcomere or cytoskeletal proteins. However, treating the cytoskeleton directly is not possible because drugs that bind to actin are not well tolerated. Mutations in the actin binding protein CAP2 can cause DCM and KO mice, either whole body (CAP2-KO) or cardiomyocyte-specific KOs (CAP2-CKO) develop DCM with cardiac conduction disease. RNA sequencing analysis of CAP2-KO hearts and isolated cardiomyocytes revealed overactivation of fetal genes, including serum response factor-regulated (SRF-regulated) genes such as Myl9 and Acta2 prior to the emergence of cardiac disease. To test if we could treat CAP2-KO mice, we synthesized and tested the SRF inhibitor CCG-1423-8u. CCG-1423-8u reduced expression of the SRF targets Myl9 and Acta2, as well as the biomarker of heart failure, Nppa. The median survival of CAP2-CKO mice was 98 days, while CCG-1423-8u-treated CKO mice survived for 116 days and also maintained normal cardiac function longer. These results suggest that some forms of sudden cardiac death and cardiac conduction disease are under cytoskeletal stress and that inhibiting signaling through SRF may benefit DCM by reducing cytoskeletal stress.