Best practices to evaluate the impact of biomedical research software-metric collection beyond citations.

MOTIVATION: Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation. RESULTS: To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software's impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress. AVAILABILITY AND IMPLEMENTATION: More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.

Resolving the Dust Bowl paradox of grassland responses to extreme drought.

During the 1930s Dust Bowl drought in the central United States, species with the C3 photosynthetic pathway expanded throughout C4-dominated grasslands. This widespread increase in C3 grasses during a decade of low rainfall and high temperatures is inconsistent with well-known traits of C3 vs. C4 pathways. Indeed, water use efficiency is generally lower, and photosynthesis is more sensitive to high temperatures in C3 than C4 species, consistent with the predominant distribution of C3 grasslands in cooler environments and at higher latitudes globally. We experimentally imposed extreme drought for 4 y in mixed C3/C4 grasslands in Kansas and Wyoming and, similar to Dust Bowl observations, also documented three- to fivefold increases in C3/C4 biomass ratios. To explain these paradoxical responses, we first analyzed long-term climate records to show that under nominal conditions in the central United States, C4 grasses dominate where precipitation and air temperature are strongly related (warmest months are wettest months). In contrast, C3 grasses flourish where precipitation inputs are less strongly coupled to warm temperatures. We then show that during extreme drought years, precipitation-temperature relationships weaken, and the proportion of precipitation falling during cooler months increases. This shift in precipitation seasonality provides a mechanism for C3 grasses to respond positively to multiyear drought, resolving the Dust Bowl paradox. Grasslands are globally important biomes and increasingly vulnerable to direct effects of climate extremes. Our findings highlight how extreme drought can indirectly alter precipitation seasonality and shift ecosystem phenology, affecting function in ways not predictable from key traits of C3 and C4 species.

Is a drought a drought in grasslands? Productivity responses to different types of drought.

Drought, defined as a marked deficiency of precipitation relative to normal, occurs as periods of below-average precipitation or complete failure of precipitation inputs, and can be limited to a single season or prolonged over multiple years. Grasslands are typically quite sensitive to drought, but there can be substantial variability in the magnitude of loss of ecosystem function. We hypothesized that differences in how drought occurs may contribute to this variability. In four native Great Plains grasslands (three C4- and one C3-dominated) spanning a ~ 500-mm precipitation gradient, we imposed drought for four consecutive years by (1) reducing each rainfall event by 66% during the growing season (chronic drought) or (2) completely excluding rainfall during a shorter portion of the growing season (intense drought). The drought treatments were similar in magnitude but differed in the following characteristics: event number, event size and length of dry periods. We observed consistent drought-induced reductions (28-37%) in aboveground net primary production (ANPP) only in the C4-dominated grasslands. In general, intense drought reduced ANPP more than chronic drought, with little evidence that drought duration altered this pattern. Conversely, belowground net primary production (BNPP) was reduced by drought in all grasslands (32-64%), with BNPP reductions greater in intense vs. chronic drought treatments in the most mesic grassland. We conclude that grassland productivity responses to drought did not strongly differ between these two types of drought, but when differences existed, intense drought consistently reduced function more than chronic drought.

Genetic and functional variation across regional and local scales is associated with climate in a foundational prairie grass.

Global change forecasts in ecosystems require knowledge of within-species diversity, particularly of dominant species within communities. We assessed site-level diversity and capacity for adaptation in Bouteloua gracilis, the dominant species in the Central US shortgrass steppe biome. We quantified genetic diversity from 17 sites across regional scales, north to south from New Mexico to South Dakota, and local scales in northern Colorado. We also quantified phenotype and plasticity within and among sites and determined the extent to which phenotypic diversity in B. gracilis was correlated with climate. Genome sequencing indicated pronounced population structure at the regional scale, and local differences indicated that gene flow and/or dispersal may also be limited. Within a common environment, we found evidence of genetic divergence in biomass-related phenotypes, plasticity, and phenotypic variance, indicating functional divergence and different adaptive potential. Phenotypes were differentiated according to climate, chiefly median Palmer Hydrological Drought Index and other aridity metrics. Our results indicate conclusive differences in genetic variation, phenotype, and plasticity in this species and suggest a mechanism explaining variation in shortgrass steppe community responses to global change. This analysis of B. gracilis intraspecific diversity across spatial scales will improve conservation and management of the shortgrass steppe ecosystem in the future.

A reality check for climate change experiments: Do they reflect the real world?

Experiments are widely used in ecology, particularly for assessing global change impacts on ecosystem function. However, results from experiments often are inconsistent with observations made under natural conditions, suggesting the need for rigorous comparisons of experimental and observational studies. We conducted such a "reality check" for a grassland ecosystem by compiling results from nine independently conducted climate change experiments. Each experiment manipulated growing season precipitation (GSP) and measured responses in aboveground net primary production (ANPP). We compared results from experiments with long-term (33-yr) annual precipitation and ANPP records to ask if collectively (n = 44 experiment-years) experiments yielded estimates of ANPP, rain-use efficiency (RUE, grams per square meter ANPP per mm precipitation), and the relationship between GSP and ANPP comparable to observations. We found that mean ANPP and RUE from experiments did not deviate from observations. Experiments and observational data also yielded similar functional relationships between ANPP and GSP, but only within the range of historically observed GSP. Fewer experiments imposed extreme levels of GSP (outside the observed 33-yr record), but when these were included, they altered the GSP-ANPP relationship. This result underscores the need for more experiments imposing extreme precipitation levels to resolve how forecast changes in climate regimes will affect ecosystem function in the future.

Underappreciated problems of low replication in ecological field studies.

The cost and difficulty of manipulative field studies makes low statistical power a pervasive issue throughout most ecological subdisciplines. Ecologists are already aware that small sample sizes increase the probability of committing Type II errors. In this article, we address a relatively unknown problem with low power: underpowered studies must overestimate small effect sizes in order to achieve statistical significance. First, we describe how low replication coupled with weak effect sizes leads to Type M errors, or exaggerated effect sizes. We then conduct a meta-analysis to determine the average statistical power and Type M error rate for manipulative field experiments that address important questions related to global change; global warming, biodiversity loss, and drought. Finally, we provide recommendations for avoiding Type M errors and constraining estimates of effect size from underpowered studies.

The value of long-term ecological research for evolutionary insights.

Scientists must have an integrative understanding of ecology and evolution across spatial and temporal scales to predict how species will respond to global change. Although comprehensively investigating these processes in nature is challenging, the infrastructure and data from long-term ecological research networks can support cross-disciplinary investigations. We propose using these networks to advance our understanding of fundamental evolutionary processes and responses to global change. For ecologists, we outline how long-term ecological experiments can be expanded for evolutionary inquiry, and for evolutionary biologists, we illustrate how observed long-term ecological patterns may motivate new evolutionary questions. We advocate for collaborative, multi-site investigations and discuss barriers to conducting evolutionary work at network sites. Ultimately, these networks offer valuable information and opportunities to improve predictions of species' responses to global change.

Motivation, inclusivity, and realism should drive data science education.

Data science education provides tremendous opportunities but remains inaccessible to many communities. Increasing the accessibility of data science to these communities not only benefits the individuals entering data science, but also increases the field's innovation and potential impact as a whole. Education is the most scalable solution to meet these needs, but many data science educators lack formal training in education. Our group has led education efforts for a variety of audiences: from professional scientists to high school students to lay audiences. These experiences have helped form our teaching philosophy which we have summarized into three main ideals: 1) motivation, 2) inclusivity, and 3) realism. 20 we also aim to iteratively update our teaching approaches and curriculum as we find ways to better reach these ideals. In this manuscript we discuss these ideals as well practical ideas for how to implement these philosophies in the classroom.

Open-source Tools for Training Resources - OTTR.

Data science and informatics tools are developing at a blistering rate, but their users often lack the educational background or resources to efficiently apply the methods to their research. Training resources and vignettes that accompany these tools often deprecate because their maintenance is not prioritized by funding, giving teams little time to devote to such endeavors. Our group has developed Open-source Tools for Training Resources (OTTR) to offer greater efficiency and flexibility for creating and maintaining these training resources. OTTR empowers creators to customize their work and allows for a simple workflow to publish using multiple platforms. OTTR allows content creators to publish training material to multiple massive online learner communities using familiar rendering mechanics. OTTR allows the incorporation of pedagogical practices like formative and summative assessments in the form of multiple choice questions and fill in the blank problems that are automatically graded. No local installation of any software is required to begin creating content with OTTR. Thus far, 15 training courses have been created with OTTR repository template. By using the OTTR system, the maintenance workload for updating these courses across platforms has been drastically reduced. For more information about OTTR and how to get started, go to ottrproject.org.

Evaluation of software impact designed for biomedical research: Are we measuring what's meaningful?

Software is vital for the advancement of biology and medicine. Through analysis of usage and impact metrics of software, developers can help determine user and community engagement. These metrics can be used to justify additional funding, encourage additional use, and identify unanticipated use cases. Such analyses can help define improvement areas and assist with managing project resources. However, there are challenges associated with assessing usage and impact, many of which vary widely depending on the type of software being evaluated. These challenges involve issues of distorted, exaggerated, understated, or misleading metrics, as well as ethical and security concerns. More attention to the nuances, challenges, and considerations involved in capturing impact across the diverse spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have comparatively compelling metrics of high usage. Although some principles are generally applicable, there is not a single perfect metric or approach to effectively evaluate a software tool's impact, as this depends on aspects unique to each tool, how it is used, and how one wishes to evaluate engagement. We propose more broadly applicable guidelines (such as infrastructure that supports the usage of software and the collection of metrics about usage), as well as strategies for various types of software and resources. We also highlight outstanding issues in the field regarding how communities measure or evaluate software impact. To gain a deeper understanding of the issues hindering software evaluations, as well as to determine what appears to be helpful, we performed a survey of participants involved with scientific software projects for the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this scientific community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that although developers recognize the utility of analyzing data related to the impact or usage of their software, they struggle to find the time or funding to support such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of the evaluations of their software so that they can more fully benefit from such assessments.

Rapid recovery of ecosystem function following extreme drought in a South African savanna grassland.

Climatic extremes, such as severe drought, are expected to increase in frequency and magnitude with climate change. Thus, identifying mechanisms of resilience is critical to predicting the vulnerability of ecosystems. An exceptional drought (

Gene expression differs in codominant prairie grasses under drought.

Grasslands of the central United States are expected to experience severe droughts and other climate extremes in the future, yet we know little about how these grasses will respond in terms of gene expression. We compared gene expression in Andropogon gerardii and Sorghastrum nutans, two closely related codominant C4 grasses responsible for the majority of ecosystem function, using RNA-seq. We compared Trinity assemblies within each species to determine annotated functions of transcripts responding to drought. Subsequently, we compared homologous annotated gene-groups across the two species using cross-species meta-level analysis and functional clustering based on key terms. The majority of variation was found between species, as opposed to between drought and watered treatments. However, there is evidence for differential responses; Andropogon allocated gene expression differently compared to Sorghastrum, suggesting Andropogon focuses on stress alleviation (such as oxygen radical scavenging) rather than prevention. In contrast, Sorghastrum may employ a drought avoidance strategy by modulating osmotic response, especially with hormonal regulation. We found Sorghastrum tended to be more sensitive within 10 key gene-groups related to stress, abscisic acid and trichomes, suggesting gene expression may mechanistically parallel sensitivity at the physiological level. Our findings corroborate phenotypic and physiological differences in the field and may help explain the phenotypic mechanisms of these two species in the tallgrass prairie community under future drought scenarios.

Codominant grasses differ in gene expression under experimental climate extremes in native tallgrass prairie.

Extremes in climate, such as heat waves and drought, are expected to become more frequent and intense with forecasted climate change. Plant species will almost certainly differ in their responses to these stressors. We experimentally imposed a heat wave and drought in the tallgrass prairie ecosystem near Manhattan, Kansas, USA to assess transcriptional responses of two ecologically important C4 grass species, Andropogon gerardii and Sorghastrum nutans. Based on previous research, we expected that S. nutans would regulate more genes, particularly those related to stress response, under high heat and drought. Across all treatments, S. nutans showed greater expression of negative regulatory and catabolism genes while A. gerardii upregulated cellular and protein metabolism. As predicted, S. nutans showed greater sensitivity to water stress, particularly with downregulation of non-coding RNAs and upregulation of water stress and catabolism genes. A. gerardii was less sensitive to drought, although A. gerardii tended to respond with upregulation in response to drought versus S. nutans which downregulated more genes under drier conditions. Surprisingly, A. gerardii only showed minimal gene expression response to increased temperature, while S. nutans showed no response. Gene functional annotation suggested that these two species may respond to stress via different mechanisms. Specifically, A. gerardii tends to maintain molecular function while S. nutans prioritizes avoidance. Sorghastrum nutans may strategize abscisic acid response and catabolism to respond rapidly to stress. These results have important implications for success of these two important grass species under a more variable and extreme climate forecast for the future.

Gene expression patterns of two dominant tallgrass prairie species differ in response to warming and altered precipitation.

To better understand the mechanisms underlying plant species responses to climate change, we compared transcriptional profiles of the co-dominant C4 grasses, Andropogon gerardii Vitman and Sorghastrum nutans (L.) Nash, in response to increased temperatures and more variable precipitation regimes in a long-term field experiment in native tallgrass prairie. We used microarray probing of a closely related model species (Zea mays) to assess correlations in leaf temperature (Tleaf) and leaf water potential (LWP) and abundance changes of ~10,000 transcripts in leaf tissue collected from individuals of both species. A greater number of transcripts were found to significantly change in abundance levels with Tleaf and LWP in S. nutans than in A. gerardii. S. nutans also was more responsive to short-term drought recovery than A. gerardii. Water flow regulating transcripts associated with stress avoidance (e.g., aquaporins), as well as those involved in the prevention and repair of damage (e.g., antioxidant enzymes, HSPs), were uniquely more abundant in response to increasing Tleaf in S. nutans. The differential transcriptomic responses of the co-dominant C4 grasses suggest that these species may cope with and respond to temperature and water stress at the molecular level in distinct ways, with implications for tallgrass prairie ecosystem function.

Use of ex vitro composite plants to study the interaction of cowpea (Vigna unguiculata L.) with the root parasitic angiosperm Striga gesnerioides.

BACKGROUND: Cowpea (Vigna unguiculata L.) is an important grain and forage legume grown throughout sub-Saharan Africa primarily by subsistence farmers on poor, drought prone soils. Genetic improvement of the crop is being actively pursued and numerous functional genomics studies are underway aimed at characterizing gene controlling key agronomic characteristics for disease and pest resistances. Unfortunately, similar to other legumes, efficient plant transformation technology is a rate-limiting step in analysis of gene function in cowpea. RESULTS: Here we describe an optimized protocol for the rapid generation of transformed hairy roots on ex vitro composite plants of cowpea using Agrobacterium rhizogenes. We further demonstrate the applicability of cowpea composite plants to study gene expression involved in the resistance response of the plant roots to attack by the root parasitic weed, Striga gesnerioides. The utility of the new system and critical parameters of the method are described and discussed herein. CONCLUSIONS: Cowpea composite plants offer a rapid alternative to methods requiring stable transformation and whole plant regeneration for studying gene expression in resistance or susceptibility responses to parasitic weeds. Their use can likely be readily adapted to look at the effects of both ectopic gene overexpression as well as gene knockdown of root associated defense responses and to the study of a broader range of root associated physiological and aphysiological processes including root growth and differentiation as well as interactions with other root pests, parasites, and symbionts.