Evaluation of software impact designed for biomedical research: Are we measuring what's meaningful?

Software is vital for the advancement of biology and medicine. Through analysis of usage and impact metrics of software, developers can help determine user and community engagement. These metrics can be used to justify additional funding, encourage additional use, and identify unanticipated use cases. Such analyses can help define improvement areas and assist with managing project resources. However, there are challenges associated with assessing usage and impact, many of which vary widely depending on the type of software being evaluated. These challenges involve issues of distorted, exaggerated, understated, or misleading metrics, as well as ethical and security concerns. More attention to the nuances, challenges, and considerations involved in capturing impact across the diverse spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have comparatively compelling metrics of high usage. Although some principles are generally applicable, there is not a single perfect metric or approach to effectively evaluate a software tool's impact, as this depends on aspects unique to each tool, how it is used, and how one wishes to evaluate engagement. We propose more broadly applicable guidelines (such as infrastructure that supports the usage of software and the collection of metrics about usage), as well as strategies for various types of software and resources. We also highlight outstanding issues in the field regarding how communities measure or evaluate software impact. To gain a deeper understanding of the issues hindering software evaluations, as well as to determine what appears to be helpful, we performed a survey of participants involved with scientific software projects for the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this scientific community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that although developers recognize the utility of analyzing data related to the impact or usage of their software, they struggle to find the time or funding to support such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of the evaluations of their software so that they can more fully benefit from such assessments.

Open-source Tools for Training Resources - OTTR.

Data science and informatics tools are developing at a blistering rate, but their users often lack the educational background or resources to efficiently apply the methods to their research. Training resources and vignettes that accompany these tools often deprecate because their maintenance is not prioritized by funding, giving teams little time to devote to such endeavors. Our group has developed Open-source Tools for Training Resources (OTTR) to offer greater efficiency and flexibility for creating and maintaining these training resources. OTTR empowers creators to customize their work and allows for a simple workflow to publish using multiple platforms. OTTR allows content creators to publish training material to multiple massive online learner communities using familiar rendering mechanics. OTTR allows the incorporation of pedagogical practices like formative and summative assessments in the form of multiple choice questions and fill in the blank problems that are automatically graded. No local installation of any software is required to begin creating content with OTTR. Thus far, 15 training courses have been created with OTTR repository template. By using the OTTR system, the maintenance workload for updating these courses across platforms has been drastically reduced. For more information about OTTR and how to get started, go to ottrproject.org.

Is a drought a drought in grasslands? Productivity responses to different types of drought.

Drought, defined as a marked deficiency of precipitation relative to normal, occurs as periods of below-average precipitation or complete failure of precipitation inputs, and can be limited to a single season or prolonged over multiple years. Grasslands are typically quite sensitive to drought, but there can be substantial variability in the magnitude of loss of ecosystem function. We hypothesized that differences in how drought occurs may contribute to this variability. In four native Great Plains grasslands (three C4- and one C3-dominated) spanning a ~ 500-mm precipitation gradient, we imposed drought for four consecutive years by (1) reducing each rainfall event by 66% during the growing season (chronic drought) or (2) completely excluding rainfall during a shorter portion of the growing season (intense drought). The drought treatments were similar in magnitude but differed in the following characteristics: event number, event size and length of dry periods. We observed consistent drought-induced reductions (28-37%) in aboveground net primary production (ANPP) only in the C4-dominated grasslands. In general, intense drought reduced ANPP more than chronic drought, with little evidence that drought duration altered this pattern. Conversely, belowground net primary production (BNPP) was reduced by drought in all grasslands (32-64%), with BNPP reductions greater in intense vs. chronic drought treatments in the most mesic grassland. We conclude that grassland productivity responses to drought did not strongly differ between these two types of drought, but when differences existed, intense drought consistently reduced function more than chronic drought.

Resolving the Dust Bowl paradox of grassland responses to extreme drought.

During the 1930s Dust Bowl drought in the central United States, species with the C3 photosynthetic pathway expanded throughout C4-dominated grasslands. This widespread increase in C3 grasses during a decade of low rainfall and high temperatures is inconsistent with well-known traits of C3 vs. C4 pathways. Indeed, water use efficiency is generally lower, and photosynthesis is more sensitive to high temperatures in C3 than C4 species, consistent with the predominant distribution of C3 grasslands in cooler environments and at higher latitudes globally. We experimentally imposed extreme drought for 4 y in mixed C3/C4 grasslands in Kansas and Wyoming and, similar to Dust Bowl observations, also documented three- to fivefold increases in C3/C4 biomass ratios. To explain these paradoxical responses, we first analyzed long-term climate records to show that under nominal conditions in the central United States, C4 grasses dominate where precipitation and air temperature are strongly related (warmest months are wettest months). In contrast, C3 grasses flourish where precipitation inputs are less strongly coupled to warm temperatures. We then show that during extreme drought years, precipitation-temperature relationships weaken, and the proportion of precipitation falling during cooler months increases. This shift in precipitation seasonality provides a mechanism for C3 grasses to respond positively to multiyear drought, resolving the Dust Bowl paradox. Grasslands are globally important biomes and increasingly vulnerable to direct effects of climate extremes. Our findings highlight how extreme drought can indirectly alter precipitation seasonality and shift ecosystem phenology, affecting function in ways not predictable from key traits of C3 and C4 species.

Genetic and functional variation across regional and local scales is associated with climate in a foundational prairie grass.

Global change forecasts in ecosystems require knowledge of within-species diversity, particularly of dominant species within communities. We assessed site-level diversity and capacity for adaptation in Bouteloua gracilis, the dominant species in the Central US shortgrass steppe biome. We quantified genetic diversity from 17 sites across regional scales, north to south from New Mexico to South Dakota, and local scales in northern Colorado. We also quantified phenotype and plasticity within and among sites and determined the extent to which phenotypic diversity in B. gracilis was correlated with climate. Genome sequencing indicated pronounced population structure at the regional scale, and local differences indicated that gene flow and/or dispersal may also be limited. Within a common environment, we found evidence of genetic divergence in biomass-related phenotypes, plasticity, and phenotypic variance, indicating functional divergence and different adaptive potential. Phenotypes were differentiated according to climate, chiefly median Palmer Hydrological Drought Index and other aridity metrics. Our results indicate conclusive differences in genetic variation, phenotype, and plasticity in this species and suggest a mechanism explaining variation in shortgrass steppe community responses to global change. This analysis of B. gracilis intraspecific diversity across spatial scales will improve conservation and management of the shortgrass steppe ecosystem in the future.

Rapid recovery of ecosystem function following extreme drought in a South African savanna grassland.

Climatic extremes, such as severe drought, are expected to increase in frequency and magnitude with climate change. Thus, identifying mechanisms of resilience is critical to predicting the vulnerability of ecosystems. An exceptional drought (

A reality check for climate change experiments: Do they reflect the real world?

Experiments are widely used in ecology, particularly for assessing global change impacts on ecosystem function. However, results from experiments often are inconsistent with observations made under natural conditions, suggesting the need for rigorous comparisons of experimental and observational studies. We conducted such a "reality check" for a grassland ecosystem by compiling results from nine independently conducted climate change experiments. Each experiment manipulated growing season precipitation (GSP) and measured responses in aboveground net primary production (ANPP). We compared results from experiments with long-term (33-yr) annual precipitation and ANPP records to ask if collectively (n = 44 experiment-years) experiments yielded estimates of ANPP, rain-use efficiency (RUE, grams per square meter ANPP per mm precipitation), and the relationship between GSP and ANPP comparable to observations. We found that mean ANPP and RUE from experiments did not deviate from observations. Experiments and observational data also yielded similar functional relationships between ANPP and GSP, but only within the range of historically observed GSP. Fewer experiments imposed extreme levels of GSP (outside the observed 33-yr record), but when these were included, they altered the GSP-ANPP relationship. This result underscores the need for more experiments imposing extreme precipitation levels to resolve how forecast changes in climate regimes will affect ecosystem function in the future.

Codominant grasses differ in gene expression under experimental climate extremes in native tallgrass prairie.

Extremes in climate, such as heat waves and drought, are expected to become more frequent and intense with forecasted climate change. Plant species will almost certainly differ in their responses to these stressors. We experimentally imposed a heat wave and drought in the tallgrass prairie ecosystem near Manhattan, Kansas, USA to assess transcriptional responses of two ecologically important C4 grass species, Andropogon gerardii and Sorghastrum nutans. Based on previous research, we expected that S. nutans would regulate more genes, particularly those related to stress response, under high heat and drought. Across all treatments, S. nutans showed greater expression of negative regulatory and catabolism genes while A. gerardii upregulated cellular and protein metabolism. As predicted, S. nutans showed greater sensitivity to water stress, particularly with downregulation of non-coding RNAs and upregulation of water stress and catabolism genes. A. gerardii was less sensitive to drought, although A. gerardii tended to respond with upregulation in response to drought versus S. nutans which downregulated more genes under drier conditions. Surprisingly, A. gerardii only showed minimal gene expression response to increased temperature, while S. nutans showed no response. Gene functional annotation suggested that these two species may respond to stress via different mechanisms. Specifically, A. gerardii tends to maintain molecular function while S. nutans prioritizes avoidance. Sorghastrum nutans may strategize abscisic acid response and catabolism to respond rapidly to stress. These results have important implications for success of these two important grass species under a more variable and extreme climate forecast for the future.

Gene expression differs in codominant prairie grasses under drought.

Grasslands of the central United States are expected to experience severe droughts and other climate extremes in the future, yet we know little about how these grasses will respond in terms of gene expression. We compared gene expression in Andropogon gerardii and Sorghastrum nutans, two closely related codominant C4 grasses responsible for the majority of ecosystem function, using RNA-seq. We compared Trinity assemblies within each species to determine annotated functions of transcripts responding to drought. Subsequently, we compared homologous annotated gene-groups across the two species using cross-species meta-level analysis and functional clustering based on key terms. The majority of variation was found between species, as opposed to between drought and watered treatments. However, there is evidence for differential responses; Andropogon allocated gene expression differently compared to Sorghastrum, suggesting Andropogon focuses on stress alleviation (such as oxygen radical scavenging) rather than prevention. In contrast, Sorghastrum may employ a drought avoidance strategy by modulating osmotic response, especially with hormonal regulation. We found Sorghastrum tended to be more sensitive within 10 key gene-groups related to stress, abscisic acid and trichomes, suggesting gene expression may mechanistically parallel sensitivity at the physiological level. Our findings corroborate phenotypic and physiological differences in the field and may help explain the phenotypic mechanisms of these two species in the tallgrass prairie community under future drought scenarios.

Gene expression patterns of two dominant tallgrass prairie species differ in response to warming and altered precipitation.

To better understand the mechanisms underlying plant species responses to climate change, we compared transcriptional profiles of the co-dominant C4 grasses, Andropogon gerardii Vitman and Sorghastrum nutans (L.) Nash, in response to increased temperatures and more variable precipitation regimes in a long-term field experiment in native tallgrass prairie. We used microarray probing of a closely related model species (Zea mays) to assess correlations in leaf temperature (Tleaf) and leaf water potential (LWP) and abundance changes of ~10,000 transcripts in leaf tissue collected from individuals of both species. A greater number of transcripts were found to significantly change in abundance levels with Tleaf and LWP in S. nutans than in A. gerardii. S. nutans also was more responsive to short-term drought recovery than A. gerardii. Water flow regulating transcripts associated with stress avoidance (e.g., aquaporins), as well as those involved in the prevention and repair of damage (e.g., antioxidant enzymes, HSPs), were uniquely more abundant in response to increasing Tleaf in S. nutans. The differential transcriptomic responses of the co-dominant C4 grasses suggest that these species may cope with and respond to temperature and water stress at the molecular level in distinct ways, with implications for tallgrass prairie ecosystem function.

Use of ex vitro composite plants to study the interaction of cowpea (Vigna unguiculata L.) with the root parasitic angiosperm Striga gesnerioides.

BACKGROUND: Cowpea (Vigna unguiculata L.) is an important grain and forage legume grown throughout sub-Saharan Africa primarily by subsistence farmers on poor, drought prone soils. Genetic improvement of the crop is being actively pursued and numerous functional genomics studies are underway aimed at characterizing gene controlling key agronomic characteristics for disease and pest resistances. Unfortunately, similar to other legumes, efficient plant transformation technology is a rate-limiting step in analysis of gene function in cowpea. RESULTS: Here we describe an optimized protocol for the rapid generation of transformed hairy roots on ex vitro composite plants of cowpea using Agrobacterium rhizogenes. We further demonstrate the applicability of cowpea composite plants to study gene expression involved in the resistance response of the plant roots to attack by the root parasitic weed, Striga gesnerioides. The utility of the new system and critical parameters of the method are described and discussed herein. CONCLUSIONS: Cowpea composite plants offer a rapid alternative to methods requiring stable transformation and whole plant regeneration for studying gene expression in resistance or susceptibility responses to parasitic weeds. Their use can likely be readily adapted to look at the effects of both ectopic gene overexpression as well as gene knockdown of root associated defense responses and to the study of a broader range of root associated physiological and aphysiological processes including root growth and differentiation as well as interactions with other root pests, parasites, and symbionts.