Hill-Chao numbers allow decomposing gamma multifunctionality into alpha and beta components.

Biodiversity-ecosystem functioning (BEF) research has provided strong evidence and mechanistic underpinnings to support positive effects of biodiversity on ecosystem functioning, from single to multiple functions. This research has provided knowledge gained mainly at the local alpha scale (i.e. within ecosystems), but the increasing homogenization of landscapes in the Anthropocene has raised the potential that declining biodiversity at the beta (across ecosystems) and gamma scales is likely to also impact ecosystem functioning. Drawing on biodiversity theory, we propose a new statistical framework based on Hill-Chao numbers. The framework allows decomposition of multifunctionality at gamma scales into alpha and beta components, a critical but hitherto missing tool in BEF research; it also allows weighting of individual ecosystem functions. Through the proposed decomposition, new BEF results for beta and gamma scales are discovered. Our novel approach is applicable across ecosystems and connects local- and landscape-scale BEF assessments from experiments to natural settings.

Mapping wild vascular plant species diversity in urban areas in California using crowdsourcing data by regression kriging: Examining socioeconomic disparities.

Biodiversity is crucial for human health, but previous methods of measuring biodiversity require intensive resources and have other limitations. Crowdsourced datasets from citizen scientists offer a cost-effective solution for characterizing biodiversity on a large spatial scale. This study has two aims: 1) to generate fine-resolution plant species diversity maps in California urban areas using crowdsourced data and extrapolation methods; and 2) to examine their associations with sociodemographic factors and identify subpopulations with low biodiversity exposure. We used iNaturalist observations from 2019 to 2022 to calculate species diversity metrics by exploring the sampling completeness in a 5 × 5-km2 grid and then computing species diversity metrics for grid cells with at least 80 % sample completeness (841 out of 4755 grid cells). A generalized additive model with ordinary kriging (GAM OK) provided moderately reliable estimates, with correlations of 0.64-0.66 between observed and extrapolated metrics, relative mean absolute errors of 21 %-23 %, and relative root mean squared errors of 27 %-30 % for grid cells with ≥80 % sample completeness from 10-fold cross-validation. GAM OK was further applied to extrapolate species diversity metrics from saturated grid cells (N = 841) to the remaining grid cells with <80 % sample completeness (N = 3914) and generate diversity maps that cover the grid. Further, generalized linear mixed models were used to examine the associations between species diversity and sociodemographic indicators at census tract level. The wild vascular plant species diversity metrics were inversely associated with neighborhood socioeconomic status (i.e., unemployment, linguistic isolation, educational attainment, and poverty rate). Minority populations (i.e., African American, Asian American, and Hispanic) and children had significantly lower diversity exposure in their neighborhoods. Crowdsourcing data offers a cost-effective solution for characterizing large-scale biodiversity in urban areas.

Sample coverage estimation, rarefaction, and extrapolation based on sample-based abundance data.

Sample coverage, the proportion of individuals that belong to observed species in a sample, is a metric used to measure the completeness of a sample. Rather than using equal sample sizes, equal sample coverage has become a widely accepted standard for comparing diversity across multiple assemblages, resulting in a more accurate representation of the true relationship between the richness of the assemblages. In practice, sample-based abundance data are the most frequently used data type for evaluating species diversity. In sample-based abundance data, the sampling unit (e.g., a plot, net, trap, or transect) is randomly selected from the target area, and the number of individuals for each species observed in the sampled unit is recorded. In this case, the individuals in the sample are no longer randomly and independently sampled, and the Good-Turing estimators of abundance-based sample coverage in reference, rarefied, and extrapolated samples may be severely biased when individuals present a highly spatially aggregated pattern. Here, I derive a novel estimator of abundance-based sample coverage based on the Good-Turing frequency formula. Additionally, a new analytical approach is introduced for enabling smooth coverage-based rarefaction and extrapolation to compare richness among assemblages. The near unbiasedness of the proposed estimator and a less biased richness ratio achieved using the newly developed coverage-based standardizing approach are demonstrated by analyzing three ForestGEO permanent forest plot data sets.

Quantifying and estimating ecological network diversity based on incomplete sampling data.

An ecological network refers to the ecological interactions among sets of species. Quantification of ecological network diversity and related sampling/estimation challenges have explicit analogues in species diversity research. A unified framework based on Hill numbers and their generalizations was developed to quantify taxonomic, phylogenetic and functional diversity. Drawing on this unified framework, we propose three dimensions of network diversity that incorporate the frequency (or strength) of interactions, species phylogenies and traits. As with surveys in species inventories, nearly all network studies are based on sampling data and thus also suffer from under-sampling effects. Adapting the sampling/estimation theory and the iNEXT (interpolation/extrapolation) standardization developed for species diversity research, we propose the iNEXT.link method to analyse network sampling data. The proposed method integrates the following four inference procedures: (i) assessment of sample completeness of networks; (ii) asymptotic analysis via estimating the true network diversity; (iii) non-asymptotic analysis based on standardizing sample completeness via rarefaction and extrapolation with network diversity; and (iv) estimation of the degree of unevenness or specialization in networks based on standardized diversity. Interaction data between European trees and saproxylic beetles are used to illustrate the proposed procedures. The software iNEXT.link has been developed to facilitate all computations and graphics. This article is part of the theme issue 'Detecting and attributing the causes of biodiversity change: needs, gaps and solutions'.

A more reliable species richness estimator based on the Gamma-Poisson model.

Background: Accurately estimating the true richness of a target community is still a statistical challenge, particularly in highly diverse communities. Due to sampling limitations or limited resources, undetected species are present in many surveys and observed richness is an underestimate of true richness. In the literature, methods for estimating the undetected richness of a sample are generally divided into two categories: parametric and nonparametric estimators. Imposing no assumptions on species detection rates, nonparametric methods demonstrate robust statistical performance and are widely used in ecological studies. However, nonparametric estimators may seriously underestimate richness when species composition has a high degree of heterogeneity. Parametric approaches, which reduce the number of parameters by assuming that species-specific detection probabilities follow a given statistical distribution, use traditional statistical inference to calculate species richness estimates. When species detection rates meet the model assumption, the parametric approach could supply a nearly unbiased estimator. However, the infeasibility and inefficiency of solving maximum likelihood functions limit the application of parametric methods in ecological studies when the model assumption is violated, or the collected data is sparse. Method: To overcome these estimating challenges associated with parametric methods, an estimator employing the moment estimation method instead of the maximum likelihood estimation method is proposed to estimate parameters based on a Gamma-Poisson mixture model. Drawing on the concept of the Good-Turing frequency formula, the proposed estimator only uses the number of singletons, doubletons, and tripletons in a sample for undetected richness estimation. Results: The statistical behavior of the new estimator was evaluated by using real and simulated data sets from various species abundance models. Simulation results indicated that the new estimator reduces the bias presented in traditional nonparametric estimators, presents more robust statistical behavior compared to other parametric estimators, and provides confidence intervals with better coverage among the discussed estimators, especially in assemblages with high species composition heterogeneity.

Diversity from genes to ecosystems: A unifying framework to study variation across biological metrics and scales.

Biological diversity is a key concept in the life sciences and plays a fundamental role in many ecological and evolutionary processes. Although biodiversity is inherently a hierarchical concept covering different levels of organization (genes, population, species, ecological communities and ecosystems), a diversity index that behaves consistently across these different levels has so far been lacking, hindering the development of truly integrative biodiversity studies. To fill this important knowledge gap, we present a unifying framework for the measurement of biodiversity across hierarchical levels of organization. Our weighted, information-based decomposition framework is based on a Hill number of order q = 1, which weights all elements in proportion to their frequency and leads to diversity measures based on Shannon's entropy. We investigated the numerical behaviour of our approach with simulations and showed that it can accurately describe complex spatial hierarchical structures. To demonstrate the intuitive and straightforward interpretation of our diversity measures in terms of effective number of components (alleles, species, etc.), we applied the framework to a real data set on coral reef biodiversity. We expect our framework will have multiple applications covering the fields of conservation biology, community genetics and eco-evolutionary dynamics.

Deciphering the enigma of undetected species, phylogenetic, and functional diversity based on Good-Turing theory.

Estimating the species, phylogenetic, and functional diversity of a community is challenging because rare species are often undetected, even with intensive sampling. The Good-Turing frequency formula, originally developed for cryptography, estimates in an ecological context the true frequencies of rare species in a single assemblage based on an incomplete sample of individuals. Until now, this formula has never been used to estimate undetected species, phylogenetic, and functional diversity. Here, we first generalize the Good-Turing formula to incomplete sampling of two assemblages. The original formula and its two-assemblage generalization provide a novel and unified approach to notation, terminology, and estimation of undetected biological diversity. For species richness, the Good-Turing framework offers an intuitive way to derive the non-parametric estimators of the undetected species richness in a single assemblage, and of the undetected species shared between two assemblages. For phylogenetic diversity, the unified approach leads to an estimator of the undetected Faith's phylogenetic diversity (PD, the total length of undetected branches of a phylogenetic tree connecting all species), as well as a new estimator of undetected PD shared between two phylogenetic trees. For functional diversity based on species traits, the unified approach yields a new estimator of undetected Walker et al.'s functional attribute diversity (FAD, the total species-pairwise functional distance) in a single assemblage, as well as a new estimator of undetected FAD shared between two assemblages. Although some of the resulting estimators have been previously published (but derived with traditional mathematical inequalities), all taxonomic, phylogenetic, and functional diversity estimators are now derived under the same framework. All the derived estimators are theoretically lower bounds of the corresponding undetected diversities; our approach reveals the sufficient conditions under which the estimators are nearly unbiased, thus offering new insights. Simulation results are reported to numerically verify the performance of the derived estimators. We illustrate all estimators and assess their sampling uncertainty with an empirical dataset for Brazilian rain forest trees. These estimators should be widely applicable to many current problems in ecology, such as the effects of climate change on spatial and temporal beta diversity and the contribution of trait diversity to ecosystem multi-functionality.

Environment-induced changes in selective constraints on social learning during the peopling of the Americas.

The weaponry technology associated with Clovis and related Early Paleoindians represents the earliest well-defined evidence of humans in Pleistocene North America. We assess the technological diversity of these fluted stone points found at archaeological sites in the western and eastern halves of North America by employing statistical tools used in the quantification of ecological biodiversity. Our results demonstrate that the earliest hunters in the environmentally heterogeneous East used a more diverse set of points than those in the environmentally homogenous West. This and other evidence shows that environmental heterogeneity in the East promoted the relaxation of selective constraints on social learning and increased experimentation with point designs.

Estimating and comparing microbial diversity in the presence of sequencing errors.

Estimating and comparing microbial diversity are statistically challenging due to limited sampling and possible sequencing errors for low-frequency counts, producing spurious singletons. The inflated singleton count seriously affects statistical analysis and inferences about microbial diversity. Previous statistical approaches to tackle the sequencing errors generally require different parametric assumptions about the sampling model or about the functional form of frequency counts. Different parametric assumptions may lead to drastically different diversity estimates. We focus on nonparametric methods which are universally valid for all parametric assumptions and can be used to compare diversity across communities. We develop here a nonparametric estimator of the true singleton count to replace the spurious singleton count in all methods/approaches. Our estimator of the true singleton count is in terms of the frequency counts of doubletons, tripletons and quadrupletons, provided these three frequency counts are reliable. To quantify microbial alpha diversity for an individual community, we adopt the measure of Hill numbers (effective number of taxa) under a nonparametric framework. Hill numbers, parameterized by an order q that determines the measures' emphasis on rare or common species, include taxa richness (q = 0), Shannon diversity (q = 1, the exponential of Shannon entropy), and Simpson diversity (q = 2, the inverse of Simpson index). A diversity profile which depicts the Hill number as a function of order q conveys all information contained in a taxa abundance distribution. Based on the estimated singleton count and the original non-singleton frequency counts, two statistical approaches (non-asymptotic and asymptotic) are developed to compare microbial diversity for multiple communities. (1) A non-asymptotic approach refers to the comparison of estimated diversities of standardized samples with a common finite sample size or sample completeness. This approach aims to compare diversity estimates for equally-large or equally-complete samples; it is based on the seamless rarefaction and extrapolation sampling curves of Hill numbers, specifically for q = 0, 1 and 2. (2) An asymptotic approach refers to the comparison of the estimated asymptotic diversity profiles. That is, this approach compares the estimated profiles for complete samples or samples whose size tends to be sufficiently large. It is based on statistical estimation of the true Hill number of any order q ≥ 0. In the two approaches, replacing the spurious singleton count by our estimated count, we can greatly remove the positive biases associated with diversity estimates due to spurious singletons and also make fair comparisons across microbial communities, as illustrated in our simulation results and in applying our method to analyze sequencing data from viral metagenomes.

Distance-based functional diversity measures and their decomposition: a framework based on Hill numbers.

Hill numbers (or the "effective number of species") are increasingly used to characterize species diversity of an assemblage. This work extends Hill numbers to incorporate species pairwise functional distances calculated from species traits. We derive a parametric class of functional Hill numbers, which quantify "the effective number of equally abundant and (functionally) equally distinct species" in an assemblage. We also propose a class of mean functional diversity (per species), which quantifies the effective sum of functional distances between a fixed species to all other species. The product of the functional Hill number and the mean functional diversity thus quantifies the (total) functional diversity, i.e., the effective total distance between species of the assemblage. The three measures (functional Hill numbers, mean functional diversity and total functional diversity) quantify different aspects of species trait space, and all are based on species abundance and species pairwise functional distances. When all species are equally distinct, our functional Hill numbers reduce to ordinary Hill numbers. When species abundances are not considered or species are equally abundant, our total functional diversity reduces to the sum of all pairwise distances between species of an assemblage. The functional Hill numbers and the mean functional diversity both satisfy a replication principle, implying the total functional diversity satisfies a quadratic replication principle. When there are multiple assemblages defined by the investigator, each of the three measures of the pooled assemblage (gamma) can be multiplicatively decomposed into alpha and beta components, and the two components are independent. The resulting beta component measures pure functional differentiation among assemblages and can be further transformed to obtain several classes of normalized functional similarity (or differentiation) measures, including N-assemblage functional generalizations of the classic Jaccard, Sørensen, Horn and Morisita-Horn similarity indices. The proposed measures are applied to artificial and real data for illustration.

An improved nonparametric lower bound of species richness via a modified good-turing frequency formula.

It is difficult to accurately estimate species richness if there are many almost undetectable species in a hyper-diverse community. Practically, an accurate lower bound for species richness is preferable to an inaccurate point estimator. The traditional nonparametric lower bound developed by Chao (1984, Scandinavian Journal of Statistics 11, 265-270) for individual-based abundance data uses only the information on the rarest species (the numbers of singletons and doubletons) to estimate the number of undetected species in samples. Applying a modified Good-Turing frequency formula, we derive an approximate formula for the first-order bias of this traditional lower bound. The approximate bias is estimated by using additional information (namely, the numbers of tripletons and quadrupletons). This approximate bias can be corrected, and an improved lower bound is thus obtained. The proposed lower bound is nonparametric in the sense that it is universally valid for any species abundance distribution. A similar type of improved lower bound can be derived for incidence data. We test our proposed lower bounds on simulated data sets generated from various species abundance models. Simulation results show that the proposed lower bounds always reduce bias over the traditional lower bounds and improve accuracy (as measured by mean squared error) when the heterogeneity of species abundances is relatively high. We also apply the proposed new lower bounds to real data for illustration and for comparisons with previously developed estimators.

Proposing a resolution to debates on diversity partitioning.

There have been intense debates about the decomposition of regional diversity (gamma) into its within-community component (alpha) and between-community component (beta). Although a recent Ecology Forum achieved consensus in the use of "numbers equivalents" (Hill numbers) as the proper choice of diversity measure, three related major issues were still left unresolved. (1) What is the precise meaning of the "independence" or "statistical independence" of alpha diversity and beta diversity? (2) Which partitioning (additive vs. multiplicative) should be used for a given application? (3) What is the proper formula for alpha diversity, as there are two formulas in the literature? This paper proposes a possible resolution to each of these issues. For the first issue, we clarify the definitions of "independence" and "statistical independence" from two perspectives so that confusion about this issue can be cleared up. We also discuss the causes of dependence, so that the dependence relationship between any two diversity components in both partitioning schemes can be rigorously justified by theory and also intuitively understood by simulation. For the second issue, both multiplicative and additive beta diversities based on Hill numbers are useful measures and quantify different aspects of communities. However, neither can be directly applied to compare relative compositional similarity or differentiation across multiple regions with different numbers of communities because multiplicative beta diversity depends on the number of communities, and additive beta diversity additionally depends on alpha (equivalently, on gamma). Such dependences should be removed. We propose transformations to remove these dependences, and we show that the transformed multiplicative beta and additive beta both lead to the same classes of measures, which are always in a range of [0, 1] and thus can be used to compare relative similarity or differentiation among communities across multiple regions. These similarity measures include multiple-community generalizations of the Sørenson, Jaccard, Horn, and Morisita-Horn measures. For the third issue, we present some observations including a finding about which alpha formula produces independent alpha and beta components. These may help to resolve the choice of a proper formula for alpha diversity. Some related issues are also briefly discussed.

Phylogenetic diversity measures based on Hill numbers.

We propose a parametric class of phylogenetic diversity (PD) measures that are sensitive to both species abundance and species taxonomic or phylogenetic distances. This work extends the conventional parametric species-neutral approach (based on 'effective number of species' or Hill numbers) to take into account species relatedness, and also generalizes the traditional phylogenetic approach (based on 'total phylogenetic length') to incorporate species abundances. The proposed measure quantifies 'the mean effective number of species' over any time interval of interest, or the 'effective number of maximally distinct lineages' over that time interval. The product of the measure and the interval length quantifies the 'branch diversity' of the phylogenetic tree during that interval. The new measures generalize and unify many existing measures and lead to a natural definition of taxonomic diversity as a special case. The replication principle (or doubling property), an important requirement for species-neutral diversity, is generalized to PD. The widely used Rao's quadratic entropy and the phylogenetic entropy do not satisfy this essential property, but a simple transformation converts each to our measures, which do satisfy the property. The proposed approach is applied to forest data for interpreting the effects of thinning.