FST and the triangle inequality for biallelic markers.

The population differentiation statistic FST, introduced by Sewall Wright, is often treated as a pairwise distance measure between populations. As was known to Wright, however, FST is not a true metric because allele frequencies exist for which it does not satisfy the triangle inequality. We prove that a stronger result holds: for biallelic markers whose allele frequencies differ across three populations, FSTnever satisfies the triangle inequality. We study the deviation from the triangle inequality as a function of the allele frequencies of three populations, identifying the frequency vector at which the deviation is maximal. We also examine the implications of the failure of the triangle inequality for four-point conditions for placement of groups of four populations on evolutionary trees. Next, we study the extent to which FST fails to satisfy the triangle inequality in human genomic data, finding that some loci produce deviations near the maximum. We provide results describing the consequences of the theory for various types of data analysis, including multidimensional scaling and inference of neighbor-joining trees from pairwise FST matrices.

On the joint distribution of tree height and tree length under the coalescent.

Many statistics that examine genetic variation depend on the underlying shapes of genealogical trees. Under the coalescent model, we investigate the joint distribution of two quantities that describe genealogical tree shape: tree height and tree length. We derive a recursive formula for their exact joint distribution under a demographic model of a constant-sized population. We obtain approximations for the mean and variance of the ratio of tree height to tree length, using them to show that this ratio converges in probability to 0 as the sample size increases. We find that as the sample size increases, the correlation coefficient for tree height and length approaches (π2-6)∕[π2π2-18]≈0.9340. Using simulations, we examine the joint distribution of height and length under demographic models with population growth and population subdivision. We interpret the joint distribution in relation to problems of interest in data analysis, including inference of the time to the most recent common ancestor. The results assist in understanding the influences of demographic histories on two fundamental features of tree shape.