Assessing model accuracy using random data split: a simulation study.

Randomization is considered a safeguard against bias and a gold standard in clinical studies. To assess the generalizability of the accuracy of a model, a common approach is to randomly split a master data set into two parts: one for training and the other for testing. In this paper, we demonstrated the limitations of random split in assessing the generalizability of the accuracy of models through simulation studies. We generated three simulation data for binary or continuous endpoints, each with large sample size (n = 10,000). In each simulation scenario, we randomly split the data into two, one for training and one for testing, and then compare the performance of the model between training and testing data. All simulations were repeated 1,000 times. When random split was used, the model performance based on training and testing data behaved similarly in terms of the true positive fraction and false positive fraction for binary data and mean-squared errors for continuous data. However, when there is a time drift effect in the data, random split will result in large differences between training and testing data. As the training and testing data are similar through a random split, assessing the generalizability of the model on similar data will generate similar results. Generalizability of the accuracy of models is thus best achieved if testing is done in a distinct and independent study.

Stepping-stone spatial structure causes slow decay of linkage disequilibrium and shifts the site frequency spectrum.

The symmetric island model with D demes and equal migration rates is often chosen for the investigation of the consequences of population subdivision. Here we show that a stepping-stone model has a more pronounced effect on the genealogy of a sample. For samples from a small geographical region commonly used in genetic studies of humans and Drosophila, there is a shift of the frequency spectrum that decreases the number of low-frequency-derived alleles and skews the distribution of statistics of Tajima, Fu and Li, and Fay and Wu. Stepping-stone spatial structure also changes the two-locus sampling distribution and increases both linkage disequilibrium and the probability that two sites are perfectly correlated. This may cause a false prediction of cold spots of recombination and may confuse haplotype tests that compute probabilities on the basis of a homogeneously mixing population.

Evaluation of heart failure biomarker tests: a survey of statistical considerations.

Biomarkers assessing cardiovascular function can encompass a wide range of biochemical or physiological measurements. Medical tests that measure biomarkers are typically evaluated for measurement validation and clinical performance in the context of their intended use. General statistical principles for the evaluation of medical tests are discussed in this paper in the context of heart failure. Statistical aspects of study design and analysis to be considered while assessing the quality of measurements and the clinical performance of tests are highlighted. A discussion of statistical considerations for specific clinical uses is also provided. The remarks in this paper mainly focus on methods and considerations for statistical evaluation of medical tests from the perspective of bias and precision. With such an evaluation of performance, healthcare professionals could have information that leads to a better understanding on the strengths and limitations of tests related to heart failure.