Triple challenges - Small sample size in both exposure and control groups to scan rare maternal outcomes in a signal identification approach: A simulation study.

There is a dearth of safety data on maternal outcomes after perinatal medication exposure. Data-mining for unexpected adverse event occurrence in existing datasets is a potentially useful approach. One method, the Poisson tree-based scan statistic (TBSS), assumes that the expected outcome counts, based on incidence of outcomes in the control group, are estimated without error. This assumption may be difficult to satisfy with a small control group. Our simulation study evaluated the effect of imprecise incidence proportions from the control group on TBSS' ability to identify maternal outcomes in pregnancy research. We simulated base case analyses with "true" expected incidence proportions and compared these to imprecise incidence proportions derived from sparse control samples. We varied parameters impacting Type I error and statistical power (exposure group size, outcome's incidence proportion, and effect size). We found that imprecise incidence proportions generated by a small control group resulted in inaccurate alerting, inflation of Type I error, and removal of very rare outcomes for TBSS analysis due to "zero" background counts. Ideally, the control size should be at least several times larger than the exposure size to limit the number of false positive alerts and retain statistical power for true alerts.

Perspective on statistical power and equivalence tests.

The use of both sexes or genders should be considered in experimental design, analysis, and reporting. Since there is no requirement to double the sample size or to have sufficient power to study sex differences, challenges for the statistical analysis can arise. In this article, we focus on the topics of statistical power and ways to increase this power. We also discuss the choice of an appropriate design and statistical method and include a separate section on equivalence tests needed to show the absence of a relevant difference.

Effect of using the Simple Erosion Narrowing Score or Sharp/van der Heijde score on power of rheumatoid arthritis clinical trials to detect differences in radiographic progression.

OBJECTIVES: The Simple Erosion Narrowing Score (SENS) is a simplification of the Sharp/van der Heijde score (SHS). Previous studies found SENS and SHS to have very similar measurement properties, but suggest that SENS has a lower discriminative ability that may result in reduced power. Therefore, we aimed to quantify the effect of using SENS rather than SHS on the power to show between-group differences in radiographic progression. METHODS: Using data from two clinical trials in rheumatoid arthritis (DRESS and BeSt), SENS was derived from the SHS. Criterion validity of the SENS in relation to the SHS was assessed by calculating the Spearman correlation. The power of both scores to show a difference between groups was compared using bootstrapping to generate 10.000 replications of each study. Then, the number of replications with a significant difference in progression (using ANCOVA adjusted for baseline scores) were compared. RESULTS: Correlations between SENS and SHS were all >0.9, indicating high criterion validity of SENS compared with SHS as a reference standard. There was one exception, the DRESS study showed a somewhat lower correlation for the change score at 18 months (0.787). The loss in power of SENS over SHS was limited to at most 19% (BeSt year 5). In addition, the difference in power between SENS and SHS is smaller at higher levels of power. CONCLUSION: SENS appears to be a reasonable alternative to SHS, with only a limited loss of power to show between-group differences in radiographic progression.

Using Alternative Definitions of Controls to Increase Statistical Power in GWAS.

Genome-wide association studies (GWAS) are often underpowered due to small effect sizes of common single nucleotide polymorphisms (SNPs) on phenotypes and extreme multiple testing thresholds. The most common approach for increasing statistical power is to increase sample size. We propose an alternative strategy of redefining case-control outcomes into ordinal case-subthreshold-asymptomatic variables. While maintaining the clinical case threshold, we subdivide controls into two groups: individuals who are symptomatic but do not meet the clinical criteria for diagnosis (subthreshold) and individuals who are effectively asymptomatic. We conducted a simulation study to examine the impact of effect size, minor allele frequency, population prevalence, and the prevalence of the subthreshold group on statistical power to detect genetic associations in three scenarios: a standard case-control, an ordinal, and a case-asymptomatic control analysis. Our results suggest the ordinal model consistently provides the greatest statistical power while the case-control model the least. Power in the case-asymptomatic control model reflects the case-control or ordinal model depending on the population prevalence and size of the subthreshold category. We then analyzed a major depression phenotype from the UK Biobank to corroborate our simulation results. Overall, the ordinal model improves statistical power in GWAS consistent with increasing the sample size by approximately 10%.

Comparison of Pocock and Simon's covariate-adaptive randomization procedures in clinical trials.

When multiple influential covariates need to be balanced during a clinical trial, stratified blocked randomization and covariate-adaptive randomization procedures are frequently used in trials to prevent bias and enhance the validity of data analysis results. The latter approach is increasingly used in practice for a study with multiple covariates and limited sample sizes. Among a group of these approaches, the covariate-adaptive procedures proposed by Pocock and Simon are straightforward to be utilized in practice. We aim to investigate the optimal design parameters for the patient treatment assignment probability of their developed three methods. In addition, we seek to answer the question related to the randomization performance when additional covariates are added to the existing randomization procedure. We conducted extensive simulation studies to address these practically important questions.

Power analyses for measurement model misspecification and response shift detection with structural equation modeling.

PURPOSE: Statistical power for response shift detection with structural equation modeling (SEM) is currently underreported. The present paper addresses this issue by providing worked-out examples and syntaxes of power calculations relevant for the statistical tests associated with the SEM approach for response shift detection. METHODS: Power calculations and related sample-size requirements are illustrated for two modelling goals: (1) to detect misspecification in the measurement model, and (2) to detect response shift. Power analyses for hypotheses regarding (exact) overall model fit and the presence of response shift are demonstrated in a step-by-step manner. The freely available and user-friendly R-package lavaan and shiny-app 'power4SEM' are used for the calculations. RESULTS: Using the SF-36 as an example, we illustrate the specification of null-hypothesis (H0) and alternative hypothesis (H1) models to calculate chi-square based power for the test on overall model fit, the omnibus test on response shift, and the specific test on response shift. For example, we show that a sample size of 506 is needed to reject an incorrectly specified measurement model, when the actual model has two-medium sized cross loadings. We also illustrate power calculation based on the RMSEA index for approximate fit, where H0 and H1 are defined in terms of RMSEA-values. CONCLUSION: By providing accessible resources to perform power analyses and emphasizing the different power analyses associated with different modeling goals, we hope to facilitate the uptake of power analyses for response shift detection with SEM and thereby enhance the stringency of response shift research.

Problems of Domain Factors with Small Factor Loadings in Bi-Factor Models.

Many measurement designs produce domain factors with small variances and factor loadings. The current study investigates the cause, prevalence, and problematic consequences of such domain factors. We collected a meta-analytic sample of empirical applications, conducted a simulation study on statistical power and estimation precision, and provide a reanalysis of an empirical example. The meta-analysis shows that about a quarter of all standardized domain factor loadings is in the range of -.2<λ<.2 and about a third of all domains is measured by five or fewer indicators, resulting in small factor variances. The simulation study examines the associated difficulties concerning statistical power, trait recovery, irregular estimates, and estimation precision for a range of such realistic cases. The empirical example illustrates the challenge to develop measures that produce clearly interpretable domain factors. Study planning and interpretation need to take the (expected) sum of squared factor loadings per domain factor into account. This is relevant even if influences of domain factors are desired to be small, and equally applies to different model variants. We propose several strategies for how researchers may better unlock the bifactor model's full potential and clarify its interpretation.

Strategy for Designing In Vivo Dose-Response Comparison Studies.

In preclinical drug discovery, at the step of lead optimization of a compound, in vivo experimentation can differentiate several compounds in terms of efficacy and potency in a biological system of whole living organisms. For the lead optimization study, it may be desirable to implement a dose-response design so that compound comparisons can be made from nonlinear curves fitted to the data. A dose-response design requires more thought relative to a simpler study design, needing parameters for the number of doses, the dose values, and the sample size per dose. This tutorial illustrates how to calculate statistical power, choose doses, and determine sample size per dose for a comparison of two or more dose-response curves for a future in vivo study.

Group sequential design with maximin efficiency robust test for immunotherapy with generalized delayed treatment effect.

The delayed treatment effect is a common feature of immunotherapy, characterized by a gradual onset of action ranging from no effect to full effect. In this study, we propose a generalized delayed treatment effect function to depict the delayed effective process precisely and flexibly. To reduce potential power loss caused by the delayed treatment effect in a group sequential trial, we employ the maximin efficiency robust test, which enhances power robustness across a range of possible delays. We present novel approaches based on the Markov chain method for determining group sequential boundaries, calculating the power function, and estimating the maximum sample size through iterative regressions between the square root of the maximum sample size and the normal quantile of power. Extensive simulation studies validate the effectiveness of our approaches, particularly in balanced trials, demonstrating the validity of group sequential boundaries and the accuracy of maximum sample size estimations. Additionally, we utilize a real trial as an example to compare our considered trial with group sequential trials using the log-rank and generalized piecewise weighted log-rank tests. The results show significantly reduced maximum sample sizes, highlighting the economic advantage of our approach.

Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies.

When comparing the performance of two or more competing tests, simulation studies commonly focus on statistical power. However, if the size of the tests being compared are either different from one another or from the nominal size, comparing tests based on power alone may be misleading. By analogy with diagnostic accuracy studies, we introduce relative positive and negative likelihood ratios to factor in both power and size in the comparison of multiple tests. We derive sample size formulas for a comparative simulation study. As an example, we compared the performance of six statistical tests for small-study effects in meta-analyses of randomized controlled trials: Begg's rank correlation, Egger's regression, Schwarzer's method for sparse data, the trim-and-fill method, the arcsine-Thompson test, and Lin and Chu's combined test. We illustrate that comparing power alone, or power adjusted or penalized for size, can be misleading, and how the proposed likelihood ratio approach enables accurate comparison of the trade-off between power and size between competing tests.

How much data do we need to estimate computational models of decision-making? The COMPASS toolbox.

How much data are needed to obtain useful parameter estimations from a computational model? The standard approach to address this question is to carry out a goodness-of-recovery study. Here, the correlation between individual-participant true and estimated parameter values determines when a sample size is large enough. However, depending on one's research question, this approach may be suboptimal, potentially leading to sample sizes that are either too small (underpowered) or too large (overcostly or unfeasible). In this paper, we formulate a generalized concept of statistical power and use this to propose a novel approach toward determining how much data is needed to obtain useful parameter estimates from a computational model. We describe a Python-based toolbox (COMPASS) that allows one to determine how many participants are needed to fit one specific computational model, namely the Rescorla-Wagner model of learning and decision-making. Simulations revealed that a high number of trials per person (more than the number of persons) are a prerequisite for high-powered studies in this particular setting.

How Many Participants? How Many Trials? Maximizing the Power of Reaction Time Studies.

Due to limitations in the resources available for carrying out reaction time (RT) experiments, researchers often have to choose between testing relatively few participants with relatively many trials each or testing relatively many participants with relatively few trials each. To compare the experimental power that would be obtained under each of these options, I simulated virtual experiments using subsets of participants and trials from eight large real RT datasets examining 19 experimental effects. The simulations compared designs using the first N T trials from N P randomly selected participants, holding constant the total number of trials across all participants, N P × N T . The [ N P , N T ] combination maximizing the power to detect each effect depended on how the mean and variability of that effect changed with practice. For most effects, power was greater in designs having many participants with few trials each rather than the reverse, suggesting that researchers should usually try to recruit large numbers of participants for short experimental sessions. In some cases, power for a fixed total number of trials across all participants was maximized by having as few as two trials per participant in each condition. Where researchers can make plausible predictions about how their effects will change over the course of a session, they can use those predictions to increase their experimental power.

Estimating statistical power for structural equation models in developmental cognitive science: A tutorial in R : Power simulation for SEMs.

Determining the compositional structure and dimensionality of psychological constructs lies at the heart of many research questions in developmental science. Structural equation modeling (SEM) provides a versatile framework for formalizing and estimating the relationships among multiple latent constructs. While the flexibility of SEM can accommodate many complex assumptions on the underlying structure of psychological constructs, it makes a priori estimation of statistical power and required sample size challenging. This difficulty is magnified when comparing non-nested SEMs, which prevents the use of traditional likelihood-ratio tests. Sample size estimates for SEM model fit comparisons typically rely on generic rules of thumb. Such heuristics can be misleading because statistical power in SEM depends on a variety of model properties. Here, we demonstrate a Monte Carlo simulation approach for estimating a priori statistical power for model selection when comparing non-nested models in an SEM framework. We provide a step-by-step guide to this approach based on an example from our memory development research in children.

Enabling analytical power calculations for multilevel models with autocorrelated errors through deriving and approximating the precision matrix.

To unravel how within-person psychological processes fluctuate in daily life, and how these processes differ between persons, intensive longitudinal (IL) designs in which participants are repeatedly measured, have become popular. Commonly used statistical models for those designs are multilevel models with autocorrelated errors. Substantive hypotheses of interest are then typically investigated via statistical hypotheses tests for model parameters of interest. An important question in the design of such IL studies concerns the determination of the number of participants and the number of measurements per person needed to achieve sufficient statistical power for those statistical tests. Recent advances in computational methods and software have enabled the computation of statistical power using Monte Carlo simulations. However, this approach is computationally intensive and therefore quite restrictive. To ease power computations, we derive simple-to-use analytical formulas for multilevel models with AR(1) within-person errors. Analytic expressions for a model family are obtained via asymptotic approximations of all sample statistics in the precision matrix of the fixed effects. To validate this analytical approach to power computation, we compare it to the simulation-based approach via a series of Monte Carlo simulations. We find comparable performances making the analytic approach a useful tool for researchers that can drastically save them time and resources.

Statistical power analysis and sample size planning for moderated mediation models.

Conditional process models, including moderated mediation models and mediated moderation models, are widely used in behavioral science research. However, few studies have examined approaches to conduct statistical power analysis for such models and there is also a lack of software packages that provide such power analysis functionalities. In this paper, we introduce new simulation-based methods for power analysis of conditional process models with a focus on moderated mediation models. These simulation-based methods provide intuitive ways for sample-size planning based on regression coefficients in a moderated mediation model as well as selected variance and covariance components. We demonstrate how the methods can be applied to five commonly used moderated mediation models using a simulation study, and we also assess the performance of the methods through the five models. We implement our approaches in the WebPower R package and also in Web apps to ease their application.

Integrated variant allele frequency analysis pipeline and R package: easyVAF.

Somatic sequence variants are associated with cancer diagnosis, prognostic stratification, and treatment response. Variant allele frequency (VAF), the percentage of sequence reads with a specific DNA variant over the read depth at that locus, has been used as a metric to quantify mutation rates in these applications. VAF has the potential for feature detection by reflecting changes in tumor clonal composition across treatments or time points. Although there are several packages, including Genome Analysis Toolkit and VarScan, designed for variant calling and rare mutation identification, there is no readily available package for comparing VAFs among and between groups to identify loci of interest. To this end, we have developed the R package easyVAF, which includes parametric and nonparametric tests to compare VAFs among multiple groups. It is accompanied by an interactive R Shiny app. With easyVAF, the investigator has the option between three statistical tests to maximize power while maintaining an acceptable type I error rate. This paper presents our proposed pipeline for VAF analysis, from quality checking to group comparison. We evaluate our method in a wide range of simulated scenarios and show that choosing the appropriate test to limit the type I error rate is critical. For situations where data is sparse, we recommend comparing VAFs with the beta-binomial likelihood ratio test over Fisher's exact test and Pearson's χ2 test.

Genome-wide hierarchical mixed model association analysis.

In genome-wide mixed model association analysis, we stratified the genomic mixed model into two hierarchies to estimate genomic breeding values (GBVs) using the genomic best linear unbiased prediction and statistically infer the association of GBVs with each SNP using the generalized least square. The hierarchical mixed model (Hi-LMM) can correct confounders effectively with polygenic effects as residuals for association tests, preventing potential false-negative errors produced with genome-wide rapid association using mixed model and regression or an efficient mixed-model association expedited (EMMAX). Meanwhile, the Hi-LMM performs the same statistical power as the exact mixed model association and the same computing efficiency as EMMAX. When the GBVs have been estimated precisely, the Hi-LMM can detect more quantitative trait nucleotides (QTNs) than existing methods. Especially under the Hi-LMM framework, joint association analysis can be made straightforward to improve the statistical power of detecting QTNs.

Statistical power in clinical trials of interventions for mood, anxiety, and psychotic disorders.

BACKGROUND: Previous research has suggested that statistical power is suboptimal in many biomedical disciplines, but it is unclear whether power is better in trials for particular interventions, disorders, or outcome types. We therefore performed a detailed examination of power in trials of psychotherapy, pharmacotherapy, and complementary and alternative medicine (CAM) for mood, anxiety, and psychotic disorders. METHODS: We extracted data from the Cochrane Database of Systematic Reviews (Mental Health). We focused on continuous efficacy outcomes and estimated power to detect predetermined effect sizes (standardized mean difference [SMD] = 0.20-0.80, primary SMD = 0.40) and meta-analytic effect sizes (ESMA). We performed meta-regression to estimate the influence of including underpowered studies in meta-analyses. RESULTS: We included 256 reviews with 10 686 meta-analyses and 47 384 studies. Statistical power for continuous efficacy outcomes was very low across intervention and disorder types (overall median [IQR] power for SMD = 0.40: 0.32 [0.19-0.54]; for ESMA: 0.23 [0.09-0.58]), only reaching conventionally acceptable levels (80%) for SMD = 0.80. Median power to detect the ESMA was higher in treatment-as-usual (TAU)/waitlist-controlled (0.49-0.63) or placebo-controlled (0.12-0.38) trials than in trials comparing active treatments (0.07-0.13). Adequately-powered studies produced smaller effect sizes than underpowered studies (B = -0.06, p ⩽ 0.001). CONCLUSIONS: Power to detect both predetermined and meta-analytic effect sizes in psychiatric trials was low across all interventions and disorders examined. Consistent with the presence of reporting bias, underpowered studies produced larger effect sizes than adequately-powered studies. These results emphasize the need to increase sample sizes and to reduce reporting bias against studies reporting null results to improve the reliability of the published literature.

Statistical inference and power analysis for direct and spillover effects in two-stage randomized experiments.

Two-stage randomized experiments become an increasingly popular experimental design for causal inference when the outcome of one unit may be affected by the treatment assignments of other units in the same cluster. In this paper, we provide a methodological framework for general tools of statistical inference and power analysis for two-stage randomized experiments. Under the randomization-based framework, we consider the estimation of a new direct effect of interest as well as the average direct and spillover effects studied in the literature. We provide unbiased estimators of these causal quantities and their conservative variance estimators in a general setting. Using these results, we then develop hypothesis testing procedures and derive sample size formulas. We theoretically compare the two-stage randomized design with the completely randomized and cluster randomized designs, which represent two limiting designs. Finally, we conduct simulation studies to evaluate the empirical performance of our sample size formulas. For empirical illustration, the proposed methodology is applied to the randomized evaluation of the Indian National Health Insurance Program. An open-source software package is available for implementing the proposed methodology.

Designing a large-scale track-based monitoring program to detect changes in species distributions in arid Australia.

Monitoring trends in animal populations in arid regions is challenging due to remoteness and low population densities. However, detecting species' tracks or signs is an effective survey technique for monitoring population trends across large spatial and temporal scales. In this study, we developed a simulation framework to evaluate the performance of alternative track-based monitoring designs at detecting change in species distributions in arid Australia. We collated presence-absence records from 550 2-ha track-based plots for 11 vertebrates over 13 years and fitted ensemble species distribution models to predict occupancy in 2018. We simulated plausible changes in species' distributions over the next 15 years and, with estimates of detectability, simulated monitoring to evaluate the statistical power of three alternative monitoring scenarios: (1) where surveys were restricted to existing 2-ha plots, (2) where surveys were optimized to target all species equally, and (3) where surveys were optimized to target two species of conservation concern. Across all monitoring designs and scenarios, we found that power was higher when detecting increasing occupancy trends compared to decreasing trends owing to the relatively low levels of initial occupancy. Our results suggest that surveying 200 of the existing plots annually (with a small subset resurveyed twice within a year) will have at least an 80% chance of detecting 30% declines in occupancy for four of the five invasive species modeled and one of the six native species. This increased to 10 of the 11 species assuming larger (50%) declines. When plots were positioned to target all species equally, power improved slightly for most compared to the existing survey network. When plots were positioned to target two species of conservation concern (crest-tailed mulgara and dusky hopping mouse), power to detect 30% declines increased by 29% and 31% for these species, respectively, at the cost of reduced power for the remaining species. The effect of varying survey frequency depended on its trade-off with the number of sites sampled and requires further consideration. Nonetheless, our research suggests that track-based surveying is an effective and logistically feasible approach to monitoring broad-scale occupancy trends in desert species with both widespread and restricted distributions.