Misstatements, misperceptions, and mistakes in controlling for covariates in observational research.

We discuss 12 misperceptions, misstatements, or mistakes concerning the use of covariates in observational or nonrandomized research. Additionally, we offer advice to help investigators, editors, reviewers, and readers make more informed decisions about conducting and interpreting research where the influence of covariates may be at issue. We primarily address misperceptions in the context of statistical management of the covariates through various forms of modeling, although we also emphasize design and model or variable selection. Other approaches to addressing the effects of covariates, including matching, have logical extensions from what we discuss here but are not dwelled upon heavily. The misperceptions, misstatements, or mistakes we discuss include accurate representation of covariates, effects of measurement error, overreliance on covariate categorization, underestimation of power loss when controlling for covariates, misinterpretation of significance in statistical models, and misconceptions about confounding variables, selecting on a collider, and p value interpretations in covariate-inclusive analyses. This condensed overview serves to correct common errors and improve research quality in general and in nutrition research specifically.

Unreliable Findings Due to Miscalculations and Errors. Comment on Nally et al. The Effectiveness of School-Based Interventions on Obesity-Related Behaviours in Primary School Children: A Systematic Review and Meta-Analysis of Randomised Controlled Trials. Children 2021, 8, 489.

Nally et al. [...].

Corrected analysis of "the effects of bright light treatment on affective symptoms in people with dementia: a 24-week cluster randomized controlled trial" that accounts for clustering and nesting verifies conclusions.

In this correspondence, we explain the reasoning for invalidity of the analysis choices by Kolberg et al., and provide the results produced using correct statistical procedures for their study design. Reassuringly, we could verify the original conclusions. That is, results of the corrected statistical models are similar to the results of the original analysis. Regardless of the magnitude of difference that corrected statistical methods make, results and conclusions that are derived from invalid methods are unsubstantiated. By verifying the results, we allow the readers to be assured that the published conclusions in the study by Kolberg et al. now rest on a sound evidential basis.

One Cluster per Condition Is Not a Valid Design in Cluster-Randomized Trials. Comment on Wang et al. The Effect of Different Physical Exercise Programs on Physical Fitness among Preschool Children: A Cluster-Randomized Controlled Trial. Int. J. Environ. Res. Public Health 2023, 20, 4254.

Wang et al. [...].

Factors associated with choice of behavioural weight loss program by adults with obesity.

We assessed the preference for two behavioural weight loss programs, Diabetes Prevention Program (DPP) and Healthy Weight for Living (HWL) in adults with obesity. A cross-sectional survey was fielded on the Amazon Mechanical Turk. Eligibility criteria included reporting BMI ≥30 and at least two chronic health conditions. Participants read about the programs, selected their preferred program, and answered follow-up questions. The estimated probability of choosing either program was not significantly different from .5 (N = 1005, 50.8% DPP and 49.2% HWL, p = .61). Participants' expectations about adherence, weight loss magnitude, and dropout likelihood were associated with their choice (p < .0001). Non-White participants (p = .040) and those with monthly income greater than $4999 (p = .002) were less likely to choose DPP. Participants who had postgraduate education (p = .007), did not report high serum cholesterol (p = .028), and reported not having tried losing weight before (p = .025) were more likely to choose DPP. Those who chose HWL were marginally more likely to report that being offered two different programs rather than one would likely affect their decision to enrol in one of the two (p = .052). The enrolment into DPP and HWL was balanced, but race, educational attainment, income, previous attempt to lose weight, and serum cholesterol levels had significant associations with the choice of weight loss program.

Only one cluster per condition is an invalid design for a cluster randomized trial and should be re-labeled a quasi-experimental study. Response to: "Effect of behavioral activation on time and frequency domain heart rate variability in older adults with subthreshold depression: a cluster randomized controlled trial in Thailand".

Ayudhaya et al. examined the effect of Behavioral Activation on daily step count and heart rate variability among older adults with depression in a study labeled a cluster randomized controlled trial (cRCT). However, only one cluster was assigned to either of the study conditions. Such a design would have zero degrees of freedom for inferential testing, because the variation due to cluster membership cannot be estimated apart from the variation due to treatment assignment. Thus, the intervention effect is completely confounded with the cluster effect. The study should be labeled a quasi-experimental study, not a cRCT. Accordingly, the numerical results should be interpreted as associations but not evidence for causal relationships.

Mixed Random and Nonrandom Allocation, and Group Randomization Have Been Mislabeled and Misanalysed, Necessitating Reanalysis. Comment on Conner et al. KiwiC for Vitality: Results of a Randomized Placebo-Controlled Trial Testing the Effects of Kiwifruit or Vitamin C Tablets on Vitality in Adults with Low Vitamin C Levels. Nutrients 2020, 12, 2898.

We read the report by Conner and colleagues that tested whether kiwifruit or vitamin C affected measures of vitality [...].

Accounting for the clustering and nesting effects verifies most conclusions. Corrected analysis of: "Randomized nutrient bar supplementation improves exercise-associated changes in plasma metabolome in adolescents and adult family members at cardiometabolic risk".

In a published randomized controlled trial, household units were randomized to a nutrient bar supplementation group or a control condition, but the non-independence of observations within the same household (i.e., the clustering effect) was not accounted for in the statistical analyses. Therefore, we reanalyzed the data appropriately by adjusting degrees of freedom using the between-within method, and accounting for household units using linear mixed effect models with random intercepts for family units and subjects nested within family units for each reported outcome. Results from this reanalysis showed that ignoring the clustering and nesting effects in the original analyses had resulted in anticonservative (i.e., too small) time x group interaction p-values. Still, majority of the conclusions remained unchanged.

Evaluation of the type I error rate when using parametric bootstrap analysis of a cluster randomized controlled trial with binary outcomes and a small number of clusters.

BACKGROUND: Cluster randomized controlled trials (cRCTs) are increasingly used but must be analyzed carefully. We conducted a simulation study to evaluate the validity of a parametric bootstrap (PB) approach with respect to the empirical type I error rate for a cRCT with binary outcomes and a small number of clusters. METHODS: We simulated a case study with a binary (0/1) outcome, four clusters, and 100 subjects per cluster. To compare the validity of the test with respect to error rate, we simulated the same experiment with K=10, 20, and 30 clusters, each with 2,000 simulated datasets. To test the null hypothesis, we used a generalized linear mixed model including a random intercept for clusters and obtained p-values based on likelihood ratio tests (LRTs) using the parametric bootstrap method as implemented in the R package "pbkrtest". RESULTS: The PB test produced error rates of 9.1%, 5.5%, 4.9%, and 5.0% on average across all ICC values for K=4, K=10, K=20, and K=30, respectively. The error rates were higher, ranging from 9.1% to 36.5% for K=4, in the models with singular fits (i.e., ignoring clustering) because the ICC was estimated to be zero. CONCLUSION: Using the parametric bootstrap for cRCTs with a small number of clusters results in inflated error rates and is not valid.