Researcher degrees of freedom in statistical software contribute to unreliable results: A comparison of nonparametric analyses conducted in SPSS, SAS, Stata, and R.

Researcher degrees of freedom can affect the results of hypothesis tests and consequently, the conclusions drawn from the data. Previous research has documented variability in accuracy, speed, and documentation of output across various statistical software packages. In the current investigation, we conducted Pearson's chi-square test of independence, Spearman's rank-ordered correlation, Kruskal-Wallis one-way analysis of variance, Wilcoxon Mann-Whitney U rank-sum tests, and Wilcoxon signed-rank tests, along with estimates of skewness and kurtosis, on large, medium, and small samples of real and simulated data in SPSS, SAS, Stata, and R and compared the results with those obtained through hand calculation using the raw computational formulas. Multiple inconsistencies were found in the results produced between statistical packages due to algorithmic variation, computational error, and statistical output. The most notable inconsistencies were due to algorithmic variations in the computation of Pearson's chi-square test conducted on 2 × 2 tables, where differences in p-values reported by different software packages ranged from .005 to .162, largely as a function of sample size. We discuss how such inconsistencies may influence the conclusions drawn from the results of statistical analyses depending on the statistical software used, and we urge researchers to analyze their data across multiple packages to check for inconsistencies and report details regarding the statistical procedure used for data analysis.

Cognitive effort avoidance in veterans with suicide attempt histories.

Suicide attempts (SA) are increasing in the United States, especially in veterans. Discovering individual cognitive features of the subset of suicide ideators who attempt suicide is critical. Cognitive theories attribute SA to facile schema-based negative interpretations of environmental events. Over-general autobiographical memory and facile solutions in problem solving tasks in SA survivors suggest that aversion to expending cognitive effort may be a neurobehavioral marker of SA risk. In veterans receiving care for mood disorder, we compared cognitive effort discounting and evidence-gathering in a beads task between veterans with (SAHx+; n = 26) versus without (SAHx-; n = 22) a history of SA. Groups did not differ in depressed mood or in a proxy metric of premorbid intelligence. Compared to SAHx- participants, SAHx+ participants self-reported significantly more severe cognitive problems in most domains, and also eschewed choice to earn higher monetary reward if earning it required a slightly increased working memory (WM) demand relative to an easy WM task. There was no group difference, however, in extent of evidence-gathering before declaring a conclusion in a beads task. These preliminary data suggest that aversion to expenditure of cognitive effort, potentially as a component of cognitive difficulties, may be a marker for SA risk.

Quantifying Dispositional Fear as Threat Sensitivity: Development and Initial Validation of a Model-Based Scale Measure.

The Research Domain Criteria initiative aims to reorient the focus of psychopathology research toward biobehavioral constructs that cut across different modalities of measurement, including self-report and neurophysiology. Constructs within the Research Domain Criteria framework are intentionally transdiagnostic, with the construct of "acute threat," for example, broadly relevant to clinical problems and associated traits involving fearfulness and stress reactivity. A potentially valuable referent for research on the construct of acute threat is a structural model of fear/fearlessness questionnaires known to predict variations in physiological threat reactivity as indexed by startle potentiation. The aim of the current work was to develop an efficient, item-based scale measure of the general factor of this structural model for use in studies of dispositional threat sensitivity and its relationship to psychopathology. A self-report scale consisting of 44 items from a conceptually relevant, nonproprietary questionnaire was first developed in a sample of 1,307 student participants, using the general factor of the fear/fearlessness model as a direct referent. This new Trait Fear scale was then evaluated for convergent and discriminant validity with measures of personality and psychopathology in a separate sample (n = 213) consisting of community adults and undergraduate students. The strong performance of the scale in this criterion-validation sample suggests that it can provide an effective means for indexing variations along a dispositional continuum of fearfulness reflecting variations in sensitivity to acute threat.

Fear-potentiated startle response as an endophenotype: Evaluating metrics and methods for genetic applications.

The modulation of the startle response (SR) by threatening stimuli (fear-potentiated startle; FPS) is a proposed endophenotype for disorders of the fearful-fearlessness spectrum. FPS has failed to show evidence of heritability, raising concerns. However, metrics used to index FPS-and, importantly, other conditional phenotypes that are dependent on a baseline-may not be suitable for the approaches used in genetic epidemiology studies. Here, we evaluated multiple metrics of FPS in a population-based sample of preadolescent twins (N = 569 from 320 twin pairs, Mage = 11.4) who completed a fear-conditioning paradigm with airpuff-elicited SR on two occasions (~1 month apart). We applied univariate and multivariate biometric modeling to estimate the heritability of FPS using several proposed standardization procedures. This was extended with data simulations to evaluate biases in heritability estimates of FPS (and similar metrics) under various scenarios. Consistent with previous studies, results indicated moderate test-retest reliability (r = 0.59) and heritability of the overall SR (h2 = 34%) but poor reliability and virtually no unique genetic influences on FPS when considering a raw or standardized differential score that removes baseline SR. Simulations demonstrated that the use of differential scores introduces bias in heritability estimates relative to jointly analyzing baseline SR and FPS in a multivariate model. However, strong dependency of FPS on baseline levels makes unique genetic influences virtually impossible to detect regardless of methodology. These findings indicate that FPS and other conditional phenotypes may not be well suited to serve as endophenotypes unless such codependency can be disentangled.

Genetic network properties of the human cortex based on regional thickness and surface area measures.

We examined network properties of genetic covariance between average cortical thickness (CT) and surface area (SA) within genetically-identified cortical parcellations that we previously derived from human cortical genetic maps using vertex-wise fuzzy clustering analysis with high spatial resolution. There were 24 hierarchical parcellations based on vertex-wise CT and 24 based on vertex-wise SA expansion/contraction; in both cases the 12 parcellations per hemisphere were largely symmetrical. We utilized three techniques-biometrical genetic modeling, cluster analysis, and graph theory-to examine genetic relationships and network properties within and between the 48 parcellation measures. Biometrical modeling indicated significant shared genetic covariance between size of several of the genetic parcellations. Cluster analysis suggested small distinct groupings of genetic covariance; networks highlighted several significant negative and positive genetic correlations between bilateral parcellations. Graph theoretical analysis suggested that small world, but not rich club, network properties may characterize the genetic relationships between these regional size measures. These findings suggest that cortical genetic parcellations exhibit short characteristic path lengths across a broad network of connections. This property may be protective against network failure. In contrast, previous research with structural data has observed strong rich club properties with tightly interconnected hub networks. Future studies of these genetic networks might provide powerful phenotypes for genetic studies of normal and pathological brain development, aging, and function.