Common, uncommon, and novel applications of random forest in psychological research.

Recent reform efforts have pushed toward a better understanding of the distinction between exploratory and confirmatory research, and appropriate use of each. As some utilize more exploratory tools, it may be tempting to employ multiple linear regression models. In this paper, we advocate for the use of random forest (RF) models. RF is able to obtain better predictive performance than traditional regression, while also inherently protecting against overfitting as well as detecting nonlinear effects and interactions among predictors. Given the advantages of RF compared to other statistical procedures, it is a tool commonly used within a plethora of industries, including stock trading, banking, pharmaceuticals, and patient healthcare planning. However, we find RF is used within the field of psychology comparatively less frequently. In the current paper, we advocate for RF as an important statistical tool within the context of behavioral and psychological research. In hopes of increasing the use of RF in the field of psychology, we provide information pertaining to the limitations one might confront in using RF and how to overcome such limitations. Moreover, we discuss various methods for how to optimally utilize RF with psychological data, such as nonparametric modeling, interaction and nonlinearity detection, variable selection, prediction and classification modeling, and assessing parameters of Monte Carlo simulations. Throughout, we illustrate the use of RF with visualization strategies, aimed to make RF models more comprehensible and intuitive.

A graph for every analysis: Mapping visuals onto common analyses using flexplot.

For decades, statisticians and methodologists have insisted researchers utilize graphical analysis much more heavily. Despite cogent and passionate recommendations, there has been no graphical revolution. Instead, researchers rely heavily on misleading graphics that violate visual processing heuristics. Perhaps the main reason for the persistence of deceptive graphics is software; most software familiar to psychological researchers suffer from poor defaults and limited capabilities. Also, visualization is ancillary to statistical analysis, providing an incentive to not produce graphics at all. In this paper, we argue that every statistical analysis must have an accompanying graphic, and we introduce the point-and-click software Flexplot, available both in JASP and Jamovi. We then present the theoretical framework that guides Flexplot, as well as show how to perform the most common statistical analyses in psychological literature.

Estimating Incremental Validity Under Missing Data.

A common form of missing data is caused by selection on an observed variable (e.g., Z). If the selection variable was measured and is available, the data are regarded as missing at random (MAR). Selection biases correlation, reliability, and effect size estimates when these estimates are computed on listwise deleted (LD) data sets. On the other hand, maximum likelihood (ML) estimates are generally unbiased and outperform LD in most situations, at least when the data are MAR. The exception is when we estimate the partial correlation. In this situation, LD estimates are unbiased when the cause of missingness is partialled out. In other words, there is no advantage of ML estimates over LD estimates in this situation. We demonstrate that under a MAR condition, even ML estimates may become biased, depending on how partial correlations are computed. Finally, we conclude with recommendations about how future researchers might estimate partial correlations even when the cause of missingness is unknown and, perhaps, unknowable.

Altered type II interferon precedes autoantibody accrual and elevated type I interferon activity prior to systemic lupus erythematosus classification.

OBJECTIVES: The relationship of immune dysregulation and autoantibody production that may contribute to systemic lupus erythematosus (SLE) pathogenesis is unknown. This study evaluates the individual and combined contributions of autoantibodies, type I interferon (IFN-α) activity, and IFN-associated soluble mediators to disease development leading to SLE. METHODS: Serial serum specimens from 55 individuals collected prior to SLE classification (average timespan=4.3 years) and unaffected healthy controls matched by age (±5 years), gender, race and time of sample procurement were obtained from the Department of Defense Serum Repository. Levels of serum IFN-α activity, IFN-associated mediators and autoantibodies were evaluated and temporal relationships assessed by growth curve modelling, path analysis, analysis of covariance and random forest models. RESULTS: In cases, but not matched controls, autoantibody specificities and IFN-associated mediators accumulated over a period of years, plateauing near the time of disease classification (p<0.001). Autoantibody positivity coincided with or followed type II IFN dysregulation, preceding IFN-α activity in growth curve models, with elevated IFN-α activity and B-lymphocyte stimulator levels occurring shortly before SLE classification (p≤0.005). Cases were distinguished by multivariate random forest models incorporating IFN-γ, macrophage chemoattractant protein (MCP)-3, anti-chromatin and anti-spliceosome antibodies (accuracy 93% >4 years pre-classification; 97% within 2 years of SLE classification). CONCLUSIONS: Years before SLE classification, enhancement of the type II IFN pathway allows for accumulation of autoantibodies and subsequent elevations in IFN-α activity immediately preceding SLE classification. Perturbations in select immunological processes may help identify at-risk individuals for further clinical evaluation or participation in prospective intervention trials.

Dysregulation of innate and adaptive serum mediators precedes systemic lupus erythematosus classification and improves prognostic accuracy of autoantibodies.

Systemic lupus erythematosus (SLE) is a complex autoimmune disease with a poorly understood preclinical stage of immune dysregulation and symptom accrual. Accumulation of antinuclear autoantibody (ANA) specificities is a hallmark of impending clinical disease. Yet, many ANA-positive individuals remain healthy, suggesting that additional immune dysregulation underlies SLE pathogenesis. Indeed, we have recently demonstrated that interferon (IFN) pathways are dysregulated in preclinical SLE. To determine if other forms of immune dysregulation contribute to preclinical SLE pathogenesis, we measured SLE-associated autoantibodies and soluble mediators in samples from 84 individuals collected prior to SLE classification (average timespan = 5.98 years), compared to unaffected, healthy control samples matched by race, gender, age (±5 years), and time of sample procurement. We found that multiple soluble mediators, including interleukin (IL)-5, IL-6, and IFN-γ, were significantly elevated in cases compared to controls more than 3.5 years pre-classification, prior to or concurrent with autoantibody positivity. Additional mediators, including innate cytokines, IFN-associated chemokines, and soluble tumor necrosis factor (TNF) superfamily mediators increased longitudinally in cases approaching SLE classification, but not in controls. In particular, levels of B lymphocyte stimulator (BLyS) and a proliferation-inducing ligand (APRIL) were comparable in cases and controls until less than 10 months pre-classification. Over the entire pre-classification period, random forest models incorporating ANA and anti-Ro/SSA positivity with levels of IL-5, IL-6, and the IFN-γ-induced chemokine, MIG, distinguished future SLE patients with 92% (±1.8%) accuracy, compared to 78% accuracy utilizing ANA positivity alone. These data suggest that immune dysregulation involving multiple pathways contributes to SLE pathogenesis. Importantly, distinct immunological profiles are predictive for individuals who will develop clinical SLE and may be useful for delineating early pathogenesis, discovering therapeutic targets, and designing prevention trials.

Model Conditioned Data Elasticity in Path Analysis: Assessing the "Confoundability" of Model/Data Characteristics.

Much research has been directed at the validity of fit indices in Path Analysis and Structural Equation Modeling (e.g., Browne, MacCallum, Kim, Andersen, & Glaser, 2002 ; Heene, Hilbert, Draxler, Ziegler, & Bühner, 2011 ; Hu & Bentler, 1999 ; Marsh, Hau, & Wen, 2004 ). Recent developments (e.g., Preacher, 2006 ; Roberts & Pashler, 2000 , 2002 ) have encouraged researchers to investigate other criteria for comparing models, including model complexity. What has not been investigated is the inherent ability of a particular data set to be fitted with a constrained set of randomly generated linear models, which we call Model Conditioned Data Elasticity (DE). In this article we show how DE can be compared with the problem of equivalent models and a more general problem of the "confoundability" of data/model combinations (see MacCallum, Wegener, Uchino, & Fabrigar, 1993 ). Using the DE package in R, we show how DE can be assessed through automated computer searches. Finally, we discuss how DE fits within the controversy surrounding the use of fit statistics.

Understanding the exploratory/confirmatory data analysis continuum: Moving beyond the "replication crisis".

In light of the "replication crisis," some advocate for stricter standards and greater transparency in research methods. These efforts push toward a data analysis approach called "confirmatory data analysis" (CDA; see Wagenmakers et al., 2012). However, some (e.g., Baumeister, 2016; Goldin-Meadow, 2016) suggest that emphasizing CDA may restrict creativity and discovery. These scholars argued (sometimes inadvertently) for greater freedom to pursue "exploratory data analysis" (EDA; see Tukey, 1977). Ironically and unfortunately, many who push against stricter CDA standards do not realize EDA exists, or misunderstand the philosophy and proper tools for exploration. In this article, the meaning, tools, philosophy, and ethics associated with EDA, CDA, and a relatively unknown but important approach called "rough CDA" are clarified. Important distinctions are developed between EDA/rough CDA/CDA and other (some problematic) analysis activities including p-hacking, HARKing, and data mining, which are situated in a (graphical) framework that clarifies relationships and ethical boundaries with each. In short, the proper data analytic approach depends on (a) intentions and (b) transparency. Most psychological research is not at a maturity level to justify CDA; researchers have historically used tools mismatched to their research agenda. In the conclusion, recommendations are presented about how these typologies can be integrated into graduate training programs and how a cumulative research program can help psychology move beyond the replication crisis. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Seeing the impossible: Visualizing latent variable models with flexplavaan.

Latent variable models (LVMs) are incredibly flexible tools that allow users to address research questions they might otherwise never be able to answer (McDonald, 2013). However, one major limitation of LVMs is evaluating model fit. There is no universal consensus about how to evaluate model fit, either globally or locally. Part of the reason evaluating these models is difficult is because fit is typically reduced to a handful of statistics that may or may not reflect the model's adequacy and/or assumptions. In this article we argue that proper evaluation of model fit must include visualizing both the raw data and the model-implied fit. Visuals reveal, at a glance, the fit of the model and whether the model's assumptions have been met. Unfortunately, tools for visualizing LVMs have historically been limited. In this article, we introduce new plots and reframe existing plots that provide necessary resources for evaluating LVMs. These plots are available in a new open-source R package called flexplavaan, which combines the model plotting capabilities of flexplot with the latent variable modeling capabilities of lavaan. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Beliefs related to health care incentives: Comparison of substance use disorder treatment providers, medical treatment providers, and a public sample.

This study surveyed substance use disorder (SUD) treatment providers, medical treatment providers, and a public sample about beliefs regarding health care incentives to explore differences among the groups and across health disorders for which research has demonstrated incentives improve outcomes. Six hundred participants (n = 200/group) completed the Provider Survey of Incentives. The study found between group differences for positive and negative beliefs. The public sample was highest on the positive beliefs subscale (M = 3.81), followed by SUD (M = 3.63) and medical treatment providers (M = 3.48; F(2, 597) = 20.09, p < .001). The medical treatment providers were highest on the negative beliefs subscale (M = 2.91), compared to the public sample (M = 2.77) and SUD treatment providers (M = 2.65; F(2, 597) = 7.521, p < .001). Endorsement of incentives to treat medical disorders was similar across the groups, with obesity the most endorsed disorder. In contrast, endorsement of incentives to treat SUDs differed across groups, except for smoking. The SUD treatment providers were almost twice as likely as the public sample (OR = 1.81, 95% CI = 1.27-2.59) and the public sample almost twice as likely as the medical treatment providers (OR = 1.74, 95% CI = 1.24-2.47) to endorse the use of incentives to treat more SUDs. Medical treatment providers were also the least likely to endorse incentives to treat both legal and illicit substance use. These findings suggest that incentive programs have good acceptability among SUD treatment providers and the public, but medical treatment providers are less accepting of incentive programs. This study provides evidence that incentive-based interventions are acceptable to the public and is the first to document specific objections that individuals disseminating incentive interventions will most likely face when introducing them in medical settings.

Selecting response anchors with equal intervals for summated rating scales.

Summated rating scales are ubiquitous in organizational research, and there are well-delineated guidelines for scale development (e.g., Hinkin, 1998). Nevertheless, there has been less research on the explicit selection of the response anchors. Constructing survey questions with equal-interval properties (i.e., interval or ratio data) is important if researchers plan to analyze their data using parametric statistics. As such, the primary objectives of the current study were to (a) determine the most common contexts in which summated rating scales are used (e.g., agreement, similarity, frequency, amount, and judgment), (b) determine the most commonly used anchors (e.g., strongly disagree, often, very good), and (c) provide empirical data on the conceptual distance between these anchors. We present the mean and standard deviation of scores for estimates of each anchor and the percentage of distribution overlap between the anchors. Our results provide researchers with data that can be used to guide the selection of verbal anchors with equal-interval properties so as to reduce measurement error and improve confidence in the results of subsequent analyses. We also conducted multiple empirical studies to examine the consequences of measuring constructs with unequal-interval anchors. A clear pattern of results is that correlations involving unequal-interval anchors are consistently weaker than correlations involving equal-interval anchors. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Reproducibility of Ocular Surface Staining in the Assessment of Sjögren Syndrome-Related Keratoconjunctivitis Sicca: Implications on Disease Classification.

OBJECTIVE: The objective of this study was to assess the performance and reproducibility of the two currently used ocular surface staining scores in the assessment of keratoconjunctivitis sicca in Sjögren syndrome (SS) research classification. METHODS: In a multidisciplinary clinic for the evaluation of sicca, we performed all tests for the American European Consensus Group (AECG) and the American College of Rheumatology (ACR)/European League Against Rheumatism (EULAR) classification criteria, including the van Bijsterveld score (vBS) and the Ocular Staining Score (OSS), in 994 participants with SS or with non-SS sicca. We analyzed the concordance between the scores, the diagnostic accuracy and correlation with clinical variables, and interrater and intrasubject reproducibility. RESULTS: A total of 308 (31.1%) participants had a discordant vBS and OSS that was due to extra corneal staining points in the OSS. The presence of one or more of the additional points was highly predictive of SS classification (odds ratio = 3.66; P = 1.65 × 10e-20) and was associated with abnormal results of all measures of autoimmunity and glandular dysfunction. Receiver operating characteristic curves showed optimal cutoff values of four for the vBS (sensitivity = 0.62; specificity = 0.71; Youden's J = 0.33) and five for the OSS (sensitivity = 0.56; specificity = 0.75; Youden's J = 0.31). Notably, there was very poor consistency in interobserver mean scores and distributions (P < 0.0001) and in intrasubject scores after a median of 5.5 years (35% changed status of the ocular criterion). CONCLUSION: Ocular surface staining scores are useful for SS research classification; however, they are subject to significant interrater and intrasubject variability, which could result in changes in classification in 5%-10% of all subjects. These results highlight the need for objective and reproducible markers of disease that have thus far remained elusive for SS.

Association of IFIH1 and pro-inflammatory mediators: Potential new clues in SLE-associated pathogenesis.

Antiviral defenses are inappropriately activated in systemic lupus erythematosus (SLE) and association between SLE and the antiviral helicase gene, IFIH1, is well established. We sought to extend the previously reported association of pathogenic soluble mediators and autoantibodies with mouse Mda5 to its human ortholog, IFIH1. To better understand the role this gene plays in human lupus, we assessed association of IFIH1 variants with soluble mediators and autoantibodies in 357 European-American SLE patients, first-degree relatives, and unrelated, unaffected healthy controls. Association between each of 135 genotyped SNPs in IFIH1 and four lupus-associated plasma mediators, IL-6, TNF-α, IFN-ß, and IP-10, were investigated via linear regression. No significant associations were found to SNPs orthologous to those identified in exon 13 of the mouse. However, outside of this region there were significant associations between IL-6 and rs76162067 (p = 0.008), as well as IP-10 and rs79711023 (p = 0.003), located in a region of IFIH1 previously shown to directly influence MDA-5 mediated IP-10 and IL-6 secretion. SLE patients and FDRs carrying the minor allele for rs79711023 demonstrated lower levels of IP-10, while only FDRs carrying the minor allele for rs76162067 demonstrated an increased level of IL-6. This would suggest that the change in IP-10 is genotypically driven, while the change in IL-6 may be reflective of SLE transition status. These data suggest that IFIH1 may contribute to SLE pathogenesis via altered inflammatory mechanisms.

Vitamin d deficiency in a multiethnic healthy control cohort and altered immune response in vitamin D deficient European-American healthy controls.

OBJECTIVE: In recent years, vitamin D has been shown to possess a wide range of immunomodulatory effects. Although there is extensive amount of research on vitamin D, we lack a comprehensive understanding of the prevalence of vitamin D deficiency or the mechanism by which vitamin D regulates the human immune system. This study examined the prevalence and correlates of vitamin D deficiency and the relationship between vitamin D and the immune system in healthy individuals. METHODS: Healthy individuals (n = 774) comprised of European-Americans (EA, n = 470), African-Americans (AA, n = 125), and Native Americans (NA, n = 179) were screened for 25-hydroxyvitamin D [25(OH)D] levels by ELISA. To identify the most noticeable effects of vitamin D on the immune system, 20 EA individuals with severely deficient (<11.3 ng/mL) and sufficient (>24.8 ng/mL) vitamin D levels were matched and selected for further analysis. Serum cytokine level measurement, immune cell phenotyping, and phosphoflow cytometry were performed. RESULTS: Vitamin D sufficiency was observed in 37.5% of the study cohort. By multivariate analysis, AA, NA, and females with a high body mass index (BMI, >30) demonstrate higher rates of vitamin D deficiency (p<0.05). Individuals with vitamin D deficiency had significantly higher levels of serum GM-CSF (p = 0.04), decreased circulating activated CD4+ (p = 0.04) and CD8+ T (p = 0.04) cell frequencies than individuals with sufficient vitamin D levels. CONCLUSION: A large portion of healthy individuals have vitamin D deficiency. These individuals have altered T and B cell responses, indicating that the absence of sufficient vitamin D levels could result in undesirable cellular and molecular alterations ultimately contributing to immune dysregulation.

Revisiting Case IV: a reassessment of bias and standard errors of Case IV under range restriction.

In 2004, Hunter and Schmidt proposed a correction (called Case IV) that seeks to estimate disattenuated correlations when selection is made on an unmeasured variable. Although Case IV is an important theoretical development in the range restriction literature, it makes an untestable assumption, namely that the partial correlation between the unobserved selection variable and the performance measure is zero. We show in this paper why this assumption may be difficult to meet and why previous simulations have failed to detect the full extent of bias. We use meta-analytic literature to investigate the plausible range of bias. We also show how Case IV performs in terms of standard errors. Finally, we give practical recommendations about how the contributions of Hunter and Schmidt (2004) can be extended without making such stringent assumptions.