A primer on method effects on observed correlations with examples.

I use a path/confirmatory factor analytic framework to present an under-the-hood introduction to (a) the effects of measurement error and method variance on observed measures, (b) the effects of common latent Trait and Method factors on relationships between observed measures in the presence of measurement error and (c) examine implications of developments presented here for the possible reinterpretation of Campbell and Fiske's original criteria for convergent and discrimainat validity and presence of common method bias. I use results from one previously published and often-cited primary study and four meta-analyses of multitrait-multimethod studies to provide empirical examples to illustrate points made here. I show analytically that (a) some common interpretations of method effects are correct, (b) others are incorrect and (c) still others are more nuanced than is usually recognised. Finally, I offer some suggested directions for future research.

Effective and adaptable: Four studies on the shortened attitude toward the color blue marker variable scale.

This research is an extension of the recent scale development efforts for the marker variable Attitude Toward the Color Blue (ATCB), which addresses the efficacy of multiple shorter permutations of the scale. The purpose of this study is to develop a shorter version of an ideal marker variable scale used to detect common method variance. Potential uses of the shorter version of ATCB include intensive longitudinal studies, implementation of experience sampling methodology, or any brief survey for which the original version might be cumbersome to implement repeatedly or appear very odd to the respondent when paired with only a few other substantive items. Study 1, uses all six-, five-, and four-item versions of ATCB in confirmatory factor analysis (CFA) marker technique tests on a bivariate relationship. Study 2 analyzes the best- and worst-performing versions of reduced lengths of the ATCB scale found in the first study on another bivariate relationship. Study 3 compares the original seven-item version, as well as randomly selected reduced length versions in a data set with 15 model relationships. Study 4 uses an experiment to determine the efficacy of providing respondents with one of three shorter ATCB scales in a model of three substantive variables. Our findings indicate that ATCB of different permutations and lengths can detect CMV successfully, and that researchers should choose the length of scale based on their survey length. We conclude that ATCB is adaptable for a variety of research situations, presenting it as a valuable tool for high-quality research.

Are Biasing Factors Idiosyncratic to Measures? A Comparison of Interpersonal Conflict, Organizational Constraints, and Workload.

Widespread concern has been raised about the possibility of potential biasing factors influencing the measurement of organizational variables and distorting inferences and conclusions reached about them. Recent research calls for a measure-centric approach in which every measure is independently evaluated to assess what factor(s) may uniquely bias it. This paper examines three popular stressor measures from this perspective. Across three studies, we examine factors that may bias three popular measures of job stressors: The Interpersonal Conflict at Work Scale (ICAWS), the Organizational Constraints Scale (OCS), and the Quantitative Workload Inventory (QWI). The first study used a two-wave design to survey 276 MTurk workers to assess the three stressor scales, four strains, and five measures of potential bias sources: hostile attribution bias, negative affectivity, mood, neutral objects satisfaction, and social desirability. The second study used an experimental design with 439 MTurk workers who were randomly assigned to a positive, negative, or no mood induction condition to assess effects on means of the three stressor measures and their correlations with strains. The third study surveyed 161 employee-supervisor dyads to explore the convergence of results involving the three stressor measures across sources. Based on several forms of evidence we conclude that potential biasing factors affect the three stressor measures differently, supporting the merits of a measure centric approach, even among measures in the same domain.

Structural Equation Modeling of Common Cognitive Abilities in Preschool-Aged Children Using WPPSI-IV and BRIEF-P.

Various studies have addressed the relationship between intelligence and executive functions (EF). There is widespread agreement that EF in preschool children is a unitary construct in which the subordinate factors of Updating, Inhibition, and Shifting are still undifferentiated and correlate moderately with a general factor of intelligence (g). The aim of this study is to investigate the common structural relationship between these two constructs using confirmatory factor analysis. Furthermore, we intend to close the gap of more daily life-associated executive functions and replicate findings in preschool-aged children. Data from a sample of N = 124 average developed children without severe impairments (aged 4 years 0 months-6 years 11 months) were analyzed using the data pool of the standardization and validation studies on the German Wechsler Preschool and Primary Scale of Intelligence-Fourth Edition. Additionally, Executive functions were assessed using a standardized parent-completed questionnaire (BRIEF-P) on their children's everyday behavior. A second-order factor solution revealed that a model with a loading of the common factor of general intelligence (g-factor) onto the EF factor fits the data best. To specify possible method effects due to different sources of measurements, a latent method factor was generated. The results indicate a heterogeneous method effect and a decreasing factor loading from g on to EF while controlling for the method factor.

Learning About Your Mental Health From Your Playlist? Investigating the Correlation Between Music Preference and Mental Health of College Students.

The present study explored the correlation between music preference and mental health of college students to make an empirical contribution to research in this field. The self-reported music preference scale and positive mental health scale of college students were adopted to conduct a questionnaire survey in college students. Common method variance was conducted to test any serious common method bias problem. No serious common method bias problem was observed. The results showed that college students' preference for pop music, Western classical music, and Chinese traditional music has a significant and positive correlation with their mental health. Furthermore, college students' preference for heavy music has a significant and inverse correlation with their mental health. This research presents a correlational study; therefore, no causality can be inferred.

Impact of Social Desirability and Common Method Variance on Two Measures of Entitlement.

Entitlement is the predisposition toward expecting or demanding more rewards than someone else regardless of one's own contribution, effort, or performance. This study examines the impact of social desirability and common method variance on the discriminant validity of the two most popular measures of entitlement: The Psychological Entitlement Scale and the Entitlement subscale of the Narcissistic Personality Inventory. Data were gathered using three independent samples. In all three studies, forcing items to load on their intended and separate constructs measuring entitlement fit the data well thus providing some evidence of discriminant validity. The correlation between the two entitlement constructs was larger than in previous studies but short of that which is necessary for collinearity providing further evidence of discriminant validity in each study. In each study the amount of shared variance between the two latent entitlement constructs was more than the variance explained in either construct thus failing to provide evidence of discriminant validity. These two measures are similarly affected by socially desirable responding but not likely contaminated by common method variance. Because of their consistently and moderately strong correlation, the PES and the entitlement subscale of the NPI likely measure different but strongly related constructs.

Editor's Choice: Consistency tendency and the theory of planned behavior: a randomized controlled crossover trial in a physical activity context.

Objective: This study examined the effects of consistency tendency on the predictive power of the theory of planned behavior (TPB) in relation to physical activity behavior.Methods: In this randomized controlled cross-over trial, we recruited 770 undergraduate students from Indonesia who were randomly assigned into two groups. Participants completed physical activity versions of TPB measures at T1 (baseline) and T2 (post 1 week), and the International Physical Activity Questionnaire at T3 (post 1 month). At T1 and T2, the TPB questions were either presented in ensemble-order (i.e., consistency tendency supressed) or alternate-order (i.e., consistency tendency facilitated).Results: The parameter estimates of the model (CFI > .92, TLI > .90, SRMR < .08, RMSEA < .08) aligned with the tenets of TPB. As compared to ensemble-order, a TPB measured in alternate-order yielded stronger cross-sectional relationships, but this pattern did not appear in the prospective relationships in TPB (i.e., intention/perceived behavioral control and behavior).Conclusions: Consistency tendency inflated the factor correlations of cross-sectionally measured TPB variables, but the inflation was not observed in the prospective prediction of behavior. Health psychology questionnaires with items presented in ensemble order may represent a viable means of reducing the confounding effect of consistency tendency.

Unstable correspondence between salivary testosterone measured with enzyme immunoassays and tandem mass spectrometry.

Although some studies reveal that saliva handling and storage practices may influence salivary testosterone concentrations measured with immunoassays, the effect of these method factors on the validity of testosterone immunoassays remains unknown. The validity of immunoassays can be assessed by comparing hormone concentrations measured with immunoassays to a standard reference method: liquid chromatography tandem mass spectrometry (MS). We previously reported the correspondence between salivary testosterone measured with enzyme immunoassays (EIAs) and with MS when there was less variance in (or more control over) method factors related to saliva handling and storage across measurement methods (Welker et al., 2016). In the present study, we expanded the original dataset and compared the correspondence between Salimetrics EIAs and MS when there was greater variance in (or less control over) method factors across EIAs and MS (high method variance), to when there was less variance in these factors (low method variance). If variance in these method factors impacts the validity of testosterone measurement, then the EIA-MS correspondence should be stronger when method variance is low compared to when it is high. Our results contradicted this hypothesis: Salimetrics EIA-MS correspondence was stronger when variance in method factors was high compared to when it was low. The composite average correlation across both method variance comparisons provides an updated estimate of Salimetrics EIA-MS correspondence, but the instability in this correspondence may pose challenges to the reproducibility of psychoneuroendocrinology research. We discuss possible explanations for the surprising pattern of results and provide recommendations for future research.

Personality Across the Life Span.

Trait stability and maturation are fundamental principles of contemporary personality psychology and have been shown to hold across many cultures. However, it has proven difficult to move beyond these general findings to a detailed account of trait development. There are pervasive and unexplained inconsistencies across studies that may be due to ( a) insufficient attention to measurement error, ( b) subtle but age-sensitive differences in alternative measures of the same trait, or ( c) different perspectives reflected in self-reports and observer ratings. Multiscale, multimethod-and ideally multinational-studies are needed. Several hypotheses have been proposed to account for trait stability and change, but supporting evidence is currently weak or indirect; trait development is a fertile if sometimes frustrating field for theory and research. Beyond traits, there are approaches to personality development that are of interest to students of adult development, and these may be fruitfully addressed from a trait perspective.

Extraversion and life satisfaction: A cross-cultural examination of student and nationally representative samples.

OBJECTIVE: Previous research on Extraversion and life satisfaction suggests that extraverted individuals are more satisfied with their lives. However, existing studies provide inflated effect sizes, as they were based on simple correlations. In five studies, the authors provide better estimates of the relationship between Extraversion and life satisfaction. METHOD: The current study examined student and nationally representative samples from Canada, the United States, the United Kingdom, Germany, and Japan (Study 1, N = 1,460; Study 2, N = 5,882; Study 3, N = 18,683; Study 4, N = 13,443; Study 5, Japan N = 952 and U.S. N = 891). The relationship between Extraversion and life satisfaction was examined using structural equation modeling by regressing life satisfaction on the Big Five traits. RESULTS: Extraversion was a unique predictor of life satisfaction in the North American student and nationally representative samples (Study 1, ß = .232; Study 2, ß = .225; Study 5, ß = .217), but the effect size was weaker or absent in other non-North American samples (Germany, United Kingdom, and Japan). CONCLUSIONS: The findings attest to the moderating role of culture on Extraversion and life satisfaction and the importance of controlling for shared method variance.

Measuring arithmetic: A psychometric approach to understanding formatting effects and domain specificity.

Using multitrait, multimethod data, and confirmatory factor analysis, the current study examined the effects of arithmetic item formatting and the possibility that across formats, abilities other than arithmetic may contribute to children's answers. Measurement hypotheses were guided by several leading theories of arithmetic cognition. With a sample of 1314 3rd grade students (age M=103.24 months, SD=5.41 months), Abstract Code Theory, Encoding Complex Theory, Triple Code Theory, and the Exact versus Approximate Calculations Hypothesis were evaluated, using 11 measures of arithmetic with symbolic problem formats (e.g., Arabic numeral and language-based formats) and various problem demands (e.g., requiring both exact and approximate calculations). In general, results provided support for both Triple Code Theory and Encoding Complex Theory. As predicted by Triple Code Theory, arithmetic outcomes with language formatting, Arabic numeral formatting, and estimation demands (across formats) were related but distinct from one another. As predicted by Encoding Complex Theory, executive attention was a direct predictor of all arithmetic outcomes. Language was no longer a direct predictor of arithmetic outcomes when executive attention was accounted for in the model; however, a strong and enduring relationship between language and executive attention suggested that language may play a facilitative role in reasoning during numeric processing. These findings have important implications for assessing arithmetic in educational settings and suggest that in addition to arithmetic-focused interventions, interventions targeting executive attention, language, and/or the interplay between them (i.e., internal speech during problem-solving) may be a promising avenues of mathematical problem-solving intervention.

We Know the Yin-But Where Is the Yang? Toward a Balanced Approach on Common Source Bias in Public Administration Scholarship.

Surveys have long been a dominant instrument for data collection in public administration. However, it has become widely accepted in the last decade that the usage of a self-reported instrument to measure both the independent and dependent variables results in common source bias (CSB). In turn, CSB is argued to inflate correlations between variables, resulting in biased findings. Subsequently, a narrow blinkered approach on the usage of surveys as single data source has emerged. In this article, we argue that this approach has resulted in an unbalanced perspective on CSB. We argue that claims on CSB are exaggerated, draw upon selective evidence, and project what should be tentative inferences as certainty over large domains of inquiry. We also discuss the perceptual nature of some variables and measurement validity concerns in using archival data. In conclusion, we present a flowchart that public administration scholars can use to analyze CSB concerns.

Isolating Trait and Method Variance in the Measurement of Callous and Unemotional Traits.

To examine hypothesized influence of method variance from negatively keyed items in measurement of callous-unemotional (CU) traits, nine a priori confirmatory factor analysis model comparisons of the Inventory of Callous-Unemotional Traits were evaluated on multiple fit indices and theoretical coherence. Tested models included a unidimensional model, a three-factor model, a three-bifactor model, an item response theory-shortened model, two item-parceled models, and three correlated trait-correlated method minus one models (unidimensional, correlated three-factor, and bifactor). Data were self-reports of 234 adolescents (191 juvenile offenders, 43 high school students; 63% male; ages 11-17 years). Consistent with hypotheses, models accounting for method variance substantially improved fit to the data. Additionally, bifactor models with a general CU factor better fit the data compared with correlated factor models, suggesting a general CU factor is important to understanding the construct of CU traits. Future Inventory of Callous-Unemotional Traits analyses should account for method variance from item keying and response bias to isolate trait variance.

The Driver Behaviour Questionnaire as accident predictor; A methodological re-meta-analysis.

INTRODUCTION: The Manchester Driver Behaviour Questionnaire (DBQ) is the most commonly used self-report tool in traffic safety research and applied settings. It has been claimed that the violation factor of this instrument predicts accident involvement, which was supported by a previous meta-analysis. However, that analysis did not test for methodological effects, or include unpublished results. METHOD: The present study re-analysed studies on prediction of accident involvement from DBQ factors, including lapses, and many unpublished effects. Tests of various types of dissemination bias and common method variance were undertaken. RESULTS: Outlier analysis showed that some effects were probably not reliable data, but excluding them did not change the results. For correlations between violations and crashes, tendencies for published effects to be larger than unpublished ones and for effects to decrease over time were observed, but were not significant. Also, using the mean of accidents as proxy for effect indicated that studies where effects for violations are not reported have smaller effect sizes. These differences indicate dissemination bias. Studies using self-reported accidents as dependent variables had much larger effects than those using recorded accident data. Also, zero-order correlations were larger than partial correlations controlled for exposure. Similarly, violations/accidents effects were strong only when there was also a strong correlation between accidents and exposure. Overall, the true effect is probably very close to zero (r<.07) for violations versus traffic accident involvement, depending upon which tendencies are controlled for. CONCLUSIONS: Methodological factors and dissemination bias have inflated the published effect sizes of the DBQ. Strong evidence of various artefactual effects is apparent. PRACTICAL APPLICATIONS: A greater level of care should be taken if the DBQ continues to be used in traffic safety research. Also, validation of self-reports should be more comprehensive in the future, taking into account the possibility of common method variance.

Examining Method Effect of Synonym and Antonym Test in Verbal Abilities Measure.

Many researchers have assumed that different methods could be substituted to measure the same attributes in assessment. Various models have been developed to accommodate the amount of variance attributable to the methods but these models application in empirical research is rare. The present study applied one of those models to examine whether method effects were presents in synonym and antonym tests. Study participants were 3,469 applicants to graduate school. The instrument used was the Graduate Academic Potential Test (PAPS), which includes synonym and antonym questions to measure verbal abilities. Our analysis showed that measurement models that using correlated trait-correlated methods minus one, CT-C(M-1), that separated trait and method effect into distinct latent constructs yielded slightly better values for multiple goodness-of-fit indices than one factor model. However, either for the synonym or antonym items, the proportion of variance accounted for by the method is smaller than trait variance. The correlation between factor scores of both methods is high (r = 0.994). These findings confirm that synonym and antonym tests represent the same attribute so that both tests cannot be treated as two unique methods for measuring verbal ability.

A generalized linear factor model approach to the hierarchical framework for responses and response times.

We show how the hierarchical model for responses and response times as developed by van der Linden (2007), Fox, Klein Entink, and van der Linden (2007), Klein Entink, Fox, and van der Linden (2009), and Glas and van der Linden (2010) can be simplified to a generalized linear factor model with only the mild restriction that there is no hierarchical model at the item side. This result is valuable as it enables all well-developed modelling tools and extensions that come with these methods. We show that the restriction we impose on the hierarchical model does not influence parameter recovery under realistic circumstances. In addition, we present two illustrative real data analyses to demonstrate the practical benefits of our approach.

The Promotion Strategy of Green Construction Materials: A Path Analysis Approach.

As one of the major materials used in construction, cement can be very resource-consuming and polluting to produce and use. Compared with traditional cement processing methods, dry-mix mortar is more environmentally friendly by reducing waste production or carbon emissions. Despite the continuous development and promotion of green construction materials, only a few of them are accepted or widely used in the market. In addition, the majority of existing research on green construction materials focuses more on their physical or chemical characteristics than on their promotion. Without effective promotion, their benefits cannot be fully appreciated and realized. Therefore, this study is conducted to explore the promotion of dry-mix mortars, one of the green materials. This study uses both qualitative and quantitative methods. First, through a case study, the potential of reducing carbon emission is verified. Then a path analysis is conducted to verify the validity and predictability of the samples based on the technology acceptance model (TAM) in this study. According to the findings of this research, to ensure better promotion results and wider application of dry-mix mortar, it is suggested that more systematic efforts be invested in promoting the usefulness and benefits of dry-mix mortar. The model developed in this study can provide helpful references for future research and promotion of other green materials.

Neuroticism explains unwanted variance in Implicit Association Tests of personality: possible evidence for an affective valence confound.

Meta-analytic data highlight the value of the Implicit Association Test (IAT) as an indirect measure of personality. Based on evidence suggesting that confounding factors such as cognitive abilities contribute to the IAT effect, this study provides a first investigation of whether basic personality traits explain unwanted variance in the IAT. In a gender-balanced sample of 204 volunteers, the Big-Five dimensions were assessed via self-report, peer-report, and IAT. By means of structural equation modeling (SEM), latent Big-Five personality factors (based on self- and peer-report) were estimated and their predictive value for unwanted variance in the IAT was examined. In a first analysis, unwanted variance was defined in the sense of method-specific variance which may result from differences in task demands between the two IAT block conditions and which can be mirrored by the absolute size of the IAT effects. In a second analysis, unwanted variance was examined in a broader sense defined as those systematic variance components in the raw IAT scores that are not explained by the latent implicit personality factors. In contrast to the absolute IAT scores, this also considers biases associated with the direction of IAT effects (i.e., whether they are positive or negative in sign), biases that might result, for example, from the IAT's stimulus or category features. None of the explicit Big-Five factors was predictive for method-specific variance in the IATs (first analysis). However, when considering unwanted variance that goes beyond pure method-specific variance (second analysis), a substantial effect of neuroticism occurred that may have been driven by the affective valence of IAT attribute categories and the facilitated processing of negative stimuli, typically associated with neuroticism. The findings thus point to the necessity of using attribute category labels and stimuli of similar affective valence in personality IATs to avoid confounding due to recoding.

Self-other agreement and assumed similarity in neuroticism, extraversion, and trait affect: distinguishing the effects of form and content.

Trait Negative Affect (NA) and Positive Affect (PA) are strongly associated with Neuroticism and Extraversion, respectively. Nevertheless, measures of the former tend to show substantially weaker self-other agreement-and stronger assumed similarity correlations-than scales assessing the latter. The current study separated the effects of item content versus format on agreement and assumed similarity using two different sets of Neuroticism and Extraversion measures and two different indicators of NA and PA (N = 381 newlyweds). Neuroticism and Extraversion consistently showed stronger agreement than NA and PA; in addition, however, scales with more elaborated items yielded significantly higher agreement correlations than those based on single adjectives. Conversely, the trait affect scales yielded stronger assumed similarity correlations than the personality scales; these coefficients were strongest for the adjectival measures of trait affect. Thus, our data establish a significant role for both content and format in assumed similarity and self-other agreement.