From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the Journal of Applied Psychology.

The psychometric soundness of measures has been a central concern of articles published in the Journal of Applied Psychology (JAP) since the inception of the journal. At the same time, it isn't clear that investigators and reviewers prioritize psychometric soundness to a degree that would allow one to have sufficient confidence in conclusions regarding constructs. The purposes of the present article are to (a) examine current scale development and evaluation practices in JAP; (b) compare these practices to recommended practices, previous practices, and practices in other journals; and (c) use these comparisons to make recommendations for reviewers, editors, and investigators regarding the creation and evaluation of measures including Excel-based calculators for various indices. Finally, given that model complexity appears to have increased the need for short scales, we offer a user-friendly R Shiny app (https://orgscience.uncc.edu/about-us/resources) that identifies the subset of items that maximize a variety of psychometric criteria rather than merely maximizing alpha. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Using the Nursing Culture Assessment Tool (NCAT) in Long-Term Care: An Update on Psychometrics and Scoring Standardization.

An effective workforce performing within the context of a positive cultural environment is central to a healthcare organization's ability to achieve quality outcomes. The Nursing Culture Assessment Tool (NCAT) provides nurses with a valid and reliable tool that captures the general aspects of nursing culture. This study extends earlier work confirming the tool's construct validity and dimensionality by standardizing the scoring approach and establishing norm-referenced scoring. Scoring standardization provides a reliable point of comparison for NCAT users. NCAT assessments support nursing's ability to evaluate nursing culture, use results to shape the culture into one that supports change, and advance nursing's best practices and care outcomes. Registered nurses, licensed practical nurses, and certified nursing assistants from 54 long-term care facilities in Kentucky, Nevada, North Carolina, and Oregon were surveyed. Confirmatory factor analysis yielded six first order factors forming the NCAT's subscales (Expectations, Behaviors, Teamwork, Communication, Satisfaction, Commitment) (Comparative Fit Index 0.93) and a second order factor-The Total Culture Score. Aggregated facility level comparisons of observed group variance with expected random variance using rwg(J) statistics is presented. Normative scores and cumulative rank percentages and how the NCAT can be used in implementing planned change are provided.

Predictive validity of the MMPI-2 among female offenders in a residential treatment program.

This study investigated the Minnesota Multiphasic Personality Inventory-2 (MMPI-2) validity, clinical, restructured clinical, supplementary, and content scales in a female correctional population. The study used archival data for a final sample of 116 women who had been evaluated for acceptance into a residential rehabilitative treatment program for nonviolent female offenders in North Carolina. MMPI-2 scale elevations are reported and assessed for predictive validity in relation to treatment success, as measured by treatment attendance and graduation status. In relation to predictive validity, logistic regression analyses revealed that elevations on Scales FRS (Fears) and R (Repression) differentiated women who attended the program from women who did not. Elevations on Scales 4, DEP (Depression), Re (Responsibility), and AAS (Addiction Admission Scale) differentiated women who graduated the program from women who did not. Implications for the rehabilitation of female offenders, as well as limitations of this exploratory study, are discussed.

Personality retesting for managing intentional distortion.

Self-report personality questionnaires often contain validity scales designed to flag individuals who intentionally distort their responses toward a more favorable characterization of themselves. Yet, there are no clear directives on how scores on these scales should be used by administrators when making high-stakes decisions about respondents. Two studies were conducted to investigate whether administrator-initiated retesting of flagged individuals represents a viable response to managing intentional distortion on personality questionnaires. We explored the effectiveness of retesting by considering whether retest responses are more accurate representations of a flagged individual's personality characteristics. A comparison of retest scores to a baseline measure of personality indicated that such scores were more accurate. Retesting should only work as a strategy for dealing with intentional distortion when individuals choose to respond more accurately the second time. Thus, we further explored the emotional reaction to being asked to retest as one possible explanation of why individuals who engage in intentional distortion respond more accurately upon retest.

An inductive exploration of the social effectiveness construct space.

There is no agreement regarding the nature or number of dimensions that make up the social effectiveness domain. We inductively explore the relationships between a set of social effectiveness measures with the intention of identifying an initial set of dimensions. An exploratory factor analysis of the Social Competence Inventory (SCI, Schneider, 2001) resulted in the identification of four factors: Social Potency, Social Appropriateness, Social Emotional Expression, and Social Reputation. A joint factor analysis between the SCI and a set of extant measures resulted in the identification of the same four factors. A fifth factor emerged when a set of scales from an emotional intelligence measure was included in the analysis, suggesting that emotional intelligence is not captured within the common factor space defined by measures of social effectiveness. This study represents a first step in the establishment of a set of common social effectiveness dimensions.

An examination of psychometric bias due to retesting on cognitive ability tests in selection settings.

Using a latent variable approach, the authors examined whether retesting on a cognitive ability measure resulted in measurement and predictive bias. A sample of 941 candidates completed a cognitive ability test in a high-stakes context. Results of both the within-group between-occasions comparison and the between-groups within-occasion comparison indicated that no measurement bias existed during the initial testing but that retesting induced both measurement and predictive bias. Specifically, the results suggest that the factor underlying the retest scores was less saturated with g and more associated with memory than the latent factor underlying initial test scores and that these changes eliminated the test's criterion-related validity. This study's implications for retesting theory, practice, and research are discussed.

Forced-choice assessments of personality for selection: evaluating issues of normative assessment and faking resistance.

Recent research suggests multidimensional forced-choice (MFC) response formats may provide resistance to purposeful response distortion on personality assessments. It remains unclear, however, whether these formats provide normative trait information required for selection contexts. The current research evaluated score correspondences between an MFC format measure and 2 Likert-type measures in honest and instructed-faking conditions. In honest response conditions, scores from the MFC measure appeared valid indicators of normative trait standing. Under faking conditions, the MFC measure showed less score inflation than the Likert measure at the group level of analysis. In the individual-level analyses, however, the MFC measure was as affected by faking as was the Likert measure. Results suggest the MFC format is not a viable method to control faking.

The predictive validity of self-efficacy in training performance: little more than past performance.

Past research on the influence of self-efficacy in training has provided mixed results. Key differences between studies pertain to whether past performance is operationalized as a residual variable or as an unadjusted variable and to the type of task used. In this study, the authors conducted and performed a reanalysis to examine the influence of self-efficacy using both operationalizations of past performance in 2 experimental tasks. Results indicate that, regardless of task version or type, self-efficacy predicted performance only when a residual measure of past performance was used, but not when past performance was unadjusted. However, when past performance was adjusted, the findings for self-efficacy were likely a statistical artifact. These results suggest that self-efficacy is a consequence rather than a cause of performance in training.

Interruption management: the use of attention-directing tactile cues.

Previous research has suggested that providing informative cues about interrupting stimuli aids management of multiple tasks. However, auditory and visual cues can be ineffective in certain situations. The objective of the present study was to explore whether attention-directing tactile cues aid or interfere with performance. A two-group posttest-only randomized experiment was conducted. Sixty-one participants completed a 30-min performance session consisting of aircraft-monitoring and gauge-reading computer tasks. Tactile signals were administered to a treatment group to indicate the arrival and location of interrupting tasks. Control participants had to remember to visually check for the interrupting tasks. Participants in the treatment group responded to more interrupting tasks and responded faster than did control participants. Groups did not differ on error rates for the interrupting tasks, performance of the primary task, or subjective workload perceptions. In the context of the tasks used in the present research, tactile cues allowed participants to effectively direct attention where needed without disrupting ongoing information processing. Tactile cues should be explored in a variety of other visual, interrupt-laden environments. Potential applications exist for aviation, user-interface design, vigilance tasks, and team environments.

Faking and selection: considering the use of personality from select-in and select-out perspectives.

The effects of faking on criterion-related validity and the quality of selection decisions are examined in the present study by combining the control of an experiment with the realism of an applicant setting. Participants completed an achievement motivation measure in either a control group or an incentive group and then completed a performance task. With respect to validity, greater prediction error was found in the incentive condition among those with scores at the high end of the predictor distribution. When selection ratios were small, those in the incentive condition were more likely to be selected and had lower mean performance than those in the control group. Implications for using personality assessments from select-in and select-out strategies are discussed.