Modeling Multidimensional Forced Choice Measures with the Zinnes and Griggs Pairwise Preference Item Response Theory Model.

This research developed a new ideal point-based item response theory (IRT) model for multidimensional forced choice (MFC) measures. We adapted the Zinnes and Griggs (ZG; 1974) IRT model and the multi-unidimensional pairwise preference (MUPP; Stark et al., 2005) model, henceforth referred to as ZG-MUPP. We derived the information function to evaluate the psychometric properties of MFC measures and developed a model parameter estimation algorithm using Markov chain Monte Carlo (MCMC). To evaluate the efficacy of the proposed model, we conducted a simulation study under various experimental conditions such as sample sizes, number of items, and ranges of discrimination and location parameters. The results showed that the model parameters were accurately estimated when the sample size was as low as 500. The empirical results also showed that the scores from the ZG-MUPP model were comparable to those from the MUPP model and the Thurstonian IRT (TIRT) model. Practical implications and limitations are further discussed.

Comparison of parameter estimation approaches for multi-unidimensional pairwise preference tests.

Multidimensional forced-choice (MFC) testing has been proposed as a way of reducing response biases in noncognitive measurement. Although early item response theory (IRT) research focused on illustrating that person parameter estimates with normative properties could be obtained using various MFC models and formats, more recent attention has been devoted to exploring the processes involved in test construction and how that influences MFC scores. This research compared two approaches for estimating multi-unidimensional pairwise preference model (MUPP; Stark et al., 2005) parameters based on the generalized graded unfolding model (GGUM; Roberts et al., 2000). More specifically, we compared the efficacy of statement and person parameter estimation based on a "two-step" process, developed by Stark et al. (2005), with a more recently developed "direct" estimation approach (Lee et al., 2019) in a Monte Carlo study that also manipulated test length, test dimensionality, sample size, and the correlations between generating person parameters for each dimension. Results indicated that the two approaches had similar scoring accuracy, although the two-step approach had better statement parameter recovery than the direct approach. Limitations, implications for MFC test construction and scoring, and recommendations for future MFC research and practice are discussed.

The Development and Validation of a Multidimensional Forced-Choice Format Character Measure: Testing the Thurstonian IRT Approach.

There has been reemerging interest within psychology in the construct of character, yet assessing it can be difficult due to social desirability of character traits. Forced-choice formats offer one way to address response bias, but traditional scoring methods (i.e., ipsative) associated with this format makes comparing scores between people problematic. Nevertheless, recent advances in modeling item responding (Thurstonian IRT) enable scoring that recovers absolute standing on latent traits and allows for score comparisons between people. Based on recent work in character measurement (CIVIC), we developed a multidimensional forced-choice measure of character (CIVIC-MFC) and scored it using Thurstonian IRT. Initial validation using a sample of 798 participants demonstrated good support for factorial, convergent, and concurrent validity for scores on the CIVIC-MFC, although they did not demonstrate more faking resistance than scores on a Likert-type format version. Potential explanations are discussed.

Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring.

Likert-type measures have been criticized in psychological assessment because they are vulnerable to response biases, including central tendency, acquiescence, leniency, halo, and socially desirable responding. As an alternative, multidimensional forced choice (MFC) testing has been proposed to address these concerns. A number of researchers have developed item response theory (IRT) models for MFC data and have examined latent trait estimation with tests of different dimensionality and length. Research has also explored the advantages of computerized adaptive testing (CAT) with MFC pair tests having as many as 25 dimensions, but there have been no published studies on CAT with MFC triplets or tetrads. Thus, in this research we aimed to address that issue. We used recently developed item information functions for an MFC ranking model to compare the benefits of CAT with MFC pair, triplet, and tetrad tests. A simulation study showed that CAT substantially outperformed nonadaptive testing for latent trait estimation across MFC formats. More importantly, CAT with MFC pairs provided estimation accuracy similar to or better than that from tests of equivalent numbers of nonadaptive MFC triplets. On the basis of these findings, implications and recommendations are further discussed for constructing MFC measures to use in psychological contexts.

Predictors of attitudes and performance in U.S. Army recruiters: Does personality matter?

Given the interpersonal nature of recruiting and the validity of personality assessments for predicting performance in a broad range of civilian and military jobs, personality traits are likely to predict the performance of recruiters in the Army as well. However, much of the research on the characteristics of successful recruiters has been conducted in civilian samples and has not examined the effects of recruiters' personality on their job-related attitudes and behaviors. Although some research has examined the prediction of recruiter performance in a military context, more research is needed to identify profiles of personality traits that will help recruiters to be successful on the job. We explored this relationship in a sample of experienced recruiters with at least six months of service in a recruiting duty assignment. Results indicated that composites of personality traits were substantial predictors of recruiter performance and attitudes. The implications of these results for the selection and assessment of recruiters in the U.S. Army will be discussed.

Adaptive vocational interest diagnostic: Informing and improving the job assignment process.

The history of vocational interests shows that these measures have great promise for use in job assignment, suggesting that individuals will be more satisfied and successful in their job when they are doing work that interests them. Recent research has provided empirical support for these predictions and demonstrated that the match between an individual's interests and his or her work activities is positively related to job performance and negatively related to attrition. Building on these positive empirical findings, the U.S. Army Research Institute is investigating vocational interest measures for personnel job assignment. Person-job fit is very important in a context such as the U.S. Army, where applicants have over 140 military occupational specialties from which to choose. This paper begins by reviewing evidence for the validity of interests and discussing how vocational interest measures may be used for assigning Soldiers in a military context followed by our recent research to develop a new measure of vocational interests to improve the process of matching Soldiers to military occupational specialties. We will conclude with the next steps for this research and potential paths of implementation.

Examining personality for the selection and classification of soldiers: Validity and differential validity across jobs.

A number of past studies have demonstrated that personality traits are modest predictors of workplace attitudes and behaviors and can provide incremental validity over cognitive ability. However, less is known about the utility of personality for job classification. In addition, concerns about the effects of faking on personality measures still remain. In this study, we examined the validity of a forced choice personality measure administered under operational conditions to explore the use of personality traits in high-stakes settings. In addition, we also examined the potential use of personality for classification into military occupational specialties (MOS). We explored these issues in a large sample of Soldiers from five different MOS to examine the prediction of performance during initial military training (IMT). Results indicated that composites of personality traits were valid predictors of performance and attrition and that these composites may be useful for classifying individuals into different military occupations. The implications of these results for Soldier selection and classification are discussed.

Validity of six openness facets in predicting work behaviors: a meta-analysis.

To illustrate the importance of facet-level investigations in predicting organizational outcomes and the need for more primary studies on this topic, we conducted a meta-analysis that took an exploratory look at differential relationships among 7 organizational criteria and Openness traits varying in breadth. Nine Openness predictors--the global dimension, 2 aspects (intellect and culture), and 6 facets (intellectual efficiency, ingenuity, curiosity, aesthetics, tolerance, and depth)--were examined in relations with various organizational criteria such as traditional performance outcomes (task performance, contextual performance, counterproductive work behavior), turnover, leadership effectiveness, training performance, and adaptive performance. Our results support the idea that Openness facets could exhibit differential validity for many organizational outcomes.

Openness to Experience: its lower level structure, measurement, and cross-cultural equivalence.

Openness to Experience is an important but relatively poorly understood personality construct. Advances in openness research require further construct clarification as well as establishment of a common framework for conceptualizing and measuring the lower level structure of the construct. In this article, we present data from 3 studies to address this research need. In Study 1, we identify 6 facets of Openness to Experience--intellectual efficiency, ingenuity, curiosity, aesthetics, tolerance, and depth--based on a factor analysis of 36 existing Openness-related scales. In Study 2, we present further validity evidence for the 6-facet structure based on a newly developed measure of Openness. Data from this study also suggest the presence of 2 intermediate-level factors (i.e., aspects) of Openness: intellect and culture. In Study 3, we present a short form of the newly developed measure, retaining items that showed the highest internal consistency and measurement invariance across 3 samples: U.S. undergraduates, Chinese MBA students, and Chinese undergraduates. Together these 3 studies offer a more nuanced understanding of the multifaceted nature of the Openness construct.

Linking Methods for Multidimensional Forced Choice Tests Using the Multi-Unidimensional Pairwise Preference Model.

Applications of multidimensional forced choice (MFC) testing have increased considerably over the last 20 years. Yet there has been little, if any, research on methods for linking the parameter estimates from different samples. This research addressed that important need by extending four widely used methods for unidimensional linking and comparing the efficacy of new estimation algorithms for MFC linking coefficients based on the Multi-Unidimensional Pairwise Preference model (MUPP). More specifically, we compared the efficacy of multidimensional test characteristic curve (TCC), item characteristic curve (ICC; Haebara, 1980), mean/mean (M/M), and mean/sigma (M/S) methods in a Monte Carlo study that also manipulated test length, test dimensionality, sample size, percentage of anchor items, and linking scenarios. Results indicated that the ICC method outperformed the M/M method, which was better than the M/S method, with the TCC method being the least effective. However, as the number of items "per dimension" and the percentage of anchor items increased, the differences between the ICC, M/M, and M/S methods decreased. Study implications and practical recommendations for MUPP linking, as well as limitations, are discussed.

Piloting a novel medical student virtual discharge counseling process in the time of the COVID-19 pandemic.

BACKGROUND: During the COVID-19 pandemic, we identified a gap in adequate discharge counseling for COVID-19 patients in the Emergency Department. This was due to high patient volumes and lack of patient education regarding a novel disease. Medical students were also restricted from clinical areas due to safety concerns, compromising their clinical experience. We piloted a novel program in which medical students served as virtual discharge counselors for COVID-19 patients via teleconference. We aimed to demonstrate an impact on patient care by examining the patient bounce back rate as well as assessing medical student education and experience. METHODS: This program was piloted in a tertiary care Emergency Department. Medical student volunteers served as virtual discharge counselors. Students were trained in discharge counseling with a standardized protocol and a discharge script. Eligible patients for virtual discharge counseling were 18 years old or greater with a diagnosis of confirmed or suspected COVID-19 and no impediment precluding them from participating in a telemedicine encounter. Counseling was provided via secure teleconference in the patient's preferred language. Counseling included diagnosis, supportive care with medication dosing, quarantine instructions, return precautions, follow up, and time to ask questions. Duration of counseling was recorded and medical students were anonymously surveyed regarding their experience. RESULTS: Over an 18-week period, 45 patients were counseled for a median of 20 min. The 72-hr ED revisit rate was 0%, versus 4.2% in similarly-matched, not counseled COVID-19 patients. 90% of medical students believed this project increased their confidence when speaking with patients while 80% indicated this was their first telemedicine experience. CONCLUSION: Our pilot discharge program provided patients with an extensive discharge counseling experience that would not otherwise be possible in an urban ED setting and demonstrated benefit to patient care. Medical students received a safe clinical experience that improved their communication skills.

Measuring vaping-related expectancies in young adults: Psychometric evaluation of the Electronic Nicotine Vaping Outcomes (ENVO) scale.

OBJECTIVE: Electronic cigarettes are the most commonly used tobacco products by young adults. Measures of beliefs about outcomes of use (i.e., expectancies) can be helpful in predicting use, as well as informing and evaluating interventions to impact use. METHODS: We surveyed young adult students (N = 2296, Mean age=20.0, SD=1.8, 64 % female, 34 % White) from a community college, a historically black university, and a state university. Students answered ENDS expectancy items derived from focus groups and expert panel refinement using Delphi methods. Factor Analysis and Item Response Theory (IRT) methods were used to understand relevant factors and identify useful items. RESULTS: A 5-factor solution [Positive Reinforcement (consists of Stimulation, Sensorimotor, and Taste subthemes, α = .92), Negative Consequences (Health Risks and Stigma, α = .94), Negative Affect Reduction (α = .95), Weight Control (α = .92), and Addiction (α = .87)] fit the data well (CFI=0.95; TLI=0.94; RMSEA=0.05) and was invariant across subgroups. Factors were significantly correlated with relevant vaping measures, including vaping susceptibility and lifetime vaping. Hierarchical linear regression demonstrated factors were significant predictors of lifetime vaping after controlling for demographics, vaping ad exposure, and peer/family vaping. IRT analyses indicated that individual items tended to be related to their underlying constructs (a parameters ranged from 1.26 to 3.18) and covered a relatively wide range of the expectancies continuum (b parameters ranged from -0.72 to 2.47). CONCLUSIONS: A novel ENDS expectancy measure appears to be a reliable measure for young adults with promising results in the domains of concurrent validity, incremental validity, and IRT characteristics. This tool may be helpful in predicting use and informing future interventions. IMPLICATIONS: Findings provide support for the future development of computerized adaptive testing of vaping beliefs. Expectancies appear to play a role in vaping similar to smoking and other substance use. Public health messaging should target expectancies to modify young adult vaping behavior.

The Explanatory Generalized Graded Unfolding Model: Incorporating Collateral Information to Improve the Latent Trait Estimation Accuracy.

Collateral information has been used to address subpopulation heterogeneity and increase estimation accuracy in some large-scale cognitive assessments. The methodology that takes collateral information into account has not been developed and explored in published research with models designed specifically for noncognitive measurement. Because the accurate noncognitive measurement is becoming increasingly important, we sought to examine the benefits of using collateral information in latent trait estimation with an item response theory model that has proven valuable for noncognitive testing, namely, the generalized graded unfolding model (GGUM). Our presentation introduces an extension of the GGUM that incorporates collateral information, henceforth called Explanatory GGUM. We then present a simulation study that examined Explanatory GGUM latent trait estimation as a function of sample size, test length, number of background covariates, and correlation between the covariates and the latent trait. Results indicated the Explanatory GGUM approach provides scoring accuracy and precision superior to traditional expected a posteriori (EAP) and full Bayesian (FB) methods. Implications and recommendations are discussed.

Bayesian Approaches for Detecting Differential Item Functioning Using the Generalized Graded Unfolding Model.

Differential item functioning (DIF) analysis is one of the most important applications of item response theory (IRT) in psychological assessment. This study examined the performance of two Bayesian DIF methods, Bayes factor (BF) and deviance information criterion (DIC), with the generalized graded unfolding model (GGUM). The Type I error and power were investigated in a Monte Carlo simulation that manipulated sample size, DIF source, DIF size, DIF location, subpopulation trait distribution, and type of baseline model. We also examined the performance of two likelihood-based methods, the likelihood ratio (LR) test and Akaike information criterion (AIC), using marginal maximum likelihood (MML) estimation for comparison with past DIF research. The results indicated that the proposed BF and DIC methods provided well-controlled Type I error and high power using a free-baseline model implementation, their performance was superior to LR and AIC in terms of Type I error rates when the reference and focal group trait distributions differed. The implications and recommendations for applied research are discussed.

Testing Measurement Invariance Across Unobserved Groups: The Role of Covariates in Factor Mixture Modeling.

Factor mixture modeling (FMM) has been increasingly used to investigate unobserved population heterogeneity. This study examined the issue of covariate effects with FMM in the context of measurement invariance testing. Specifically, the impact of excluding and misspecifying covariate effects on measurement invariance testing and class enumeration was investigated via Monte Carlo simulations. Data were generated based on FMM models with (1) a zero covariate effect, (2) a covariate effect on the latent class variable, and (3) covariate effects on both the latent class variable and the factor. For each population model, different analysis models that excluded or misspecified covariate effects were fitted. Results highlighted the importance of including proper covariates in measurement invariance testing and evidenced the utility of a model comparison approach in searching for the correct specification of covariate effects and the level of measurement invariance. This approach was demonstrated using an empirical data set. Implications for methodological and applied research are discussed.

Item Parameter Estimation With the General Hyperbolic Cosine Ideal Point IRT Model.

Over the last decade, researchers have come to recognize the benefits of ideal point item response theory (IRT) models for noncognitive measurement. Although most applied studies have utilized the Generalized Graded Unfolding Model (GGUM), many others have been developed. Most notably, David Andrich and colleagues published a series of papers comparing dominance and ideal point measurement perspectives, and they proposed ideal point models for dichotomous and polytomous single-stimulus responses, known as the Hyperbolic Cosine Model (HCM) and the General Hyperbolic Cosine Model (GHCM), respectively. These models have item response functions resembling the GGUM and its more constrained forms, but they are mathematically simpler. Despite the apparent impact of Andrich's work on ensuing investigations, the HCM and GHCM have been largely overlooked by applied researchers. This may stem from questions about the compatibility of the parameter metric with other ideal point estimation and model-data fit software or seemingly unrealistic parameter estimates sometimes produced by the original joint maximum likelihood (JML) estimation software. Given the growing list of ideal point applications and variations in sample and scale characteristics, the authors believe these HCMs warrant renewed consideration. To address this need and overcome potential JML estimation difficulties, this study developed a marginal maximum likelihood (MML) estimation algorithm for the GHCM and explored parameter estimation requirements in a Monte Carlo study manipulating sample size, scale length, and data types. The authors found a sample size of 400 was adequate for parameter estimation and, in accordance with GGUM studies, estimation was superior in polytomous conditions.

GGUM-RANK Statement and Person Parameter Estimation With Multidimensional Forced Choice Triplets.

Historically, multidimensional forced choice (MFC) measures have been criticized because conventional scoring methods can lead to ipsativity problems that render scores unsuitable for interindividual comparisons. However, with the recent advent of item response theory (IRT) scoring methods that yield normative information, MFC measures are surging in popularity and becoming important components in high-stake evaluation settings. This article aims to add to burgeoning methodological advances in MFC measurement by focusing on statement and person parameter recovery for the GGUM-RANK (generalized graded unfolding-RANK) IRT model. Markov chain Monte Carlo (MCMC) algorithm was developed for estimating GGUM-RANK statement and person parameters directly from MFC rank responses. In simulation studies, it was examined that how the psychometric properties of statements composing MFC items, test length, and sample size influenced statement and person parameter estimation; and it was explored for the benefits of measurement using MFC triplets relative to pairs. To demonstrate this methodology, an empirical validity study was then conducted using an MFC triplet personality measure. The results and implications of these studies for future research and practice are discussed.

Exploring the Test of Covariate Moderation Effects in Multilevel MIMIC Models.

In multilevel multiple-indicator multiple-cause (MIMIC) models, covariates can interact at the within level, at the between level, or across levels. This study examines the performance of multilevel MIMIC models in estimating and detecting the interaction effect of two covariates through a simulation and provides an empirical demonstration of modeling the interaction in multilevel MIMIC models. The design factors include the location of the interaction effect (i.e., between, within, or across levels), cluster number, cluster size, intraclass correlation (ICC) level, magnitude of the interaction effect, and cross-level measurement invariance status. Type I error, power, relative bias, and root mean square of error of the interaction effects are examined. The results showed that multilevel MIMIC models performed well in detecting the interaction effect at the within or across levels. However, when the interaction effect was at the between level, the performance of multilevel MIMIC models depended on the magnitude of the interaction effect, ICC, and sample size, especially cluster number. Overall, cross-level measurement noninvariance did not make a notable impact on the estimation of interaction in the structural part of multilevel MIMIC models when factor loadings were allowed to be different across levels.

Thoracoscopic Left Atrial Appendage Clipping: A Multicenter Cohort Analysis.

OBJECTIVES: This study sought to document the closure rate, safety, and stroke rate after thoracoscopic left atrial appendage (LAA) clipping. BACKGROUND: The LAA is the main source of stroke in patients with atrial fibrillation, and thoracoscopic clipping may provide a durable and safe closure technique. METHODS: The investigators studied consecutive patients undergoing clipping as part of a thoracoscopic maze procedure in 4 referral centers (the Netherlands and the United States) from 2012 to 2016. Completeness of LAA closure was assessed by either computed tomography (n = 100) or transesophageal echocardiography (n = 122). The primary outcome was complete LAA closure (absence of residual LAA flow and pouch <10 mm). The secondary outcomes were 30-day complications; the composite of ischemic stroke, hemorrhagic stroke, or transient ischemic attack; and all-cause mortality. RESULTS: A total of 222 patients were included, with a mean age of 66 ± 9 years, and 68.5% were male. The mean CHA2DS2-VASc (congestive heart failure, hypertension, age ≥75 years, diabetes mellitus, prior stroke or transient ischemic attack or thromboembolism, vascular disease, age 65 to 74 years, sex category [female]) score was 2.3 ± 1.0. Complete LAA closure was achieved in 95.0% of patients. There were no intraoperative or clip-related complications, and the overall 30-day freedom from any complication rate was 96.4%. The freedom from cerebrovascular events after surgery was 99.1% after median follow-up of 20 months (interquartile range: 14 to 25 months; 369 patient-years of follow-up), and overall survival was 98.6%. The observed rate of cerebrovascular events after LAA clipping was low (0.5 per 100 patient-years). CONCLUSIONS: LAA clipping during thoracoscopic ablation is a feasible and safe technique for closure of the LAA in patients with atrial fibrillation. The lower than expected rate of cerebrovascular events after deployment was likely multifactorial, including not only LAA closure, but also the effect of oral anticoagulation and rhythm control.

Constructing personality scales under the assumptions of an ideal point response process: toward increasing the flexibility of personality measures.

The main aim of this article is to explicate why a transition to ideal point methods of scale construction is needed to advance the field of personality assessment. The study empirically demonstrated the substantive benefits of ideal point methodology as compared with the dominance framework underlying traditional methods of scale construction. Specifically, using a large, heterogeneous pool of order items, the authors constructed scales using traditional classical test theory, dominance item response theory (IRT), and ideal point IRT methods. The merits of each method were examined in terms of item pool utilization, model-data fit, measurement precision, and construct and criterion-related validity. Results show that adoption of the ideal point approach provided a more flexible platform for creating future personality measures, and this transition did not adversely affect the validity of personality test scores.