A Multidimensional Model to Facilitate Within Person Comparison of Attributes.

In psychological research and practice, a person's scores on two different traits or abilities are often compared. Such within-person comparisons require that measurements have equal units (EU) and/or equal origins: an assumption rarely validated. We describe a multidimensional SEM/IRT model from the literature and, using principles of conjoint measurement, show that its expected response variables satisfy the axioms of additive conjoint measurement for measurement on a common scale. In an application to Quality of Life data, the EU analysis is used as a pre-processing step to derive a simple structure Quality of Life model with three dimensions expressed in equal units. The results are used to address questions that can only be addressed by scores expressed in equal units. When the EU model fits the data, scores in the corresponding simple structure model will have added validity in that they can address questions that cannot otherwise be addressed. Limitations and the need for further research are discussed.

Linear equality constraints: Reformulations of criterion related profile analysis with extensions to moderated regression for multiple groups.

Criterion-related profile analysis (CPA) is a least squares linear regression technique for identifying a criterion-related pattern (CRP) among predictor variables and for quantifying the variance accounted for by the pattern. A CRP is a pattern, described by a vector of contrast coefficients, such that predictor profiles with higher similarity to the pattern have higher expected criterion scores. A review of applications shows that researchers have extended the analysis to meta-analyses, logit regression, canonical regression, and structural equation modeling. It also reveals a need for better methods of comparing CRPs across populations. While the original method for identifying the CRP tends to underestimate the variance accounted for by pattern only, both the pattern identified by the original method and the pattern identified by the new method proposed here have useful and complementary interpretations. Imposing linear equality constraints on regression coefficients yields a more accurate method of estimating the variance accounted for by pattern only, and this constrained approach leads to moderated regression models for investigating whether the CRP is the same in two or more populations. Finally, we show how the elements in Cronbach and Gleser's (1953) classic profile decomposition are related to the linear regression model and the CPA model. Academic ability tests as predictors of college GPA are used to illustrate the analyses. Implications of the profile pattern models for psychological theory and applied decision-making are discussed. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

MOCCA-College: Preliminary Validity Evidence of a Cognitive Diagnostic Reading Comprehension Assessment.

As access to higher education increases, it is important to monitor students with special needs to facilitate the provision of appropriate resources and support. Although metrics such as the "reading readiness" ACT (formerly American College Testing) of provide insight into how many students may need such resources, they do not specify why a student may need support or how to provide that support. Increasingly, students are bringing reading comprehension struggles to college. Multiple-choice Online Causal Comprehension Assessment-College (MOCCA-College) is a new diagnostic reading comprehension assessment designed to identify who is a poor comprehender and also diagnose why they are a poor comprehender. Using reliability coefficients, receiver-operating characteristic curve analysis, and correlations, this study reports findings from the first year of a 3-year study to validate the assessment with 988 postsecondary students who took MOCCA-College, a subset of whom also provided data on other reading assessments (i.e., ACT, n = 377; Scholastic Aptitude Test [SAT], n = 192; and Nelson-Denny Reading Test [NDRT], n = 78). Despite some limitations (e.g., the sample is predominantly females from 4-year institutions), results indicate that MOCCA-College has good internal reliability, and scores are correlated with other reading assessments. Through a series of analyses of variance (ANOVAs), we also report how students identified by MOCCA-College as good and poor comprehenders differ in terms of demographics, cognitive processes used while reading, overall comprehension ability, and scores on admissions tests. Findings are discussed in terms of using MOCCA-College to help gauge which students may be at risk of reading comprehension difficulties, identify why they may be struggling, and inform directions in actionable instructional changes based on comprehension processing data.

Regression with reduced rank predictor matrices: A model of trade-offs.

A regression model of predictor trade-offs is described. Each regression parameter equals the expected change in Y obtained by trading 1 point from one predictor to a second predictor. The model applies to predictor variables that sum to a constant T for all observations; for example, proportions summing to T = 1.0 or percentages summing to T = 100 for each observation. If predictor variables sum to a constant T for all observations and if a least squares solution exists, the predicted values for the criterion variable Y will be uniquely determined, but there will be an infinite set of linear regression weights and the familiar interpretation of regression weights does not apply. However, the regression weights are determined up to an additive constant and thus differences in regression weights ßv-ßv∗ are uniquely determined, readily estimable, and interpretable. ßv-ßv∗ is the expected increase in Y given a transfer of 1 point from variable v∗ to variable v. The model is applied to multiple-choice test items that have four response categories, one correct and three incorrect. Results indicate that the expected outcome depends, not just on the student's number of correct answers, but also on how the student's incorrect responses are distributed over the three incorrect response types. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Bayesian modeling of associations in bivariate piecewise linear mixed-effects models.

Longitudinal processes rarely occur in isolation; often the growth curves of 2 or more variables are interdependent. Moreover, growth curves rarely exhibit a constant pattern of change. Many educational and psychological phenomena are comprised of different developmental phases (segments). Bivariate piecewise linear mixed-effects models (BPLMEM) are a useful and flexible statistical framework that allow simultaneous modeling of 2 processes that portray segmented change and investigates their associations over time. The purpose of the present study was to develop a BPLMEM using a Bayesian inference approach allowing the estimation of the association between the error variances and providing a more robust modeling choice for the joint random-effects of the 2 processes. This study aims to improve upon the limitations of the prior literature on bivariate piecewise mixed-effects models, such as only allowing the modeling of uncorrelated residual errors across the 2 longitudinal processes and restricting modeling choices for the random effects. The performance of the BPLMEM was investigated via a Monte Carlo simulation study. Furthermore, the utility of BPLMEM was illustrated by using a national educational dataset, Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K), where we examined the joint development of mathematics and reading achievement scores and the association between their trajectories over 7 measurement occasions. The findings obtained shed new light on the relationship between these 2 prominent educational domains over time. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Addressing Quantitative and Qualitative Hypotheses Using Regression Models with Equality Restrictions and Predictors Measured in Common Units.

In regression, some or all of the predictors may be measured in common units: e.g. X1 = carbohydrate calories, X2 = protein calories, X3 = fat calories. Such predictors can occur in disciplines as diverse as business, economics, education, medicine, nutrition, psychology, sport science, etc. Predictors in common units can lead to unique quantitative and qualitative hypotheses that can be addressed by imposing equality restrictions on the regression weights (e.g. b1=b2=b3). A simple device, total score substitution, is available for constraining regression coefficients to be equal in a variety of regression applications. Applications to linear, moderated linear, and polynomial models are described, but extensions to generalized linear models and multilevel linear models are also possible. Total score substitution in linear and moderated regression is illustrated using high school coursework and mathematics achievement data. Data, code (R, SPSS, SAS), and output are publicly available.

Item and scale properties of the Oral Health Literacy Adults Questionnaire assessed by item response theory.

OBJECTIVES: a) To evaluate the item and scale properties of the Oral Health Literacy Adults Questionnaire (OHL-AQ) in an adult general population. b) To determine precision or accuracy of the respondents' estimated scores along the Oral Health Literacy (OHL) spectrum using item response theory (IRT) modeling. METHODS: Survey data were collected from a convenience sample of 405 adult attendees of the 2014 Minnesota State Fair. We used the two-parameter logistic (2PL) model for the item response theory (IRT) analyses of OHL-AQ data and calibrated items to estimate model-based item difficulty and discrimination parameters. Item and scale properties were also assessed by plotting and interpreting item characteristic curves (ICCs), test characteristic curve (TCC), and test information function (TIF). RESULTS: Based on interpretation of model coefficients, statistical testing, and model fit criteria, we deemed the 2PL model superior and selected this model to examine item and scale properties. Scale reliability was shown to be good through the test information function (TIF). TIF from our analysis showed that higher levels of OHL were measured less precisely than lower levels of OHL. CONCLUSION: We demonstrated OHL-AQ as a whole has promising psychometric properties. However, for equiprecise measurement across the scale range, the scale needs more items for measuring higher levels of OHL.

Meta-analytic criterion profile analysis.

Intraindividual patterns or configurations are intuitive explanations for phenomena, and popular in both lay and research contexts. Criterion profile analysis (CPA; Davison & Davenport, 2002) is a well-established, regression-based pattern matching procedure that identifies a pattern of predictors that optimally relate to a criterion of interest and quantifies the strength of that association. Existing CPA methods require individual-level data, limiting opportunities for reanalysis of published work, including research synthesis via meta-analysis and associated corrections for psychometric artifacts. In this article, we develop methods for meta-analytic criterion profile analysis (MACPA), including new methods for estimating cross-validity and fungibility of criterion patterns. We also review key methodological considerations for applying MACPA, including homogeneity of studies in meta-analyses, corrections for statistical artifacts, and second-order sampling error. Finally, we present example applications of MACPA to published meta-analyses from organizational, educational, personality, and clinical psychological literatures. R code implementing these methods is provided in the configural package, available at https://cran.r-project.org/package=configural and at https://doi.org/10.17605/osf.io/aqmpc. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Constructing Subscores That Add Validity: A Case Study of Identifying Students at Risk.

Prior research suggests that subscores from a single achievement test seldom add value over a single total score. Such scores typically correspond to subcontent areas in the total content domain, but content subdomains might not provide a sound basis for subscores. Using scores on an inferential reading comprehension test from 625 third, fourth, and fifth graders, two new methods of creating subscores were explored. Three subscores were based on the types of incorrect answers given by students. The fourth was based on temporal efficiency in giving correct answers. All four scores were reliable. The three subscores based on incorrect answers added value and validity. In logistic regression analyses predicting failure to reach proficiency on a statewide test, models including subscores fit better than the model with a single total score. Including the pattern of incorrect responses improved fit in all three grades, whereas including the comprehension efficiency score only modestly improved fit in fourth and fifth grades, but not third grade. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curves based on the various models were higher for models including subscores than those without subscores. Implications for using models with and without subscores are illustrated and discussed.

Estimating Between-Person and Within-Person Subscore Reliability with Profile Analysis.

Subscores are of increasing interest in educational and psychological testing due to their diagnostic function for evaluating examinees' strengths and weaknesses within particular domains of knowledge. Previous studies about the utility of subscores have mostly focused on the overall reliability of individual subscores and ignored the fact that subscores should be distinct and have added value over the total score. This study introduces a profile reliability approach that partitions the overall subscore reliability into within-person and between-person subscore reliability. The estimation of between-person reliability and within-person reliability coefficients is demonstrated using subscores from number-correct scoring, unidimensional and multidimensional item response theory scoring, and augmented scoring approaches via a simulation study and a real data study. The effects of various testing conditions, such as subtest length, correlations among subscores, and the number of subtests, are examined. Results indicate that there is a substantial trade-off between within-person and between-person reliability of subscores. Profile reliability coefficients can be useful in determining the extent to which subscores provide distinct and reliable information under various testing conditions.

Maximum likelihood versus multiple imputation for missing data in small longitudinal samples with nonnormality.

The study examined the performance of maximum likelihood (ML) and multiple imputation (MI) procedures for missing data in longitudinal research when fitting latent growth models. A Monte Carlo simulation study was conducted with conditions of small sample size, intermittent missing data, and nonnormality. The results indicated that ML tended to display slightly smaller degrees of bias than MI across missing completely at random (MCAR) and missing at random (MAR) conditions. Although specification of prior information in the MI imputation-posterior (I-P) phase influenced the performance of MI, especially with nonnormal small samples and missing not at random (MNAR), the impact of this tight specification was not dramatic. Several corrected ML test statistics showed proper rejections rates across research designs, whereas posterior predictive p values for MI methods were more likely to be influenced by distribution shape and yielded higher rejection rates in MCAR and MAR than in MNAR. In conclusion, ML appears to be preferable to MI in research conditions with small missing samples and multivariate nonnormality whether or not strong prior information for the I-P phase of MI analysis is available. (PsycINFO Database Record

Fitting a linear-linear piecewise growth mixture model with unknown knots: A comparison of two common approaches to inference.

A linear-linear piecewise growth mixture model (PGMM) is appropriate for analyzing segmented (disjointed) change in individual behavior over time, where the data come from a mixture of 2 or more latent classes, and the underlying growth trajectories in the different segments of the developmental process within each latent class are linear. A PGMM allows the knot (change point), the time of transition from 1 phase (segment) to another, to be estimated (when it is not known a priori) along with the other model parameters. To assist researchers in deciding which estimation method is most advantageous for analyzing this kind of mixture data, the current research compares 2 popular approaches to inference for PGMMs: maximum likelihood (ML) via an expectation-maximization (EM) algorithm, and Markov chain Monte Carlo (MCMC) for Bayesian inference. Monte Carlo simulations were carried out to investigate and compare the ability of the 2 approaches to recover the true parameters in linear-linear PGMMs with unknown knots. The results show that MCMC for Bayesian inference outperformed ML via EM in nearly every simulation scenario. Real data examples are also presented, and the corresponding computer codes for model fitting are provided in the Appendix to aid practitioners who wish to apply this class of models.

Development of an adverse-event coping scale (AECS) using item response theory.

BACKGROUND: Adverse drug events (ADEs) cause significant morbidity and mortality to patients. A brief questionnaire asking patients how they coped with such problems could be a useful tool for providing timely interventions. OBJECTIVE: The aim of this study was to develop an adverse-event coping scale (AECS) to measure patients' coping responses to their ADE. METHODS: Data were collected from subjects recruited from community pharmacies. Psychometric analyses based on item response theory (IRT) were performed to calibrate items and assess reliability. Convergent validity was evaluated by testing a priori formulated hypotheses about expected correlations between the coping scores and other related scales. RESULTS: A total of 140 patients participated in this study by answering the developed items. Confirmatory factor analysis supported a one-dimensional item bank with 11 items. The developed scale was reliable with the reliability coefficient of 0.82. Coping scores were positively correlated with seriousness of the ADE and health literacy, but not coping self-efficacy. Overall, results suggest that the score reflects problem magnitude and coping effort rather than coping efficacy. CONCLUSION: A high score on the AECS indicates an ADE serious enough to prompt a patient to invest substantial efforts to cope with it. The final AECS item bank and its short-form can help clinicians better understand their patients' ADE-coping efforts.

Automated assembly of optimally spaced and balanced paired comparisons: controlling order effects.

To control order effects in questionnaires containing paired comparisons, Ross (1934) described an optimal ordering of the pairings. The pairs can also be balanced so that every stimulus appears equal numbers of times as the first and the second member of a pair. First, we describe and illustrate the optimally spaced, balanced ordering of pairings. Then we show how the optimally spaced, balanced order can be used to implement a matrix-sampling design or a fully incomplete design when the number of stimuli n is so large that respondents cannot reasonably be expected to judge all n(n - 1)/2 pairs. The algorithm for balancing and optimally spacing the list of pairs is described.

Predictors of patients' perceived need for medication to prevent fracture.

BACKGROUND: Adherence with medications to prevent fractures is suboptimal. Patients' perceived need for medication is an important predictor of medication-use behavior. OBJECTIVE: Estimate the associations of patients' perceived need of medication for fracture prevention with objective indicators of fracture risk, patients' concerns about medications, and the quality of the patient-physician relationship. RESEARCH DESIGN: Cross-sectional medical record review and mailed survey. A multivariate path model was used to estimate the associations of predictor variables with perceived need for fracture prevention medication. SUBJECTS: A total of 1155 individuals were prescribed an oral bisphosphonate medication between January 1, 2006 and March 31, 2007 at a large urban multispecialty clinic in the United States. RESULTS: Trust in the prescribing physician, prevalent vertebral fracture on spine imaging, patients' self-reported susceptibility to and perceived severity of fractures, and medication concerns were independently associated with perceived need for medication. Bone mineral density and fracture history were only weakly associated with perceived need for medication. Trust in the physician was associated with perceived severity of fractures but not with self-reported susceptibility to fractures. Patients' perceptions that their physician communicates openly with them and their satisfaction with their physician's decision-making style are strongly associated with their trust in that physician. CONCLUSIONS: Documenting prevalent vertebral fracture may influence patients' perceived need for fracture prevention medication. Patients' trust in their physicians influences perceived need for fracture prevention medication. Patients' perceptions of open physician communication and their satisfaction with their physician's decision-making style are indirectly associated with perceived need for fracture prevention medication.

Confirmatory Factor Analysis and Profile Analysis via Multidimensional Scaling.

This paper describes the Confirmatory Factor Analysis (CFA) parameterization of the Profile Analysis via Multidimensional Scaling (PAMS) model to demonstrate validation of profile pattern hypotheses derived from multidimensional scaling (MDS). Profile Analysis via Multidimensional Scaling (PAMS) is an exploratory method for identifying major profiles in a multi-subtest test battery. Major profile patterns are represented as dimensions extracted from a MDS analysis. PAMS represents an individual observed score as a linear combination of dimensions where the dimensions are the most typical profile patterns present in a population. While the PAMS approach was initially developed for exploratory purposes, its results can later be confirmed in a different sample by CFA. Since CFA is often used to verify results from an exploratory factor analysis, the present paper makes the connection between a factor model and the PAMS model, and then illustrates CFA with a simulated example (that was generated by the PAMS model) and at the same time with a real example. The real example demonstrates confirmation of PAMS exploratory results by using a different sample. Fit indexes can be used to indicate whether the CFA reparameterization as a confirmatory approach works for the PAMS exploratory results.

Estimating Cognitive Profiles Using Profile Analysis via Multidimensional Scaling (PAMS).

Two of the most popular methods of profile analysis, cluster analysis and modal profile analysis, have limitations. First, neither technique is adequate when the sample size is large. Second, neither method will necessarily provide profile information in terms of both level and pattern. A new method of profile analysis, called Profile Analysis via Multidimensional Scaling (PAMS; Davison, 1996), is introduced to meet the challenge. PAMS extends the use of simple multidimensional scaling methods to identify latent profiles in a multi-test battery. Application of PAMS to profile analysis is described. The PAMS model is then used to identify latent profiles from a subgroup (N = 357) within the sample of the Woodcock-Johnson Psychoeducational Battery-Revised (WJ-R; McGrew, Werder, & Woodcock, 1991; Woodcock & Johnson, 1989), followed by a discussion of procedures for interpreting participants' observed score profiles from the latent PAMS profiles. Finally, advantages and limitations of the PAMS technique are discussed.

Identifying criterion-related patterns of predictor scores using multiple regression.

Along with examples involving vocational interests and mathematics achievement, the authors describe a multiple regression based, pattern recognition procedure that can be used to identify a pattern of predictor scores associated with high scores on a criterion variable. This pattern is called the criterion pattern. After the criterion pattern has been identified, a second regression procedure can be used to estimate the proportion of variation attributable to the criterion pattern. Cross-validation can then be used to estimate the variation attributable to a criterion pattern derived from regression weights estimated in another sample. Finally, issues of criterion pattern invariance and interpretation are discussed.