A Psychometric Perspective on the Associations between Response Accuracy and Response Time Residuals.

We provide an alternative psychometric perspective on the empirical statistical dependencies observed between response accuracy residuals (RARs) and response time residuals (RTRs) in the context of the van der Linden model. This perspective emphasizes the RAR (or parts of the RAR) as being exogenous and having a directional influence on response time. Our simple and theoretically justifiable perspective adds to previous joint response time/accuracy models and comports with recent generalizations of the D-diffusion IRT model incorporating person-by-item interactions, and thus similarly reproduces many of the recently highlighted empirical findings concerning the associations between RARs and RTRs. Using both empirical and simulation-based results, we show how our psychometric perspective has both applied and interpretational implications. Specifically, it would suggest that (1) studies of item parameter estimate heterogeneity in relation to response times may reflect more of a psychometric artifact (due to the exogenous effects of the RARs) as opposed to providing insights about the response process (e.g., the application of different response strategies) and that (2) efforts to use RTRs as indicators of latent proficiency should attend to the anticipated interactions between the latent proficiency and RAR on response times. The validity of our psychometric perspective against alternatives likely relies on appeals to theory; the best perspective to take may vary depending on the test setting.

Predicting response time on self-report rating scale assessments of noncognitive constructs.

Methodological studies of response time on noncognitive assessments have separately demonstrated the relevance of content trait level and response styles as predictive factors. In this paper we examine the simultaneous relevance of both factors as well as the potential for omitted predictor bias when ignoring either factor. Using response time data from several different noncognitive assessments, we demonstrate how a multilevel regression model that attends simultaneously to content and response style factors leads to consistent findings that support the simultaneous relevance of both factors. The average effects of response style consistently emerge as stronger, although also show greater respondent-level variability, possibly due to the multiple different underlying causes of response style behavior. Some implications for the use of response times in noncognitive measurement are considered.

Rejoinder to Commentaries on Lyu, Bolt and Westby's "Exploring the Effects of Item Specific Factors in Sequential and IRTree Models".

We respond to the commentaries on Lyu, Bolt and Westby's "Exploring the effects of item specific factors in sequential and IRTree models." The commentaries raise important points that allow us to clarify our theoretical expectation for item specific factors in many educational and psychological test items. At the same time, we agree with the commentaries in acknowledging challenges associated with providing empirical evidence for their presence and reflect on strategies that might support their estimation. We maintain that the principal concern is the ambiguity item specific factors create in attempting to interpret or use the parameters beyond the first node.

Exploring the Effects of Item-Specific Factors in Sequential and IRTree Models.

Test items for which the item score reflects a sequential or IRTree modeling outcome are considered. For such items, we argue that item-specific factors, although not empirically measurable, will often be present across stages of the same item. In this paper, we present a conceptual model that incorporates such factors. We use the model to demonstrate how the varying conditional distributions of item-specific factors across stages become absorbed into the stage-specific item discrimination and difficulty parameters, creating ambiguity in the interpretations of item and person parameters beyond the first stage. We discuss implications in relation to various applications considered in the literature, including methodological studies of (1) repeated attempt items; (2) answer change/review, (3) on-demand item hints; (4) item skipping behavior; and (5) Likert scale items. Our own empirical applications, as well as several examples published in the literature, show patterns of violations of item parameter invariance across stages that are highly suggestive of item-specific factors. For applications using sequential or IRTree models as analytical models, or for which the resulting item score might be viewed as outcomes of such a process, we recommend (1) regular inspection of data or analytic results for empirical evidence (or theoretical expectations) of item-specific factors; and (2) sensitivity analyses to evaluate the implications of item-specific factors for the intended inferences or applications.

Bayesian Dynamic Borrowing of Historical Information with Applications to the Analysis of Large-Scale Assessments.

The purpose of this paper is to demonstrate and evaluate the use of Bayesian dynamic borrowing (Viele et al, in Pharm Stat 13:41-54, 2014) as a means of systematically utilizing historical information with specific applications to large-scale educational assessments. Dynamic borrowing via Bayesian hierarchical models is a special case of a general framework of historical borrowing where the degree of borrowing depends on the heterogeneity among historical data and current data. A joint prior distribution over the historical and current data sets is specified with the degree of heterogeneity across the data sets controlled by the variance of the joint distribution. We apply Bayesian dynamic borrowing to both single-level and multilevel models and compare this approach to other historical borrowing methods such as complete pooling, Bayesian synthesis, and power priors. Two case studies using data from the Program for International Student Assessment reveal the utility of Bayesian dynamic borrowing in terms of predictive accuracy. This is followed by two simulation studies that reveal the utility of Bayesian dynamic borrowing over simple pooling and power priors in cases where the historical data is heterogeneous compared to the current data based on bias, mean squared error, and predictive accuracy. In cases of homogeneous historical data, Bayesian dynamic borrowing performs similarly to data pooling, Bayesian synthesis, and power priors. In contrast, for heterogeneous historical data, Bayesian dynamic borrowing performed at least as well, if not better, than other methods of borrowing with respect to mean squared error, percent bias, and leave-one-out cross-validation.

A psychometric model for respondent-level anchoring on self-report rating scale instruments.

Among the various forms of response bias that can emerge with self-report rating scale assessments are those related to anchoring, the tendency for respondents to select categories in close proximity to the rating category used for the immediately preceding item. In this study we propose a psychometric model based on a multidimensional nominal model for response style that also simultaneously accommodates a respondent-level anchoring tendency. The model is estimated using a fully Bayesian estimation procedure. By applying this model to a real test data set measuring extraversion, we explore a theory that both response styles and anchoring might be viewed as evidence of a lack of effortful responding. Empirical results show that there is a positive correlation between the strength of midpoint response style and the anchoring effect; further, responses indicative of either anchoring or response style both negatively correlate with response time, consistent with a theory that both phenomena reflect reduced respondent effort. The results support attending to both anchoring and midpoint response style as ways of assessing respondent engagement.