A Time-Varying Dynamic Partial Credit Model to Analyze Polytomous and Multivariate Time Series Data.

The accessibility to electronic devices and the novel statistical methodologies available have allowed researchers to comprehend psychological processes at the individual level. However, there are still great challenges to overcome as, in many cases, collected data are more complex than the available models are able to handle. For example, most methods assume that the variables in the time series are measured on an interval scale, which is not the case when Likert-scale items were used. Ignoring the scale of the variables can be problematic and bias the results. Additionally, most methods also assume that the time series are stationary, which is rarely the case. To tackle these disadvantages, we propose a model that combines the partial credit model (PCM) of the item response theory framework and the time-varying autoregressive model (TV-AR), which is a popular model used to study psychological dynamics. The proposed model is referred to as the time-varying dynamic partial credit model (TV-DPCM), which allows to appropriately analyze multivariate polytomous data and nonstationary time series. We test the performance and accuracy of the TV-DPCM in a simulation study. Lastly, by means of an example, we show how to fit the model to empirical data and interpret the results.

The Crit coefficient in Mokken scale analysis: a simulation study and an application in quality-of-life research.

PURPOSE: In Mokken scaling, the Crit index was proposed and is sometimes used as evidence (or lack thereof) of violations of some common model assumptions. The main goal of our study was twofold: To make the formulation of the Crit index explicit and accessible, and to investigate its distribution under various measurement conditions. METHODS: We conducted two simulation studies in the context of dichotomously scored item responses. We manipulated the type of assumption violation, the proportion of violating items, sample size, and quality. False positive rates and power to detect assumption violations were our main outcome variables. Furthermore, we used the Crit coefficient in a Mokken scale analysis to a set of responses to the General Health Questionnaire (GHQ-12), a self-administered questionnaire for assessing current mental health. RESULTS: We found that the false positive rates of Crit were close to the nominal rate in most conditions, and that power to detect misfit depended on the sample size, type of violation, and number of assumption-violating items. Overall, in small samples Crit lacked the power to detect misfit, and in larger samples power differed considerably depending on the type of violation and proportion of misfitting items. Furthermore, we also found in our empirical example that even in large samples the Crit index may fail to detect assumption violations. DISCUSSION: Even in large samples, the Crit coefficient showed limited usefulness for detecting moderate and severe violations of monotonicity. Our findings are relevant to researchers and practitioners who use Mokken scaling for scale and questionnaire construction and revision.

Searching for the optimal number of response alternatives for the distress scale of the four-dimensional symptom questionnaire.

BACKGROUND: The Four-Dimensional Symptom Questionnaire (4DSQ) is a self-report questionnaire designed to measure distress, depression, anxiety, and somatization. Prior to computing scale scores from the item scores, the three highest response alternatives ('Regularly', 'Often', and 'Very often or constantly present') are usually collapsed into one category to reduce the influence of extreme responding on item- and scale scores. In this study, we evaluate the usefulness of this transformation for the distress scale based on a variety of criteria. METHODS: Specifically, by using the Graded Response Model, we investigated the effect of this transformation on model fit, local measurement precision, and various indicators of the scale's validity to get an indication on whether the current practice of recoding should be advocated or not. In particular, the effect on the convergent- (operationalized by the General Health Questionnaire and the Maastricht Questionnaire), divergent- (operationalized by the Neuroticism scale of the NEO-FFI), and predictive validity (operationalized as obtrusion with daily chores and activities, the Biographical Problem list and the Utrecht Burnout Scale) of the distress scale was investigated. RESULTS: Results indicate that recoding leads to (i) better model fit as indicated by lower mean probabilities of exact test statistics assessing item fit, (ii) small (<.02) losses in the sizes of various validity coefficients, and (iii) a decrease (DIFF (SE's) = .10-.25) in measurement precision for medium and high levels of distress. CONCLUSIONS: For clinical applications and applications in longitudinal research, the current practice of recoding should be avoided because recoding decreases measurement precision for medium and high levels of distress. It would be interesting to see whether this advice also holds for the three other domains of the 4DSQ.

Person-fit feedback on inconsistent symptom reports in clinical depression care.

BACKGROUND: Depressive patients can present with complex and different symptom patterns in clinical care. Of these, some may report patterns that are inconsistent with typical patterns of depressive symptoms. This study aimed to evaluate the validity of person-fit statistics to identify inconsistent symptom reports and to assess the clinical usefulness of providing clinicians with person-fit score feedback during depression assessment. METHODS: Inconsistent symptom reports on the Inventory of Depressive Symptomatology Self-Report (IDS-SR) were investigated quantitatively with person-fit statistics for both intake and follow-up measurements in the Groningen University Center of Psychiatry (n = 2036). Subsequently, to investigate the causes and clinical usefulness of on-the-fly person-fit alerts, qualitative follow-up assessments were conducted with three psychiatrists about 20 of their patients that were randomly selected. RESULTS: Inconsistent symptom reports at intake (12.3%) were predominantly characterized by reporting of severe symptoms (e.g. psychomotor slowing) without mild symptoms (e.g. irritability). Person-fit scores at intake and follow-up were positively correlated (r = 0.45). Qualitative interviews with psychiatrists resulted in an explanation for the inconsistent response behavior (e.g. complex comorbidity, somatic complaints, and neurological abnormalities) for 19 of 20 patients. Psychiatrists indicated that if provided directly after the assessment, a person-fit alert would have led to new insights in 60%, and be reason for discussion with the patient in 75% of the cases. CONCLUSIONS: Providing clinicians with automated feedback when inconsistent symptom reports occur is informative and can be used to support clinical decision-making.

Identifying levels of general distress in first line mental health services: can GP- and eHealth clients' scores be meaningfully compared?

BACKGROUND: The Four-Dimensional Symptom Questionnaire (4DSQ) (Huisarts Wetenschap 39: 538-47, 1996) is a self-report questionnaire developed in the Netherlands to distinguish non-specific general distress from depression, anxiety, and somatization. This questionnaire is often used in different populations and settings and there is a paper-and-pencil and computerized version. METHODS: We used item response theory to investigate whether the 4DSQ measures the same construct (structural equivalence) in the same way (scalar equivalence) in two samples comprised of primary mental health care attendees: (i) clients who visited their General Practitioner responded to the 4DSQ paper-and-pencil version, and (ii) eHealth clients responded to the 4DSQ computerized version. Specifically, we investigated whether the distress items functioned differently in eHealth clients compared to General Practitioners' clients and whether these differences lead to substantial differences at scale level. RESULTS: Results showed that in general structural equivalence holds for the distress scale. This means that the distress scale measures the same construct in both General Practitioners' clients and eHealth clients. Furthermore, although eHealth clients have higher observed distress scores than General Practitioners' clients, application of a multiple group generalized partial credit response model suggests that scalar equivalence holds. CONCLUSIONS: The same cutoff scores can be used for classifying respondents as having low, moderate and high levels of distress in both settings.

Investigating the Practical Consequences of Model Misfit in Unidimensional IRT Models.

In this article, the practical consequences of violations of unidimensionality on selection decisions in the framework of unidimensional item response theory (IRT) models are investigated based on simulated data. The factors manipulated include the severity of violations, the proportion of misfitting items, and test length. The outcomes that were considered are the precision and accuracy of the estimated model parameters, the correlations of estimated ability ([Formula: see text]) and number-correct ([Formula: see text]) scores with the true ability ([Formula: see text]), the ranks of the examinees and the overlap between sets of examinees selected based on either [Formula: see text], [Formula: see text], or [Formula: see text] scores, and the bias in criterion-related validity estimates. Results show that the [Formula: see text] values were unbiased by violations of unidimensionality, but their precision decreased as multidimensionality and the proportion of misfitting items increased; the estimated item parameters were robust to violations of unidimensionality. The correlations between [Formula: see text], [Formula: see text], and [Formula: see text] scores, the agreement between the three selection criteria, and the accuracy of criterion-related validity estimates are all negatively affected, to some extent, by increasing levels of multidimensionality and the proportion of misfitting items. However, removing the misfitting items only improved the results in the case of severe multidimensionality and large proportion of misfitting items, and deteriorated them otherwise.

Assessment of fit of the time-varying dynamic partial credit model using the posterior predictive model checking method.

Several new models based on item response theory have recently been suggested to analyse intensive longitudinal data. One of these new models is the time-varying dynamic partial credit model (TV-DPCM; Castro-Alvarez et al., Multivariate Behavioral Research, 2023, 1), which is a combination of the partial credit model and the time-varying autoregressive model. The model allows the study of the psychometric properties of the items and the modelling of nonlinear trends at the latent state level. However, there is a severe lack of tools to assess the fit of the TV-DPCM. In this paper, we propose and develop several test statistics and discrepancy measures based on the posterior predictive model checking (PPMC) method (PPMC; Rubin, The Annals of Statistics, 1984, 12, 1151) to assess the fit of the TV-DPCM. Simulated and empirical data are used to study the performance of and illustrate the effectiveness of the PPMC method.

Measuring individual significant change on the Beck Depression Inventory-II through IRT-based statistics.

Several researchers have emphasized that item response theory (IRT)-based methods should be preferred over classical approaches in measuring change for individual patients. In the present study we discuss and evaluate the use of IRT-based statistics to measure statistical significant individual change on the Beck Depression Inventory-II (BDI-II, Beck, Steer, & Brown, 1996). We compare results obtained with a simple IRT-based statistical test (Z-test) to those obtained with the Reliable Change Index (RCI) in a sample of clinical outpatients. Mean group differences between the Z-test and the RCI were similar, but for some individuals change classifications differed. Differences were most evident for change scores within the lower range of depression scores. We show that this may have consequences for the measurement of individual change in psychotherapy outcome research and clinical practice.

Item response theory: how Mokken scaling can be used in clinical practice.

AIMS: To demonstrate the principles and application of Mokken scaling. BACKGROUND: The history and development of Mokken scaling is described, some examples of applications are given, and some recent development of the method are summarised. DESIGN: Secondary analysis of data obtained by cross-sectional survey methods, including self-report and observation. METHODS: Data from the Edinburgh Feeding Evaluation in Dementia scale and the Townsend Functional Ability Scale were analysed using the Mokken scaling procedure within the 'R' statistical package. Specifically, invariant item ordering (the extent to which the order of the items in terms of difficulty was the same for all respondents whatever their total scale score) was studied. RESULTS: The Edinburgh Feeding Evaluation in Dementia scale and the Townsend Functional Ability Scale showed no violations of invariant item ordering, although only the Townsend Functional Ability Scale showed a medium accuracy. CONCLUSION: Mokken scaling is an established method for item response theory analysis with wide application in the social sciences. It provides psychometricians with an additional tool in the development of questionnaires and in the study of individuals and their responses to latent traits. Specifically, with regard to the analyses conducted in this study, the Edinburgh Feeding Evaluation in Dementia scale requires further development and study across different levels of severity of dementia and feeding difficulty. RELEVANCE TO CLINICAL PRACTICE: Good scales are required for assessment in clinical practice and the present paper shows how a relatively recently developed method for analysing Mokken scales can contribute to this. The two scales used as examples for analysis are highly clinically relevant.

Education increases decision-rule use: An investigation of education and incentives to improve decision making.

Robust scientific evidence shows that human performance predictions are more valid when information is combined mechanically (with a decision rule) rather than holistically (in the decision-maker's mind). Yet, information is often combined holistically in practice. One reason is that decision makers lack the knowledge of evidence-based decision making. In a performance prediction task, we tested whether watching an educational video on evidence-based decision making increased decision-makers' use of a decision rule and their prediction accuracy immediately after the manipulation and a month later. Furthermore, we manipulated whether participants earned incentives for accurate predictions. Existing research showed that incentives decrease decision-rule use and prediction accuracy. We hypothesized that this is the case for decision makers who did not receive educational information about evidence-based decision making, but that incentives increase decision-rule use and prediction accuracy for participants who received educational information. Our results showed that educational information increased decision-rule use. This resulted in increased prediction accuracy, but only immediately after receiving the educational information. In contrast to the existing literature, incentives slightly increased decision-rule use. We did not find evidence that this effect was larger for educated participants. Providing decision makers with educational information may be effective to increase decision-rule use in practice. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Using structural equation modeling to study traits and states in intensive longitudinal data.

Traditionally, researchers have used time series and multilevel models to analyze intensive longitudinal data. However, these models do not directly address traits and states which conceptualize the stability and variability implicit in longitudinal research, and they do not explicitly take into account measurement error. An alternative to overcome these drawbacks is to consider structural equation models (state-trait SEMs) for longitudinal data that represent traits and states as latent variables. Most of these models are encompassed in the latent state-trait (LST) theory. These state-trait SEMs can be problematic when the number of measurement occasions increases. As they require the data to be in wide format, these models quickly become overparameterized and lead to nonconvergence issues. For these reasons, multilevel versions of state-trait SEMs have been proposed, which require the data in long format. To study how suitable state-trait SEMs are for intensive longitudinal data, we carried out a simulation study. We compared the traditional single level to the multilevel version of three state-trait SEMs. The selected models were the multistate-singletrait (MSST) model, the common and unique trait-state (CUTS) model, and the trait-state-occasion (TSO) model. Furthermore, we also included an empirical application. Our results indicated that the TSO model performed best in both the simulated and the empirical data. To conclude, we highlight the usefulness of state-trait SEMs to study the psychometric properties of the questionnaires used in intensive longitudinal data. Yet, these models still have multiple limitations, some of which might be overcome by extending them to more general frameworks. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

How soccer scouts identify talented players.

Scouts of soccer clubs are often the first to identify talented players. However, there is a lack of research on how these scouts assess and predict overall soccer performance. Therefore, we conducted a large-scaled study to examine the process of talent identification among 125 soccer scouts. Through an online self-report questionnaire, scouts were asked about (1) the players' age at which they can predict players' soccer performance, (2) the attributes they consider relevant, and (3) the extent to which they predict performance in a structured manner. The most important results are as follows. First, scouts who observed 12-year-old and younger players perceived they could predict at older ages (13.6 years old, on average) whether a player has the potential to become a professional soccer player. This suggests that scouts are aware of the idea that early indicators of later performance are often lacking, yet do advise on selection of players at younger ages. Second, when identifying talented players, scouts considered more easily observable attributes, such as technical attributes. However, scouts described these often in a broad sense rather than in terms of specific predictors of future performance. Finally, scouts reported that they assess attributes of players in a structured manner. Yet, they ultimately based their prediction (i.e. final score) on an intuitive integration of different performance attributes, which is a suboptimal strategy according to existing literature. Taken together, these outcomes provide specific clues to improve the reliability and validity of the scouting process. HighlightsBased on a large sample of soccer scouts, we examine three issues that are important in the process of identifying talented soccer players: The age at which good performance predictions can be made, which attributes are relevant predictors, and how performance predictions are formed.Scouts who observe players in young age cohorts believe that the age at which they can predict performance is older than the players they typically scout, suggesting that they are aware that early indicators of performance are often lacking.Technical performance attributes are considered as most important performance predictors by scouts, but these are often described in a broad - rather than specific - sense.Scouts indicate that they predict performance in a structured manner, but form their overall performance prediction on an intuitive integration of different performance attributes, which can be a suboptimal approach.

On the Practical Consequences of Misfit in Mokken Scaling.

Mokken scale analysis is a popular method to evaluate the psychometric quality of clinical and personality questionnaires and their individual items. Although many empirical papers report on the extent to which sets of items form Mokken scales, there is less attention for the effect of violations of commonly used rules of thumb. In this study, the authors investigated the practical consequences of retaining or removing items with psychometric properties that do not comply with these rules of thumb. Using simulated data, they concluded that items with low scalability had some influence on the reliability of test scores, person ordering and selection, and criterion-related validity estimates. Removing the misfitting items from the scale had, in general, a small effect on the outcomes. Although important outcome variables were fairly robust against scale violations in some conditions, authors conclude that researchers should not rely exclusively on algorithms allowing automatic selection of items. In particular, content validity must be taken into account to build sensible psychometric instruments.

The validity of small-sided games in predicting 11-vs-11 soccer game performance.

Predicting performance in soccer games has been a major focus within talent identification and development. Past research has mainly used performance levels, such as elite vs. non-elite players, as the performance to predict (i.e. the criterion). Moreover, these studies have mainly focused on isolated performance attributes as predictors of soccer performance levels. However, there has been an increasing interest in finer grained criterion measures of soccer performance, as well as representative assessments at the level of performance predictors. In this study, we first determined the degree to which 7-vs-7 small-sided games can be considered as representative of 11-vs-11 games. Second, we assessed the validity of individual players' small-sided game performance in predicting their 11-vs-11 game performance on a continuous scale. Moreover, we explored the predictive validity for 11-vs-11 game performance of several physiological and motor tests in isolation. Sixty-three elite youth players of a professional soccer academy participated in 11 to 17 small-sided games and six 11-vs-11 soccer games. In-game performance indicators were assessed through notational analysis and combined into an overall offensive and defensive performance measure, based on their relationship with game success. Physiological and motor abilities were assessed using a sprint, endurance, and agility test. Results showed that the small-sided games were faster paced, but representative of 11-vs-11 games, with the exception of aerial duels. Furthermore, individual small-sided game performance yielded moderate predictive validities with 11-vs-11 game performance. In contrast, the physiological and motor tests yielded small to trivial relations with game performance. Altogether, this study provides novel insights into the application of representative soccer assessments and the use of continuous criterion measures of soccer performance.

Reprint of: Negative symptoms predict high relapse rates and both predict less favorable functional outcome in first episode psychosis, independent of treatment strategy.

BACKGROUND: In first episode psychosis (FEP) baseline negative symptoms (BNS) and relapse both predict less favorable functional outcome. Relapse-prevention is one of the most important goals of treatment. Apart from discontinuation of antipsychotics, natural causes of relapse are unexplained. We hypothesized that BNS, apart from predicting worse functional outcome, might also increase relapse risk. METHODS: We performed a post-hoc analysis of 7-year follow-up data of a FEP cohort (n = 103) involved in a dose-reduction/discontinuation (DR) vs. maintenance treatment (MT) trial. We examined: 1) what predicted relapse, 2) what predicted functional outcome, and 3) if BNS predicted relapse, whether MT reduced relapse rates compared to DR. After remission patients were randomly assigned to DR or MT for 18 months. Thereafter, treatment was uncontrolled. OUTCOMES: BNS and duration of untreated psychosis (DUP) predicted relapse. Number of relapses, BNS, and treatment strategy predicted functional outcome. BNS was the strongest predictor of relapse, while number of relapses was the strongest predictor of functional outcome above BNS and treatment strategy. Overall and within MT, but not within DR, more severe BNS predicted significantly higher relapse rates. Treatment strategies did not make a difference in relapse rates, regardless of BNS severity. INTERPRETATION: BNS not only predicted worse functional outcome, but also relapses during follow-up. Since current low dose maintenance treatment strategies did not prevent relapse proneness in patients with more severe BNS, resources should be deployed to find optimal treatment strategies for this particular group of patients.

Negative symptoms predict high relapse rates and both predict less favorable functional outcome in first episode psychosis, independent of treatment strategy.

BACKGROUND: In first episode psychosis (FEP) baseline negative symptoms (BNS) and relapse both predict less favorable functional outcome. Relapse-prevention is one of the most important goals of treatment. Apart from discontinuation of antipsychotics, natural causes of relapse are unexplained. We hypothesized that BNS, apart from predicting worse functional outcome, might also increase relapse risk. METHODS: We performed a post-hoc analysis of 7-year follow-up data of a FEP cohort (n = 103) involved in a dose-reduction/discontinuation (DR) vs. maintenance treatment (MT) trial. We examined: 1) what predicted relapse, 2) what predicted functional outcome, and 3) if BNS predicted relapse, whether MT reduced relapse rates compared to DR. After remission patients were randomly assigned to DR or MT for 18 months. Thereafter, treatment was uncontrolled. OUTCOMES: BNS and duration of untreated psychosis (DUP) predicted relapse. Number of relapses, BNS, and treatment strategy predicted functional outcome. BNS was the strongest predictor of relapse, while number of relapses was the strongest predictor of functional outcome above BNS and treatment strategy. Overall and within MT, but not within DR, more severe BNS predicted significantly higher relapse rates. Treatment strategies did not make a difference in relapse rates, regardless of BNS severity. INTERPRETATION: BNS not only predicted worse functional outcome, but also relapses during follow-up. Since current low dose maintenance treatment strategies did not prevent relapse proneness in patients with more severe BNS, resources should be deployed to find optimal treatment strategies for this particular group of patients.