Interrater reliability for multilevel data: A generalizability theory approach.

Current interrater reliability (IRR) coefficients ignore the nested structure of multilevel observational data, resulting in biased estimates of both subject- and cluster-level IRR. We used generalizability theory to provide a conceptualization and estimation method for IRR of continuous multilevel observational data. We explain how generalizability theory decomposes the variance of multilevel observational data into subject-, cluster-, and rater-related components, which can be estimated using Markov chain Monte Carlo (MCMC) estimation. We explain how IRR coefficients for each level can be derived from these variance components, and how they can be estimated as intraclass correlation coefficients (ICC). We assessed the quality of MCMC point and interval estimates with a simulation study, and showed that small numbers of raters were the main source of bias and inefficiency of the ICCs. In a follow-up simulation, we showed that a planned missing data design can diminish most estimation difficulties in these conditions, yielding a useful approach to estimating multilevel interrater reliability for most social and behavioral research. We illustrated the method using data on student-teacher relationships. All software code and data used for this article is available on the Open Science Framework: https://osf.io/bwk5t/. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Testing hypotheses involving Cronbach's alpha using marginal models.

We discuss the statistical testing of three relevant hypotheses involving Cronbach's alpha: one where alpha equals a particular criterion; a second testing the equality of two alpha coefficients for independent samples; and a third testing the equality of two alpha coefficients for dependent samples. For each of these hypotheses, various statistical tests have been proposed. Over the years, these tests have depended on progressively fewer assumptions. We propose a new approach to testing the three hypotheses that relies on even fewer assumptions, is especially suited for discrete item scores, and can be applied easily to tests containing large numbers of items. The new approach uses marginal modelling. We compared the Type I error rate and the power of the marginal modelling approach to several of the available tests in a simulation study using realistic conditions. We found that the marginal modelling approach had the most accurate Type I error rates, whereas the power was similar across the statistical tests.

Methodological artifacts in dimensionality assessment of the hospital anxiety and depression scale (HADS).

OBJECTIVE: The hospital anxiety and depression scale (HADS) is a brief, self-administered questionnaire for the assessment of anxiety and depression in hospital patients. A recent review discussed the disagreement among different studies with respect to the dimensionality of the HADS, leading Coyne and Van Sonderen to conclude from this disagreement that the HADS must be abandoned. Our study argues that the disagreement is mainly due to a methodological artifact, and that the HADS needs revision rather than abandonment. METHOD: We used Mokken scale analysis (MSA) to investigate the dimensionality of the HADS items in a representative sample from the Dutch non-clinical population (N=3643) and compared the dimensionality structure with the results that Emons, Sijtsma, and Pedersen obtained in a Dutch cardiac-patient sample. RESULTS: We demonstrated how MSA can retrieve either one scale, two subscales, or three subscales, and that the result not only depends on the data structure but also on choices that the researcher makes. Two 5-item HADS scales for anxiety and depression seemed adequate. Four HADS items constituted a weak scale and contributed little to reliable measurement. CONCLUSIONS: We argued that several psychometric methods show only one level of a hierarchical dimensionality structure and that users of psychometric methods are often unaware of this phenomenon and miss information about other levels. In addition, we argued that a theory about the attribute may guide the researcher but that well-tested theories are often absent.