Sample size planning for complex study designs: A tutorial for the mlpwr package.

A common challenge in designing empirical studies is determining an appropriate sample size. When more complex models are used, estimates of power can only be obtained using Monte Carlo simulations. In this tutorial, we introduce the R package mlpwr to perform simulation-based power analysis based on surrogate modeling. Surrogate modeling is a powerful tool in guiding the search for study design parameters that imply a desired power or meet a cost threshold (e.g., in terms of monetary cost). mlpwr can be used to search for the optimal allocation when there are multiple design parameters, e.g., when balancing the number of participants and the number of groups in multilevel modeling. At the same time, the approach can take into account the cost of each design parameter, and aims to find a cost-efficient design. We introduce the basic functionality of the package, which can be applied to a wide range of statistical models and study designs. Additionally, we provide two examples based on empirical studies for illustration: one for sample size planning when using an item response theory model, and one for assigning the number of participants and the number of countries for a study using multilevel modeling.

Development and validation of a patient-reported outcome measure for systemic sclerosis: the EULAR Systemic Sclerosis Impact of Disease (ScleroID) questionnaire.

OBJECTIVES: Patient-reported outcome measures (PROMs) are important for clinical practice and research. Given the high unmet need, our aim was to develop a comprehensive PROM for systemic sclerosis (SSc), jointly with patient experts. METHODS: This European Alliance of Associations for Rheumatology (EULAR)-endorsed project involved 11 European SSc centres. Relevant health dimensions were chosen and prioritised by patients. The resulting Systemic Sclerosis Impact of Disease (ScleroID) questionnaire was subsequently weighted and validated by Outcome Measures in Rheumatology criteria in an observational cohort study, cross-sectionally and longitudinally. As comparators, SSc-Health Assessment Questionnaire (HAQ), EuroQol Five Dimensional (EQ-5D), Short Form-36 (SF-36) were included. RESULTS: Initially, 17 health dimensions were selected and prioritised. The top 10 health dimensions were selected for the ScleroID questionnaire. Importantly, Raynaud's phenomenon, impaired hand function, pain and fatigue had the highest patient-reported disease impact. The validation cohort study included 472 patients with a baseline visit, from which 109 had a test-retest reliability visit and 113 had a follow-up visit (85% female, 38% diffuse SSc, mean age 58 years, mean disease duration 9 years). The total ScleroID score showed strong Pearson correlation coefficients with comparators (SSc-HAQ, 0.73; Patient's global assessment, Visual Analogue Scale 0.77; HAQ-Disability Index, 0.62; SF-36 physical score, -0.62; each p<0.001). The internal consistency was strong: Cronbach's alpha was 0.87, similar to SSc-HAQ (0.88) and higher than EQ-5D (0.77). The ScleroID had excellent reliability and good sensitivity to change, superior to all comparators (intraclass correlation coefficient 0.84; standardised response mean 0.57). CONCLUSIONS: We have developed and validated the EULAR ScleroID, which is a novel, brief, disease-specific, patient-derived, disease impact PROM, suitable for research and clinical use in SSc.

Engagement With a Mobile Phone-Based Life Skills Intervention for Adolescents and Its Association With Participant Characteristics and Outcomes: Tree-Based Analysis.

BACKGROUND: Mobile phone-delivered life skills programs are an emerging and promising way to promote mental health and prevent substance use among adolescents, but little is known about how adolescents actually use them. OBJECTIVE: The aim of this study is to determine engagement with a mobile phone-based life skills program and its different components, as well as the associations of engagement with adolescent characteristics and intended substance use and mental health outcomes. METHODS: We performed secondary data analysis on data from the intervention group (n=750) from a study that compared a mobile phone-based life skills intervention for adolescents recruited in secondary and upper secondary school classes with an assessment-only control group. Throughout the 6-month intervention, participants received 1 SMS text message prompt per week that introduced a life skills topic or encouraged participation in a quiz or individual life skills training or stimulated sharing messages with other program participants through a friendly contest. Decision trees were used to identify predictors of engagement (use and subjective experience). The stability of these decision trees was assessed using a resampling method and by graphical representation. Finally, associations between engagement and intended substance use and mental health outcomes were examined using logistic and linear regression analyses. RESULTS: The adolescents took part in half of the 50 interactions (mean 23.6, SD 15.9) prompted by the program, with SMS text messages being the most used and contests being the least used components. Adolescents who did not drink in a problematic manner and attended an upper secondary school were the ones to use the program the most. Regarding associations between engagement and intended outcomes, adolescents who used the contests more frequently were more likely to be nonsmokers at follow-up than those who did not (odds ratio 0.86, 95% CI 0.76-0.98; P=.02). In addition, adolescents who read the SMS text messages more attentively were less likely to drink in a problematic manner at follow-up (odds ratio 0.43, 95% CI 1.29-3.41; P=.003). Finally, participants who used the program the most and least were more likely to increase their well-being from baseline to 6-month follow-up compared with those with average engagement (ßs=.39; t586=2.66; P=.008; R2=0.24). CONCLUSIONS: Most of the adolescents participating in a digital life skills program that aimed to prevent substance use and promote mental health engaged with the intervention. However, measures to increase engagement in problem drinkers should be considered. Furthermore, efforts must be made to ensure that interventions are engaging and powerful across different educational levels. First results indicate that higher engagement with digital life skills programs could be associated with intended outcomes. Future studies should apply further measures to improve the reach of lower-engaged participants at follow-up to establish such associations with certainty.

An R toolbox for score-based measurement invariance tests in IRT models.

The detection of differential item functioning (DIF) is a central topic in psychometrics and educational measurement. In the past few years, a new family of score-based tests of measurement invariance has been proposed, which allows the detection of DIF along arbitrary person covariates in a variety of item response theory (IRT) models. This paper illustrates the application of these tests within the R system for statistical computing, making them accessible to a broad range of users. This presentation also includes IRT models for which these tests have not previously been investigated, such as the generalized partial credit model. The paper has three goals: First, we review the ideas behind score-based tests of measurement invariance. Second, we describe the implementation of these tests within the R system for statistical computing, which is based on the interaction of the R packages mirt, psychotools and strucchange. Third, we illustrate the application of this software and the interpretation of its output in two empirical datasets. The complete R code for reproducing our results is reported in the paper.

With a little help from familiar interlocutors: real-world language use in young and older adults.

OBJECTIVES: Functional psychologists are concerned with the performance of cognitive activities in the real world in relation to cognitive changes in older age. Conversational contexts may mitigate the influence of cognitive aging on the cognitive activity of language production. This study examined effects of familiarity with interlocutors, as a context, on language production in the real world. METHOD: We collected speech samples using iPhones, where an audio recording app (i.e. Electronically Activated Recorder [EAR]) was installed. Over 31,300 brief audio files (30-second long) were randomly collected across four days from 61 young and 48 healthy older adults in Switzerland. We transcribed the audio files that included participants' speech and manually coded for familiar interlocutors (i.e. significant other, friends, family members) and strangers. We computed scores of vocabulary richness and grammatical complexity from the transcripts using computational linguistics techniques. RESULTS: Bayesian multilevel analyses showed that participants used richer vocabulary and more complex grammar when talking with familiar interlocutors than with strangers. Young adults used more diverse vocabulary than older adults and the age effects remained stable across contexts. Furthermore, older adults produced equally complex grammar as young adults did with the significant other, but simpler grammar than young adults with friends and family members. CONCLUSION: Familiarity with interlocutors is a promising contextual factor for research on aging and language complexity in the real world. Results were discussed in the context of cognitive aging.

Model Selection of Nested and Non-Nested Item Response Models Using Vuong Tests.

In this paper, we apply Vuong's general approach of model selection to the comparison of nested and non-nested unidimensional and multidimensional item response theory (IRT) models. Vuong's approach of model selection is useful because it allows for formal statistical tests of both nested and non-nested models. However, only the test of non-nested models has been applied in the context of IRT models to date. After summarizing the statistical theory underlying the tests, we investigate the performance of all three distinct Vuong tests in the context of IRT models using simulation studies and real data. In the non-nested case we observed that the tests can reliably distinguish between the graded response model and the generalized partial credit model. In the nested case, we observed that the tests typically perform as well as or sometimes better than the traditional likelihood ratio test. Based on these results, we argue that Vuong's approach provides a useful set of tools for researchers and practitioners to effectively compare competing nested and non-nested IRT models.

Simulation-based design optimization for statistical power: Utilizing machine learning.

The planning of adequately powered research designs increasingly goes beyond determining a suitable sample size. More challenging scenarios demand simultaneous tuning of multiple design parameter dimensions and can only be addressed using Monte Carlo simulation if no analytical approach is available. In addition, cost considerations, for example, in terms of monetary costs, are a relevant target for optimization. In this context, optimal design parameters can imply a desired level of power at minimum cost or maximum power at a cost threshold. We introduce a surrogate modeling framework based on machine learning predictions to solve these optimization tasks. In a simulation study, we demonstrate the efficiency for a wide range of hypothesis testing scenarios with single- and multidimensional design parameters, including t tests, analysis of variance, item response theory models, multilevel models, and multiple imputations. Our framework provides an algorithmic solution for optimizing study designs when no analytic power analysis is available, handling multiple design dimensions and cost considerations. Our implementation is publicly available in the R package mlpwr. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Power Analysis for the Wald, LR, Score, and Gradient Tests in a Marginal Maximum Likelihood Framework: Applications in IRT.

The Wald, likelihood ratio, score, and the recently proposed gradient statistics can be used to assess a broad range of hypotheses in item response theory models, for instance, to check the overall model fit or to detect differential item functioning. We introduce new methods for power analysis and sample size planning that can be applied when marginal maximum likelihood estimation is used. This allows the application to a variety of IRT models, which are commonly used in practice, e.g., in large-scale educational assessments. An analytical method utilizes the asymptotic distributions of the statistics under alternative hypotheses. We also provide a sampling-based approach for applications where the analytical approach is computationally infeasible. This can be the case with 20 or more items, since the computational load increases exponentially with the number of items. We performed extensive simulation studies in three practically relevant settings, i.e., testing a Rasch model against a 2PL model, testing for differential item functioning, and testing a partial credit model against a generalized partial credit model. The observed distributions of the test statistics and the power of the tests agreed well with the predictions by the proposed methods in sufficiently large samples. We provide an openly accessible R package that implements the methods for user-supplied hypotheses.

A New Stopping Criterion for Rasch Trees Based on the Mantel-Haenszel Effect Size Measure for Differential Item Functioning.

To detect differential item functioning (DIF), Rasch trees search for optimal splitpoints in covariates and identify subgroups of respondents in a data-driven way. To determine whether and in which covariate a split should be performed, Rasch trees use statistical significance tests. Consequently, Rasch trees are more likely to label small DIF effects as significant in larger samples. This leads to larger trees, which split the sample into more subgroups. What would be more desirable is an approach that is driven more by effect size rather than sample size. In order to achieve this, we suggest to implement an additional stopping criterion: the popular Educational Testing Service (ETS) classification scheme based on the Mantel-Haenszel odds ratio. This criterion helps us to evaluate whether a split in a Rasch tree is based on a substantial or an ignorable difference in item parameters, and it allows the Rasch tree to stop growing when DIF between the identified subgroups is small. Furthermore, it supports identifying DIF items and quantifying DIF effect sizes in each split. Based on simulation results, we conclude that the Mantel-Haenszel effect size further reduces unnecessary splits in Rasch trees under the null hypothesis, or when the sample size is large but DIF effects are negligible. To make the stopping criterion easy-to-use for applied researchers, we have implemented the procedure in the statistical software R. Finally, we discuss how DIF effects between different nodes in a Rasch tree can be interpreted and emphasize the importance of purification strategies for the Mantel-Haenszel procedure on tree stopping and DIF item classification.

What Affects the Quality of Score Transformations? Potential Issues in True-Score Equating Using the Partial Credit Model.

This simulation study investigated to what extent departures from construct similarity as well as differences in the difficulty and targeting of scales impact the score transformation when scales are equated by means of concurrent calibration using the partial credit model with a common person design. Practical implications of the simulation results are discussed with a focus on scale equating in health-related research settings. The study simulated data for two scales, varying the number of items and the sample sizes. The factor correlation between scales was used to operationalize construct similarity. Targeting of the scales was operationalized through increasing departure from equal difficulty and by varying the dispersion of the item and person parameters in each scale. The results show that low similarity between scales goes along with lower transformation precision. In cases with equal levels of similarity, precision improves in settings where the range of the item parameters is encompassing the person parameters range. With decreasing similarity, score transformation precision benefits more from good targeting. Difficulty shifts up to two logits somewhat increased the estimation bias but without affecting the transformation precision. The observed robustness against difficulty shifts supports the advantage of applying a true-score equating methods over identity equating, which was used as a naive baseline method for comparison. Finally, larger sample size did not improve the transformation precision in this study, longer scales improved only marginally the quality of the equating. The insights from the simulation study are used in a real-data example.

Interpretable machine learning for psychological research: Opportunities and pitfalls.

In recent years, machine learning methods have become increasingly popular prediction methods in psychology. At the same time, psychological researchers are typically not only interested in making predictions about the dependent variable, but also in learning which predictor variables are relevant, how they influence the dependent variable, and which predictors interact with each other. However, most machine learning methods are not directly interpretable. Interpretation techniques that support researchers in describing how the machine learning technique came to its prediction may be a means to this end. We present a variety of interpretation techniques and illustrate the opportunities they provide for interpreting the results of two widely used black box machine learning methods that serve as our examples: random forests and neural networks. At the same time, we illustrate potential pitfalls and risks of misinterpretation that may occur in certain data settings. We show in which way correlated predictors impact interpretations with regard to the relevance or shape of predictor effects and in which situations interaction effects may or may not be detected. We use simulated didactic examples throughout the article, as well as an empirical data set for illustrating an approach to objectify the interpretation of visualizations. We conclude that, when critically reflected, interpretable machine learning techniques may provide useful tools when describing complex psychological relationships. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Score-based measurement invariance checks for Bayesian maximum-a-posteriori estimates in item response theory.

A family of score-based tests has been proposed in recent years for assessing the invariance of model parameters in several models of item response theory (IRT). These tests were originally developed in a maximum likelihood framework. This study discusses analogous tests for Bayesian maximum-a-posteriori estimates and multiple-group IRT models. We propose two families of statistical tests, which are based on an approximation using a pooled variance method, or on a simulation approach based on asymptotic results. The resulting tests were evaluated by a simulation study, which investigated their sensitivity against differential item functioning with respect to a categorical or continuous person covariate in the two- and three-parametric logistic models. Whereas the method based on pooled variance was found to be useful in practice with maximum likelihood as well as maximum-a-posteriori estimates, the simulation-based approach was found to require large sample sizes to lead to satisfactory results.

Testing the Local Independence Assumption of the Rasch Model With Q 3-Based Nonparametric Model Tests.

Local independence is a central assumption of commonly used item response theory models. Violations of this assumption are usually tested using test statistics based on item pairs. This study presents two quasi-exact tests based on the Q 3 statistic for testing the hypothesis of local independence in the Rasch model. The proposed tests do not require the estimation of item parameters and can also be applied to small data sets. The authors evaluate the tests with three simulation studies. Their results indicate that the quasi-exact tests hold their alpha level under the Rasch model and have higher power against different forms of local dependence than several alternative parametric and nonparametric model tests for local independence.

Investigating Measurement Invariance by Means of Parameter Instability Tests for 2PL and 3PL Models.

M-fluctuation tests are a recently proposed method for detecting differential item functioning in Rasch models. This article discusses a generalization of this method to two additional item response theory models: the two-parametric logistic model and the three-parametric logistic model with a common guessing parameter. The Type I error rate and the power of this method were evaluated by a variety of simulation studies. The results suggest that the new method allows the detection of various forms of differential item functioning in these models, which also includes differential discrimination and differential guessing effects. It is also robust against moderate violations of several assumptions made in the item parameter estimation.

An Evaluation of Overall Goodness-of-Fit Tests for the Rasch Model.

For assessing the fit of item response theory models, it has been suggested to apply overall goodness-of-fit tests as well as tests for individual items and item pairs. Although numerous goodness-of-fit tests have been proposed in the literature for the Rasch model, their relative power against several model violations has not been investigated so far. This study compares four of these tests, which are all available in R software: T 10, T 11, M 2, and the LR test. Results on the Type I error rate and the sensitivity to violations of different assumptions of the Rasch model (unidimensionality, local independence on the level of item pairs, equal item discrimination, zero as a lower asymptote for the item characteristic curves, invariance of the item parameters) are reported. The results indicate that the T 11 test is comparatively most powerful against violations of the assumption of parallel item characteristic curves, which includes the presence of unequal item discriminations and a non-zero lower asymptote. Against the remaining model violations, which can be summarized as local dependence, M 2 is found to be most powerful. T 10 and LR are found to be sensitive against violations of the assumption of parallel item characteristic curves, but are insensitive against local dependence.

Psychobiological Protective Factors Modifying the Association Between Age and Sexual Health in Men: Findings From the Men's Health 40+ Study.

Sexual health severely decreases with age. For males older than 40 years, erectile dysfunction (ED) is the most common sexual disorder. Although physical and psychological risk factors for ED have been identified, protective factors are yet to be determined. To date, no study has examined endocrine and psychosocial factors in parallel with regard to their modifying effect on the age-related increase in ED. Two hundred and seventy-one self-reporting healthy men aged between 40 and 75 years provided both psychometric data on sexual function and a set of potential psychosocial protective factors, and saliva samples for the analysis of steroid hormones and proinflammatory cytokines. Around 35% of the participants reported at least a mild form of ED. Direct associations with ED were identified for perceived general health, emotional support, relationship quality, intimacy motivation but not for steroid hormones or proinflammatory markers. Moderation analyses for the association between age and ED revealed positive effects for testosterone (T), dehydroepiandrosterone (DHEA), perceived general health, emotional support, intimacy motivation, and a negative effect for interleukin-6 (all p < .05; f2 > .17). Group differences between older men with and without ED emerged for T, DHEA, and psychometric measures such as perceived general health, emotional support, satisfaction with life, and intimacy motivation (all p < .05; d > .3). Both psychosocial and endocrine parameters moderated the association between age and sexual health. Perceived general health, emotional support, intimacy motivation, and relationship quality emerged as psychosocial protective factors against ED. Higher T and DHEA and lower interleukin-6 levels also buffered against an age-related increase in ED.

Comparing the Effects of Different Smoothing Algorithms on the Assessment of Dimensionality of Ordered Categorical Items with Parallel Analysis.

The analysis of polychoric correlations via principal component analysis and exploratory factor analysis are well-known approaches to determine the dimensionality of ordered categorical items. However, the application of these approaches has been considered as critical due to the possible indefiniteness of the polychoric correlation matrix. A possible solution to this problem is the application of smoothing algorithms. This study compared the effects of three smoothing algorithms, based on the Frobenius norm, the adaption of the eigenvalues and eigenvectors, and on minimum-trace factor analysis, on the accuracy of various variations of parallel analysis by the means of a simulation study. We simulated different datasets which varied with respect to the size of the respondent sample, the size of the item set, the underlying factor model, the skewness of the response distributions and the number of response categories in each item. We found that a parallel analysis and principal component analysis of smoothed polychoric and Pearson correlations led to the most accurate results in detecting the number of major factors in simulated datasets when compared to the other methods we investigated. Of the methods used for smoothing polychoric correlation matrices, we recommend the algorithm based on minimum trace factor analysis.

Assessment of planning ability: Psychometric analyses on the unidimensionality and construct validity of the Tower of London Task (TOL-F).

OBJECTIVE: The Tower of London (TOL) is widely used to assess planning ability as a prototypical executive function in healthy and clinical populations. Despite its popularity, there is still no consensus on (a) whether the TOL measures a psychometrically unidimensional trait, and (b) how differences in problem structure relate to the difficulty of individual items. METHOD: Employing the framework of factor analysis and item response theory we investigated these issues of construct validity in a sample of 798 participants (443 female; 16 to 84 years). Participants worked on the TOL-Freiburg version (TOL-F) comprising a set of 24 4- to 6-move problems, which--based on comprehensive cognitive task analyses-systematically differ with regard to several structural problem parameters (minimum number of moves, search depth, goal hierarchy). RESULTS: Results revealed that TOL-F performance is mainly explained by 1 major factor, while further minor factors additionally account for smaller, but possibly still informative, shares of variance. Individual item difficulties can be predicted by the experimentally varied problem parameters to a high degree (r = .89) and can be considered stable across different levels of age, sex, education, and planning ability. CONCLUSION: These findings demonstrate the TOL-F's construct validity as measuring planning ability in terms of an essentially unidimensional cognitive function while adhering to theory-driven concepts of task difficulty. TOL-F task performance hence represents an accurate, robust, and theoretically grounded estimation of a participant's planning ability. The results further highlight the merit of using established concepts from experimental psychology for improving neuropsychological assessment.

Assessing Planning Ability Across the Adult Life Span: Population-Representative and Age-Adjusted Reliability Estimates for the Tower of London (TOL-F).

Planning ahead the consequences of future actions is a prototypical executive function. In clinical and experimental neuropsychology, disc-transfer tasks like the Tower of London (TOL) are commonly used for the assessment of planning ability. Previous psychometric evaluations have, however, yielded a poor reliability of measuring planning performance with the TOL. Based on theory-grounded task analyses and a systematic problem selection, the computerized TOL-Freiburg version (TOL-F) was developed to improve the task's psychometric properties for diagnostic applications. Here, we report reliability estimates for the TOL-F from two large samples collected in Mainz, Germany (n = 3,770; 40-80 years) and in Vienna, Austria (n = 830; 16-84 years). Results show that planning accuracy on the TOL-F possesses an adequate internal consistency and split-half reliability (>0.7) that are stable across the adult life span while the TOL-F covers a broad range of graded difficulty even in healthy adults, making it suitable for both research and clinical application.