Evaluating When Subscores Add Value in Psychological and Health Applications.

Many scales used in psychological and health research are designed to yield subscores, yet it is common to see total scores reported instead. One challenge of using subscores is they can lack adequate reliability due to their shortened length. However, methods originally developed for educational measurement have shown that augmenting subscores can improve reliability estimates. Augmented subscores blend the individual score with other sources of information. The present study sought to understand (a) the costs of ignoring subscores in favor of total scores and (b) the extent to which augmentation can help alleviate challenges encountered when using subscores. Data were simulated to examine when subscores should be preferred to total scores and the magnitude of improvement from using augmented subscores over non-augmented subscores. Results suggested that when a scale is designed to yield subscores, there is practical benefit to using them. In situations where subscore reliability is low, we recommend using augmentation.

Toward better outcome measurement for insomnia in children with autism spectrum disorder.

LAY ABSTRACT: Insomnia, trouble falling asleep or staying asleep, is common in autistic children. In a previous report, we described the results of focus groups with parents of autistic children toward the development of the Pediatric Autism Insomnia Rating Scale. In this article, we report on the steps taken to complete the Pediatric Autism Insomnia Rating Scale. With help from the Simons Foundation registry, we collected information from parents on 1185 children with autism spectrum disorder to test the new measure. These results were evaluated using standard statistical methods such as factor analysis. To confirm the validity of the new measure, we enrolled a separate sample of 134 autistic children for a detailed assessment by video conference. This step showed that the Pediatric Autism Insomnia Rating Scale is clearly measuring symptoms of insomnia in children with autism spectrum disorder and not related problems such as hyperactivity, repetitive behavior, or anxiety. We also showed that the total score on the Pediatric Autism Insomnia Rating Scale is stable when repeated over a brief period of time. This is important because a measure that is not stable over a brief period of time would not be suitable as an outcome measure. In summary, the Pediatric Autism Insomnia Rating Scale is a brief and valid measure of insomnia in children with autism spectrum disorder that provides reliable scores.

Measurement Invariance in Intellectual and Developmental Disability Research.

Measurement invariance (MI) is a psychometric property of an instrument indicating the degree to which scores from an instrument are comparable across groups. In recent years, there has been a marked uptick in publications using MI in intellectual and developmental disability (IDD) samples. Our goal here is to provide an overview of why MI is important to IDD researchers and to describe some challenges to evaluating it, with an eye towards nudging our subfield into a more thoughtful and measured interpretation of studies using MI.

Corrigendum: The relationship between metamotivational knowledge and performance.

[This corrects the article DOI: 10.3389/fpsyg.2023.1124171.].

The relationship between metamotivational knowledge and performance.

Self-regulation research highlights the performance trade-offs of different motivational states. For instance, within the context of regulatory focus theory, promotion motivation enhances performance on eager tasks and prevention motivation enhances performance on vigilant tasks (i.e., regulatory focus task-motivation fit). Work on metamotivation-people's understanding and regulation of their motivational states-reveals that, on average, people demonstrate knowledge of how to create such task-motivation fit; at the same time, there is substantial variability in this normative accuracy. The present research examines whether having accurate normative metamotivational knowledge predicts performance. Results revealed that more accurate metamotivational knowledge predicts better performance on brief, single-shot tasks (Study 1) and in a consequential setting (course grades; Study 2). The effect was more robust in Study 2; potential implications of this variability are discussed for understanding when and why knowledge may be associated with performance.

Evaluating avoidable heterogeneity in exploratory factor analysis results.

Meaningful interpretations of scores derived from psychological scales depend on the replicability of psychometric properties. Despite this, and unexpected inconsistencies in psychometric results across studies, psychometrics has often been overlooked in the replication literature. In this article, we begin to address replication issues in exploratory factor analysis (EFA). We use a Monte Carlo simulation to investigate methodological choices made throughout the EFA process that have the potential to add heterogeneity to results. Our findings show that critical decision points for EFA include the method for determining the number of factors as well as rotation. The results also demonstrate the relevancy of data characteristics, as some contexts are more susceptible to the effects of methodological choice on the heterogeneity of results. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Gel Cohesivity and Breast Augmentation: Applications to Clinical Practice.

Matching breast implant gel cohesivity to the patient's needs based on anatomical and soft-tissue considerations and patient preferences is essential to optimizing aesthetic outcomes and patient satisfaction. Generally, gel cohesivity contributes to the feel and shape of the breast, which reflects volume distribution within the breast. Studies on the physical properties of silicone gel breast implants have established the relative cohesivity of implants from several manufacturers. Comprehensive patient education, tissue-based preoperative planning, and a thorough understanding of how implant properties may affect outcomes are recommended before breast augmentation. This review summarizes the literature on breast implant physical properties with a focus on gel cohesivity and translates gel cohesivity to practical considerations in choosing the right implant for the right patient in primary augmentation. Based on clinical experience, the authors generally recommend lower cohesivity implants for patients who desire a tapered upper pole or a natural appearance/"anatomic shape" as well as for those with a high BMI or a large amount of native breast tissue. Medium-cohesivity implants are most widely used and are appropriate for those with ptosis/augmentation mastopexy, patients with tight skin envelope, and for those seeking upper pole fullness. A high-cohesivity implant is recommended for patients desiring a round shape, patients with constricted lower poles, and patients with a postpartum loose envelope, a thin, soft tissue envelope, or fibrous breast tissue.

Examining the Robustness of the Graded Response and 2-Parameter Logistic Models to Violations of Construct Normality.

When fitting unidimensional item response theory (IRT) models, the population distribution of the latent trait (θ) is often assumed to be normally distributed. However, some psychological theories would suggest a nonnormal θ. For example, some clinical traits (e.g., alcoholism, depression) are believed to follow a positively skewed distribution where the construct is low for most people, medium for some, and high for few. Failure to account for nonnormality may compromise the validity of inferences and conclusions. Although corrections have been developed to account for nonnormality, these methods can be computationally intensive and have not yet been widely adopted. Previous research has recommended implementing nonnormality corrections when θ is not "approximately normal." This research focused on examining how far θ can deviate from normal before the normality assumption becomes untenable. Specifically, our goal was to identify the type(s) and degree(s) of nonnormality that result in unacceptable parameter recovery for the graded response model (GRM) and 2-parameter logistic model (2PLM).

Using validity theory and psychometrics to evaluate and support expanded uses of existing scales.

BACKGROUND: Scale development is a complex activity requiring significant investments of time and money to produce evidence of a scale's ability to produce reliable scores and valid inferences. With increasing use of clinical outcome assessments (COAs) in medical product development, evidentiary expectations of regulatory bodies to support inferences are a key consideration. The goal of this paper is to demonstrate how existing methods in measurement science can be used to identify and fill evidence gaps when considering re-purposing an existing scale for a new use case (e.g., new patient population, altering the recall period), rather than creating a new COA tool. METHODS: We briefly review select validity theory and psychometric concepts, linking them to the nomenclature in the COA/regulated space. Four examples (two in-text and two in online supplemental materials) of modifications are presented to demonstrate these ideas in practice for quality of life (QOL)-related measures. RESULTS: Each example highlights the initial process of evaluating the desired validity claims, identifying gaps in evidence to support these claims, and determining how such gaps could be filled, often without having to develop a new measure. CONCLUSIONS: If an existing scale, with minimal modification or additional evidence, can be shown to be fit for a new purpose, considerable effort can be saved and research waste avoided. In many cases, a new instrument is simply unnecessary. Far better to recycle an "old" scale for a new use-with sufficient evidence that it is fit for that purpose-than to "buy" a new one.

A revised and expanded taxonomy for understanding heterogeneity in research and reporting practices.

Concerns about replication failures can be partially recast as concerns about excessive heterogeneity in research results. Although this heterogeneity is an inherent part of science (e.g., sampling variability; studying different conditions), not all heterogeneity results from unavoidable sources. In particular, the flexibility researchers have when designing studies and analyzing data adds additional heterogeneity. This flexibility has been the topic of considerable discussion in the last decade. Ideas, and corresponding phrases, have been introduced to help unpack researcher behaviors, including researcher degrees of freedom and questionable research practices. Using these concepts and phrases, methodological and substantive researchers have considered how researchers' choices impact statistical conclusions and reduce clarity in the research literature. While progress has been made, inconsistent, vague, and overlapping use of the terminology surrounding these choices has made it difficult to have clear conversations about the most pressing issues. Further refinement of the language conveying the underlying concepts can catalyze further progress. We propose a revised, expanded taxonomy for assessing research and reporting practices. In addition, we redefine several crucial terms in a way that reduces overlap and enhances conceptual clarity, with particular focus on distinguishing practices along two lines: research versus reporting practices and choices involving multiple empirically supported options versus choices known to be subpar. We illustrate the effectiveness of these changes using conceptual and simulated demonstrations, and we discuss how this taxonomy can be valuable to substantive researchers by helping to navigate this flexibility and to methodological researchers by motivating research toward areas of greatest need. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Enabling patient-reported outcome measures in clinical trials, exemplified by cardiovascular trials.

OBJECTIVES: There has been limited success in achieving integration of patient-reported outcomes (PROs) in clinical trials. We describe how stakeholders envision a solution to this challenge. METHODS: Stakeholders from academia, industry, non-profits, insurers, clinicians, and the Food and Drug Administration convened at a Think Tank meeting funded by the Duke Clinical Research Institute to discuss the challenges of incorporating PROs into clinical trials and how to address those challenges. Using examples from cardiovascular trials, this article describes a potential path forward with a focus on applications in the United States. RESULTS: Think Tank members identified one key challenge: a common understanding of the level of evidence that is necessary to support patient-reported outcome measures (PROMs) in trials. Think Tank participants discussed the possibility of creating general evidentiary standards depending upon contextual factors, but such guidelines could not be feasibly developed because many contextual factors are at play. The attendees posited that a more informative approach to PROM evidentiary standards would be to develop validity arguments akin to courtroom briefs, which would emphasize a compelling rationale (interpretation/use argument) to support a PROM within a specific context. Participants envisioned a future in which validity arguments would be publicly available via a repository, which would be indexed by contextual factors, clinical populations, and types of claims. CONCLUSIONS: A publicly available repository would help stakeholders better understand what a community believes constitutes compelling support for a specific PROM in a trial. Our proposed strategy is expected to facilitate the incorporation of PROMs into cardiovascular clinical trials and trials in general.

The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation.

Model fit indices are being increasingly recommended and used to select the number of factors in an exploratory factor analysis. Growing evidence suggests that the recommended cutoff values for common model fit indices are not appropriate for use in an exploratory factor analysis context. A particularly prominent problem in scale evaluation is the ubiquity of correlated residuals and imperfect model specification. Our research focuses on a scale evaluation context and the performance of four standard model fit indices: root mean square error of approximate (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker-Lewis index (TLI), and two equivalence test-based model fit indices: RMSEAt and CFIt. We use Monte Carlo simulation to generate and analyze data based on a substantive example using the positive and negative affective schedule (N = 1,000). We systematically vary the number and magnitude of correlated residuals as well as nonspecific misspecification, to evaluate the impact on model fit indices in fitting a two-factor exploratory factor analysis. Our results show that all fit indices, except SRMR, are overly sensitive to correlated residuals and nonspecific error, resulting in solutions that are overfactored. SRMR performed well, consistently selecting the correct number of factors; however, previous research suggests it does not perform well with categorical data. In general, we do not recommend using model fit indices to select number of factors in a scale evaluation framework.

A Psychometric Analysis of the Brief Self-Control Scale.

The Brief Self-Control Scale (BSCS) is a widely used measure of self-control, a construct associated with beneficial psychological outcomes. Several studies have investigated the psychometric properties of the BSCS but have failed to reach consensus. This has resulted in an unstable and ambiguous understanding of the scale and its psychometric properties. The current study sought resolution by implementing scale evaluation approaches guided by modern psychometric literature. Additionally, our goal was to provide a more comprehensive item analysis via the item response theory (IRT) framework. Results from the current study support both unidimensional and multidimensional factor structures for the 13-item version of the BSCS. The addition of an IRT analysis provided a new perspective on item- and test-level functioning. The goal of a more defensible psychometric grounding for the BSCS is to promote greater consistency, stability, and trust in future results.

Bayes factor in one-sample tests of means with a sensitivity analysis: A discussion of separate prior distributions.

Due to some widely known critiques of traditional hypothesis testing, Bayesian hypothesis testing using the Bayes factor has been considered as a better alternative. Previous research about the influence of the prior focuses on the prior for the effect size and there is a debate about how to specify the prior. Thus, the focus of this paper is to explore the impact of different priors on the population mean and variance separately (separate priors) on the Bayes factor, and compare the separate priors with the priors on the effect size. Our simulation results show that both the prior distributions on mean and variance have a considerable influence on the Bayes factor, and different types of priors (different separate priors and priors on the effect size) have different influence patterns. We also find that regardless of separate priors or priors on the effect size, and shapes and centers of the priors, different priors could yield similar Bayes factors. Because noninformative prior distributions bias the Bayes factor in support of the null hypothesis, and very informative priors could be risky, we suggest that researchers use weakly informative priors as reasonable priors and they are expected to provide similar conclusions across different shapes and centers of prior distributions. Conducting sensitivity analysis is helpful in examining the influence of prior distributions and specifying reasonable prior distributions for the Bayes factor. A real data example is used to illustrate how to choose reasonable priors by a sensitivity analysis. We hope our results will help researchers choose prior distributions when conducting Bayesian hypothesis testing.

Development of the Parent-Rated Anxiety Scale for Youth With Autism Spectrum Disorder.

OBJECTIVE: Anxiety is common in youth with autism spectrum disorder (ASD). There is no accepted outcome measure for anxiety in this population. METHOD: Following a series of focus groups with parents of youth with ASD, we generated 72 items (scored 0-3). Parents of 990 youth with ASD (aged 5-17 years; 80.8% male) completed an online survey. Factor analysis and item response theory analyses reduced the content to a single factor with 25 items. Youth with at least mild anxiety (n = 116; aged 5-17 years; 79.3% male) participated in a comprehensive clinical assessment to evaluate the validity and reliability of the 25-item Parent-Rated Anxiety Scale for ASD (PRAS-ASD). RESULTS: In the online sample, the mean PRAS-ASD score was 29.04 ± 14.9 (range, 0-75). The coefficient α was 0.93. The item response theory results indicated excellent reliability across a wide range of scores with low standard errors. In the clinical sample (n = 116), the PRAS-ASD mean was 31.0 ± 15.6 (range, 1-65). Pearson correlations with parent ratings of ASD symptom severity, repetitive behavior, and disruptive behavior ranged 0.33 to 0.66, supporting divergent validity of the PRAS-ASD. Pearson correlation with a parent-rated measure of anxiety used in the general pediatric population of 0.83 supported convergent validity. A total of 40 participants (32 boys, 8 girls; mean age, 11.9 ± 3.4 years) returned at time 2 (mean, 12.2 days) and time 3 (mean, 24.2 days). Intraclass correlation showed test-retest reliabilities of 0.88 and 0.86 at time 2 and time 3, respectively. CONCLUSION: The 25-item PRAS-ASD is a reliable and valid scale for measuring anxiety in youth with ASD.

Lactation Outcomes in More Than 3500 Women Following Primary Augmentation: 5-Year Data From the Breast Implant Follow-Up Study.

BACKGROUND: Women with breast implants may have concerns about their ability to successfully breast-feed. The Breast Implant Follow-up Study (BIFS-001) is a large, 10-year observational study evaluating the performance and safety of Natrelle round silicone gel-filled breast implants. OBJECTIVES: This analysis compared lactation outcomes in women enrolled in BIFS-001 who gave birth after they underwent primary augmentation with Natrelle round silicone implants or saline implants. METHODS: At baseline and annually after surgery (>5-year visit window), patients completed questionnaires regarding pregnancy and lactation. Comparisons were made using summary statistics and odds ratios with 90% confidence intervals (OR [90% CI]). RESULTS: A total of 4679 subjects gave birth at least once after primary augmentation for a total of 5736 live births during the study (silicone, 3695 births; saline, 2041 births). Of these, 3715 (79.4%) women breast-fed at least 1 child, resulting in 80.0% (silicone) and 75.9% (saline) of babies being breast-fed. The most common complication was insufficient milk production, which was reported for 19.6% (silicone) and 19.8% (saline) of single births (OR, 0.94 [0.83, 1.06]). Complications occurred at similar rates in each group when evaluated by incision type, implant size, pocket location, and age. CONCLUSIONS: In this large group of women who gave birth after primary breast augmentation with Natrelle round silicone implants or saline implants, most were able to breast-feed their infants without complications. Lactation complications were comparable between the silicone and saline cohorts, and the incidence was comparable to reports in the general population of women who breast-feed.

The Rutgers Alcohol Problem Index for Adolescent Alcohol and Drug Problems: A Comprehensive Modern Psychometric Study.

OBJECTIVE: The Rutgers Alcohol Problem Index (RAPI) has been used extensively as a measure of alcohol-related problems experienced by adolescents and young adults. The present study aimed to comprehensively examine the psychometric profile and criterion-related validity of an 18-item RAPI adapted to measure negative consequences resulting from alcohol and other drug use in an at-risk adolescent population. METHOD: Categorical confirmatory factor analysis and the graded response model were used for evaluations of the latent factor structure, item properties, test information, and item invariance across gender, age groups (middle vs. late adolescence), and drug use profiles (readily available drugs vs. other illicit drugs), as well as computation of the item response theory scale scores for each of 617 alternative high school students (44% female). RESULTS: A reduced set of 16 RAPI items provided plausible evidence of unidimensionality and good measurement precision at a relatively wide range of the latent trait continuum. All but two items were invariant across the drug use profiles. The other illicit drug group had higher levels of the latent substance-related problems than the readily available drug group. Criterion-related validity was supported using a measure of past-year polysubstance use. CONCLUSIONS: The RAPI can be effectively modified to measure negative consequences resulting from alcohol and other drug use and reflects a sufficiently wide range of the latent continuum of problem use.

Trajectories of cannabis-related associative memory among vulnerable adolescents: Psychometric and longitudinal evaluations.

Indirect tests of memory associations relevant to cannabis have been shown to be useful in explaining and predicting adolescent cannabis use habits. This study sought to increase the understanding of adolescent cannabis-related associative memory and cannabis use behavior over time. A longitudinal sample of alternative high school students (N = 775) was assessed yearly for 3 years. The study first conducted extensive longitudinal measurement analyses of the cannabis-related word association test (WAT) applying contemporary psychometric models. Second, the study examined the longitudinal trajectories of cannabis-related associative memory and cannabis use and their contemporaneous and longitudinal relationships. Results showed that the cannabis-related WAT provided strong evidence of sound psychometric properties. Longitudinal change in cannabis-related associative memory was best described by modeling either a linearly decreasing trajectory or two separate trajectories: During middle adolescence, levels of cannabis-related associative memory were highest and stable but then gradually decreased toward late adolescence. Moreover, cannabis-related associative memory was contemporaneously predictive of cannabis use within ages 15 to 19 while controlling for the underlying growth process of cannabis use and time-invariant covariates (TICs) of gender and lifetime concurrent use of alcohol and cigarettes. Partial support of longitudinal prediction of cannabis use was also obtained from age 17 to 18 and age 19 to 20 while adjusting for growth in cannabis use and the TICs. These results demonstrated that predictive effects of cannabis-related memory associations on cannabis use were detected within some of the 1-year age spans and were consistent within ages across adolescent years. (PsycINFO Database Record

Scale development with small samples: a new application of longitudinal item response theory.

PURPOSE: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. METHODS: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model's ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. RESULTS: Simulation study results demonstrate that the longitudinal IRT model's ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. CONCLUSION: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.