Using validity theory and psychometrics to evaluate and support expanded uses of existing scales.

BACKGROUND: Scale development is a complex activity requiring significant investments of time and money to produce evidence of a scale's ability to produce reliable scores and valid inferences. With increasing use of clinical outcome assessments (COAs) in medical product development, evidentiary expectations of regulatory bodies to support inferences are a key consideration. The goal of this paper is to demonstrate how existing methods in measurement science can be used to identify and fill evidence gaps when considering re-purposing an existing scale for a new use case (e.g., new patient population, altering the recall period), rather than creating a new COA tool. METHODS: We briefly review select validity theory and psychometric concepts, linking them to the nomenclature in the COA/regulated space. Four examples (two in-text and two in online supplemental materials) of modifications are presented to demonstrate these ideas in practice for quality of life (QOL)-related measures. RESULTS: Each example highlights the initial process of evaluating the desired validity claims, identifying gaps in evidence to support these claims, and determining how such gaps could be filled, often without having to develop a new measure. CONCLUSIONS: If an existing scale, with minimal modification or additional evidence, can be shown to be fit for a new purpose, considerable effort can be saved and research waste avoided. In many cases, a new instrument is simply unnecessary. Far better to recycle an "old" scale for a new use-with sufficient evidence that it is fit for that purpose-than to "buy" a new one.

Enabling patient-reported outcome measures in clinical trials, exemplified by cardiovascular trials.

OBJECTIVES: There has been limited success in achieving integration of patient-reported outcomes (PROs) in clinical trials. We describe how stakeholders envision a solution to this challenge. METHODS: Stakeholders from academia, industry, non-profits, insurers, clinicians, and the Food and Drug Administration convened at a Think Tank meeting funded by the Duke Clinical Research Institute to discuss the challenges of incorporating PROs into clinical trials and how to address those challenges. Using examples from cardiovascular trials, this article describes a potential path forward with a focus on applications in the United States. RESULTS: Think Tank members identified one key challenge: a common understanding of the level of evidence that is necessary to support patient-reported outcome measures (PROMs) in trials. Think Tank participants discussed the possibility of creating general evidentiary standards depending upon contextual factors, but such guidelines could not be feasibly developed because many contextual factors are at play. The attendees posited that a more informative approach to PROM evidentiary standards would be to develop validity arguments akin to courtroom briefs, which would emphasize a compelling rationale (interpretation/use argument) to support a PROM within a specific context. Participants envisioned a future in which validity arguments would be publicly available via a repository, which would be indexed by contextual factors, clinical populations, and types of claims. CONCLUSIONS: A publicly available repository would help stakeholders better understand what a community believes constitutes compelling support for a specific PROM in a trial. Our proposed strategy is expected to facilitate the incorporation of PROMs into cardiovascular clinical trials and trials in general.

Lactation Outcomes in More Than 3500 Women Following Primary Augmentation: 5-Year Data From the Breast Implant Follow-Up Study.

BACKGROUND: Women with breast implants may have concerns about their ability to successfully breast-feed. The Breast Implant Follow-up Study (BIFS-001) is a large, 10-year observational study evaluating the performance and safety of Natrelle round silicone gel-filled breast implants. OBJECTIVES: This analysis compared lactation outcomes in women enrolled in BIFS-001 who gave birth after they underwent primary augmentation with Natrelle round silicone implants or saline implants. METHODS: At baseline and annually after surgery (>5-year visit window), patients completed questionnaires regarding pregnancy and lactation. Comparisons were made using summary statistics and odds ratios with 90% confidence intervals (OR [90% CI]). RESULTS: A total of 4679 subjects gave birth at least once after primary augmentation for a total of 5736 live births during the study (silicone, 3695 births; saline, 2041 births). Of these, 3715 (79.4%) women breast-fed at least 1 child, resulting in 80.0% (silicone) and 75.9% (saline) of babies being breast-fed. The most common complication was insufficient milk production, which was reported for 19.6% (silicone) and 19.8% (saline) of single births (OR, 0.94 [0.83, 1.06]). Complications occurred at similar rates in each group when evaluated by incision type, implant size, pocket location, and age. CONCLUSIONS: In this large group of women who gave birth after primary breast augmentation with Natrelle round silicone implants or saline implants, most were able to breast-feed their infants without complications. Lactation complications were comparable between the silicone and saline cohorts, and the incidence was comparable to reports in the general population of women who breast-feed.

Bayes factor in one-sample tests of means with a sensitivity analysis: A discussion of separate prior distributions.

Due to some widely known critiques of traditional hypothesis testing, Bayesian hypothesis testing using the Bayes factor has been considered as a better alternative. Previous research about the influence of the prior focuses on the prior for the effect size and there is a debate about how to specify the prior. Thus, the focus of this paper is to explore the impact of different priors on the population mean and variance separately (separate priors) on the Bayes factor, and compare the separate priors with the priors on the effect size. Our simulation results show that both the prior distributions on mean and variance have a considerable influence on the Bayes factor, and different types of priors (different separate priors and priors on the effect size) have different influence patterns. We also find that regardless of separate priors or priors on the effect size, and shapes and centers of the priors, different priors could yield similar Bayes factors. Because noninformative prior distributions bias the Bayes factor in support of the null hypothesis, and very informative priors could be risky, we suggest that researchers use weakly informative priors as reasonable priors and they are expected to provide similar conclusions across different shapes and centers of prior distributions. Conducting sensitivity analysis is helpful in examining the influence of prior distributions and specifying reasonable prior distributions for the Bayes factor. A real data example is used to illustrate how to choose reasonable priors by a sensitivity analysis. We hope our results will help researchers choose prior distributions when conducting Bayesian hypothesis testing.

Measurement invariance, the lack thereof, and modeling change.

PURPOSE: Measurement invariance issues should be considered during test construction. In this paper, we provide a conceptual overview of measurement invariance and describe how the concept is implemented in several different statistical approaches. Typical applications look for invariance over things such as mode of administration (paper and pencil vs. computer based), language/translation, age, time, and gender, to cite just a few examples. To the extent that the relationships between items and constructs are stable/invariant, we can be more confident in score interpretations. METHODS: A series of simulated examples are reported which highlight different kinds of non-invariance, the impact it can have, and the effect of appropriately modeling a lack of invariance. One example focuses on the longitudinal context, where measurement invariance is critical to understanding trends over time. Software syntax is provided to help researchers apply these models with their own data. RESULTS: The simulation studies demonstrate the negative impact an erroneous assumption of invariance may have on scores and substantive conclusions drawn from naively analyzing those scores. CONCLUSIONS: Measurement invariance implies that the links between the items and the construct of interest are invariant over some domain, grouping, or classification. Examining a new or existing test for measurement invariance should be part of any test construction/implementation plan. In addition to reviewing implications of the simulation study results, we also provide a discussion of the limitations of current approaches and areas in need of additional research.

Fit for purpose and modern validity theory in clinical outcomes assessment.

PURPOSE: The US Food and Drug Administration (FDA), as part of its regulatory mission, is charged with determining whether a clinical outcome assessment (COA) is "fit for purpose" when used in clinical trials to support drug approval and product labeling. In this paper, we will provide a review (and some commentary) on the current state of affairs in COA development/evaluation/use with a focus on one aspect: How do you know you are measuring the right thing? In the psychometric literature, this concept is referred to broadly as validity and has itself evolved over many years of research and application. REVIEW: After a brief introduction, the first section will review current ideas about "fit for purpose" and how it has been viewed by FDA. This section will also describe some of the unique challenges to COA development/evaluation/use in the clinical trials space. Following this, we provide an overview of modern validity theory as it is currently understood in the psychometric tradition. This overview will focus primarily on the perspective of validity theorists such as Messick and Kane whose work forms the backbone for the bulk of high-stakes assessment in areas such as education, psychology, and health outcomes. CONCLUSIONS: We situate the concept of fit for purpose within the broader context of validity. By comparing and contrasting the approaches and the situations where they have traditionally been applied, we identify areas of conceptual overlap as well as areas where more discussion and research are needed.

Scale development with small samples: a new application of longitudinal item response theory.

PURPOSE: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. METHODS: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model's ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. RESULTS: Simulation study results demonstrate that the longitudinal IRT model's ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. CONCLUSION: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.

Molecular Characterization and Functional Analysis of PR-1-Like Proteins Identified from the Wheat Head Blight Fungus Fusarium graminearum.

The group 1 pathogenesis-related (PR-1) proteins originally identified from plants and their homologs are also found in other eukaryotic kingdoms. Studies on nonplant PR-1-like (PR-1L) proteins have been pursued widely in humans and animals but rarely in filamentous ascomycetes. Here, we report the characterization of four PR-1L proteins identified from the ascomycete fungus Fusarium graminearum, the primary cause of Fusarium head blight of wheat and barley (designated FgPR-1L). Molecular cloning revealed that the four FgPR-1L proteins are all encoded by small open reading frames (612 to 909 bp) that are often interrupted by introns, in contrast to plant PR-1 genes that lack introns. Sequence analysis indicated that all FgPR-1L proteins contain the PR-1-specific three-dimensional structure, and one of them features a C-terminal transmembrane (TM) domain that has not been reported for any stand-alone PR-1 proteins. Transcriptional analysis revealed that the four FgPR-1L genes are expressed in axenic cultures and in planta with different spatial or temporal expression patterns. Phylogenetic analysis indicated that fungal PR-1L proteins fall into three major groups, one of which harbors FgPR-1L-2-related TM-containing proteins from both phytopathogenic and human-pathogenic ascomycetes. Low-temperature sodium dodecyl sulfate polyacrylamide gel electrophoresis and proteolytic assays indicated that the recombinant FgPR-1L-4 protein exists as a monomer and is resistant to subtilisin of the serine protease family. Functional analysis confirmed that deletion of the FgPR-1L-4 gene from the fungal genome results in significantly reduced virulence on susceptible wheat. This study provides the first example that the F. graminearum-wheat interaction involves a pathogen-derived PR-1L protein that affects fungal virulence on the host.

Molecular cloning and characterization of two novel genes from hexaploid wheat that encode double PR-1 domains coupled with a receptor-like protein kinase.

Hexaploid wheat (Triticum aestivum L.) contains at least 23 TaPr-1 genes encoding the group 1 pathogenesis-related (PR-1) proteins as identified in our previous work. Here, we report the cloning and characterization of TaPr-1-rk1 and TaPr-1-rk2, two novel genes closely related to the wheat PR-1 family. The two TaPr-1-rk genes are located on homoeologous chromosomes 3D and 3A, respectively, and each contains a large open reading frame (7385 or 6060 bp) that is interrupted by seven introns and subjected to alternative splicing (AS) with five or six isoforms of mRNA transcripts. The deduced full-length TaPR-1-RK1 and TaPR-1-RK2 proteins (95% identity) contain two repeat PR-1 domains, the second of which is fused via a transmembrane helix to a serine/threonine kinase catalytic (STKc) domain characteristic of receptor-like protein kinases. Phylogenetic analysis indicated that the two PR-1 domains of the TaPR-1-RK proteins form sister clades with their homologues identified in other monocot plants and are well separated from stand-alone PR-1 proteins, whereas the STKc domains may have originated from cysteine-rich receptor-like kinases (CRKs). Reverse-transcriptase-PCR analysis revealed that the TaPr-1-rk genes are predominantly expressed in wheat leaves and their expression levels are elevated in response to pathogen attack, such as infection by barley stripe mosaic virus (BSMV), and also to stress conditions, most obviously, to soil salinity. This is the first report of PR-1-CRK hybrid proteins in wheat. The data may shed new insights into the function/evolutionary origin of the PR-1 family and the STKc-mediated defense/stress response pathways in plants.

ORF43 of maize rayado fino virus is dispensable for systemic infection of maize and transmission by leafhoppers.

Maize rayado fino virus (MRFV) possesses an open reading frame (ORF43) predicted to encode a 43 kDa protein (p43) that has been postulated to be a viral movement protein. Using a clone of MRFV (pMRFV-US) from which infectious RNA can be produced, point mutations were introduced to either prevent initiation from three potential AUG initiation codons near the 5'-end of ORF43 or prematurely terminate translation of ORF43. Inoculation of maize seed via vascular puncture inoculation (VPI) resulted in plants exhibiting symptoms typical of MRFV infection for all mutants tested. Furthermore, corn leafhoppers (Dalbulus maidis) transmitted the virus mutants to healthy plants at a frequency similar to that for wild-type MRFV-US. Viral RNA recovered from plants infected with mutants both prior to and after leafhopper transmission retained mutations blocking ORF43 expression. The results indicate that ORF43 of MRFV is dispensable for both systemic infection of maize and transmission by leafhoppers.

Item response theory analysis of Centers for Disease Control and Prevention Health-Related Quality of Life (CDC HRQOL) items in adults with arthritis.

BACKGROUND: Examine the feasibility of performing an item response theory (IRT) analysis on two of the Centers for Disease Control and Prevention health-related quality of life (CDC HRQOL) modules - the 4-item Healthy Days Core Module (HDCM) and the 5-item Healthy days Symptoms Module (HDSM). Previous principal components analyses confirm that the two scales both assess a mix of mental (CDC-MH) and physical health (CDC-PH). The purpose is to conduct item response theory (IRT) analysis on the CDC-MH and CDC-PH scales separately. METHODS: 2182 patients with self-reported or physician-diagnosed arthritis completed a cross-sectional survey including HDCM and HDSM items. Besides global health, the other 8 items ask the number of days that some statement was true; we chose to recode the data into 8 categories based on observed clustering. The IRT assumptions were assessed using confirmatory factor analysis and the data could be modeled using an unidimensional IRT model. The graded response model was used for IRT analyses and CDC-MH and CDC-PH scales were analyzed separately in flexMIRT. RESULTS: The IRT parameter estimates for the five-item CDC-PH all appeared reasonable. The three-item CDC-MH did not have reasonable parameter estimates. CONCLUSIONS: The CDC-PH scale is amenable to IRT analysis but the existing The CDC-MH scale is not. We suggest either using the 4-item Healthy Days Core Module (HDCM) and the 5-item Healthy days Symptoms Module (HDSM) as they currently stand or the CDC-PH scale alone if the primary goal is to measure physical health related HRQOL.

A review of empirical research related to the use of small quantitative samples in clinical outcome scale development.

INTRODUCTION: There has been a notable increase in the advocacy of using small-sample designs as an initial quantitative assessment of item and scale performance during the scale development process. This is particularly true in the development of clinical outcome assessments (COAs), where Rasch analysis has been advanced as an appropriate statistical tool for evaluating the developing COAs using a small sample. METHODS: We review the benefits such methods are purported to offer from both a practical and statistical standpoint and detail several problematic areas, including both practical and statistical theory concerns, with respect to the use of quantitative methods, including Rasch-consistent methods, with small samples. CONCLUSIONS: The feasibility of obtaining accurate information and the potential negative impacts of misusing large-sample statistical methods with small samples during COA development are discussed.

Genome-Wide Analysis of Small Secreted Cysteine-Rich Proteins Identifies Candidate Effector Proteins Potentially Involved in Fusarium graminearum-Wheat Interactions.

Pathogen-derived, small secreted cysteine-rich proteins (SSCPs) are known to be a common source of fungal effectors that trigger resistance or susceptibility in specific host plants. This group of proteins has not been well studied in Fusarium graminearum, the primary cause of Fusarium head blight (FHB), a devastating disease of wheat. We report here a comprehensive analysis of SSCPs encoded in the genome of this fungus and selection of candidate effector proteins through proteomics and sequence/transcriptional analyses. A total of 190 SSCPs were identified in the genome of F. graminearum (isolate PH-1) based on the presence of N-terminal signal peptide sequences, size (≤200 amino acids), and cysteine content (≥2%) of the mature proteins. Twenty-five (approximately 13%) SSCPs were confirmed to be true extracellular proteins by nanoscale liquid chromatography-tandem mass spectrometry (nanoLC-MS/MS) analysis of a minimal medium-based in vitro secretome. Sequence analysis suggested that 17 SSCPs harbor conserved functional domains, including two homologous to Ecp2, a known effector produced by the tomato pathogen Cladosporium fulvum. Transcriptional analysis revealed that at least 34 SSCPs (including 23 detected in the in vitro secretome) are expressed in infected wheat heads; about half are up-regulated with expression patterns correlating with the development of FHB. This work provides a solid candidate list for SSCP-derived effectors that may play roles in mediating F. graminearum-wheat interactions. The in vitro secretome-based method presented here also may be applicable for identifying candidate effectors in other ascomycete pathogens of crop plants.

Development of the Contact Lens User Experience: CLUE Scales.

PURPOSE: The field of optometry has become increasingly interested in patient-reported outcomes, reflecting a common trend occurring across the spectrum of healthcare. This article reviews the development of the Contact Lens User Experience: CLUE system designed to assess patient evaluations of contact lenses. CLUE was built using modern psychometric methods such as factor analysis and item response theory. METHODS: The qualitative process through which relevant domains were identified is outlined as well as the process of creating initial item banks. Psychometric analyses were conducted on the initial item banks and refinements were made to the domains and items. Following this data-driven refinement phase, a second round of data was collected to further refine the items and obtain final item response theory item parameters estimates. RESULTS: Extensive qualitative work identified three key areas patients consider important when describing their experience with contact lenses. Based on item content and psychometric dimensionality assessments, the developing CLUE instruments were ultimately focused around four domains: comfort, vision, handling, and packaging. Item response theory parameters were estimated for the CLUE item banks (377 items), and the resulting scales were found to provide precise and reliable assignment of scores detailing users' subjective experiences with contact lenses. CONCLUSIONS: The CLUE family of instruments, as it currently exists, exhibits excellent psychometric properties.

REMAP-a Resilience Resources Measure for Prediction and Management of Somatic Symptoms.

PURPOSE: Patients with symptoms but without an identified disease are a challenge to primary care providers. A 22-item measure is introduced and evaluated to offer medical care providers with an instrument to assess and discuss possible deficiencies in resilience resources that may contribute to symptoms without identifiable pathology. This instrument highlights psychosocial and lifestyle resources that serve as buffers to life's stressors rather than focusing on stress and its related symptoms. METHODS: The measure included items from five resilience domains-relational engagement, emotional sensibility, meaningful action, awareness of self and others, and physical health behaviors (REMAP). Its structure and function were evaluated using two different samples. RESULTS: Results suggest that scores from the REMAP have reasonable psychometric properties. Higher REMAP scores were predictive of fewer health symptoms in a sample representative of the US population. In a second sample, REMAP was positively associated with perceived resilience, ego strength and mindfulness attention and negatively related to perceived stress, depression, sleep disturbances, and loneliness, providing evidence of convergent and divergent validity. Furthermore, the REMAP scale was sensitive to change following a life style intervention. CONCLUSION: This suggests that REMAP can be a useful tool in practice settings for counseling patients with unexplained symptoms. With insight into the biopsychosocial aspect of their symptoms, patients may become more receptive to cognitive behavioral options to improve their resilience resources and lifestyle choices.

A ToxA-like protein from Cochliobolus heterostrophus induces light-dependent leaf necrosis and acts as a virulence factor with host selectivity on maize.

ToxA, the first discovered fungal proteinaceous host-selective toxin (HST), was originally identified in 1989 from the tan spot fungus Pyrenophora tritici-repentis (Ptr). About 25years later, a homolog was identified in the leaf/glume blotch fungus Stagonospora nodorum (Parastagonospora nodorum), also a pathogen of wheat. Here we report the identification and function of a ToxA-like protein from the maize pathogen Cochliobolus heterostrophus (Ch) that possesses necrosis-inducing activity specifically against maize. ChToxA is encoded by a 535-bp open reading frame featuring a ToxA-specific intron with unusual splicing sites (5'-ATAAGT…TAC-3') at conserved positions relative to PtrToxA. The protein shows 64% similarity to PtrToxA and is predicted to adopt a similar three-dimensional structure, although lacking the arginyl-glycyl-aspartic acid (RGD) motif reported to be required for internalization into sensitive wheat mesophyll cells. Reverse-transcriptase PCR revealed that the ChTOXA gene expression is up-regulated in planta, relative to axenic culture. Plant assays indicated that the recombinant ChToxA protein induces light-dependent leaf necrosis in a host-selective manner on maize inbred lines. Gene deletion experiments confirmed that ChtoxA mutants are reduced in virulence on specific ChToxA-sensitive maize lines, relative to virulence caused by wild-type strains. Database searches identified potential ChToxA homologues in other plant-pathogenic ascomycetes. Sequence and phylogenetic analyses revealed that the corresponding ToxA-like proteins include one member recently shown to be associated with formation of penetration hypha. These results provide the first evidence that C. heterostrophus is capable of producing proteinaceous HSTs as virulence factors in addition to well-known secondary metabolite-type toxins produced biosynthetically by polyketide synthase megaenzymes. Further studies on ChToxA may provide new insights into effector evolution in host-pathogen interactions.

Infectious Maize rayado fino virus from Cloned cDNA.

A full-length cDNA clone was produced from a U.S. isolate of Maize rayado fino virus (MRFV), the type member of the genus Marafivirus within the family Tymoviridae. Infectivity of transcripts derived from cDNA clones was demonstrated by infection of maize plants and protoplasts, as well as by transmission via the known leafhopper vectors Dalbulus maidis and Graminella nigrifrons that transmit the virus in a persistent-propagative manner. Infection of maize plants through vascular puncture inoculation of seed with transcript RNA resulted in the induction of fine stipple stripe symptoms typical of those produced by wild-type MRFV and a frequency of infection comparable with that of the wild type. Northern and Western blotting confirmed the production of MRFV-specific RNAs and proteins in infected plants and protoplasts. An unanticipated increase in subgenomic RNA synthesis over levels in infected plants was observed in protoplasts infected with either wild-type or cloned virus. A conserved cleavage site motif previously demonstrated to function in both Oat blue dwarf virus capsid protein and tymoviral nonstructural protein processing was identified near the amino terminus of the MRFV replicase polyprotein, suggesting that cleavage at this site also may occur.

Measurement Invariance in Intellectual and Developmental Disability Research.

Measurement invariance (MI) is a psychometric property of an instrument indicating the degree to which scores from an instrument are comparable across groups. In recent years, there has been a marked uptick in publications using MI in intellectual and developmental disability (IDD) samples. Our goal here is to provide an overview of why MI is important to IDD researchers and to describe some challenges to evaluating it, with an eye towards nudging our subfield into a more thoughtful and measured interpretation of studies using MI.

Factor analytic and item response theory evaluation of the Penn State Worry Questionnaire in women with cancer.

PURPOSE: Cancer survivors frequently experience worry about a variety of topics, including fear of recurrence. However, general measures of worry still require examination of reliability for this vulnerable population. This study utilized modern psychometric methods to examine the reliability of a worry measure in women with breast or gynecologic cancer. METHODS: Women with cancer (n = 332) completed the 16-item Penn State Worry Questionnaire (PSWQ), which has an abbreviated 8-item version (PSWQ-A). Categorical confirmatory factor analysis (CCFA) was used to determine the factor structure and item response theory (IRT) was used to examine score reliability. RESULTS: CCFA supported a two-factor structure with 11 positively worded items and the 5 negatively worded items loading on different factors. IRT analysis of the 11 positively worded items showed that each was contributing meaningful information to the overall scores. The 11 positively worded items and the PSWQ-A produced the most reliable scores for levels of worry ranging from one θ below to two θ above the mean. CONCLUSIONS: The 11 positively worded items of the PSWQ and the 8-item PSWQ-A were suitable for use in cancer patients while the full PSWQ was unsuitable due to inclusion of the negatively worded items. Future research should consider measuring worry when examining distress in cancer survivors.