Using validity theory and psychometrics to evaluate and support expanded uses of existing scales.

BACKGROUND: Scale development is a complex activity requiring significant investments of time and money to produce evidence of a scale's ability to produce reliable scores and valid inferences. With increasing use of clinical outcome assessments (COAs) in medical product development, evidentiary expectations of regulatory bodies to support inferences are a key consideration. The goal of this paper is to demonstrate how existing methods in measurement science can be used to identify and fill evidence gaps when considering re-purposing an existing scale for a new use case (e.g., new patient population, altering the recall period), rather than creating a new COA tool. METHODS: We briefly review select validity theory and psychometric concepts, linking them to the nomenclature in the COA/regulated space. Four examples (two in-text and two in online supplemental materials) of modifications are presented to demonstrate these ideas in practice for quality of life (QOL)-related measures. RESULTS: Each example highlights the initial process of evaluating the desired validity claims, identifying gaps in evidence to support these claims, and determining how such gaps could be filled, often without having to develop a new measure. CONCLUSIONS: If an existing scale, with minimal modification or additional evidence, can be shown to be fit for a new purpose, considerable effort can be saved and research waste avoided. In many cases, a new instrument is simply unnecessary. Far better to recycle an "old" scale for a new use-with sufficient evidence that it is fit for that purpose-than to "buy" a new one.

Constructing and evaluating a validity argument for a performance outcome measure for clinical trials: An example using the Multi-luminance Mobility Test.

BACKGROUND: Clinical trials that evaluate new medical products often use clinical outcome assessments to measure how patients feel or function. Determining the evidentiary support needed for clinical outcome assessments is challenging but necessary to ensure scores from a clinical outcome assessment reflect the relevant aspects of patients' health. Modern validity theory-from educational and psychological testing-addresses the challenge by requiring that investigators state key assumptions underlying the proposed use of a clinical outcome assessment and collect evidence for or against those assumptions. METHODS: This article describes the argument-based approach to validity using an example of a performance outcome measure-the Multi-luminance Mobility Test-designed to assess patients with inherited retinal dystrophy that causes progressive loss of night vision. For the proposed interpretation and use of a performance outcome measure to be reasonable, several key assumptions need to be plausible. I describe the assumptions along with examples of supporting evidence from the published literature to evaluate each assumption within the rationale. RESULTS: This article provides an example of a validity rationale to evaluate a clinical outcome assessment using the Multi-luminance Mobility Test as an example. CONCLUSION: The demonstration illustrates the use of the argument-based approach to validity evaluation and the challenges in supporting parts of a validity rationale for clinical outcome assessments that measure how patients feel and function in a more indirect way. By making clinical outcome assessment validation practices consistent with modern validity theory, investigators, sponsors, and regulators should be able to communicate more clearly and direct resources more efficiently to support the creation of patient-centered endpoints in clinical trials.

Development of the Diabetic Wound Assessment Learning Tool (DiWALT) and validity evidence.

OBJECTIVE: Diabetic foot wounds account for up to one-third of diabetes-related health care expenditure and are the greatest cause of extremity amputation in Canada. Physicians encounter patients with such wounds in all specialties, particularly as generalists in medical wards and emergency departments. However, there is a dearth of literature on the optimal way to teach and to assess the management of these patients. Given the importance of assessment for learning in the shift toward competency-based medical education, we aimed to develop an assessment tool and to build validity evidence for its use in this context. METHODS: A consensus process involving nine Canadian experts in diabetic wound management was used to develop the Diabetic Wound Assessment Learning Tool (DiWALT) items and two 10-minute simulation-based testing scenarios. The simulators used were modified from commercially available models to serve the testing scenarios. Validity evidence for the DiWALT was subsequently evaluated by assessing 24 physician participants' performance during the two scenarios. All participants were novices (<50 cases managed). Two assessors independently rated participants using the DiWALT. Evidence was organized using Kane's validity framework and included Cronbach α for interitem consistency as well as test-retest and inter-rater reliability using the intra-class correlation coefficient (ICC). RESULTS: Cronbach α was 0.92, implying high internal consistency. Test-retest reliability was also excellent with ICC of 0.89 (confidence interval [CI], 0.76-0.95) for single measures and ICC of 0.94 (CI, 0.86-0.98) for average measures. Inter-rater reliability was fair for single measures with ICC of 0.68 (CI, 0.65-0.71) and good for average measures with ICC of 0.81 (CI, 0.79-0.83). CONCLUSIONS: These results demonstrate that the DiWALT consistently and reliably evaluates competence in diabetic wound management during simulated cases using a small, homogeneous sample of physicians. Further work is necessary to quantify sources of error in the assessment scores, to establish validity evidence when it is used to assess larger and more heterogeneous participants, and to identify how well the DiWALT differentiates between different experience levels.

Constructing arguments for the interpretation and use of patient-reported outcome measures in research: an application of modern validity theory.

The past 100 years have witnessed an evolution of the meaning of validity and validation within the fields of education and psychology. Validity was once viewed as a property of tests and scales, but is now viewed as the extent to which theory and evidence support proposed interpretations and uses of test scores. Uncertainty about what types of validity evidence were needed motivated the current "argument-based" approach, as reflected in the 2014 Standards for Educational and Psychological Testing. According to this approach, investigators should delineate the assumptions required in order for a proposed interpretation or use to be plausible and then seek evidence that supports or refutes those assumptions. Though validation practices within the field of patient-reported outcome measurement have implicitly included many elements of the argument-based approach, the approach has yet to be explicitly adopted. To facilitate adoption, this article proposes an initial set of assumptions that might be included in most arguments for research-related interpretations and uses of scores from patient-reported outcome measures. The article also includes brief descriptions of the types of evidence that would be best suited for evaluating each assumption. It is hoped that these generic assumptions will stimulate further discussion and debate among quality of life researchers regarding how best to adopt modern validity theory to patient-reported outcome measures.

Fostering implementation of knowledge into health practice: study protocol for the validation and redevelopment of the Knowledge Uptake and Utilization Tool.

BACKGROUND: Measurement of what knowledge is taken-up and how that information is used to inform practice and policies can provide an understanding about the effectiveness of knowledge uptake and utilization processes. In 2007, the Knowledge Uptake and Utilization Tool (KUUT) was developed to evaluate the implementation of knowledge into practice. The KUUT has been used by numerous large health organizations despite limited validity evidence and a narrow understanding about how the tool is used in practice and interpreted by users. As such, the overall purpose of this protocol is to redevelop the KUUT and gather validity evidence to examine and support its use in various health-related organizations. This protocol paper outlines a validation and redevelopment procedure for the KUUT using the unitary view of validity. METHODS: The protocol outlined in this article proceeds through four phases, starting with redeveloping the tool, then evaluating validity evidence based on: test content, response processes and internal structure. The initial phase gathers information to redevelop the tool, and evaluates item content and response format. The second phase evaluates response process validity evidence by examining how a variety of users interact with the tool. In the third phase, the tool will be pilot tested with knowledge users and, in the final phase, psychometric properties of the tool will be examined and a final scoring structure will be determined. A knowledge translation plan described herein outlines where the final tool will be housed and how the information about the tool will be disseminated. DISCUSSION: This protocol outlines a procedure to gather different sources of validity evidence for the KUUT. By addressing limitations in the original KUUT, such as complexities with scoring, a redeveloped KUUT supporting validity evidence will enhance the ability of health-related organizations to effectively use this tool for its intended purpose.

Measurement Invariance in Intellectual and Developmental Disability Research.

Measurement invariance (MI) is a psychometric property of an instrument indicating the degree to which scores from an instrument are comparable across groups. In recent years, there has been a marked uptick in publications using MI in intellectual and developmental disability (IDD) samples. Our goal here is to provide an overview of why MI is important to IDD researchers and to describe some challenges to evaluating it, with an eye towards nudging our subfield into a more thoughtful and measured interpretation of studies using MI.

A Primer on Observational Measurement.

Observational measurement plays an integral role in a variety of scientific endeavors within biology, psychology, sociology, education, medicine, and marketing. The current article provides an interdisciplinary primer on observational measurement; in particular, it highlights recent advances in observational methodology and the challenges that accompany such growth. First, we detail the various types of instrument that can be used to standardize measurements across observers. Second, we argue for the importance of validity in observational measurement and provide several approaches to validation based on contemporary validity theory. Third, we outline the challenges currently faced by observational researchers pertaining to measurement drift, observer reactivity, reliability analysis, and time/expense. Fourth, we describe recent advances in computer-assisted measurement, fully automated measurement, and statistical data analysis. Finally, we identify several key directions for future observational research to explore.