Comparing patient global impression of severity and patient global impression of change to evaluate test-retest reliability of depression, non-small cell lung cancer, and asthma measures.

PURPOSE: Score reproducibility is an important measurement property of fit-for-purpose patient-reported outcome (PRO) measures. It is commonly assessed via test-retest reliability, and best evaluated with a stable participant sample, which can be challenging to identify in diseases with highly variable symptoms. To provide empirical evidence comparing the retrospective (patient global impression of change [PGIC]) and current state (patient global impression of severity [PGIS]) approaches to identifying a stable subgroup for test-retest analyses, 3 PRO Consortium working groups collected data using both items as anchor measures. METHODS: The PGIS was completed on Day 1 and Day 8 + 3 for the depression and non-small cell lung cancer (NSCLC) studies, and daily for the asthma study and compared between Day 3 and 10. The PGIC was completed on the final day in each study. Scores were compared using an intraclass correlation coefficient (ICC) for participants who reported "no change" between timepoints for each anchor. RESULTS: ICCs using the PGIS "no change" group were higher for depression (0.84 vs. 0.74), nighttime asthma (0.95 vs. 0.53) and daytime asthma (0.86 vs. 0.68) compared to the PGIC "no change" group. ICCs were similar for NSCLC (PGIS: 0.87; PGIC: 0.85). CONCLUSION: When considering anchor measures to identify a stable subgroup for test-retest reliability analyses, current state anchors perform better than retrospective anchors. Researchers should carefully consider the type of anchor selected, the time period covered, and should ensure anchor content is consistent with the target measure concept, as well as inclusion of both current and retrospective anchor measures.

Using validity theory and psychometrics to evaluate and support expanded uses of existing scales.

BACKGROUND: Scale development is a complex activity requiring significant investments of time and money to produce evidence of a scale's ability to produce reliable scores and valid inferences. With increasing use of clinical outcome assessments (COAs) in medical product development, evidentiary expectations of regulatory bodies to support inferences are a key consideration. The goal of this paper is to demonstrate how existing methods in measurement science can be used to identify and fill evidence gaps when considering re-purposing an existing scale for a new use case (e.g., new patient population, altering the recall period), rather than creating a new COA tool. METHODS: We briefly review select validity theory and psychometric concepts, linking them to the nomenclature in the COA/regulated space. Four examples (two in-text and two in online supplemental materials) of modifications are presented to demonstrate these ideas in practice for quality of life (QOL)-related measures. RESULTS: Each example highlights the initial process of evaluating the desired validity claims, identifying gaps in evidence to support these claims, and determining how such gaps could be filled, often without having to develop a new measure. CONCLUSIONS: If an existing scale, with minimal modification or additional evidence, can be shown to be fit for a new purpose, considerable effort can be saved and research waste avoided. In many cases, a new instrument is simply unnecessary. Far better to recycle an "old" scale for a new use-with sufficient evidence that it is fit for that purpose-than to "buy" a new one.

Measuring the impact of chronic low back pain on everyday functioning: content validity of the Roland Morris disability questionnaire.

BACKGROUND: Robust outcome measures are needed to assess and monitor the impact of chronic low back pain (CLBP) on physical functioning. The Roland Morris Disability Questionnaire (RMDQ) is a well-established measure designed to capture the impacts of back pain on everyday functioning, with a particular emphasis on physical functioning. It has documented evaluation of psychometric properties. However, there is no documented qualitative evidence to confirm the content validity of the tool, nor have changes made for electronic administration been debriefed in participants with CLBP. METHODS: In-depth, semi-structured, concept elicitation and cognitive debriefing interviews were conducted with 23 US participants with confirmed CLBP. Interviews allowed participants to describe the impact of CLBP on their day-to-day functioning and discuss comprehension and suitability of the RMDQ. Interviews were transcribed verbatim and analyzed using thematic analysis. RESULTS: Concept elicitation and cognitive debriefing revealed the substantial burden associated with CLBP, highlighting 15 key areas of functional impact. These were grouped into overarching themes of mobility (walking, stairs, sitting/standing, bending/kneeling, lifting, lying down), activities (chores/housework, dressing, washing, driving, work) and other (relationships/socializing, mood, sleep, appetite), which are consistent with those evaluated within the RMDQ. All participants found the RMDQ to be relevant with most reporting that the instructions, recall period, and response options were suitable. A few suggested minor changes, however, none were consistent or necessary to support content validity. Updates to the measure for electronic administration and to clarify the response options were well received. CONCLUSION: The qualitative data from individuals with CLBP confirmed that the RMDQ has content validity and, alongside documented psychometric evidence, supports the use of the RMDQ as a reliable and valid tool to assess the impact of CLBP on physical functioning.

Why Reinvent the Wheel? Use or Modification of Existing Clinical Outcome Assessment Tools in Medical Product Development.

Assessment of clinical benefit in treatment trials can be made through report by a clinician, a patient, or a nonclinician observer (eg, caregiver) or through a performance-based assessment. The US Food and Drug Administration (FDA) published a final guidance for industry for one type of clinical outcome assessment (COA)-patient-reported outcome (PRO) measures-in 2009 that described how FDA reviews PRO measures for their adequacy to support medical product-labeling claims. Many of the principles described in the PRO Guidance could be applicable to the other types of COAs, including instruments completed by clinicians (ie, clinician-reported outcome assessments) and nonclinician observers (ie, observer-reported outcome assessments). FDA guidance describing the regulatory expectations for all COA types including performance outcome assessments, which are based on the patient's performance of a defined task or activity, is in progress to meet requirements described within the 21st Century Cures Act and PDUFA VI. This communication highlights potential ways in which existing instruments might be modified or used "as is" to conform to good measurement principles. An industry and a regulatory perspective are described.

Symptoms of Major Depressive Disorder Scale: Performance of a Novel Patient-Reported Symptom Measure.

BACKGROUND: The Symptoms of Major Depressive Disorder Scale (SMDDS) was expressly developed on the basis of qualitative data to directly incorporate patients' voices into evaluation of treatment benefit in major depressive disorder (MDD) clinical trials. OBJECTIVES: To collect quantitative data necessary to refine/optimize the SMDDS and document its psychometric properties. METHODS: In this multicenter, observational study, participants with clinically diagnosed MDD completed questionnaires in 2 waves. Wave 1 was designed to refine the SMDDS using Rasch measurement evaluations and item reduction analyses. On a subset of wave 1 subjects, 7 to 12 months later, wave 2 further examined item performance and measurement properties. Exploratory factor analyses and assessments of construct validity and reliability (internal consistency and reproducibility) were completed. RESULTS: Using wave 1 data (N = 315; females = 71%, white = 81%, mean age = 44 years), the SMDDS was revised from 36 to 16 items. The Rasch item threshold map indicated that all but 1 item (suicidal ideation) were appropriately ordered. The 207 wave 2 participants were 74% females, 82% white, with a mean age of 45 years. The exploratory factor analyses resulted in a single component (all standardized factor loadings >0.46). Cronbach α was 0.93 and the 7-day test-retest intraclass correlation coefficient (n = 93) was 0.84 (95% confidence interval 0.77-0.89). SMDDS scores discriminated between MDD severity levels. CONCLUSIONS: The 16-item SMDDS generated highly reliable scores with substantial evidence of construct validity. On the basis of the evidence of appropriate content validity and sound psychometric performance, the Food and Drug Administration qualified the SMDDS as an outcome measure to support exploratory efficacy endpoints in MDD clinical trials.

Developing and Implementing Performance Outcome Assessments: Evidentiary, Methodologic, and Operational Considerations.

The use of performance outcome (PerfO) assessments to measure cognitive or physical function in drug trials presents several challenges for both sponsors and regulators, owing in part to a relative lack of scientific guidance on their development, implementation, and interpretation. In December 2016, the Duke-Margolis Center for Health Policy convened a 2-day workshop to explore the evidentiary, methodologic, and operational challenges associated with PerfO measures, and to identify potential paths to addressing these challenges. This paper presents both a summary of the discussion as well as additional input from a working group of experts from FDA, industry, academia, and public-private consortia. It is intended to advance the discussion around the development and use of PerfO measures to assess patient functioning in clinical trials intended to support registration of new treatments, and to highlight the key gaps in knowledge where additional research, collaboration, and discussion are needed.

Evaluation of the Measurement Properties of Four Performance Outcome Measures in Patients with Elective Hip Replacements, Elective Knee Replacements, or Hip Fractures.

OBJECTIVES: To evaluate the measurement properties of four performance outcome (PerfO) measures (timed up and go, four-step stair climb, long stair climb, and repeated chair stand) in three patient populations(elective total hip replacement [eTHR], elective total knee replacement [eTKR], and hip fracture [HF]). METHODS: A cross-sectional and longitudinal design was used to assess the PerfO measurement properties using the US Food and Drug Administration guidance for industry around patient-reported outcome measures to support labeling claims. Patient-reported outcome measures and patient- and clinician-reported global concept items were completed along with four PerfO measures at visit 1 and two follow-up visits. Measurement properties assessed included reliability, construct validity, ability to detect change, and estimates of meaningful change. RESULTS: A total of 280 patients (100 eTHR, 105 eTKR, and 75 HF) were recruited, with most (n = 276) providing data at visit 1. Most of the patients were female (64%) and retired (64%), and had at least one comorbidity (91%). Inter-rater and test-retest reliability ranged from good to excellent (0.73 ≤ intraclass correlation coefficient ≤ 0.95) for each PerfO measure. Known-groups validity was demonstrated for all PerfO measures, with those reporting less pain better physical functioning and those who did not use an assistive device having quicker mean completion times. Construct validity and ability to detect change were demonstrated and estimates of meaningful change derived. CONCLUSIONS: This study found the measurement properties of four PerfO measures in samples of patients with eTHR, eTKR, and HF to be supported for consideration of future use, and provided estimates for interpretation of change.

Evaluating the Content Validity of Four Performance Outcome Measures in Patients with Elective Hip Replacements and Hip Fractures.

OBJECTIVES: To assess the content validity of performance outcome (PerfO) measures for use with patients undergoing hip fracture (HF) surgery and elective total hip replacement (eTHR). METHODS: This study was a substudy of a broader evaluation of measurement properties of PerfO measures. The PerfO measures assessed were timed up and go (TUG), four-step stair climb (4SC), long stair climb (LSC), and repeated chair stand (RCS). For this substudy, HF and eTHR participants were interviewed to evaluate the relevance and difficulty of each PerfO measure. Qualitative analysis was conducted on interview transcripts, and summaries of coded data were produced to assess saturation. RESULTS: All 18 HF participants related the PerfO measures (TUG, 4SC, and RSC) to activities they completed in daily life, with slight variations in some specific aspects. For the eight eTHR participants, the correspondence between the PerfO measures (TUG, 4SC, and LSC) and activities in daily life varied: all participants saw similarity in the movements for the TUG; most undertook short stair climbs in daily life, but most did not regularly undertake LSC in daily life. Nevertheless, all HF and eTHR participants reported that the PerfO measures were relevant and had a level of difficulty similar to daily activities. CONCLUSIONS: This study contributes novel methods that adapt US regulatory guidance for patient-reported outcome measures to the evaluation of PerfO measures. A structured approach was used to explore specific details of each measure and correspondence to everyday life. This study demonstrates how content validity of PerfO measures can be meaningfully assessed.

Response scale selection in adult pain measures: results from a literature review.

BACKGROUND: The purpose of this literature review was to examine the existing patient-reported outcome measurement literature to understand the empirical evidence supporting response scale selection in pain measurement for the adult population. METHODS: The search strategy involved a comprehensive, structured, literature review with multiple search objectives and search terms. RESULTS: The searched yielded 6918 abstracts which were reviewed against study criteria for eligibility across the adult pain objective. The review included 42 review articles, consensus guidelines, expert opinion pieces, and primary research articles providing insights into optimal response scale selection for pain assessment in the adult population. Based on the extensive and varied literature on pain assessments, the adult pain studies typically use simple response scales with single-item measures of pain-a numeric rating scale, visual analog scale, or verbal rating scale. Across 42 review articles, consensus guidelines, expert opinion pieces, and primary research articles, the NRS response scale was most often recommended in these guidance documents. When reviewing the empirical basis for these recommendations, we found that the NRS had slightly superior measurement properties (e.g., reliability, validity, responsiveness) across a wide variety of contexts of use as compared to other response scales. CONCLUSIONS: Both empirical studies and review articles provide evidence that the 11-point NRS is likely the optimal response scale to evaluate pain among adult patients without cognitive impairment.

Patient-Centered Research to Support the Development of the Symptoms of Major Depressive Disorder Scale (SMDDS): Initial Qualitative Research.

BACKGROUND: Content valid, patient-reported outcome (PRO) measures of major depressive disorder (MDD) symptoms are needed to assess MDD treatment benefit. While a range of questionnaires are currently available to evaluate aspects of depression from the patient's perspective, their comprehensiveness and qualitative development histories are unclear. OBJECTIVE: The objective of this study was to describe the process and results of the preliminary qualitative development of a new symptom-based PRO measure intended to assess treatment benefit in MDD clinical trials. METHODS: Qualitative interviews were conducted with adult MDD patients in the USA who recently experienced a major depressive episode. Experienced interviewers conducted concept elicitation (CE) and cognitive interviews using semi-structured interview guides. The CE interview guide was used to elicit spontaneous reports of symptom experiences along with probing to further explore and confirm concepts. The cognitive interview guide was developed to evaluate concept relevance, understandability, and structure of the draft items, and to facilitate further instrument refinement. RESULTS: Forty patients participated in the CE interviews. A total of 3022 symptom codes, representing 84 different concepts were derived from the transcripts. Data from the CE interviews were considered alongside existing literature and clinical expert opinion during an item-generation process, leading to development of a preliminary version of the Symptoms of Major Depressive Disorder Scale (SMDDS). Fifteen patients participated in three waves of cognitive interviews, during which the SMDDS was further refined. CONCLUSIONS: The SMDDS is a 35-item PRO measure intended for use as an endpoint in MDD clinical trials to support medical product labeling. The SMDDS uses a 7-day recall period and verbal rating scales. It was developed in accordance with the US Food and Drug Administration (FDA)'s PRO Guidance and best practices. Qualitative interviews have provided evidence for content validity. Future quantitative studies will confirm the SMDDS's measurement properties and support FDA qualification.

Clinical and economic characteristics of total hip replacement patients with high health care costs and high health care use.

OBJECTIVE: The aims of this study were to confirm whether total hip arthroplasty (THA) patients with muscle atrophy/weakness (MAW) have high health care costs and resource use and to identify the characteristics that contribute to these high costs and use. DESIGN: This study analyzed claims from United States patients who underwent THA identified from commercial (n = 25,249) and Medicare (n = 22,472) insurance databases to compare demographics, health care costs, and resource use among patients with or without MAW. The patients were classified into three separate cohorts: pre-MAW (having MAW during the 12 mos before THA), post-MAW (having MAW during the 12 mos after THA, and no-MAW (no MAW claim). Characteristics of the THA patients associated with high health care costs were examined by multiple logistic regression, and subgroups of patients with high cost and high resource use were identified by classification and regression tree analyses. RESULTS: Health care use and costs were significantly higher for the THA patients with MAW, who had greater likelihood of inpatient and emergency department use and stays at skilled nursing facilities than the no-MAW patients. Classification and regression tree identified subgroups of high-cost patients as those with MAW having extended hospital stays and more outpatient visits. CONCLUSIONS: THA patients with MAW are at greater risk for high health care costs and resource consumption, including longer hospital stays, increased outpatient visits, and stays at skilled nursing and inpatient rehabilitation facilities.

Characteristics of hip fracture patients with and without muscle atrophy/weakness: predictors of negative economic outcomes.

OBJECTIVE: Hip fractures have negative humanistic and economic consequences. Predictors and sub-groups of negative post-fracture outcomes (high costs and extensive healthcare utilization) were identified in patients with and without muscle atrophy/weakness (MAW). METHODS: Truven Health MarketScan data identified patients ≥50 years old with inpatient hospitalizations for hip fracture. Patients had ≥12 months of continuous healthcare insurance prior to and following index hospitalization and no hip fracture diagnoses between 7 days and 1 year prior to admission. Predictors and sub-groups of negative outcomes were identified via multiple logistic regression analyses and classification and regression tree (CART) analyses, respectively. RESULTS: Post-fracture 1-year all-cause healthcare costs (USD$31,430) were higher than costs for the prior year ($18,091; p < 0.0001). Patients with MAW had greater post-fracture healthcare utilization and costs than those without MAW (p < 0.05). Greater post-fracture costs were associated with a higher number of prior hospitalizations and emergency room visits, length of index hospitalization, Charlson Comorbidity Index (CCI), and discharge status; diagnosis of rheumatoid arthritis, osteoarthritis, or osteoporosis; and prior use of antidepressants, anticonvulsants, muscle relaxants, benzodiazepines, opioids, and oral corticosteroids (all p < 0.009). High-cost patient sub-groups included those with MAW and high CCI scores. CONCLUSIONS: Negative post-fracture outcomes were associated with MAW vs no MAW, prior hospitalizations, comorbidities, and medications.