Estimating meaningful thresholds for multi-item questionnaires using item response theory.

PURPOSE: Meaningful thresholds are needed to interpret patient-reported outcome measure (PROM) results. This paper introduces a new method, based on item response theory (IRT), to estimate such thresholds. The performance of the method is examined in simulated datasets and two real datasets, and compared with other methods. METHODS: The IRT method involves fitting an IRT model to the PROM items and an anchor item indicating the criterion state of interest. The difficulty parameter of the anchor item represents the meaningful threshold on the latent trait. The latent threshold is then linked to the corresponding expected PROM score. We simulated 4500 item response datasets to a 10-item PROM, and an anchor item. The datasets varied with respect to the mean and standard deviation of the latent trait, and the reliability of the anchor item. The real datasets consisted of a depression scale with a clinical depression diagnosis as anchor variable and a pain scale with a patient acceptable symptom state (PASS) question as anchor variable. RESULTS: The new IRT method recovered the true thresholds accurately across the simulated datasets. The other methods, except one, produced biased threshold estimates if the state prevalence was smaller or greater than 0.5. The adjusted predictive modeling method matched the new IRT method (also in the real datasets) but showed some residual bias if the prevalence was smaller than 0.3 or greater than 0.7. CONCLUSIONS: The new IRT method perfectly recovers meaningful (interpretational) thresholds for multi-item questionnaires, provided that the data satisfy the assumptions for IRT analysis.

How strong should my anchor be for estimating group and individual level meaningful change? A simulation study assessing anchor correlation strength and the impact of sample size, distribution of change scores and methodology on establishing a true meaningful change threshold.

PURPOSE: Treatment benefit as assessed using clinical outcome assessments (COAs), is a key endpoint in many clinical trials at both the individual and group level. Anchor-based methods can aid interpretation of COA change scores beyond statistical significance, and help derive a meaningful change threshold (MCT). However, evidence-based guidance on the selection of appropriately related anchors is lacking. METHODS: A simulation was conducted which varied sample size, change score variability and anchor correlation strength to assess the impact of these variables on recovering the simulated MCT for interpreting individual and group-level results. To assess MCTs derived at the individual-level (i.e. responder definitions; RDs), Receiver Operating Characteristic (ROC) curves and Predictive Modelling (PM) analyses were conducted. To assess MCTs for interpreting change at the group-level, the mean change method was conducted. RESULTS: Sample sizes, change score variability and magnitude of anchor correlation affected accuracy of the estimated MCT. For individual-level RDs, ROC curves were less accurate than PM methods at recovering the true MCT. For both methods, smaller samples led to higher variability in the returned MCT, but higher variability still using ROC. Anchors with weaker correlations with COA change scores had increased variability in the estimated MCT. An anchor correlation of around 0.50-0.60 identified a true MCT cut-point under certain conditions using ROC. However, anchor correlations as low as 0.30 were appropriate when using PM under certain conditions. For interpreting group-level results, the MCT derived using the mean change method was consistently underestimated regardless of the anchor correlation. CONCLUSION: Sample size and change score variability influence the necessary anchor correlation strength when recovering individual-level RDs. Often, this needs to be higher than the commonly accepted threshold of 0.30. Stronger correlations than 0.30 are required when using the mean change method. Results can assist researchers selecting and assessing the quality of anchors.

Establishing thresholds for meaningful within-individual change using longitudinal item response theory.

PURPOSE: Thresholds for meaningful within-individual change (MWIC) are useful for interpreting patient-reported outcome measures (PROM). Transition ratings (TR) have been recommended as anchors to establish MWIC. Traditional statistical methods for analyzing MWIC such as mean change analysis, receiver operating characteristic (ROC) analysis, and predictive modeling ignore problems of floor/ceiling effects and measurement error in the PROM scores and the TR item. We present a novel approach to MWIC estimation for multi-item scales using longitudinal item response theory (LIRT). METHODS: A Graded Response LIRT model for baseline and follow-up PROM data was expanded to include a TR item measuring latent change. The LIRT threshold parameter for the TR established the MWIC threshold on the latent metric, from which the observed PROM score MWIC threshold was estimated. We compared the LIRT approach and traditional methods using an example data set with baseline and three follow-up assessments differing by magnitude of score improvement, variance of score improvement, and baseline-follow-up score correlation. RESULTS: The LIRT model provided good fit to the data. LIRT estimates of observed PROM MWIC varied between 3 and 4 points score improvement. In contrast, results from traditional methods varied from 2 to 10 points-strongly associated with proportion of self-rated improvement. Best agreement between methods was seen when approximately 50% rated their health as improved. CONCLUSION: Results from traditional analyses of anchor-based MWIC are impacted by study conditions. LIRT constitutes a promising and more robust analytic approach to identifying thresholds for MWIC.

GOunder/Desmoid Tumor Research Foundation DEsmoid Symptom/Impact Scale (GODDESS©): psychometric properties and clinically meaningful thresholds as assessed in the Phase 3 DeFi randomized controlled clinical trial.

PURPOSE: The GODDESS© tool was developed to assess Desmoid Tumor/Aggressive Fibromatosis (DT/AF) symptom severity and impact on patients' lives. This study evaluated GODDESS©'s cross-sectional and longitudinal measurement properties. METHODS: The Phase 3, randomized placebo-controlled, DeFi study (NCT03785964) of nirogacestat in DT/AF was used to assess GODDESS©'s reliability, construct validity, responsiveness, and estimate of meaningful change thresholds (MCTs). Other patient-reported outcome (PRO) measures included Patient Global Impression of Severity (PGIS) in DT/AF symptoms, EORTC QLQ-C30, Brief Pain Inventory Short Form, and PROMIS Physical Function short-form 10a v2.0 plus 3 items. RESULTS: DeFi participants (N = 142) had a median age of 34 years (range: 18-76) and were mostly female (64.8%), with extra-abdominal (76.8%) or intra-abdominal tumors (23.2%). The GODDESS© symptom/impact scales showed internal consistency at baseline, cycles 4 and 7 (Cronbach's α > 0.70) and test-retest reliability (intra-class correlation coefficient > 0.85). GODDESS© scales correlated moderately to highly with PRO measures capturing similar content and differentiated among PGIS and Eastern Cooperative Oncology Group groups. GODDESS© scales detected improvement over time. For the total symptom score, a 1.30-point decrease was estimated as the within-person MCT and a 1.00-point decrease as the between-group MCT. For the physical functioning impact score, estimated within- and between-group MCTs were 0.60-point and 0.50-point decreases, respectively. Few participants exhibited symptom worsening. CONCLUSION: GODDESS© was found to be reliable, valid, responsive, and interpretable as a clinical trial endpoint in the pooled sample of DT/AF patients. Estimated MCTs can be used to define responders and assess group-level differences in future, unblinded, efficacy analyses. TRIAL REGISTRATION NUMBER AND REGISTRATION DATE: NCT03785964; December 24, 2018.

Validity of a single-item indicator of treatment side effect bother in a diverse sample of cancer patients.

PURPOSE: With higher efficacy of cancer therapies, the numbers and types of side effects experienced by patients have also increased, evidencing a need for brief assessments of side effect bother. The Functional Assessment of Cancer Therapy-General (FACT-G) includes the item "I am bothered by side effects of treatment" (GP5). This study aimed to confirm GP5's validity in a large, diverse, real-world patient sample. METHODS: Real-world data were drawn from 10 Adelphi Disease Specific Programmes (DSP™) conducted between 2015 and 2019 in France, Germany, Italy, Spain, the UK and the USA, covering 10 cancer sites. We examined correlations between GP5 responses and varied measures of patient-reported global health and the number of side effects experienced. We explored whether more advanced patients and those with worse Eastern Cooperative Oncology Group Performance Status Rating (ECOG PSR) reported greater side effect bother. Finally, we conducted differential item functioning (DIF) assessment using the Mantel-Haenszel approach. RESULTS: The sample included 6755 advanced cancer patients. GP5 responses were distributed similarly across most cancer sites. A moderate, negative correlation (rpolyserial = - 0.43) between GP5 responses and global health evidenced convergent validity. Known groups validity was evidenced by dichotomised distributions of GP5, showing expected results between cancer stage 2 vs. 3 and 4 and with ECOG PSR (p < 0.001). Little evidence of DIF was found. CONCLUSION: GP5 exhibited evidence of validity across cancer sites and countries and appeared to measure the same construct across these countries. GP5 has significant promise as a summary indicator of side effect bother.

How do the number of missing daily diary days impact the psychometric properties and meaningful change thresholds arising from a weekly average summary score?

PURPOSE: Quality of life research often collects daily information and averages this over a week, producing a summary score. When data are missing, arbitrary rules (such as requiring at least 4/7 observations) are used to determine whether a patient's summary score is created or set to missing. This simulation work aimed to assess the impact of missing data on the estimates produced by summary scores, the psychometric properties of the resulting summary score estimates and the impact on interpretation thresholds. METHODS: Complete longitudinal data were simulated for 1000 samples of 400 patients with different day-to-day variability. Data were deleted from these samples in line with missingness mechanisms to create scenarios with up to six days of missing data. Summary scores were created for complete and missing data scenarios. Summary score estimates, psychometric properties and meaningful change estimates were assessed for missing data scenarios compared to complete data. RESULTS: In most cases, the 4/7 day rule was supported, but this depended on daily variability. Fewer days of data were sometimes acceptable, but this was also dependent on the proportion of patients with missing data. Tables and figures allow researchers to assess the potential impact of missing data in their own studies. CONCLUSIONS: This work suggests that the missing data rule used to create summary scores impacts on the estimate, measurement properties and interpretation thresholds. Although a general rule of 4/7 days is supported, the way the summary score is derived does not have a uniform impact across psychometric analyses. Recommendations are to use the 4/7 rule, but plan for sensitivity analyses with other missing data rules.

Triangulation of multiple meaningful change thresholds for patient-reported outcome scores.

PURPOSE: The notion of what constitutes meaningful differences or changes in patient-reported outcome scores is represented by meaningful change thresholds (MCTs). Applying multiple methods to estimate MCTs inevitably results in a range of estimates; however, a single estimate or small range is sought in practice to enable consistent interpretation of scores. While current recommendations for triangulation are appropriate in principle, the vital step of moving from all estimates to a value or small range lacks clarity and is subjective in nature. This article aims to review current triangulation approaches and provide more robust recommendations than what is currently available. METHODS: Current approaches to perform triangulation are described and discussed. Anchor-based estimates are focussed upon due to their recognition as the most valid and developed approach. Recommendations for triangulation are provided. RESULTS: A correlation-weighted average of MCT estimates is recommended to triangulate multiple MCT estimates derived from a single study into a single value, where increased weighting is given to stronger anchor measures. The choice of method to triangulate estimates from several published studies is highly dependent on the availability of information within the publications. MCTs designed for between-group differences, within-group changes, and within-individual changes should be considered separately. CONCLUSION: The recommendations within this article provide a reliable and transparent approach to triangulation when a single value is sought, based on meta-analytic approaches. This approach is preferable to a simple mean of estimates where all are weighted equally, or through 'eyeballing' plotted estimates which is unreliable. We encourage researchers to adopt these methods, but to remain aware of the limitations within each method and further nuances in study design that result in heterogeneity. Sensitivity analyses with a range of plausible values are encouraged; however, the recommendations provide a suitable starting value for inferences. Unresolved issues in triangulation, requiring further exploration, are highlighted.

Minimal important change (MIC): a conceptual clarification and systematic review of MIC estimates of PROMIS measures.

We define the minimal important change (MIC) as a threshold for a minimal within-person change over time above which patients perceive themselves importantly changed. There is a lot of confusion about the concept of MIC, particularly about the concepts of minimal important change and minimal detectable change, which questions the validity of published MIC values. The aims of this study were: (1) to clarify the concept of MIC and how to use it; (2) to provide practical guidance for estimating methodologically sound MIC values; and (3) to improve the applicability of PROMIS by summarizing the available evidence on plausible PROMIS MIC values. We discuss the concept of MIC and how to use it and provide practical guidance for estimating MIC values. In addition, we performed a systematic review in PubMed on MIC values of any PROMIS measure from studies using recommended approaches. A total of 50 studies estimated the MIC of a PROMIS measure, of which 19 studies used less appropriate methods. MIC values of the remaining 31 studies ranged from 0.1 to 12.7 T-score points. We recommend to use the predictive modeling method, possibly supplemented with the vignette-based method, in future MIC studies. We consider a MIC value of 2-6 T-score points for PROMIS measures reasonable to assume at this point. For surgical interventions a higher MIC value might be appropriate. We recommend more high-quality studies estimating MIC values for PROMIS.

A powerful partnership: researchers and patients working together to develop a patient-facing summary of clinical trial outcome data.

OBJECTIVE: Availability of easy-to-understand patient-reported outcome (PRO) trial data may help individuals make more informed healthcare decisions. Easily interpretable, patient-centric PRO data summaries and visualizations are therefore needed. This three-stage study explored graphical format preferences, understanding, and interpretability of clinical trial PRO data presented to people with prostate cancer (PC). MATERIALS AND METHODS: A 7-day online survey exploring people with PC's preferences for different PRO data presentations (stage 1; n = 30) informed development of a draft plain-language resource sheet containing PRO data. After refining for clarity during cognitive debriefing interviews (stage 2; n = 18), the final resource sheet was circulated to people with PC for broader feedback (stage 3; n = 45). RESULTS: Although participants expressed preferences for certain graphical formats (pie charts and bar charts), preference did not always associate with interpretability and overall message clarity. Iterative development (stages 1 and 2) led to a final resource sheet, which 91.1% of participants in stage 3 considered useful and informative, and 88.9% expressed interest in receiving similar resources in the future. DISCUSSION: Findings demonstrate PRO data are relevant to people with PC and highlights that targeted resource sheets can support patient-clinician discussions. Appropriate graphical formatting and use of plain-language text is essential for conveying interpretable PRO data. Data visualization preferences are context dependent. CONCLUSION: Resource sheets summarizing clinical trial PRO data can be helpful for decision-making in PC. Researchers and patients can work together to develop clear, relevant, sensitive, and understandable resource sheets, which equally consider patient priorities as well as those of scientists.

From Meaningful Outcomes to Meaningful Change Thresholds: A Path to Progress for Establishing Digital Endpoints.

This paper examines the use of digital endpoints (DEs) derived from digital health technologies (DHTs), focusing primarily on the specific considerations regarding the determination of meaningful change thresholds (MCT). Using DHTs in drug development is becoming more commonplace. There is general acceptance of the value of DHTs supporting patient-centric trial design, capturing data outside the traditional clinical trial setting, and generating DEs with the potential to be more sensitive to change than conventional assessments. However, the transition from exploratory endpoints to primary and secondary endpoints capable of supporting labeling claims requires these endpoints to be substantive with reproducible population-specific values. Meaningful change represents the amount of change in an endpoint measure perceived as important to patients and should be determined for each digital endpoint and given population under consideration. This paper examines existing approaches to determine meaningful change thresholds and explores examples of these methodologies and their use as part of DE development: emphasizing the importance of determining what aspects of health are important to patients and ensuring the DE captures these concepts of interest and aligns with the overarching endpoint strategy. Examples are drawn from published DE qualification documentation and responses to qualification submissions under review by the various regulatory authorities. It is the hope that these insights will inform and strengthen the development and validation of DEs as drug development tools, particularly for those new to the approaches to determine MCTs.

Development and testing of an alternative responder definition for EULAR Sjögren's Syndrome Patient Reported Index (ESSPRI).

OBJECTIVES: Dryness, fatigue and joint/muscle pain are typically assessed in Sjögren's trials using European Alliance of Associations for Rheumatology Sjögren's Syndrome Patient Reported Index (ESSPRI). A Patient Acceptable Symptom State of <5 and a Minimal Clinically Important Improvement (MCII)/responder definition (RD) of ≥1 point or 15% on ESSPRI have previously been defined. This study explored alternative RDs to better discriminate between active treatment and placebo in trials. METHODS: Anchor-based and distribution-based methods were used to derive RD thresholds in blinded phase IIb trial data (N=190) and confirm these in blinded data pooled from three early phase II trials (N=126). The populations consisted of individuals with moderate-to-severe systemic primary Sjögren's. Anchors were prioritised by ESSPRI correlations and used in similar conditions. Triangulated estimates were discussed with experts (N=3). The revised RD was compared with the original using unblinded data to assess placebo and treatment responder rates. RESULTS: Patients were predominantly female (>90%), white (90%), with mean age of 50 years. Receiver operating characteristic estimates supported an MCII threshold of 1.5-1.6 in the phase II data, whereas correlation-weighted mean change estimates supported a low/minimal symptom severity threshold of ≥2. A low/minimal symptom severity of ≤3 showed the greatest sensitivity/specificity balance. Analyses in the pooled data supported these thresholds (MCII: 1.5-2.1; low/minimal symptom severity: 2.7-3.7). Unblinded analyses confirmed the revised RD reduced placebo rates. CONCLUSIONS: Completing a trial with an improvement of ≥1.5 points compared with baseline and an ESSPRI score of ≤3 points is a relevant RD for moderate-to-severe systemic Sjögren's and reduces placebo rates.

The Patient Matters in the End(point).

Digital health technologies such as wearable sensors are increasingly being used in clinical trials. However, the endpoints created from these useful tools are wide and varied. Often, digital health technologies such as wearable sensors are used either to collect a raw metric like "step count" or with artificial intelligence algorithms to define a biomarker for improvement. In the case of the former, improvements in such a raw metric is difficult to attribute to the patient health in a meaningful way. In the case of the latter, despite the potential predictive accuracies of machine learning and artificial intelligence approaches, the resulting biomarkers are a black box, which has limited direct interpretability to the patient's specific health concerns. The paper represents a call to arms to really place the patient at the heart of the endpoint. By designing trial endpoints which are measured by digital health technologies using a patient centered approach from the outset, the patient benefits from understanding the implications of approved medication for their life.

A confirmatory factor analysis approach was found to accurately estimate the reliability of transition ratings.

INTRODUCTION: Transition ratings (TRs) are single item measures which ask patients to report on their health change. They allow for a simple assessment of improvement or deterioration and are frequently used as an "anchor" to determine interpretation thresholds on a patient-reported outcome measure (PROM). Despite their widespread use, a routinely applicable method to assess their reliability is lacking. This paper introduces a method to estimate the reliability of TRs based on confirmatory factor analysis (CFA) for categorical data. METHOD: We modelled longitudinal PROM data as independent factors representing Time 1 and Time 2 in a CFA model. PROM items taken at Time 1 (T1) loaded on the first factor, although the same items taken at Time 2 (T2) loaded on the second. The TR item loaded onto both T1 and T2 factors. Three models with various constraints on the loadings and thresholds were examined. The communality (R2) statistic was used as a measure of the TR reliability. The approach was evaluated using simulated data and exemplified in four empirical datasets. RESULTS: The simplest CFA model without constraints on the item loadings and thresholds performed equivalently to models with constraints on loadings and thresholds over time. Further constraints on the TR item loadings to be equal and opposite over time caused biased TR reliability estimates if the T1 and T2 loadings differed in magnitude. In the four empirical datasets, reliability of TRs ranged from 0.27 to 0.48. In three examples the TR had numerically stronger loading on T2 than on T1. DISCUSSION AND CONCLUSIONS: Results support the use of the proposed method in understanding the reliability of TRs. Empirical study results reflect the typical range of reliability that has previously been reported for single items. Methodological considerations to improve TR reliability are presented, and developments of this method, are posited.

Reliability and Validity of an Instrument of COVID-19 Patient-Reported Symptoms in Outpatients.

Importance: Patient-reported outcome instruments are key in assessing COVID-19-related symptoms and associated burden. However, a valid and reliable instrument to assess symptom severity and progression among outpatients with COVID-19 is not yet available. Objectives: To assess the extent to which the Symptoms Evolution of COVID-19 (SE-C19) instrument is valid, reliable, and able to detect symptom changes in outpatients with COVID-19, as well as to establish a definition of symptom resolution. Design, Setting, and Participants: In this diagnostic/prognostic study, psychometric properties of SE-C19 were assessed in participants recruited into an ongoing, adaptive, phase 1/2/3, randomized, double-blind, placebo-controlled clinical trial, during 2020 to 2022. Adult outpatients with symptomatic COVID-19 were randomized 1:1:1 to receive 2.4 g or 8.0 g intravenous casirivimab and imdevimab or placebo, in outpatient centers at 114 sites, from 2 countries (US and Mexico). Main Outcomes and Measures: Reliability, validity, and sensitivity to change of the SE-C19 were assessed. SE-C19 and Patient Global Impression of Severity (PGIS) were administered daily from predose at day 1 to day 29. Results: Analysis was conducted on 657 adult outpatients (342 female patients [52.1%], 562 White patients [85.5%]), and 337 non-Hispanic patients [51.3%]. At baseline, patients reported a mean (SD) of 6.6 (3.9) symptoms (ie, rated as at least mild) with a mean (SD) of 3.8 (3.3) of these symptoms being rated as moderate or severe. Stable patients according to PGIS showed scores with intraclass correlation values indicating moderate-to-good test-retest reliability (ie, 0.50-0.90). At baseline, 20 item scores (87%) varied significantly across PGIS-defined groups, supporting the validity of the SE-C19. A symptom-resolution end point was defined after excluding the item sneezing due to its low ability to discriminate severity levels, and excluding confusion, rash, and vomiting, due to their low prevalence in this population. Symptom resolution required complete absence of all remaining items, except cough, fatigue, and headache, which could be mild or moderate in severity. A total of 19 of 23 items from the SE-C19 instrument were identified as valid and reliable to measure disease-related symptoms in outpatients with COVID-19. Conclusions and Relevance: This study identified 19 items that are valid and reliable to measure disease-related symptoms in outpatients with COVID-19, and proposed a definition of symptom resolution for potential use in future clinical trials.