Statistical properties of items and summary scores from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE®) in a diverse cancer sample.

BACKGROUND/AIMS: The Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE®) was developed to capture symptomatic adverse events from the patient perspective. We aim to describe statistical properties of PRO-CTCAE items and summary scores and to provide evidence for recommendations regarding PRO-CTCAE administration and reporting. METHODS: Using data from the PRO-CTCAE validation study (NCT02158637), prevalence, means, and standard deviations of PRO-CTCAE items, composite scores, and mean and maximum scores across attributes (frequency, severity, and/or interference) of symptomatic adverse events were calculated. For each adverse event, correlations and agreement between attributes, correlations between attributes and composite scores, and correlations between composite, mean, and maximum scores were estimated. RESULTS: PRO-CTCAE items were completed by 899 patients with various cancer types. Most patients reported experiencing one or more adverse events, with the most prevalent being fatigue (87.7%), sad/unhappy feelings (66.0%), anxiety (63.6%), pain (63.2%), insomnia (61.8%), and dry mouth (60.0%). Attributes were moderately to strongly correlated within an adverse event (r = 0.53 to 0.77, all p < 0.001) but not fully concordant (κweighted = 0.26 to 0.60, all p < 0.001), with interference demonstrating lowest mean scores and prevalence among attributes of the same adverse event. Attributes were moderately to strongly correlated with composite scores (r = 0.67 to 0.97, all p < 0.001). Composite scores were moderately to strongly correlated with mean and maximum scores for the same adverse event (r = 0.69 to 0.94, all p < 0.001). Correlations between composite scores of different adverse events varied widely (r = 0.04 to 0.68) but were moderate to strong for conceptually related adverse events. CONCLUSIONS: Results provide evidence for PRO-CTCAE administration and reporting recommendations that the full complement of attributes be administered for each adverse event, and that attributes as well as summary scores be reported.

Optimization of alert notifications in electronic patient-reported outcome (ePRO) remote symptom monitoring systems (AFT-39).

PURPOSE: Clinical benefits result from electronic patient-reported outcome (ePRO) systems that enable remote symptom monitoring. Although clinically useful, real-time alert notifications for severe or worsening symptoms can overburden nurses. Thus, we aimed to algorithmically identify likely non-urgent alerts that could be suppressed. METHODS: We evaluated alerts from the PRO-TECT trial (Alliance AFT-39) in which oncology practices implemented remote symptom monitoring. Patients completed weekly at-home ePRO symptom surveys, and nurses received real-time alert notifications for severe or worsening symptoms. During parts of the trial, patients and nurses each indicated whether alerts were urgent or could wait until the next visit. We developed an algorithm for suppressing alerts based on patient assessment of urgency and model-based predictions of nurse assessment of urgency. RESULTS: 593 patients participated (median age = 64 years, 61% female, 80% white, 10% reported never using computers/tablets/smartphones). Patients completed 91% of expected weekly surveys. 34% of surveys generated an alert, and 59% of alerts prompted immediate nurse actions. Patients considered 10% of alerts urgent. Of the remaining cases, nurses considered alerts urgent more often when patients reported any worsening symptom compared to the prior week (33% of alerts with versus 26% without any worsening symptom, p = 0.009). The algorithm identified 38% of alerts as likely non-urgent that could be suppressed with acceptable discrimination (sensitivity = 80%, 95% CI [76%, 84%]; specificity = 52%, 95% CI [49%, 55%]). CONCLUSION: An algorithm can identify remote symptom monitoring alerts likely to be considered non-urgent by nurses, and may assist in fostering nurse acceptance and implementation feasibility of ePRO systems.

Financial Toxicity Monitoring in a Randomized Controlled Trial of Patient-Reported Outcomes During Cancer Treatment (Alliance AFT-39).

PURPOSE: Financial toxicity (FT) affects 20% of cancer survivors and is associated with poor clinical outcomes. No large-scale programs have been implemented to mitigate FT. We evaluated the effect of monthly FT screening as part of a larger patient-reported outcomes (PROs) digital monitoring intervention. METHODS: PRO-TECT (AFT-39) is a cluster-randomized trial of patients undergoing systemic therapy for metastatic cancer. Practices were randomly assigned 1:1 to digital symptom monitoring (PRO practices) or usual care (control practices). Digital monitoring consisted of between-visit online or automated telephone patient surveys about symptoms, functioning, and FT (single-item screening question from Functional Assessment of Chronic Illness Therapy-COmprehensive Score for financial Toxicity) for up to 1 year, with automated alerts sent to practice nurses for concerning survey scores. Clinical team actions in response to alerts were not mandated. The primary outcome of this planned secondary analysis was development or worsening of financial difficulties, assessed via the European Organisation for Research and Treatment of Cancer QLQ-C30 financial difficulties measure, at any time compared with baseline. A randomly selected subset of patients and nurses were interviewed about their experiences with the intervention. RESULTS: One thousand one hundred ninety-one patients were enrolled (593 PRO; 598 control) at 52 US community oncology practices. Overall, 30.2% of patients treated at practices that received the FT screening intervention developed, or experienced worsening of, financial difficulties, compared with 39.0% treated at control practices (P = .004). Patients and nurses interviewed stated that FT screening identified patients for financial counseling who otherwise would be reluctant to seek, or unaware of the availability of, assistance. CONCLUSION: In this report of a secondary outcome from a randomized clinical trial, FT screening as part of routine digital patient monitoring with PROs reduced the development, or worsening, of financial difficulties among patients undergoing systemic cancer therapy.

An Exploratory Analysis of the "Was It Worth It?" Questionnaire as a Novel Metric to Capture Patient Perceptions of Cancer Treatment.

OBJECTIVES: Asking "Was it worth it?" (WIWI) potentially captures the patient perception of a treatment's benefit weighed against its harms. This exploratory analysis evaluates the WIWI questionnaire as a metric of patients' perspectives on the worthwhileness of cancer treatment. METHODS: A 3-item WIWI questionnaire was assessed at end of treatment in patients with cancer on the COMET-2 trial (NCT01522443). WIWI items were evaluated to determine their association with quality of life (QOL), treatment duration, end-of-treatment reason, patient-reported adverse events (AEs), and disease response. RESULTS: A total of 65 patients completed the questionnaire; 40 (62%), 16 (25%), and 9 (14%) patients replied yes, uncertain, and no to "Was it worthwhile for you to receive the cancer treatment given in this study?" (item 1), respectively; 39 (60%), 12 (18%), and 14 (22%) to "If you had to do it over again, would you choose to have this cancer treatment?"; and 40 (62%), 14 (22%), and 11 (17%) to "Would you recommend this cancer treatment to others?" Patients responding yes to item 1 remained on treatment longer than those responding uncertain or no (mean 23.0 vs 11.3 weeks, P<.001). Patients responding uncertain/no to item 1 discontinued treatment because of AEs more frequently than those responding yes (36% vs 7.5%, P=.004) and demonstrated meaningful decline in QOL from baseline (-2.5 vs -0.2 mean change, P<.001). Associations between WIWI responses and most patient-reported AEs or treatment efficacy did not reach statistical significance. CONCLUSIONS: Patients who responded affirmatively on WIWI items remained on therapy longer, were less likely to stop treatment because of AEs, and demonstrated superior QOL. The WIWI may inform clinical practice, oncology research, and value frameworks.

Effect of Electronic Symptom Monitoring on Patient-Reported Outcomes Among Patients With Metastatic Cancer: A Randomized Clinical Trial.

Importance: Electronic systems that facilitate patient-reported outcome (PRO) surveys for patients with cancer may detect symptoms early and prompt clinicians to intervene. Objective: To evaluate whether electronic symptom monitoring during cancer treatment confers benefits on quality-of-life outcomes. Design, Setting, and Participants: Report of secondary outcomes from the PRO-TECT (Alliance AFT-39) cluster randomized trial in 52 US community oncology practices randomized to electronic symptom monitoring with PRO surveys or usual care. Between October 2017 and March 2020, 1191 adults being treated for metastatic cancer were enrolled, with last follow-up on May 17, 2021. Interventions: In the PRO group, participants (n = 593) were asked to complete weekly surveys via an internet-based or automated telephone system for up to 1 year. Severe or worsening symptoms triggered care team alerts. The control group (n = 598) received usual care. Main Outcomes and Measures: The 3 prespecified secondary outcomes were physical function, symptom control, and health-related quality of life (HRQOL) at 3 months, measured by the European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire (QLQ-C30; range, 0-100 points; minimum clinically important difference [MCID], 2-7 for physical function; no MCID defined for symptom control or HRQOL). Results on the primary outcome, overall survival, are not yet available. Results: Among 52 practices, 1191 patients were included (mean age, 62.2 years; 694 [58.3%] women); 1066 (89.5%) completed 3-month follow-up. Compared with usual care, mean changes on the QLQ-C30 from baseline to 3 months were significantly improved in the PRO group for physical function (PRO, from 74.27 to 75.81 points; control, from 73.54 to 72.61 points; mean difference, 2.47 [95% CI, 0.41-4.53]; P = .02), symptom control (PRO, from 77.67 to 80.03 points; control, from 76.75 to 76.55 points; mean difference, 2.56 [95% CI, 0.95-4.17]; P = .002), and HRQOL (PRO, from 78.11 to 80.03 points; control, from 77.00 to 76.50 points; mean difference, 2.43 [95% CI, 0.90-3.96]; P = .002). Patients in the PRO group had significantly greater odds of experiencing clinically meaningful benefits vs usual care for physical function (7.7% more with improvements of ≥5 points and 6.1% fewer with worsening of ≥5 points; odds ratio [OR], 1.35 [95% CI, 1.08-1.70]; P = .009), symptom control (8.6% and 7.5%, respectively; OR, 1.50 [95% CI, 1.15-1.95]; P = .003), and HRQOL (8.5% and 4.9%, respectively; OR, 1.41 [95% CI, 1.10-1.81]; P = .006). Conclusions and Relevance: In this report of secondary outcomes from a randomized clinical trial of adults receiving cancer treatment, use of weekly electronic PRO surveys to monitor symptoms, compared with usual care, resulted in statistically significant improvements in physical function, symptom control, and HRQOL at 3 months, with mean improvements of approximately 2.5 points on a 0- to 100-point scale. These findings should be interpreted provisionally pending results of the primary outcome of overall survival. Trial Registration: ClinicalTrials.gov Identifier: NCT03249090.

Missing data strategies for the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) in Alliance A091105 and COMET-2.

PURPOSE: Missing scores complicate analysis of the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) because patients with and without missing scores may systematically differ. We focus on optimal analysis methods for incomplete PRO-CTCAE items, with application to two randomized, double-blind, placebo-controlled, phase III trials. METHODS: In Alliance A091105 and COMET-2, patients completed PRO-CTCAE items before randomization and several times post-randomization (N = 64 and 107, respectively). For each trial, we conducted between-arm comparisons on the PRO-CTCAE via complete-case two-sample t-tests, mixed modeling with contrast, and multiple imputation followed by two-sample t-tests. Because interest lies in whether CTCAE grades can inform missing PRO-CTCAE scores, we performed multiple imputation with and without CTCAE grades as auxiliary variables to assess the added benefit of including them in the imputation model relative to only including PRO-CTCAE scores across all cycles. RESULTS: PRO-CTCAE completion rates ranged from 100.0 to 71.4% and 100.0 to 77.1% across time in A091105 and COMET-2, respectively. In both trials, mixed modeling and multiple imputation provided the most similar estimates of the average treatment effects. Including CTCAE grades in the imputation model did not consistently narrow confidence intervals of the average treatment effects because correlations for the same PRO-CTCAE item between different cycles were generally stronger than correlations between each PRO-CTCAE item and its corresponding CTCAE grade at the same cycle. CONCLUSION: For between-arm comparisons, mixed modeling and multiple imputation are informative techniques for handling missing PRO-CTCAE scores. CTCAE grades do not provide added benefit for informing missing PRO-CTCAE scores. CLINICALTRIALS: gov Identifiers: NCT02066181 (Alliance A091105); NCT01522443 (COMET-2).

Evaluating Treatment Tolerability Using the Toxicity Index With Patient-Reported Outcomes Data.

CONTEXT: Summarizing longitudinal symptomatic adverse events during clinical trials is necessary for understanding treatment tolerability. The Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) provides insight for capturing treatment tolerability within trials. Tolerability summary measures, such as the maximum score, are often used to communicate the potential negative symptoms both in the medical literature and directly to patients. Commonly, the proportions of present and severe symptomatic adverse events are used and reported between treatment arms among adverse event types. The toxicity index is also a summary measure previously applied to clinician-reported CTCAE data. OBJECTIVES: Apply the toxicity index to PRO-CTCAE data from the COMET-2 trial alongside the maximum score, then present and discuss considerations for using the toxicity index as a summary measure for communicating tolerability to patients and clinicians. METHODS: Proportions of maximum PRO-CTCAE severity levels and median toxicity index were computed by arm using all trial data and adjusting for baseline symptoms. RESULTS: Group-wise statistical differences were similar whether using severity level proportions or the toxicity index. The impact of adjusting for baseline symptoms was equivalently seen when comparing arms using severity rates or the toxicity index. CONCLUSION: The toxicity index is a useful method when ranking patients from those with the least to most symptomatic adverse event burden. This study showed the toxicity index can be applied to PRO-CTCAE data. Though as a tolerability summary measure, further study is needed to provide a clear clinical or patient-facing interpretation of the toxicity index.

The experience of financial toxicity among advanced melanoma patients treated with immunotherapy.

Purpose To measure financial toxicity and explore its association with quality of life (QOL) in an emerging population of survivors: advanced melanoma patients treated with immunotherapy. Design Cross-sectional survey and medical record review. Sample 106 survivors (39% response). Median time since start of immunotherapy was 36.4 months (range: 14.2-133.9). Methods The Comprehensive Score for Financial Toxicity measured financial toxicity, and the EORTC-QLQ30 assessed QOL and functioning across five domains. Data were collected online, by phone, or in clinic. Findings: Younger patients (<65 years) reported higher financial toxicity (p < .001) than older patients. Controlling for age, financial toxicity was correlated with QOL (p < .001), financial difficulties (p < .001), and EORTC-QLQ30 functioning subscales. Conclusions Given the demonstrated association between financial toxicity and QOL, our study highlights the importance of addressing financial toxicity, particularly among patients receiving high-cost treatments. Implications for Psychosocial Providers: Providers should educate patients and their caregivers about cost-management techniques, link them with available resources, and provide psychosocial counseling to alleviate related distress.

Composite grading algorithm for the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE).

BACKGROUND: The Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events is an item library designed for eliciting patient-reported adverse events in oncology. For each adverse event, up to three individual items are scored for frequency, severity, and interference with daily activities. To align the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events with other standardized tools for adverse event assessment including the Common Terminology Criteria for Adverse Events, an algorithm for mapping individual items for any given adverse event to a single composite numerical grade was developed and tested. METHODS: A five-step process was used: (1) All 179 possible Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events score combinations were presented to 20 clinical investigators to subjectively map combinations to single numerical grades ranging from 0 to 3. (2) Combinations with <75% agreement were presented to investigator committees at a National Clinical Trials Network cooperative group meeting to gain majority consensus via anonymous voting. (3) The resulting algorithm was refined via graphical and tabular approaches to assure directional consistency. (4) Validity, reliability, and sensitivity were assessed in a national study dataset. (5) Accuracy for delineating adverse events between study arms was measured in two Phase III clinical trials (NCT02066181 and NCT01522443). RESULTS: In Step 1, 12/179 score combinations had <75% initial agreement. In Step 2, majority consensus was reached for all combinations. In Step 3, five grades were adjusted to assure directional consistency. In Steps 4 and 5, composite grades performed well and comparably to individual item scores on validity, reliability, sensitivity, and between-arm delineation. CONCLUSION: A composite grading algorithm has been developed and yields single numerical grades for adverse events assessed via the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events, and can be useful in analyses and reporting.

Clinical Utility and User Perceptions of a Digital System for Electronic Patient-Reported Symptom Monitoring During Routine Cancer Care: Findings From the PRO-TECT Trial.

PURPOSE: There is increasing interest in implementing digital systems for remote monitoring of patients' symptoms during routine oncology practice. Information is limited about the clinical utility and user perceptions of these systems. METHODS: PRO-TECT is a multicenter trial evaluating implementation of electronic patient-reported outcomes (ePROs) among adults with advanced and metastatic cancers receiving treatment at US community oncology practices (ClinicalTrials.gov identifier: NCT03249090). Questions derived from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) are administered weekly by web or automated telephone system, with alerts to nurses for severe or worsening symptoms. To elicit user feedback, surveys were administered to participating patients and clinicians. RESULTS: Among 496 patients across 26 practices, the majority found the system and questions easy to understand (95%), easy to use (93%), and relevant to their care (91%). Most patients reported that PRO information was used by their clinicians for care (70%), improved discussions with clinicians (73%), made them feel more in control of their own care (77%), and would recommend the system to other patients (89%). Scores for most patient feedback questions were significantly positively correlated with weekly PRO completion rates in both univariate and multivariable analyses. Among 57 nurses, most reported that PRO information was helpful for clinical documentation (79%), increased efficiency of patient discussions (84%), and was useful for patient care (75%). Among 39 oncologists, most found PRO information useful (91%), with 65% using PROs to guide patient discussions sometimes or often and 65% using PROs to make treatment decisions sometimes or often. CONCLUSION: These findings support the clinical utility and value of implementing digital systems for monitoring PROs, including the PRO-CTCAE, in routine cancer care.

Quality of life in long-term survivors of advanced melanoma treated with checkpoint inhibitors.

BACKGROUND: Immune checkpoint inhibitors (CIs) have revolutionized treatment of advanced melanoma, leading to an emerging population of long-term survivors. Survivors' quality of life (QOL) and symptom burden are poorly understood. We set out to evaluate symptom burden and QOL in patients with advanced melanoma alive more than 1 year after initiating CI therapy. METHODS: Cross-sectional surveys, accompanied by chart review of patients with advanced melanoma treated with CIs at Memorial Sloan Kettering Cancer Center, completed therapy, and were alive >1 year after treatment initiation. Surveys were administered between February and August 2018. Surveys included: European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-Core 30, EuroQOL, items from Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events and Fatigue Severity Scale. RESULTS: We included 90 patients. The most common CI regimens were ipilimumab plus nivolumab (53%) and pembrolizumab (41%); most patients (71%) were not treated in clinical trials. Median time from CI therapy initiation was 40 months and from last dose was 28 months. Fatigue was reported by 28%, with higher fatigue scores in women than men; 12% reported difficulty sleeping. Aching joints (17%) and muscles (12%) were fairly common. Level of functioning was generally high. Overall QOL was excellent though 40% reported 'some or moderate' problems with anxiety/depression and 31% with pain/discomfort. CONCLUSIONS: After CI therapy, long-surviving advanced melanoma patients commonly report fatigue but otherwise have moderate symptom burden and good QOL. Ensuring appropriate symptom management will optimize clinical outcomes for these patients.

Assessment of Adverse Events From the Patient Perspective in a Phase 3 Metastatic Castration-Resistant Prostate Cancer Clinical Trial.

Importance: Standard adverse event (AE) reporting in oncology clinical trials has historically relied on clinician grading, which prior research has shown can lead to underestimation of rates of symptomatic AEs. Industry sponsors are beginning to implement in trials the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE), which was developed to allow patients to self-report symptomatic AEs and improve the quality of symptomatic AE detection. Objectives: To evaluate the feasibility of implementing PRO-CTCAE in a prespecified correlative analysis of the phase 3 COMET-2 trial and enumerate statistically significant between-group differences in symptomatic AEs using PRO-CTCAE and the CTCAE. Design, Setting, and Participants: This correlative study of 119 men in the randomized, double-blind, placebo-controlled phase 3 COMET-2 trial with metastatic castration-resistant prostate cancer who had undergone at least 2 prior lines of systemic treatment was conducted from March 2012 to July 2014. Participants completed PRO-CTCAE items using an automated telephone system from home prior to treatment and every 3 weeks during treatment. Statistical analysis was performed from May 2018 to June 2019. Main Outcomes and Measures: The proportion of patients who completed expected PRO-CTCAE self-reports was computed as a measure of feasibility. Results: Among the 119 men in the study (median age, 65 years [range, 44-80 years]), 534 of 587 (91.0%) expected PRO-CTCAE self-reports were completed, with consistently high rates of completion throughout participation. Rates of self-report adherence were similar between groups (cabozantinib s-maleate, 286 of 317 [90.2%]; and mitoxantrone hydrochloride-prednisone, 248 of 270 [91.9%]). Of 12 measured, patient-reported PRO-CTCAE symptomatic AEs, 4 reached statistical significance when comparing the proportion of patients with at least 1 postbaseline score greater than 0 between groups (differences ranged from 20.1% to 34.1% with higher proportions in the cabozantinib group; all P < .05), and use of a method for accounting for preexisting symptoms at baseline yielded 7 AEs with statistically significant differences between groups (differences ranged from 20.5% to 41.2% with higher proportions in the cabozantinib group; all P < .05). In the same analysis using investigator-reported CTCAE data, no statistically significant differences were found between groups for any symptomatic AEs. Conclusions and Relevance: PRO-CTCAE data collection was feasible and improved the accuracy of symptomatic AE detection in a phase 3 cancer trial. This analysis adds to mounting evidence of the feasibility and value of patient-reported AEs in oncology, which should be considered for inclusion in cancer trials that incorporate AE evaluation. Trial Registration: ClinicalTrials.gov identifier: NCT01522443.

Patient free text reporting of symptomatic adverse events in cancer clinical research using the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE).

OBJECTIVE: The study sought to describe patient-entered supplemental information on symptomatic adverse events (AEs) in cancer clinical research reported via a National Cancer Institute software system and examine the feasibility of mapping these entries to established terminologies. MATERIALS AND METHODS: Patients in 3 multicenter trials electronically completed surveys during cancer treatment. Each survey included a prespecified subset of items from the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Upon completion of the survey items, patients could add supplemental symptomatic AE information in a free text box. As patients typed into the box, structured dropdown terms could be selected from the PRO-CTCAE item library or Medical Dictionary for Regulatory Activities (MedDRA), or patients could type unstructured free text for submission. RESULTS: Data were pooled from 1760 participants (48% women; 78% White) who completed 8892 surveys, of which 2387 (26.8%) included supplemental symptomatic AE information. Overall, 1024 (58%) patients entered supplemental information at least once, with an average of 2.3 per patient per study. This encompassed 1474 of 8892 (16.6%) dropdowns and 913 of 8892 (10.3%) unstructured free text entries. One-third of the unstructured free text entries (32%) could be mapped post hoc to a PRO-CTCAE term and 68% to a MedDRA term. DISCUSSION: Participants frequently added supplemental information beyond study-specific survey items. Almost half selected a structured dropdown term, although many opted to submit unstructured free text entries. Most free text entries could be mapped post hoc to PRO-CTCAE or MedDRA terms, suggesting opportunities to enhance the system to perform real-time mapping for AE reporting. CONCLUSIONS: Patient reporting of symptomatic AEs using a text box functionality with mapping to existing terminologies is both feasible and informative.

Application of a Bayesian graded response model to characterize areas of disagreement between clinician and patient grading of symptomatic adverse events.

BACKGROUND: Traditional concordance metrics have shortcomings based on dataset characteristics (e.g., multiple attributes rated, missing data); therefore it is necessary to explore supplemental approaches to quantifying agreement between independent assessments. The purpose of this methodological paper is to apply an Item Response Theory (IRT) -based framework to an existing dataset that included unidimensional clinician and multiple attribute patient ratings of symptomatic adverse events (AEs), and explore the utility of this method in patient-reported outcome (PRO) and health-related quality of life (HRQOL) research. METHODS: Data were derived from a National Cancer Institute-sponsored study examining the validity of a measurement system (PRO-CTCAE) for patient self-reporting of AEs in cancer patients receiving treatment (N = 940). AEs included 13 multiple attribute patient-reported symptoms that had corresponding unidimensional clinician AE grades. A Bayesian IRT Model was fitted to calculate the latent grading thresholds between raters. The posterior mean values of the model-fitted item responses were calculated to represent model-based AE grades obtained from patients and clinicians. RESULTS: Model-based AE grades showed a general pattern of clinician underestimation relative to patient-graded AEs. However, the magnitude of clinician underestimation was associated with AE severity, such that clinicians' underestimation was more pronounced for moderate/very severe model-estimated AEs, and less so with mild AEs. CONCLUSIONS: The Bayesian IRT approach reconciles multiple symptom attributes and elaborates on the patterns of clinician-patient non-concordance beyond that provided by traditional metrics. This IRT-based technique may be used as a supplemental tool to detect and characterize nuanced differences in patient-, clinician-, and proxy-based ratings of HRQOL and patient-centered outcomes. TRIAL REGISTRATION: ClinicalTrials.gov NCT01031641 . Registered 1 December 2009.

Feasibility of Implementing the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events in a Multicenter Trial: NCCTG N1048.

Purpose The US National Cancer Institute (NCI) Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) was developed to enable patient reporting of symptomatic adverse events in oncology clinical research. This study was designed to assess the feasibility and resource requirements associated with implementing PRO-CTCAE in a multicenter trial. Methods Patients with locally advanced rectal cancer enrolled in the National Cancer Institute-sponsored North Central Cancer Treatment Group (Alliance) Preoperative Radiation or Selective Preoperative Radiation and Evaluation before Chemotherapy and Total Mesorectal Excision trial were asked to self-report 30 PRO-CTCAE items weekly from home during preoperative therapy, and every 6 months after surgery, via either the Web or an automated telephone system. If participants did not self-report within 3 days, a central coordinator called them to complete the items. Compliance was defined as the proportion of participants who completed PRO-CTCAE assessments at expected time points. Results The prespecified PRO-CTCAE analysis was conducted after the 500th patient completed the 6-month follow-up (median age, 56 years; 33% female; 12% nonwhite; 43% high school education or less; 5% Spanish speaking), across 165 sites. PRO-CTCAE was reported by participants at 4,491 of 4,882 expected preoperative time points (92.0% compliance), of which 3,771 (77.2%) were self-reported by participants and 720 (14.7%) were collected via central coordinator backup. Compliance at 6-month post-treatment follow-up was 333 of 468 (71.2%), with 122 (26.1%) via backup. Site research associates spent a median of 15 minutes on PRO-CTCAE work for each patient visit. Work by a central coordinator required a 50% time commitment. Conclusion Home-based reporting of PRO-CTCAE in a multicenter trial is feasible, with high patient compliance and low site administrative requirements. PRO-CTCAE data capture is improved through centralized backup calls.

Software for Administering the National Cancer Institute's Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events: Usability Study.

BACKGROUND: The US National Cancer Institute (NCI) developed software to gather symptomatic adverse events directly from patients participating in clinical trials. The software administers surveys to patients using items from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) through Web-based or automated telephone interfaces and facilitates the management of survey administration and the resultant data by professionals (clinicians and research associates). OBJECTIVE: The purpose of this study was to iteratively evaluate and improve the usability of the PRO-CTCAE software. METHODS: Heuristic evaluation of the software functionality was followed by semiscripted, think-aloud protocols in two consecutive rounds of usability testing among patients with cancer, clinicians, and research associates at 3 cancer centers. We conducted testing with patients both in clinics and at home (remotely) for both Web-based and telephone interfaces. Furthermore, we refined the software between rounds and retested. RESULTS: Heuristic evaluation identified deviations from the best practices across 10 standardized categories, which informed initial software improvement. Subsequently, we conducted user-based testing among 169 patients and 47 professionals. Software modifications between rounds addressed identified issues, including difficulty using radio buttons, absence of survey progress indicators, and login problems (for patients) as well as scheduling of patient surveys (for professionals). The initial System Usability Scale (SUS) score for the patient Web-based interface was 86 and 82 (P=.22) before and after modifications, respectively, whereas the task completion score was 4.47, which improved to 4.58 (P=.39) after modifications. Following modifications for professional users, the SUS scores improved from 71 to 75 (P=.47), and the mean task performance improved significantly (4.40 vs 4.02; P=.001). CONCLUSIONS: Software modifications, informed by rigorous assessment, rendered a usable system, which is currently used in multiple NCI-sponsored multicenter cancer clinical trials. TRIAL REGISTRATION: ClinicalTrials.gov NCT01031641; https://clinicaltrials.gov/ct2/show/NCT01031641 (Archived by WebCite at http://www.webcitation.org/708hTjlTl).

Feasibility of Patient Reporting of Symptomatic Adverse Events via the Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) in a Chemoradiotherapy Cooperative Group Multicenter Clinical Trial.

PURPOSE: To assess the feasibility of measuring symptomatic adverse events (AEs) in a multicenter clinical trial using the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). METHODS AND MATERIALS: Patients enrolled in NRG Oncology's RTOG 1012 (Prophylactic Manuka Honey for Reduction of Chemoradiation Induced Esophagitis-Related Pain during Treatment of Lung Cancer) were asked to self-report 53 PRO-CTCAE items representing 30 symptomatic AEs at 6 time points (baseline; weekly ×4 during treatment; 12 weeks after treatment). Reporting was conducted via wireless tablet computers in clinic waiting areas. Compliance was defined as the proportion of visits when an expected PRO-CTCAE assessment was completed. RESULTS: Among 226 study sites participating in RTOG 1012, 100% completed 35-minute PRO-CTCAE training for clinical research associates (CRAs); 80 sites enrolled patients, of which 34 (43%) required tablet computers to be provided. All 152 patients in RTOG 1012 agreed to self-report using the PRO-CTCAE (median age 66 years; 47% female; 84% white). Median time for CRAs to learn the system was 60 minutes (range, 30-240 minutes), and median time for CRAs to teach a patient to self-report was 10 minutes (range, 2-60 minutes). Compliance was high, particularly during active treatment, when patients self-reported at 86% of expected time points, although compliance was lower after treatment (72%). Common reasons for noncompliance were institutional errors, such as forgetting to provide computers to participants; patients missing clinic visits; Internet connectivity; and patients feeling "too sick." CONCLUSIONS: Most patients enrolled in a multicenter chemoradiotherapy trial were willing and able to self-report symptomatic AEs at visits using tablet computers. Minimal effort was required by local site staff to support this system. The observed causes of missing data may be obviated by allowing patients to self-report electronically between visits, and by using central compliance monitoring. These approaches are being incorporated into ongoing studies.

Evaluation of different recall periods for the US National Cancer Institute's PRO-CTCAE.

AIMS: The US National Cancer Institute recently developed the PRO-CTCAE (Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events). PRO-CTCAE is a library of questions for clinical trial participants to self-report symptomatic adverse events (e.g. nausea). The objective of this study is to inform evidence-based selection of a recall period when PRO-CTCAE is included in a trial. We evaluated differences between 1-, 2-, 3-, and 4-week recall periods, using daily reporting as the reference. METHODS: English-speaking patients with cancer receiving chemotherapy and/or radiotherapy were enrolled at four US cancer centers and affiliated community clinics. Participants completed 27 PRO-CTCAE items electronically daily for 28 days, and then weekly over 4 weeks, using 1-, 2-, 3-, and 4-week recall periods. For each recall period, mean differences, effect sizes, and intraclass correlation coefficients were calculated to evaluate agreement between the maximum of daily ratings and the corresponding ratings obtained using longer recall periods (e.g. maximum of daily scores over 7 days vs 1-week recall). Analyses were repeated using the average of daily scores within each recall period rather than the maximum of daily scores. RESULTS: A total of 127 subjects completed questionnaires (57% male; median age: 57). The median of the 27 mean differences in scores on the PRO-CTCAE 5-point response scale comparing the maximum daily versus the longer recall period (and corresponding effect size) was -0.20 (-0.20) for 1-week recall, -0.36 (-0.31) for 2-week recall, -0.45 (-0.39) for 3-week recall, and -0.47 (-0.40) for 4-week recall. The median intraclass correlation across 27 items between the maximum of daily ratings and the corresponding longer recall ratings for 1-week recall was 0.70 (range: 0.54-0.82), for 2-week recall was 0.74 (range: 0.58-0.83), for 3-week recall was 0.72 (range: 0.61-0.84), and for 4-week recall was 0.72 (range: 0.64-0.86). Similar results were observed for all analyses using the average of daily scores rather than the maximum of daily scores. CONCLUSION: A 1-week recall corresponds best to daily reporting. Although intraclass correlations remain stable over time, there are small but progressively larger differences between daily and longer recall periods at 2, 3, and 4 weeks, respectively. The preferred recall period for the PRO-CTCAE is the past 7 days, although investigators may opt for recall periods of 2, 3, or 4 weeks with an understanding that there may be some information loss.