Monitoring metrics over time: Why clinical trialists need to systematically collect site performance metrics.

Background: Over the last decade, there has been an increasing interest in risk-based monitoring (RBM) in clinical trials, resulting in a number of guidelines from regulators and its inclusion in ICH GCP. However, there is a lack of detail on how to approach RBM from a practical perspective, and insufficient understanding of best practice. Purpose: We present a method for clinical trials units to track their metrics within clinical trials using descriptive statistics and visualisations. Research Design: We suggest descriptive statistics and visualisations within a SWAT methodology. Study Sample: We illustrate this method using the metrics from TEMPER, a monitoring study carried out in three trials at the MRC Clinical Trials Unit at UCL. Data Collection: The data collection for TEMPER is described in DOI: 10.1177/1740774518793379. Results: We show the results and discuss a protocol for a Study-Within-A-Trial (SWAT 167) for those wishing to use the method. Conclusions: The potential benefits metric tracking brings to clinical trials include enhanced assessment of sites for potential corrective action, improved evaluation and contextualisation of the influence of metrics and their thresholds, and the establishment of best practice in RBM. The standardisation of the collection of such monitoring data would benefit both individual trials and the clinical trials community.

Site staff perspectives on communicating trial results to participants: Cost and feasibility results from the Show RESPECT cluster randomised, factorial, mixed-methods trial.

BACKGROUND/AIMS: Sharing trial results with participants is an ethical imperative but often does not happen. Show RESPECT (ISRCTN96189403) tested ways of sharing results with participants in an ovarian cancer trial (ISRCTN10356387). Sharing results via a printed summary improved patient satisfaction. Little is known about staff experience and the costs of communicating results with participants. We report the costs of communication approaches used in Show RESPECT and the views of site staff on these approaches. METHODS: We allocated 43 hospitals (sites) to share results with trial participants through one of eight intervention combinations (2 × 2 × 2 factorial; enhanced versus basic webpage, printed summary versus no printed summary, email list invitation versus no invitation). Questionnaires elicited data from staff involved in sharing results. Open- and closed-ended questions covered resources used to share results and site staff perspectives on the approaches used. Semi-structured interviews were conducted. Interview and free-text data were analysed thematically. The mean additional site costs per participant from each intervention were estimated jointly as main effects by linear regression. RESULTS: We received questionnaires from 68 staff from 41 sites and interviewed 11 site staff. Sites allocated to the printed summary had mean total site costs of sharing results £13.71/patient higher (95% confidence interval (CI): -3.19, 30.60; p = 0.108) than sites allocated no printed summary. Sites allocated to the enhanced webpage had mean total site costs £1.91/patient higher (95% CI: -14, 18.74; p = 0.819) than sites allocated to the basic webpage. Sites allocated to the email list had costs £2.87/patient lower (95% CI: -19.70, 13.95; p = 0.731) than sites allocated to no email list. Most of these costs were staff time for mailing information and handling patients' queries. Most site staff reported no concerns about how they had shared results (88%) and no challenges (76%). Most (83%) found it easy to answer queries from patients about the results and thought the way they were allocated to share results with participants would be an acceptable standard approach (76%), with 79% saying they would follow the same approach for future trials. There were no significant effects of the randomised interventions on these outcomes. Site staff emphasised the importance of preparing patients to receive the results, including giving opt-in/opt-out options, and the need to offer further support, particularly if the results could confuse or distress some patients. CONCLUSIONS: Adding a printed summary to a webpage (which significantly improved participant satisfaction) may increase costs to sites by ~£14/patient, which is modest in relation to the cost of trials. The Show RESPECT communication interventions were feasible to implement. This information could help future trials ensure they have sufficient resources to share results with participants.

Abiraterone acetate plus prednisolone with or without enzalutamide for patients with metastatic prostate cancer starting androgen deprivation therapy: final results from two randomised phase 3 trials of the STAMPEDE platform protocol.

BACKGROUND: Abiraterone acetate plus prednisolone (herein referred to as abiraterone) or enzalutamide added at the start of androgen deprivation therapy improves outcomes for patients with metastatic prostate cancer. Here, we aimed to evaluate long-term outcomes and test whether combining enzalutamide with abiraterone and androgen deprivation therapy improves survival. METHODS: We analysed two open-label, randomised, controlled, phase 3 trials of the STAMPEDE platform protocol, with no overlapping controls, conducted at 117 sites in the UK and Switzerland. Eligible patients (no age restriction) had metastatic, histologically-confirmed prostate adenocarcinoma; a WHO performance status of 0-2; and adequate haematological, renal, and liver function. Patients were randomly assigned (1:1) using a computerised algorithm and a minimisation technique to either standard of care (androgen deprivation therapy; docetaxel 75 mg/m2 intravenously for six cycles with prednisolone 10 mg orally once per day allowed from Dec 17, 2015) or standard of care plus abiraterone acetate 1000 mg and prednisolone 5 mg (in the abiraterone trial) orally or abiraterone acetate and prednisolone plus enzalutamide 160 mg orally once a day (in the abiraterone and enzalutamide trial). Patients were stratified by centre, age, WHO performance status, type of androgen deprivation therapy, use of aspirin or non-steroidal anti-inflammatory drugs, pelvic nodal status, planned radiotherapy, and planned docetaxel use. The primary outcome was overall survival assessed in the intention-to-treat population. Safety was assessed in all patients who started treatment. A fixed-effects meta-analysis of individual patient data was used to compare differences in survival between the two trials. STAMPEDE is registered with ClinicalTrials.gov (NCT00268476) and ISRCTN (ISRCTN78818544). FINDINGS: Between Nov 15, 2011, and Jan 17, 2014, 1003 patients were randomly assigned to standard of care (n=502) or standard of care plus abiraterone (n=501) in the abiraterone trial. Between July 29, 2014, and March 31, 2016, 916 patients were randomly assigned to standard of care (n=454) or standard of care plus abiraterone and enzalutamide (n=462) in the abiraterone and enzalutamide trial. Median follow-up was 96 months (IQR 86-107) in the abiraterone trial and 72 months (61-74) in the abiraterone and enzalutamide trial. In the abiraterone trial, median overall survival was 76·6 months (95% CI 67·8-86·9) in the abiraterone group versus 45·7 months (41·6-52·0) in the standard of care group (hazard ratio [HR] 0·62 [95% CI 0·53-0·73]; p<0·0001). In the abiraterone and enzalutamide trial, median overall survival was 73·1 months (61·9-81·3) in the abiraterone and enzalutamide group versus 51·8 months (45·3-59·0) in the standard of care group (HR 0·65 [0·55-0·77]; p<0·0001). We found no difference in the treatment effect between these two trials (interaction HR 1·05 [0·83-1·32]; pinteraction=0·71) or between-trial heterogeneity (I2 p=0·70). In the first 5 years of treatment, grade 3-5 toxic effects were higher when abiraterone was added to standard of care (271 [54%] of 498 vs 192 [38%] of 502 with standard of care) and the highest toxic effects were seen when abiraterone and enzalutamide were added to standard of care (302 [68%] of 445 vs 204 [45%] of 454 with standard of care). Cardiac causes were the most common cause of death due to adverse events (five [1%] with standard of care plus abiraterone and enzalutamide [two attributed to treatment] and one (<1%) with standard of care in the abiraterone trial). INTERPRETATION: Enzalutamide and abiraterone should not be combined for patients with prostate cancer starting long-term androgen deprivation therapy. Clinically important improvements in survival from addition of abiraterone to androgen deprivation therapy are maintained for longer than 7 years. FUNDING: Cancer Research UK, UK Medical Research Council, Swiss Group for Clinical Cancer Research, Janssen, and Astellas.

Lack of transparent reporting of trial monitoring approaches in randomised controlled trials: A systematic review of contemporary protocol papers.

BACKGROUND: Monitoring is essential to ensure patient safety and data integrity in clinical trials as per Good Clinical Practice. The Standard Protocol Items: Recommendations for Interventional Trials Statement and its checklist guides authors to include monitoring in their protocols. We investigated how well monitoring was reported in published 'protocol papers' for contemporary randomised controlled trials. METHODS: A systematic search was conducted in PubMed to identify eligible protocol papers published in selected journals between 1 January 2020 and 31 May 2020. Protocol papers were classified by whether they reported monitoring and, if so, by the details of monitoring. Data were summarised descriptively. RESULTS: Of 811 protocol papers for randomised controlled trials, 386 (48%; 95% CI: 44%-51%) explicitly reported some monitoring information. Of these, 20% (77/386) reported monitoring information consistent with an on-site monitoring approach, and 39% (152/386) with central monitoring, 26% (101/386) with a mixed approach, while 14% (54/386) did not provide sufficient information to specify an approach. Only 8% (30/386) of randomised controlled trials reported complete details about all of scope, frequency and organisation of monitoring; frequency of monitoring was the least reported. However, 6% (25/386) of papers used the term 'audit' to describe 'monitoring'. DISCUSSION: Monitoring information was reported in only approximately half of the protocol papers. Suboptimal reporting of monitoring hinders the clinical community from having the full information on which to judge the validity of a trial and jeopardises the value of protocol papers and the credibility of the trial itself. Greater efforts are needed to promote the transparent reporting of monitoring to journal editors and authors.

Abiraterone acetate plus prednisolone for metastatic patients starting hormone therapy: 5-year follow-up results from the STAMPEDE randomised trial (NCT00268476).

Abiraterone acetate plus prednisolone (AAP) previously demonstrated improved survival in STAMPEDE, a multiarm, multistage platform trial in men starting long-term hormone therapy for prostate cancer. This long-term analysis in metastatic patients was planned for 3 years after the first results. Standard-of-care (SOC) was androgen deprivation therapy. The comparison randomised patients 1:1 to SOC-alone with or without daily abiraterone acetate 1000 mg + prednisolone 5 mg (SOC + AAP), continued until disease progression. The primary outcome measure was overall survival. Metastatic disease risk group was classified retrospectively using baseline CT and bone scans by central radiological review and pathology reports. Analyses used Cox proportional hazards and flexible parametric models, accounting for baseline stratification factors. One thousand and three patients were contemporaneously randomised (November 2011 to January 2014): median age 67 years; 94% newly-diagnosed; metastatic disease risk group: 48% high, 44% low, 8% unassessable; median PSA 97 ng/mL. At 6.1 years median follow-up, 329 SOC-alone deaths (118 low-risk, 178 high-risk) and 244 SOC + AAP deaths (75 low-risk, 145 high-risk) were reported. Adjusted HR = 0.60 (95% CI: 0.50-0.71; P = 0.31 × 10-9 ) favoured SOC + AAP, with 5-years survival improved from 41% SOC-alone to 60% SOC + AAP. This was similar in low-risk (HR = 0.55; 95% CI: 0.41-0.76) and high-risk (HR = 0.54; 95% CI: 0.43-0.69) patients. Median and current maximum time on SOC + AAP was 2.4 and 8.1 years. Toxicity at 4 years postrandomisation was similar, with 16% patients in each group reporting grade 3 or higher toxicity. A sustained and substantial improvement in overall survival of all metastatic prostate cancer patients was achieved with SOC + abiraterone acetate + prednisolone, irrespective of metastatic disease risk group.

Testing approaches to sharing trial results with participants: The Show RESPECT cluster randomised, factorial, mixed methods trial.

BACKGROUND: Sharing trial results with participants is an ethical imperative but often does not happen. We tested an Enhanced Webpage versus a Basic Webpage, Mailed Printed Summary versus no Mailed Printed Summary, and Email List Invitation versus no Email List Invitation to see which approach resulted in the highest patient satisfaction with how the results were communicated. METHODS AND FINDINGS: We carried out a cluster randomised, 2 by 2 by 2 factorial, nonblinded study within a trial, with semistructured qualitative interviews with some patients (ISRCTN96189403). Each cluster was a UK hospital participating in the ICON8 ovarian cancer trial. Interventions were shared with 384 ICON8 participants who were alive and considered well enough to be contacted, at 43 hospitals. Hospitals were allocated to share results with participants through one of the 8 intervention combinations based on random permutation within blocks of 8, stratified by number of participants. All interventions contained a written plain English summary of the results. The Enhanced Webpage also contained a short video. Both the Enhanced Webpage and Email contained links to further information and support. The Mailed Printed Summary was opt-out. Follow-up questionnaires were sent 1 month after patients had been offered the interventions. Patients' reported satisfaction was measured using a 5-point scale, analysed by ordinal logistic regression estimating main effects for all 3 interventions, with random effects for site, restricted to those who reported receiving the results and assuming no interaction. Data collection took place in 2018 to 2019. Questionnaires were sent to 275/384 randomly selected participants and returned by 180: 90/142 allocated Basic Webpage, 90/133 Enhanced Webpage; 91/141 no Mailed Printed Summary, 89/134 Mailed Printed Summary; 82/129 no Email List Invitation, 98/146 Email List Invitation. Only 3 patients opted out of receiving the Mailed Printed Summary; no patients signed up to the email list. Patients' satisfaction was greater at sites allocated the Mailed Printed Summary, where 65/81 (80%) were quite or very satisfied compared to sites with no Mailed Printed Summary 39/64 (61%), ordinal odds ratio (OR) = 3.15 (1.66 to 5.98, p < 0.001). We found no effect on patient satisfaction from the Enhanced Webpage, OR = 1.47 (0.78 to 2.76, p = 0.235) or Email List Invitation, OR = 1.38 (0.72 to 2.63, p = 0.327). Interviewees described the results as interesting, important, and disappointing (the ICON8 trial found no benefit). Finding out the results made some feel their trial participation had been more worthwhile. Regardless of allocated group, patients who received results generally reported that the information was easy to understand and find, were glad and did not regret finding out the results. The main limitation of our study is the 65% response rate. CONCLUSIONS: Nearly all respondents wanted to know the results and were glad to receive them. Adding an opt-out Mailed Printed Summary alongside a webpage yielded the highest reported satisfaction. This study provides evidence on how to share results with other similar trial populations. Further research is needed to look at different results scenarios and patient populations. TRIAL REGISTRATION: ISRCTN: ISRCTN96189403.

Access to routinely collected health data for clinical trials - review of successful data requests to UK registries.

BACKGROUND: Clinical trials generally each collect their own data despite routinely collected health data (RCHD) increasing in quality and breadth. Our aim is to quantify UK-based randomised controlled trials (RCTs) accessing RCHD for participant data, characterise how these data are used and thereby recommend how more trials could use RCHD. METHODS: We conducted a systematic review of RCTs accessing RCHD from at least one registry in the UK between 2013 and 2018 for the purposes of informing or supplementing participant data. A list of all registries holding RCHD in the UK was compiled. In cases where registries published release registers, these were searched for RCTs accessing RCHD. Where no release register was available, registries were contacted to request a list of RCTs. For each identified RCT, information was collected from all publicly available sources (release registers, websites, protocol etc.). The search and data extraction were undertaken between January and May 2019. RESULTS: We identified 160 RCTs accessing RCHD between 2013 and 2018 from a total of 22 registries; this corresponds to only a very small proportion of all UK RCTs (about 3%). RCTs accessing RCHD were generally large (median sample size 1590), commonly evaluating treatments for cancer or cardiovascular disease. Most of the included RCTs accessed RCHD from NHS Digital (68%), and the most frequently accessed datasets were mortality (76%) and hospital visits (55%). RCHD was used to inform the primary trial (82%) and long-term follow-up (57%). There was substantial variation in how RCTs used RCHD to inform participant outcome measures. A limitation was the lack of information and transparency from registries and RCTs with respect to which datasets have been accessed and for what purposes. CONCLUSIONS: In the last five years, only a small minority of UK-based RCTs have accessed RCHD to inform participant data. We ask for improved accessibility, confirmed data quality and joined-up thinking between the registries and the regulatory authorities. TRIAL REGISTRATION: PROSPERO CRD42019123088.

Changing platforms without stopping the train: experiences of data management and data management systems when adapting platform protocols by adding and closing comparisons.

BACKGROUND: There is limited research and literature on the data management challenges encountered in multi-arm, multi-stage platform and umbrella protocols. These trial designs allow both (1) seamless addition of new research comparisons and (2) early stopping of accrual to individual comparisons that do not show sufficient activity. FOCUS4 (colorectal cancer) and STAMPEDE (prostate cancer), run from the Medical Research Council Clinical Trials Unit (CTU) at UCL, are two leading UK examples of clinical trials implementing adaptive platform protocol designs. To date, STAMPEDE has added five new research comparisons, closed two research comparisons following pre-planned interim analysis (lack of benefit), adapted the control arm following results from STAMPEDE and other relevant trials, and completed recruitment to six research comparisons. FOCUS4 has closed one research comparison following pre-planned interim analysis (lack of benefit) and added one new research comparison, with a number of further comparisons in the pipeline. We share our experiences from the operational aspects of running these adaptive trials, focusing on data management. METHODS: We held discussion groups with STAMPEDE and FOCUS4 CTU data management staff to identify data management challenges specific to adaptive platform protocols. We collated data on a number of case report form (CRF) changes, database amendments and database growth since each trial began. DISCUSSION: We found similar adaptive protocol-specific challenges in both trials. Adding comparisons to and removing them from open trials provides extra layers of complexity to CRF and database development. At the start of an adaptive trial, CRFs and databases must be designed to be flexible and scalable in order to cope with the continuous changes, ensuring future data requirements are considered where possible. When adding or stopping a comparison, the challenge is to incorporate new data requirements while ensuring data collection within ongoing comparisons is unaffected. Some changes may apply to all comparisons; others may be comparison-specific or applicable only to patients recruited during a specific time period. We discuss the advantages and disadvantages of the different approaches to CRF and database design we implemented in these trials, particularly in relation to use and maintenance of generic versus comparison-specific CRFs and databases. The work required to add or remove a comparison, including the development and testing of changes, updating of documentation, and training of sites, must be undertaken alongside data management of ongoing comparisons. Adequate resource is required for these competing data management tasks, especially in trials with long follow-up. A plan is needed for regular and pre-analysis data cleaning for multiple comparisons that could recruit at different rates and periods of time. Data-cleaning activities may need to be split and prioritised, especially if analyses for different comparisons overlap in time. CONCLUSIONS: Adaptive trials offer an efficient model to run randomised controlled trials, but setting up and conducting the data management activities in these trials can be operationally challenging. Trialists and funders must plan for scalability in data collection and the resource required to cope with additional competing data management tasks.

Early warnings and repayment plans: novel trial management methods for monitoring and managing data return rates in a multi-centre phase III randomised controlled trial with paper Case Report Forms.

BACKGROUND: Monitoring and managing data returns in multi-centre randomised controlled trials is an important aspect of trial management. Maintaining consistently high data return rates has various benefits for trials, including enhancing oversight, improving reliability of central monitoring techniques and helping prepare for database lock and trial analyses. Despite this, there is little evidence to support best practice, and current standard methods may not be optimal. METHODS: We report novel methods from the Trial of Imaging and Schedule in Seminoma Testis (TRISST), a UK-based, multi-centre, phase III trial using paper Case Report Forms to collect data over a 6-year follow-up period for 669 patients. Using an automated database report which summarises the data return rate overall and per centre, we developed a Microsoft Excel-based tool to allow observation of per-centre trends in data return rate over time. The tool allowed us to distinguish between forms that can and cannot be completed retrospectively, to inform understanding of issues at individual centres. We reviewed these statistics at regular trials unit team meetings. We notified centres whose data return rate appeared to be falling, even if they had not yet crossed the pre-defined acceptability threshold of an 80% data return rate. We developed a set method for agreeing targets for gradual improvement with centres having persistent data return problems. We formalised a detailed escalation policy to manage centres who failed to meet agreed targets. We conducted a post-hoc, descriptive analysis of the effectiveness of the new processes. RESULTS: The new processes were used from April 2015 to September 2016. By May 2016, data return rates were higher than they had been at any time previously, and there were no centres with return rates below 80%, which had never been the case before. In total, 10 centres out of 35 were contacted regarding falling data return rates. Six out of these 10 showed improved rates within 6-8 weeks, and the remainder within 4 months. CONCLUSIONS: Our results constitute preliminary effectiveness evidence for novel methods in monitoring and managing data return rates in randomised controlled trials. We encourage other researchers to work on generating better evidence-based methods in this area, whether through more robust evaluation of our methods or of others.

Implementing monitoring triggers and matching of triggered and control sites in the TEMPER study: a description and evaluation of a triggered monitoring management system.

BACKGROUND: Triggered monitoring in clinical trials is a risk-based monitoring approach where triggers (centrally monitored, predefined key risk and performance indicators) drive the extent, timing, and frequency of monitoring visits. The TEMPER study used a prospective, matched-pair design to evaluate the use of a triggered monitoring strategy, comparing findings from triggered monitoring visits with those from matched control sites. To facilitate this study, we developed a bespoke risk-based monitoring system: the TEMPER Management System. METHODS: The TEMPER Management System comprises a web application (the front end), an SQL server database (the back end) to store the data generated for TEMPER, and a reporting function to aid users in study processes such as the selection of triggered sites. Triggers based on current practice were specified for three clinical trials and were implemented in the system. Trigger data were generated in the system using data extracted from the trial databases to inform the selection of triggered sites to visit. Matching of the chosen triggered sites with untriggered control sites was also performed in the system, while data entry screens facilitated the collection and management of the data from findings gathered at monitoring visits. RESULTS: There were 38 triggers specified for the participating trials. Using these, 42 triggered sites were chosen and matched with control sites. Monitoring visits were carried out to all sites, and visit findings were entered into the TEMPER Management System. Finally, data extracted from the system were used for analysis. CONCLUSIONS: The TEMPER Management System made possible the completion of the TEMPER study. It implemented an approach of standardising the automation of current-practice triggers, and the generation of trigger data to inform the selection of triggered sites to visit. It also implemented a matching algorithm informing the selection of matched control sites. We hope that by publishing this paper it encourages other trialists to share their approaches to, and experiences of, triggered monitoring and other risk-based monitoring systems.

Triggered or routine site monitoring visits for randomised controlled trials: results of TEMPER, a prospective, matched-pair study.

BACKGROUND/AIMS: In multi-site clinical trials, where trial data and conduct are scrutinised centrally with pre-specified triggers for visits to sites, targeted monitoring may be an efficient way to prioritise on-site monitoring. This approach is widely used in academic trials, but has never been formally evaluated. METHODS: TEMPER assessed the ability of targeted monitoring, as used in three ongoing phase III randomised multi-site oncology trials, to distinguish sites at which higher and lower rates of protocol and/or Good Clinical Practice violations would be found during site visits. Using a prospective, matched-pair design, sites that had been prioritised for visits after having activated 'triggers' were matched with a control ('untriggered') site, which would not usually have been visited at that time. The paired sites were visited within 4 weeks of each other, and visit findings are recorded and categorised according to the seriousness of the deviation. The primary outcome measure was the proportion of sites with ≥1 'Major' or 'Critical' finding not previously identified centrally. The study was powered to detect an absolute difference of ≥30% between triggered and untriggered visits. A sensitivity analysis, recommended by the study's blinded endpoint review committee, excluded findings related to re-consent. Additional analyses assessed the prognostic value of individual triggers and data from pre-visit questionnaires completed by site and trials unit staff. RESULTS: In total, 42 matched pairs of visits took place between 2013 and 2016. In the primary analysis, 88.1% of triggered visits had ≥1 new Major/Critical finding, compared to 81.0% of untriggered visits, an absolute difference of 7.1% (95% confidence interval -8.3%, +22.5%; p = 0.365). When re-consent findings were excluded, these figures reduced to 85.7% versus 59.5%, (difference = 26.2%, 95% confidence interval 8.0%, 44.4%; p = 0.007). Individual triggers had modest prognostic value but knowledge of the trial-related activities carried out by site staff may be useful. CONCLUSION: Triggered monitoring approaches, as used in these trials, were not sufficiently discriminatory. The rate of Major and Critical findings was higher than anticipated, but the majority related to consent and re-consent with no indication of systemic problems that would impact trial-wide safety issues or integrity of the results in any of the three trials. Sensitivity analyses suggest triggered monitoring may be of potential use, but needs improvement and investigation of further central monitoring triggers is warranted. TEMPER highlights the need to question and evaluate methods in trial conduct, and should inform further developments in this area.