Getting our ducks in a row: The need for data utility comparisons of healthcare systems data for clinical trials.

BACKGROUND: Better use of healthcare systems data, collected as part of interactions between patients and the healthcare system, could transform planning and conduct of randomised controlled trials. Multiple challenges to widespread use include whether healthcare systems data captures sufficiently well the data traditionally captured on case report forms. "Data Utility Comparison Studies" (DUCkS) assess the utility of healthcare systems data for RCTs by comparison to data collected by the trial. Despite their importance, there are few published UK examples of DUCkS. METHODS-AND-RESULTS: Building from ongoing and selected recent examples of UK-led DUCkS in the literature, we set out experience-based considerations for the conduct of future DUCkS. Developed through informal iterative discussions in many forums, considerations are offered for planning, protocol development, data, analysis and reporting, with comparisons at "patient-level" or "trial-level", depending on the item of interest and trial status. DISCUSSION: DUCkS could be a valuable tool in assessing where healthcare systems data can be used for trials and in which trial teams can play a leading role. There is a pressing need for trials to be more efficient in their delivery and research waste must be reduced. Trials have been making inconsistent use of healthcare systems data, not least because of an absence of evidence of utility. DUCkS can also help to identify challenges in using healthcare systems data, such as linkage (access and timing) and data quality. We encourage trial teams to incorporate and report DUCkS in trials and funders and data providers to support them.

Monitoring metrics over time: Why clinical trialists need to systematically collect site performance metrics.

Background: Over the last decade, there has been an increasing interest in risk-based monitoring (RBM) in clinical trials, resulting in a number of guidelines from regulators and its inclusion in ICH GCP. However, there is a lack of detail on how to approach RBM from a practical perspective, and insufficient understanding of best practice. Purpose: We present a method for clinical trials units to track their metrics within clinical trials using descriptive statistics and visualisations. Research Design: We suggest descriptive statistics and visualisations within a SWAT methodology. Study Sample: We illustrate this method using the metrics from TEMPER, a monitoring study carried out in three trials at the MRC Clinical Trials Unit at UCL. Data Collection: The data collection for TEMPER is described in DOI: 10.1177/1740774518793379. Results: We show the results and discuss a protocol for a Study-Within-A-Trial (SWAT 167) for those wishing to use the method. Conclusions: The potential benefits metric tracking brings to clinical trials include enhanced assessment of sites for potential corrective action, improved evaluation and contextualisation of the influence of metrics and their thresholds, and the establishment of best practice in RBM. The standardisation of the collection of such monitoring data would benefit both individual trials and the clinical trials community.

Lack of transparent reporting of trial monitoring approaches in randomised controlled trials: A systematic review of contemporary protocol papers.

BACKGROUND: Monitoring is essential to ensure patient safety and data integrity in clinical trials as per Good Clinical Practice. The Standard Protocol Items: Recommendations for Interventional Trials Statement and its checklist guides authors to include monitoring in their protocols. We investigated how well monitoring was reported in published 'protocol papers' for contemporary randomised controlled trials. METHODS: A systematic search was conducted in PubMed to identify eligible protocol papers published in selected journals between 1 January 2020 and 31 May 2020. Protocol papers were classified by whether they reported monitoring and, if so, by the details of monitoring. Data were summarised descriptively. RESULTS: Of 811 protocol papers for randomised controlled trials, 386 (48%; 95% CI: 44%-51%) explicitly reported some monitoring information. Of these, 20% (77/386) reported monitoring information consistent with an on-site monitoring approach, and 39% (152/386) with central monitoring, 26% (101/386) with a mixed approach, while 14% (54/386) did not provide sufficient information to specify an approach. Only 8% (30/386) of randomised controlled trials reported complete details about all of scope, frequency and organisation of monitoring; frequency of monitoring was the least reported. However, 6% (25/386) of papers used the term 'audit' to describe 'monitoring'. DISCUSSION: Monitoring information was reported in only approximately half of the protocol papers. Suboptimal reporting of monitoring hinders the clinical community from having the full information on which to judge the validity of a trial and jeopardises the value of protocol papers and the credibility of the trial itself. Greater efforts are needed to promote the transparent reporting of monitoring to journal editors and authors.

What is the purpose of clinical trial monitoring?

BACKGROUND: The sources of information on clinical trial monitoring do not give information in an accessible language and do not give detailed guidance. In order to enable communication and to build clinical trial monitoring tools on a strong easily communicated foundation, we identified the need to define monitoring in accessible language. METHODS: In a three-step process, the material from sources that describe clinical trial monitoring were synthesised into principles of monitoring. A poll regarding their applicability was run at a UK national academic clinical trials monitoring meeting. RESULTS: The process derived 5 key principles of monitoring: keeping participants safe and respecting their rights, having data we can trust, making sure the trial is being run as it was meant to be, improving the way the trial is run and preventing problems before they happen. CONCLUSION: From the many sources mentioning monitoring of clinical trials, the purpose of monitoring can be summarised simply as 5 principles. These principles, given in accessible language, should form a firm basis for discussion of monitoring of clinical trials.

Use of routinely collected health data in randomised clinical trials: comparison of trial-specific death data in the BOSS trial with NHS Digital data.

BACKGROUND: A promising approach to reduce the increasing costs of clinical trials is the use of routinely collected health data as participant data. However, the quality of this data could limit its usability as trial participant data. METHODS: The BOSS trial is a randomised controlled trial comparing regular endoscopies versus endoscopies at need in patients with Barrett's oesophagus with primary endpoint death. Data on death and cancer collected every 2 years after randomisation (trial-specific data) were compared to data received annually (all patients on one date) from the routinely collected health data source National Health Service (NHS) Digital. We investigated completeness, agreement and timeliness and looked at the implications for the primary trial outcome. Completeness and agreement were assessed by evaluating the number of reported and missing cases and any disparities between reported dates. Timeliness was considered by graphing the year a death was first reported in the trial-specific data against that for NHS Digital data. Implications on the primary trial outcome, overall survival, of using one of the data sources alone were investigated using Kaplan-Meier graphs. To assess the utility of cause of death and cancer diagnoses, oesophageal cancer cases were compared. RESULTS: NHS Digital datasets included more deaths and often reported them sooner than the trial-specific data. The number reported as being from oesophageal cancer was similar in both datasets. Due to time lag in reporting and missing cases, the event rate appeared higher using the NHS Digital data. CONCLUSION: NHS Digital death data is useful for calculating overall survival where trial-specific follow-up is only every 2 years from randomisation and the follow-up requires patient response. The cancer data was not a large enough sample to assess usability. We suggest that this assessment of registry data is done for more phase III RCTs and for more registry data to get a more complete picture of when RCHD would be useful in phase III RCT. TRIAL REGISTRATION: ISRCTN54190466 (BOSS) 1 Oct 2009.

Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: A scoping review.

BACKGROUND/AIMS: It is increasingly recognised that reliance on frequent site visits for monitoring clinical trials is inefficient. Regulators and trialists have recently encouraged more risk-based monitoring. Risk assessment should take place before a trial begins to define the overarching monitoring strategy. It can also be done on an ongoing basis, to target sites for monitoring activity. Various methods have been proposed for such prioritisation, often using terms like 'central statistical monitoring', 'triggered monitoring' or, as in the International Conference on Harmonization Good Clinical Practice guidance, 'targeted on-site monitoring'. We conducted a scoping review to identify such methods, to establish if any were supported by adequate evidence to allow wider implementation, and to guide future developments in this field of research. METHODS: We used seven publication databases, two sets of methodological conference abstracts and an Internet search engine to identify methods for using centrally held trial data to assess site conduct during a trial. We included only reports in English, and excluded reports published before 1996 or not directly relevant to our research question. We used reference and citation searches to find additional relevant reports. We extracted data using a predefined template. We contacted authors to request additional information about included reports. RESULTS: We included 30 reports in our final dataset, of which 21 were peer-reviewed publications. In all, 20 reports described central statistical monitoring methods (of which 7 focussed on detection of fraud or misconduct) and 9 described triggered monitoring methods; 21 reports included some assessment of their methods' effectiveness, typically exploring the methods' characteristics using real trial data without known integrity issues. Of the 21 with some effectiveness assessment, most contained limited information about whether or not concerns identified through central monitoring constituted meaningful problems. Several reports demonstrated good classification ability based on more than one classification statistic, but never without caveats of unclear reporting or other classification statistics being low or unavailable. Some reports commented on cost savings from reduced on-site monitoring, but none gave detailed costings for the development and maintenance of central monitoring methods themselves. CONCLUSION: Our review identified various proposed methods, some of which could be combined within the same trial. The apparent emphasis on fraud detection may not be proportionate in all trial settings. Despite some promising evidence and some self-justifying benefits for data cleaning activity, many proposed methods have limitations that may currently prevent their routine use for targeting trial monitoring activity. The implementation costs, or uncertainty about these, may also be a barrier. We make recommendations for how the evidence-base supporting these methods could be improved.

Assessing the potential for prevention or earlier detection of on-site monitoring findings from randomised controlled trials: Further analyses of findings from the prospective TEMPER triggered monitoring study.

BACKGROUND/AIMS: Clinical trials should be designed and managed to minimise important errors with potential to compromise patient safety or data integrity, employ monitoring practices that detect and correct important errors quickly, and take robust action to prevent repetition. Regulators highlight the use of risk-based monitoring, making greater use of centralised monitoring and reducing reliance on centre visits. The TEMPER study was a prospective evaluation of triggered monitoring (a risk-based monitoring method), whereby centres are prioritised for visits based on central monitoring results. Conducted in three UK-based randomised cancer treatment trials of investigational medicine products with time-to-event outcomes, it found high levels of serious findings at triggered centre visits but also at visits to matched control centres that, based on central monitoring, were not of concern. Here, we report a detailed review of the serious findings from TEMPER centre visits. We sought to identify feasible, centralised processes which might detect or prevent these findings without a centre visit. METHODS: The primary outcome of this study was the proportion of all 'major' and 'critical' TEMPER centre visit findings theoretically detectable or preventable through a feasible, centralised process. To devise processes, we considered a representative example of each finding type through an internal consensus exercise. This involved (a) agreeing the potential, by some described process, for each finding type to be centrally detected or prevented and (b) agreeing a proposed feasibility score for each proposed process. To further assess feasibility, we ran a consultation exercise, whereby the proposed processes were reviewed and rated for feasibility by invited external trialists. RESULTS: In TEMPER, 312 major or critical findings were identified at 94 visits. These findings comprised 120 distinct issues, for which we proposed 56 different centralised processes. Following independent review of the feasibility of the proposed processes by 87 consultation respondents across eight different trial stakeholder groups, we conclude that 306/312 (98%) findings could theoretically be prevented or identified centrally. Of the processes deemed feasible, those relating to informed consent could have the most impact. Of processes not currently deemed feasible, those involving use of electronic health records are among those with the largest potential benefit. CONCLUSIONS: This work presents a best-case scenario, where a large majority of monitoring findings were deemed theoretically preventable or detectable by central processes. Caveats include the cost of applying all necessary methods, and the resource implications of enhanced central monitoring for both centre and trials unit staff. Our results will inform future monitoring plans and emphasise the importance of continued critical review of monitoring processes and outcomes to ensure they remain appropriate.

Access to routinely collected health data for clinical trials - review of successful data requests to UK registries.

BACKGROUND: Clinical trials generally each collect their own data despite routinely collected health data (RCHD) increasing in quality and breadth. Our aim is to quantify UK-based randomised controlled trials (RCTs) accessing RCHD for participant data, characterise how these data are used and thereby recommend how more trials could use RCHD. METHODS: We conducted a systematic review of RCTs accessing RCHD from at least one registry in the UK between 2013 and 2018 for the purposes of informing or supplementing participant data. A list of all registries holding RCHD in the UK was compiled. In cases where registries published release registers, these were searched for RCTs accessing RCHD. Where no release register was available, registries were contacted to request a list of RCTs. For each identified RCT, information was collected from all publicly available sources (release registers, websites, protocol etc.). The search and data extraction were undertaken between January and May 2019. RESULTS: We identified 160 RCTs accessing RCHD between 2013 and 2018 from a total of 22 registries; this corresponds to only a very small proportion of all UK RCTs (about 3%). RCTs accessing RCHD were generally large (median sample size 1590), commonly evaluating treatments for cancer or cardiovascular disease. Most of the included RCTs accessed RCHD from NHS Digital (68%), and the most frequently accessed datasets were mortality (76%) and hospital visits (55%). RCHD was used to inform the primary trial (82%) and long-term follow-up (57%). There was substantial variation in how RCTs used RCHD to inform participant outcome measures. A limitation was the lack of information and transparency from registries and RCTs with respect to which datasets have been accessed and for what purposes. CONCLUSIONS: In the last five years, only a small minority of UK-based RCTs have accessed RCHD to inform participant data. We ask for improved accessibility, confirmed data quality and joined-up thinking between the registries and the regulatory authorities. TRIAL REGISTRATION: PROSPERO CRD42019123088.

Monitoring in practice - How are UK academic clinical trials monitored? A survey.

BACKGROUND: Despite the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) encouraging the use of risk-based monitoring for trials in 2013, there remains a lack of evidence-based guidelines on how to monitor. We surveyed the academic United Kingdom Clinical Research Collaboration (UKCRC) registered clinical trials units (CTUs) to find out their policy on monitoring of phase III randomised clinical trials of an investigational medicinal product (CTIMPs). METHODS: An online survey of monitoring policy with sections on the CTU, central monitoring and on-site monitoring was sent to all 50 UKCRC registered CTUs in November 2018. Descriptive data analysis and tabulations are reported using the total number answering each question. RESULTS: A total of 43/50 (86%) of CTUs responded with 38 conducting phase III randomised CTIMP trials. Of these 38 CTUs, 34 finished the survey. Most CTUs (36/37, 97%) use a central monitoring process to guide, target or supplement site visits. More than half (19/36, 53%) of CTUs do not use an automated monitoring report when centrally monitoring trials and all units use trial team knowledge to make a final decision on whether an on-site visit is required. A total of 31/34 (91%) CTUs used triggers to decide whether or not to conduct an on-site monitoring visit. On-site, a mixture of source data verification and checking of processes was carried out. The CTUs overwhelmingly (27/34, 79%) selected optimising central monitoring as their most pressing concern. CONCLUSION: The survey showed a wide variation in phase III randomised CTIMP trial monitoring practices by academic clinical trials units within a single research-active country. We urgently need to develop evidence-based regulator-agreed guidance for CTUs on best practice for both central and on-site monitoring and to develop tools for all CTUs to use.

Early warnings and repayment plans: novel trial management methods for monitoring and managing data return rates in a multi-centre phase III randomised controlled trial with paper Case Report Forms.

BACKGROUND: Monitoring and managing data returns in multi-centre randomised controlled trials is an important aspect of trial management. Maintaining consistently high data return rates has various benefits for trials, including enhancing oversight, improving reliability of central monitoring techniques and helping prepare for database lock and trial analyses. Despite this, there is little evidence to support best practice, and current standard methods may not be optimal. METHODS: We report novel methods from the Trial of Imaging and Schedule in Seminoma Testis (TRISST), a UK-based, multi-centre, phase III trial using paper Case Report Forms to collect data over a 6-year follow-up period for 669 patients. Using an automated database report which summarises the data return rate overall and per centre, we developed a Microsoft Excel-based tool to allow observation of per-centre trends in data return rate over time. The tool allowed us to distinguish between forms that can and cannot be completed retrospectively, to inform understanding of issues at individual centres. We reviewed these statistics at regular trials unit team meetings. We notified centres whose data return rate appeared to be falling, even if they had not yet crossed the pre-defined acceptability threshold of an 80% data return rate. We developed a set method for agreeing targets for gradual improvement with centres having persistent data return problems. We formalised a detailed escalation policy to manage centres who failed to meet agreed targets. We conducted a post-hoc, descriptive analysis of the effectiveness of the new processes. RESULTS: The new processes were used from April 2015 to September 2016. By May 2016, data return rates were higher than they had been at any time previously, and there were no centres with return rates below 80%, which had never been the case before. In total, 10 centres out of 35 were contacted regarding falling data return rates. Six out of these 10 showed improved rates within 6-8 weeks, and the remainder within 4 months. CONCLUSIONS: Our results constitute preliminary effectiveness evidence for novel methods in monitoring and managing data return rates in randomised controlled trials. We encourage other researchers to work on generating better evidence-based methods in this area, whether through more robust evaluation of our methods or of others.

Efficacy and safety of a 4-drug fixed-dose combination regimen compared with separate drugs for treatment of pulmonary tuberculosis: the Study C randomized controlled trial.

CONTEXT: Fixed-dose combinations (FDCs) of drugs for treatment of tuberculosis have been advocated to prevent the emergence of drug resistance. OBJECTIVE: To assess the efficacy and safety of a 4-drug FDC for the treatment of tuberculosis. DESIGN, SETTING, AND PATIENTS: The Study C trial, a parallel-group, open-label, noninferiority, randomized controlled trial conducted in 11 sites in Africa, Asia, and Latin America between 2003 and 2008. Patients were 1585 adults with newly diagnosed smear-positive pulmonary tuberculosis. INTERVENTIONS: Patients were randomized to receive daily treatment with 4 drugs (rifampicin, isoniazid, pyrazinamide, ethambutol) given as an FDC (n = 798 patients) or separately (n = 787) in the 8-week intensive phase of treatment. MAIN OUTCOME MEASURE: Favorable treatment outcome, defined as negative culture result at 18 months post randomization and not having already been classified as unfavorable. Noninferiority was dependent on consistent results from a per-protocol and modified intention-to-treat analysis, using 2 different models for the latter, classifying all changes of treatment or refusal to continue treatment (eg, bacteriological failure/relapse, adverse event, default, drug resistance) as unfavorable (model 1) and classifying changes of treatment for reasons other than therapeutic outcomes according to their 18-month bacteriological outcome if available (post hoc model 2). The prespecified noninferiority margin was 4%. RESULTS: In the per-protocol analysis, 555 of 591 patients (93.9%) had a favorable outcome in the FDC group vs 548 of 579 (94.6%) in the separate-drugs group (risk difference, -0.7% [90% confidence interval {CI}, -3.0% to 1.5%]). In the model 1 analysis, 570 of 684 patients (83.3%) had a favorable outcome in the FDC group vs 563 of 664 (84.8%) in the separate-drugs group (risk difference, -1.5% [90% CI, -4.7% to 1.8%]). In the post hoc model 2 analysis, 591 of 658 patients (89.8%) in the FDC group and 589 of 647 (91.0%) in the separate-drugs group had a favorable outcome (risk difference, -1.2% [90% CI, -3.9% to 1.5%]). Adverse events related to trial drugs were similarly distributed among treatment groups. CONCLUSIONS: Compared with a regimen of separately administered drugs, a 4-drug FDC regimen for treatment of tuberculosis satisfied prespecified noninferiority criteria in 2 of 3 analyses. Although the results do not demonstrate full noninferiority of the FDCs compared with single drugs given separately using the strict definition applied in this trial, use of FDCs is preferred because of potential advantages associated with the administration of FDCs compared with separate-drug formulations. TRIAL REGISTRATION: clinicaltrials.gov Identifier: NCT00216333.